Mathematical Reflections 2013, Issue 3 - Problem S265

Problem:
Find all pairs $(m,n)$ of positive integers such that $m^2 + 5n$ and $n^2 + 5m$ are both perfect squares.

Proposed by Titu Andreescu.

Solution:
Suppose that $m=n$. Then $m^2+5m=m(m+5)$ must be a perfect square. Observe that $\gcd(m,m+5)=1$, otherwise $\gcd(m,m+5)=5$, which gives $m=5k$ for some $k \in \mathbb{N}^*$ and $m+5=5(k+1)$, but $m(m+5)=25k(k+1)$ cannot be a perfect square, since $k$ and $k+1$ are coprimes and must be perfect squares. Therefore, $m$ and $m+5$ are both perfect squares, and it's easy to see that this happens if and only if $m=4$.
Suppose without loss of generality that $m>n$. By an easy check, we can see that the only solution for $m \leq 4$ or $n \leq 4$ is $m=n=4$, so suppose that $m>n>4$. We have $m^2<m^2+5n<(m+3)^2$, so we obtain two cases.

(i) $m^2+5n=(m+1)^2$, which gives
\begin{equation}
5n=2m+1.                (1)
\end{equation}
So, $6n>2m$, i.e. $3n>m$. Hence,
$$(n+3)^2<n^2+10n=n^2+4m+2<n^2+5m<n^2+15n<(n+8)^2.$$
We obtain four cases

(a) $n^2+5m=(n+4)^2$, i.e. $5m=8n+16$, and summing up this equation with (1), we get $3(m-n)=17$, contradiction. So, no solution in this case.
(b) $n^2+5m=(n+5)^2$, i.e. $5m=10n+25$, which is $m=2n+5$. From equation (1) we get $n=11$ and $m=27$.
(c) $n^2+5m=(n+6)^2$, i.e. $5m=12n+36$, which is $10m=24n+72$. Multiplying by $5$ equation (1), and summing up, we get $n=77$ and $m=192$.
(d) $n^2+5m=(n+7)^2$, i.e. $5m=14n+49$, and summing up this equation with equation (1), we get $3(m-3n)=50$, contradiction. So, no solution in this case.

(ii) $m^2+5n=(m+2)^2$, which gives
\begin{equation}
5n=4m+4.                (2)
\end{equation}
So, $8n>4m$, i.e. $2n>m$. Hence,
$$(n+2)^2<n^2+5n<n^2+5m<n^2+10n<(n+5)^2.$$
We obtain two cases.

(a) $n^2+5m=(n+3)^2$, i.e. $5m=6n+9$, which is $20m=24n+36$. Multiplying by $5$ equation (2), and summing up, we get $n=56$ and $m=69$.
(b) $n^2+5m=(n+4)^2$, i.e. $5m=8n+16$, and summing up this equation with equation (2) we get $m-3n=20$, but $m-3n<0$, contradiction. So, no solution in this case.

In conclusion, all the pairs $(m,n)$ which satisfy the required conditions are
$$(4,4),(11,27),(27,11),(77,192),(192,77),(56,69),(69,56).$$