Mathematical Reflections 2012, Issue 6 - Problem O247

Problem:
Solve in positive integers the equation $$xy+yz+zx-5\sqrt{x^2+y^2+z^2}=1.$$

Proposed by Titu Andreescu.

Solution:
Clearly, $xy+yz+zx \geq 3$. Rewriting the equation in the form $$(xy+yz+zx-1)^2=25(x^2+y^2+z^2),$$ we put $xy+yz+zx=5t+1, t > 0$, so that $x^2+y^2+z^2=t^2$. Then $$(x+y+z)^2=x^2+y^2+z^2+2(xy+yz+zx)=t^2+2(5t+1)=(t+5)^2-23,$$ which gives $$(t+5-x-y-z)(t+5+x+y+z)=23.$$
Since $x,y,z,t$ are positive integers, it must be
$$\left\{\begin{array}{rcl} t+5-x-y-z & = & 1 \\ t+5+x+y+z & = & 23. \end{array} \right.$$
Summing up the two equations, we get $t=7$. So, $x+y+z=11$ and $xy+yz+zx=36$. Suppose without loss of generality that $x \leq y \leq z$. Clearly, $x\leq3$, so we have three cases.

(i) $x=1$. Therefore, $y+z=10, yz=26$, so there are no solution.
(ii) $x=2$. Therefore, $y+z=9, yz=18$, so $y=3, z=6$.
(iii) $x=3$. Therefore, $y+z=8, yz=12$, so there are no solution.

In conclusion, all the positive integer solutions are $$(2,3,6), (2,6,3), (3,2,6), (3,6,2), (6,2,3), (6,3,2).$$