Mathematical Reflections 2017, Issue 1 - Problem O397

Problem:
Solve in integers the equation: $$(x^3-1)(y^3-1)=3(x^2y^2+2).$$

Proposed by Titu Andreescu, University of Texas at Dallas, USA

Solution:
The given equation can be written as $$x^3y^3-(x^3+y^3)-3x^2y^2=5.$$ Let $s=x+y$ and $t=xy$. Observe that $x^3+y^3=(x+y)^3-3xy(x+y)=s^3-3st$, so the given equation becomes $$(t^3-s^3)-3t(t-s)=5,$$ i.e. $$(t-s)(t^2+ts+s^2-3t)=5.$$
We obtain the four systems of equations: $$\begin{array}{rcl} t-s&=&\pm 1 \\ t^2+ts+s^2-3t&=& \pm 5, \end{array} \qquad \begin{array}{rcl} t-s&=&\pm 5 \\ t^2+ts+s^2-3t&=& \pm 1 \end{array}$$
If $t-s=\pm 1$, then $t^2-2ts+s^2=1$ and subtracting this equation to the second equation, we get $3ts-3t=4$ or $3ts-3t=-6$. The first equation is impossible, the second gives $t(s-1)=-2$. So, $(s,t) \in \{(-1,1),(0,2),(2,-2),(3,-1)\}$. It's easy to see that none of these pairs satisfies $t-s=\pm 1$, so there are no solutions in this case. If $t-s=\pm 5$, then $t^2-2ts+s^2=25$ and subtracting this equation to the second equation, we get $3ts-3t=-24$ or $3ts-3t=-26$. The second equation is impossible, the first gives $t(s-1)=-8$. So, $(s,t) \in \{(-7,1),(-3,2),(-1,4),(0,8),(2,-8),(3,-4),(5,-2),(9,-1)\}$. As $t-s=\pm 5$, we obtain $(s,t) \in \{(-3,2),(-1,4)\}$. Since $s$ and $t$ also satisfy the condition $s^2-4t=n^2$ for some $n \in \mathbb{Z}$, we obtain $(s,t)=(-3,2)$. So, $x+y=-3$ and $xy=2$, which gives $(x,y) \in \{(-1,-2),(-2,-1)\}$.