\[ \int x^2 \coth ^{-1}(\tanh (a+b x))^2 \, dx \]
Optimal antiderivative \[ \frac {b^{2} x^{5}}{30}-\frac {b \,x^{4} \mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )}{6}+\frac {x^{3} \mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )^{2}}{3} \]
command
integrate(x^2*arccoth(tanh(b*x+a))^2,x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{5} \, b^{2} x^{5} - \frac {1}{4} \, {\left (-i \, \pi b - 2 \, a b\right )} x^{4} - \frac {1}{12} \, {\left (\pi ^{2} - 4 i \, \pi a - 4 \, a^{2}\right )} x^{3} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{2} \operatorname {arcoth}\left (\tanh \left (b x + a\right )\right )^{2}\,{d x} \]________________________________________________________________________________________