\[ \int x^3 \coth ^{-1}(a x)^2 \, dx \]
Optimal antiderivative \[ \frac {x^{2}}{12 a^{2}}+\frac {x \,\mathrm {arccoth}\left (a x \right )}{2 a^{3}}+\frac {x^{3} \mathrm {arccoth}\left (a x \right )}{6 a}-\frac {\mathrm {arccoth}\left (a x \right )^{2}}{4 a^{4}}+\frac {x^{4} \mathrm {arccoth}\left (a x \right )^{2}}{4}+\frac {\ln \left (-a^{2} x^{2}+1\right )}{3 a^{4}} \]
command
integrate(x^3*arccoth(a*x)^2,x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{6} \, {\left (\frac {3 \, {\left (\frac {{\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} + \frac {a x + 1}{a x - 1}\right )} \log \left (\frac {a x + 1}{a x - 1}\right )^{2}}{\frac {{\left (a x + 1\right )}^{4} a^{5}}{{\left (a x - 1\right )}^{4}} - \frac {4 \, {\left (a x + 1\right )}^{3} a^{5}}{{\left (a x - 1\right )}^{3}} + \frac {6 \, {\left (a x + 1\right )}^{2} a^{5}}{{\left (a x - 1\right )}^{2}} - \frac {4 \, {\left (a x + 1\right )} a^{5}}{a x - 1} + a^{5}} + \frac {2 \, {\left (\frac {3 \, {\left (a x + 1\right )}^{2}}{{\left (a x - 1\right )}^{2}} - \frac {3 \, {\left (a x + 1\right )}}{a x - 1} + 2\right )} \log \left (\frac {a x + 1}{a x - 1}\right )}{\frac {{\left (a x + 1\right )}^{3} a^{5}}{{\left (a x - 1\right )}^{3}} - \frac {3 \, {\left (a x + 1\right )}^{2} a^{5}}{{\left (a x - 1\right )}^{2}} + \frac {3 \, {\left (a x + 1\right )} a^{5}}{a x - 1} - a^{5}} + \frac {2 \, {\left (a x + 1\right )}}{{\left (\frac {{\left (a x + 1\right )}^{2} a^{5}}{{\left (a x - 1\right )}^{2}} - \frac {2 \, {\left (a x + 1\right )} a^{5}}{a x - 1} + a^{5}\right )} {\left (a x - 1\right )}} - \frac {4 \, \log \left (\frac {a x + 1}{a x - 1} - 1\right )}{a^{5}} + \frac {4 \, \log \left (\frac {a x + 1}{a x - 1}\right )}{a^{5}}\right )} a \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{3} \operatorname {arcoth}\left (a x\right )^{2}\,{d x} \]________________________________________________________________________________________