\[ \int x^2 \coth ^{-1}(a+b x) \, dx \]
Optimal antiderivative \[ -\frac {a x}{b^{2}}+\frac {\left (b x +a \right )^{2}}{6 b^{3}}+\frac {x^{3} \mathrm {arccoth}\left (b x +a \right )}{3}+\frac {\left (1-a \right )^{3} \ln \left (-b x -a +1\right )}{6 b^{3}}+\frac {\left (1+a \right )^{3} \ln \left (b x +a +1\right )}{6 b^{3}} \]
command
integrate(x^2*arccoth(b*x+a),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{6} \, {\left ({\left (a + 1\right )} b - {\left (a - 1\right )} b\right )} {\left (\frac {{\left (3 \, a^{2} + 1\right )} \log \left (\frac {{\left | b x + a + 1 \right |}}{{\left | b x + a - 1 \right |}}\right )}{b^{4}} - \frac {{\left (3 \, a^{2} + 1\right )} \log \left ({\left | \frac {b x + a + 1}{b x + a - 1} - 1 \right |}\right )}{b^{4}} - \frac {2 \, {\left (\frac {{\left (b x + a + 1\right )} {\left (3 \, a - 1\right )}}{b x + a - 1} - 3 \, a\right )}}{b^{4} {\left (\frac {b x + a + 1}{b x + a - 1} - 1\right )}^{2}} + \frac {{\left (\frac {3 \, {\left (b x + a + 1\right )}^{2} a^{2}}{{\left (b x + a - 1\right )}^{2}} - \frac {6 \, {\left (b x + a + 1\right )} a^{2}}{b x + a - 1} + 3 \, a^{2} - \frac {6 \, {\left (b x + a + 1\right )}^{2} a}{{\left (b x + a - 1\right )}^{2}} + \frac {6 \, {\left (b x + a + 1\right )} a}{b x + a - 1} + \frac {3 \, {\left (b x + a + 1\right )}^{2}}{{\left (b x + a - 1\right )}^{2}} + 1\right )} \log \left (-\frac {\frac {1}{a - \frac {{\left (\frac {{\left (b x + a + 1\right )} {\left (a - 1\right )}}{b x + a - 1} - a - 1\right )} b}{\frac {{\left (b x + a + 1\right )} b}{b x + a - 1} - b}} + 1}{\frac {1}{a - \frac {{\left (\frac {{\left (b x + a + 1\right )} {\left (a - 1\right )}}{b x + a - 1} - a - 1\right )} b}{\frac {{\left (b x + a + 1\right )} b}{b x + a - 1} - b}} - 1}\right )}{b^{4} {\left (\frac {b x + a + 1}{b x + a - 1} - 1\right )}^{3}}\right )} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{2} \operatorname {arcoth}\left (b x + a\right )\,{d x} \]________________________________________________________________________________________