\[ \int (a+b x)^2 \coth ^{-1}(a+b x) \, dx \]
Optimal antiderivative \[ \frac {\left (b x +a \right )^{2}}{6 b}+\frac {\left (b x +a \right )^{3} \mathrm {arccoth}\left (b x +a \right )}{3 b}+\frac {\ln \left (1-\left (b x +a \right )^{2}\right )}{6 b} \]
command
integrate((b*x+a)^2*arccoth(b*x+a),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{6} \, {\left ({\left (a + 1\right )} b - {\left (a - 1\right )} b\right )} {\left (\frac {\log \left (\frac {{\left | b x + a + 1 \right |}}{{\left | b x + a - 1 \right |}}\right )}{b^{2}} - \frac {\log \left ({\left | \frac {b x + a + 1}{b x + a - 1} - 1 \right |}\right )}{b^{2}} + \frac {{\left (\frac {3 \, {\left (b x + a + 1\right )}^{2}}{{\left (b x + a - 1\right )}^{2}} + 1\right )} \log \left (-\frac {\frac {1}{a - \frac {{\left (\frac {{\left (b x + a + 1\right )} {\left (a - 1\right )}}{b x + a - 1} - a - 1\right )} b}{\frac {{\left (b x + a + 1\right )} b}{b x + a - 1} - b}} + 1}{\frac {1}{a - \frac {{\left (\frac {{\left (b x + a + 1\right )} {\left (a - 1\right )}}{b x + a - 1} - a - 1\right )} b}{\frac {{\left (b x + a + 1\right )} b}{b x + a - 1} - b}} - 1}\right )}{b^{2} {\left (\frac {b x + a + 1}{b x + a - 1} - 1\right )}^{3}} + \frac {2 \, {\left (b x + a + 1\right )}}{{\left (b x + a - 1\right )} b^{2} {\left (\frac {b x + a + 1}{b x + a - 1} - 1\right )}^{2}}\right )} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int {\left (b x + a\right )}^{2} \operatorname {arcoth}\left (b x + a\right )\,{d x} \]________________________________________________________________________________________