\[ \int \frac {\coth ^{-1}(\tanh (a+b x))^2}{x^5} \, dx \]
Optimal antiderivative \[ \frac {b \mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )^{3}}{12 x^{3} \left (b x -\mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )\right )^{2}}+\frac {\mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )^{3}}{4 x^{4} \left (b x -\mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )\right )} \]
command
integrate(arccoth(tanh(b*x+a))^2/x^5,x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ -\frac {24 \, b^{2} x^{2} + 16 i \, \pi b x + 32 \, a b x - 3 \, \pi ^{2} + 12 i \, \pi a + 12 \, a^{2}}{48 \, x^{4}} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int \frac {\operatorname {arcoth}\left (\tanh \left (b x + a\right )\right )^{2}}{x^{5}}\,{d x} \]________________________________________________________________________________________