\[ \int x^{5/2} \tanh ^{-1}(\tanh (a+b x))^2 \, dx \]
Optimal antiderivative \[ \frac {16 b^{2} x^{\frac {11}{2}}}{693}-\frac {8 b \,x^{\frac {9}{2}} \arctanh \left (\tanh \left (b x +a \right )\right )}{63}+\frac {2 x^{\frac {7}{2}} \arctanh \left (\tanh \left (b x +a \right )\right )^{2}}{7} \]
command
integrate(x**(5/2)*atanh(tanh(b*x+a))**2,x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \frac {16 b^{2} x^{\frac {11}{2}}}{693} - \frac {8 b x^{\frac {9}{2}} \operatorname {atanh}{\left (\tanh {\left (a + b x \right )} \right )}}{63} + \frac {2 x^{\frac {7}{2}} \operatorname {atanh}^{2}{\left (\tanh {\left (a + b x \right )} \right )}}{7} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {Timed out} \]________________________________________________________________________________________