\[ \int \cot ^4(c+d x) (a+b \tan (c+d x))^{5/2} \, dx \]
Optimal antiderivative \[ -\frac {i \left (-i b +a \right )^{\frac {5}{2}} \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {-i b +a}}\right )}{d}+\frac {i \left (i b +a \right )^{\frac {5}{2}} \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {i b +a}}\right )}{d}+\frac {5 b \left (8 a^{2}-b^{2}\right ) \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {a}}\right )}{8 d \sqrt {a}}+\frac {\left (8 a^{2}-11 b^{2}\right ) \cot \left (d x +c \right ) \sqrt {a +b \tan \left (d x +c \right )}}{8 d}-\frac {13 a b \left (\cot ^{2}\left (d x +c \right )\right ) \sqrt {a +b \tan \left (d x +c \right )}}{12 d}-\frac {a^{2} \left (\cot ^{3}\left (d x +c \right )\right ) \sqrt {a +b \tan \left (d x +c \right )}}{3 d} \]
command
integrate(cot(d*x+c)^4*(a+b*tan(d*x+c))^(5/2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \text {output too large to display} \]
Fricas 1.3.7 via sagemath 9.3 output \[ \text {Timed out} \]