\[ \int \frac {\cot ^2(c+d x)}{(a+b \tan (c+d x))^{5/2}} \, dx \]
Optimal antiderivative \[ \frac {5 b \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {a}}\right )}{a^{\frac {7}{2}} d}+\frac {i \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {-i b +a}}\right )}{\left (-i b +a \right )^{\frac {5}{2}} d}-\frac {i \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {i b +a}}\right )}{\left (i b +a \right )^{\frac {5}{2}} d}-\frac {b \left (a^{4}+10 a^{2} b^{2}+5 b^{4}\right )}{a^{3} \left (a^{2}+b^{2}\right )^{2} d \sqrt {a +b \tan \left (d x +c \right )}}-\frac {b \left (3 a^{2}+5 b^{2}\right )}{3 a^{2} \left (a^{2}+b^{2}\right ) d \left (a +b \tan \left (d x +c \right )\right )^{\frac {3}{2}}}-\frac {\cot \left (d x +c \right )}{a d \left (a +b \tan \left (d x +c \right )\right )^{\frac {3}{2}}} \]
command
integrate(cot(d*x+c)^2/(a+b*tan(d*x+c))^(5/2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \text {output too large to display} \]
Fricas 1.3.7 via sagemath 9.3 output \[ \text {Timed out} \]