\[ \int \cot ^2(c+d x) (a+i a \tan (c+d x))^4 (A+B \tan (c+d x)) \, dx \]
Optimal antiderivative \[ -8 a^{4} \left (-i B +A \right ) x +\frac {a^{4} \left (4 i A +7 B \right ) \ln \left (\cos \left (d x +c \right )\right )}{d}+\frac {a^{4} \left (4 i A +B \right ) \ln \left (\sin \left (d x +c \right )\right )}{d}-\frac {a A \cot \left (d x +c \right ) \left (a +i a \tan \left (d x +c \right )\right )^{3}}{d}+\frac {\left (2 i A -B \right ) \left (a^{2}+i a^{2} \tan \left (d x +c \right )\right )^{2}}{2 d}-\frac {3 B \left (a^{4}+i a^{4} \tan \left (d x +c \right )\right )}{d} \]
command
integrate(cot(d*x+c)**2*(a+I*a*tan(d*x+c))**4*(A+B*tan(d*x+c)),x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \frac {i a^{4} \cdot \left (4 A - 7 i B\right ) \log {\left (e^{2 i d x} + \frac {\left (- 4 i A a^{4} - 4 B a^{4} + i a^{4} \cdot \left (4 A - 7 i B\right )\right ) e^{- 2 i c}}{3 B a^{4}} \right )}}{d} + \frac {i a^{4} \cdot \left (4 A - i B\right ) \log {\left (e^{2 i d x} + \frac {\left (- 4 i A a^{4} - 4 B a^{4} + i a^{4} \cdot \left (4 A - i B\right )\right ) e^{- 2 i c}}{3 B a^{4}} \right )}}{d} + \frac {- 4 i A a^{4} + 10 B a^{4} e^{4 i c} e^{4 i d x} - 8 B a^{4} + \left (- 4 i A a^{4} e^{2 i c} - 2 B a^{4} e^{2 i c}\right ) e^{2 i d x}}{d e^{6 i c} e^{6 i d x} + d e^{4 i c} e^{4 i d x} - d e^{2 i c} e^{2 i d x} - d} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {Exception raised: NotInvertible} \]________________________________________________________________________________________