\[ \int \cos ^{\frac {5}{2}}(c+d x) (a+b \sec (c+d x)) \left (A+B \sec (c+d x)+C \sec ^2(c+d x)\right ) \, dx \]
Optimal antiderivative \[ \frac {2 \left (3 a A +5 b B +5 a C \right ) \sqrt {\frac {\cos \left (d x +c \right )}{2}+\frac {1}{2}}\, \EllipticE \left (\sin \left (\frac {d x}{2}+\frac {c}{2}\right ), \sqrt {2}\right )}{5 \cos \left (\frac {d x}{2}+\frac {c}{2}\right ) d}+\frac {2 \left (A b +B a +3 C b \right ) \sqrt {\frac {\cos \left (d x +c \right )}{2}+\frac {1}{2}}\, \EllipticF \left (\sin \left (\frac {d x}{2}+\frac {c}{2}\right ), \sqrt {2}\right )}{3 \cos \left (\frac {d x}{2}+\frac {c}{2}\right ) d}+\frac {2 a A \left (\cos ^{\frac {3}{2}}\left (d x +c \right )\right ) \sin \left (d x +c \right )}{5 d}+\frac {2 \left (A b +B a \right ) \sin \left (d x +c \right ) \left (\sqrt {\cos }\left (d x +c \right )\right )}{3 d} \]
command
integrate(cos(d*x+c)^(5/2)*(a+b*sec(d*x+c))*(A+B*sec(d*x+c)+C*sec(d*x+c)^2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \frac {2 \, {\left (3 \, A a \cos \left (d x + c\right ) + 5 \, B a + 5 \, A b\right )} \sqrt {\cos \left (d x + c\right )} \sin \left (d x + c\right ) - 5 \, \sqrt {2} {\left (i \, B a + i \, {\left (A + 3 \, C\right )} b\right )} {\rm weierstrassPInverse}\left (-4, 0, \cos \left (d x + c\right ) + i \, \sin \left (d x + c\right )\right ) - 5 \, \sqrt {2} {\left (-i \, B a - i \, {\left (A + 3 \, C\right )} b\right )} {\rm weierstrassPInverse}\left (-4, 0, \cos \left (d x + c\right ) - i \, \sin \left (d x + c\right )\right ) - 3 \, \sqrt {2} {\left (-i \, {\left (3 \, A + 5 \, C\right )} a - 5 i \, B b\right )} {\rm weierstrassZeta}\left (-4, 0, {\rm weierstrassPInverse}\left (-4, 0, \cos \left (d x + c\right ) + i \, \sin \left (d x + c\right )\right )\right ) - 3 \, \sqrt {2} {\left (i \, {\left (3 \, A + 5 \, C\right )} a + 5 i \, B b\right )} {\rm weierstrassZeta}\left (-4, 0, {\rm weierstrassPInverse}\left (-4, 0, \cos \left (d x + c\right ) - i \, \sin \left (d x + c\right )\right )\right )}{15 \, d} \]
Fricas 1.3.7 via sagemath 9.3 output
\[ {\rm integral}\left ({\left (C b \cos \left (d x + c\right )^{2} \sec \left (d x + c\right )^{3} + {\left (C a + B b\right )} \cos \left (d x + c\right )^{2} \sec \left (d x + c\right )^{2} + A a \cos \left (d x + c\right )^{2} + {\left (B a + A b\right )} \cos \left (d x + c\right )^{2} \sec \left (d x + c\right )\right )} \sqrt {\cos \left (d x + c\right )}, x\right ) \]