\[ \int \left (c+d x^2\right )^3 \coth ^{-1}(a x) \, dx \]
Optimal antiderivative \[ \frac {d \left (35 a^{4} c^{2}+21 a^{2} c d +5 d^{2}\right ) x^{2}}{70 a^{5}}+\frac {d^{2} \left (21 a^{2} c +5 d \right ) x^{4}}{140 a^{3}}+\frac {d^{3} x^{6}}{42 a}+c^{3} x \,\mathrm {arccoth}\left (a x \right )+c^{2} d \,x^{3} \mathrm {arccoth}\left (a x \right )+\frac {3 c \,d^{2} x^{5} \mathrm {arccoth}\left (a x \right )}{5}+\frac {d^{3} x^{7} \mathrm {arccoth}\left (a x \right )}{7}+\frac {\left (35 a^{6} c^{3}+35 a^{4} c^{2} d +21 a^{2} c \,d^{2}+5 d^{3}\right ) \ln \left (-a^{2} x^{2}+1\right )}{70 a^{7}} \]
command
integrate((d*x^2+c)^3*arccoth(a*x),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \text {output too large to display} \]
Giac 1.7.0 via sagemath 9.3 output \[ \int {\left (d x^{2} + c\right )}^{3} \operatorname {arcoth}\left (a x\right )\,{d x} \]_______________________________________________________