\[ \int x^{11} \left (a+b \text {ArcTan}\left (c x^3\right )\right )^2 \, dx \]
Optimal antiderivative \[ \frac {a b \,x^{3}}{6 c^{3}}+\frac {b^{2} x^{6}}{36 c^{2}}+\frac {b^{2} x^{3} \arctan \left (c \,x^{3}\right )}{6 c^{3}}-\frac {b \,x^{9} \left (a +b \arctan \left (c \,x^{3}\right )\right )}{18 c}-\frac {\left (a +b \arctan \left (c \,x^{3}\right )\right )^{2}}{12 c^{4}}+\frac {x^{12} \left (a +b \arctan \left (c \,x^{3}\right )\right )^{2}}{12}-\frac {b^{2} \ln \left (c^{2} x^{6}+1\right )}{9 c^{4}} \]
command
integrate(x**11*(a+b*atan(c*x**3))**2,x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \begin {cases} \frac {a^{2} x^{12}}{12} + \frac {a b x^{12} \operatorname {atan}{\left (c x^{3} \right )}}{6} - \frac {a b x^{9}}{18 c} + \frac {a b x^{3}}{6 c^{3}} - \frac {a b \operatorname {atan}{\left (c x^{3} \right )}}{6 c^{4}} + \frac {b^{2} x^{12} \operatorname {atan}^{2}{\left (c x^{3} \right )}}{12} - \frac {b^{2} x^{9} \operatorname {atan}{\left (c x^{3} \right )}}{18 c} + \frac {b^{2} x^{6}}{36 c^{2}} + \frac {b^{2} x^{3} \operatorname {atan}{\left (c x^{3} \right )}}{6 c^{3}} + \frac {2 b^{2} \sqrt {- \frac {1}{c^{2}}} \operatorname {atan}{\left (c x^{3} \right )}}{9 c^{3}} - \frac {2 b^{2} \log {\left (x - \sqrt [6]{- \frac {1}{c^{2}}} \right )}}{9 c^{4}} - \frac {2 b^{2} \log {\left (4 x^{2} + 4 x \sqrt [6]{- \frac {1}{c^{2}}} + 4 \sqrt [3]{- \frac {1}{c^{2}}} \right )}}{9 c^{4}} - \frac {b^{2} \operatorname {atan}^{2}{\left (c x^{3} \right )}}{12 c^{4}} & \text {for}\: c \neq 0 \\\frac {a^{2} x^{12}}{12} & \text {otherwise} \end {cases} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {Timed out} \]________________________________________________________________________________________