Let \(\bar{h}_{O}\) be the angular momentum of \(A\) w.r.t to \(O\). Therefore apply the definition\begin{align*} \bar{h}_{O} & =\bar{r}_{A/O}\times m\bar{v}_{A}\\ & =3\begin{vmatrix} \hat{\imath } & \hat{\jmath } & \hat{k}\\ 1.5 & 5.8 & -1.2\\ 4.1 & 4.4 & 5.3 \end{vmatrix} \\ & =3\left ( 36.02\hat{\imath }-12.87\hat{\jmath }-17.18\hat{k}\right ) \\ & =108.06\hat{\imath }-38.61\hat{\jmath }-51.54\hat{k} \end{align*}
Hence \begin{align*} \left \vert \bar{h}_{O}\right \vert & =\sqrt{108^{2}+38.61^{2}+51.54^{2}}\\ & =125.794\text{ kg-m}^{2}\text{/sec} \end{align*}
By conservation of angular momentum\begin{align*} r_{0}mv_{A} & =2.29r_{0}mv_{2}\\ v_{2} & =\frac{v_{A}}{2.29}\\ & =\frac{9.87}{2.29}\\ & =4.31\text{ m/s} \end{align*}
Angular momentum initially \begin{align} \bar{h}_{1} & =\bar{r}_{A/O}\times m_{A}\bar{v}_{A}\nonumber \\ & =9.8\begin{vmatrix} \hat{\imath } & \hat{\jmath } & \hat{k}\\ 5 & 0 & 0\\ 0 & -3.7 & 0 \end{vmatrix} \nonumber \\ & =-181.3\hat{k}\tag{1} \end{align}
In new state, we first note that \(\left \vert \bar{v}_{A_{2}}\right \vert =\left \vert \bar{v}_{B}\right \vert \) since both are the same radius from origin. This means \(\left \vert v_{A_{y}}\right \vert =\left \vert v_{B_{y}}\right \vert \) since they move only in \(y\) direction. Then \begin{align} \bar{h}_{2} & =\bar{r}_{A/O}\times m_{A}\bar{v}_{A_{2}}+\bar{r}_{B/O}\times m_{B}\bar{v}_{B}\nonumber \\ & =9.8\begin{vmatrix} \hat{\imath } & \hat{\jmath } & \hat{k}\\ 2.5 & 0 & 0\\ 0 & -v_{A_{y}} & 0 \end{vmatrix} +8.3\begin{vmatrix} \hat{\imath } & \hat{\jmath } & \hat{k}\\ -2.5 & 0 & 0\\ 0 & v_{B_{y}} & 0 \end{vmatrix} \nonumber \\ & =9.8\begin{vmatrix} \hat{\imath } & \hat{\jmath } & \hat{k}\\ 2.5 & 0 & 0\\ 0 & -v_{B_{y}} & 0 \end{vmatrix} +8.3\begin{vmatrix} \hat{\imath } & \hat{\jmath } & \hat{k}\\ -2.5 & 0 & 0\\ 0 & v_{B_{y}} & 0 \end{vmatrix} \nonumber \\ & =-46.5v_{B_{y}}\hat{k}\tag{2} \end{align}
Since (1) and (2) are equal (conservation of angular momentum) then \[ v_{B_{y}}=\frac{-181.3}{-46.5}=3.899\text{ m/s}\]