Quiz 7

In the method of moments we formulate the moments of the probability law of the distribution in which the random variables belong to and equate these moments to the moments obtained from the sample at hand and solve for the unknown parameters.

\begin{align*} \mu _{1} & =E\left ( X\right ) \\ & =\int _{-\infty }^{\infty }xf\left ( x\right ) dx\\ & =\frac {1}{2\sigma }\int _{-\infty }^{\infty }x\ e^{-\frac {\left \vert x\right \vert }{\sigma }}dx \end{align*}

But $e^{-\frac {\left \vert x\right \vert }{\sigma }}$is symmtric around the $x=0$ due to absolute $x$ in the power of the $\exp .$(This assumes $\sigma $ positive, which is ofcourse true) but it is multiplied by negative $x$ to the left of y-axis and multiplied by positive $x$ on the right of the y-axis, hence the area of the left of the y-axis will be equal but negative to the area on the right of the y-axis. Hence the above integral is zero. Hence $\mu _{1}=0$

\begin{align*} \mu _{2} & =E\left ( X^{2}\right ) \\ & =\frac {1}{2\sigma }\int _{-\infty }^{\infty }x^{2}e^{-\frac {\left \vert x\right \vert }{\sigma }}dx \end{align*}

Due to the symmetry of $e^{-\frac {\left \vert x\right \vert }{\sigma }}$ and also $x^{2}$ is even and symmetrical around $x=0\,$, the above integral is then twice the integral from $x=0\cdots \infty $ and it becomes

\begin{equation} \sigma =\sqrt {\frac {\mu _{2}}{2}} \tag {1}\end{equation}

Now ﬁnd $\mu _{2}$ from the sample itself and substitute for it in the above. From the sample,

\begin{align*} \mu _{2} & =Var(sample)+Mean(Sample)\\ & =\frac {1}{n}{\displaystyle \sum \limits _{i=1}^{n}} \left ( X_{i}-\bar {X}\right ) ^{2}+\bar {X}\end{align*}

Since the mean of the population was found to be zero, we can take the mean of the sample $\bar {X}=0$

___________________________________________________________________________________________

The MLE of $\sigma $ is found as follows. Since i.i.d. random variables we write (Where $L\left ( \sigma \right ) $ mean $lik\left ( \sigma \right ) $ and $l\left ( \sigma \right ) $ means $\log \left ( lik\left ( \sigma \right ) \right ) $

\begin{align*} L_{n}\left ( \sigma \right ) & ={\displaystyle \prod \limits _{i=1}^{n}} f\left ( X_{i}|\sigma \right ) \\ l_{n}\left ( \sigma \right ) & ={\displaystyle \sum \limits _{i=1}^{n}} \log f\left ( X_{i}|\sigma \right ) \\ & ={\displaystyle \sum \limits _{i=1}^{n}} \log \left ( \frac {1}{2\sigma }e^{-\frac {\left \vert X_{i}\right \vert }{\sigma }}\right ) \\ & ={\displaystyle \sum \limits _{i=1}^{n}} \left ( -\log \left ( 2\sigma \right ) -\frac {\left \vert X_{i}\right \vert }{\sigma }\right ) \end{align*}

\[ \fbox {$l_{n}\left ( \sigma \right ) =-n\log \left ( 2\sigma \right ) -\frac {1}{\sigma }{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert $}\]

Now we ﬁnd the MLE, which is the value of $\sigma \,$which maximizes the above function.

\begin{align*} l_{n}^{\prime }\left ( \sigma \right ) & =0\\ \frac {-n}{\sigma }+\frac {1}{\sigma ^{2}}{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert & =0\\ -\sigma n+{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert & =0 \end{align*}

The asymptotic distribution of the MLE $\hat {\sigma }$ is normal with mean $\sigma $ and variance $\frac {1}{nI\left ( \sigma \right ) }$where

\begin{align*} l_{n}^{\prime }\left ( \sigma \right ) & =\frac {\partial }{\partial \sigma }\left [ -n\log \left ( 2\sigma \right ) -\frac {1}{\sigma }{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert \right ] \\ & =\frac {-n}{\sigma }+\frac {1}{\sigma ^{2}}{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert \end{align*}

\[ l_{n}^{\prime \prime }\left ( \sigma \right ) =\frac {n}{\sigma ^{2}}-\frac {2}{\sigma ^{3}}{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert \]

\begin{align} I\left ( \sigma \right ) & =-E\left [ \frac {n}{\sigma ^{2}}-\frac {2}{\sigma ^{3}}{\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert \right ] \nonumber \\ & =-E\left [ \frac {n}{\sigma ^{2}}\right ] +\frac {2}{\sigma ^{3}}E\left [ {\displaystyle \sum \limits _{i=1}^{n}} \left \vert X_{i}\right \vert \right ] \nonumber \\ & =-\frac {n}{\sigma ^{2}}+\frac {2}{\sigma ^{3}}{\displaystyle \sum \limits _{i=1}^{n}} E\left \vert X_{i}\right \vert \tag {2}\end{align}

Need to ﬁnd $E\left \vert X_{i}\right \vert $, since i.i.d. all random variables has the same expected value as $X$, hence

\begin{align*} E\left \vert X\right \vert & =\frac {1}{2\sigma }\int _{-\infty }^{\infty }\left \vert x\right \vert e^{-\frac {\left \vert x\right \vert }{\sigma }}dx\\ & =\frac {1}{\sigma }\int _{0}^{\infty }xe^{-\frac {x}{\sigma }}dx\\ & =\sigma \end{align*}

\begin{align*} I\left ( \sigma \right ) & =-\frac {n}{\sigma ^{2}}+\frac {2}{\sigma ^{3}}{\displaystyle \sum \limits _{i=1}^{n}} \sigma \\ & =-\frac {n}{\sigma ^{2}}+\frac {2n}{\sigma ^{2}}\end{align*}

Hence MLE $\hat {\sigma }_{n}\ $has an asymptotic distribution $\sim N\left ( \sigma ,\frac {1}{nI\left ( \sigma \right ) }\right ) $

\begin{align*} Var\left ( \hat {\sigma }_{n}\right ) & =\frac {1}{nI\left ( \sigma \right ) }\\ & =\frac {\sigma ^{2}}{n^{2}}\end{align*}

In this problem the contribution to the likelihood function of $\lambda $ comes from only one random variable. Hence we need to ﬁnd the pdf of this random observation, which is an order statistics. It is the minimum random variable among $n$ random variables where $n=5$ here.

Since this is an exponential distribution, we know that the distribution of $X_{\left ( 1\right ) }$ is given by (from section 3.7, chapter 3, textbook)

Where in the above, the $t$ is the time of the ﬁrst failures in each sample taken. (sample size is 5 in this problem).

Hence we need to ﬁnd the maximum of the above function. Since we have only one r.v., no need to take logs, use standard method:

\begin{align*} L_{n}^{\prime }\left ( \lambda \right ) & =ne^{-n\lambda t}-n^{2}t\lambda e^{-n\lambda t}\\ & =0 \end{align*}

\begin{align*} 1-nt\hat {\lambda } & =0\\ \hat {\lambda } & =\frac {1}{nt}\end{align*}

But here $n=5$ and time of ﬁrst failure is $t=100$ hence the above becomes ( write $T=100$ ) then we have

Since $\hat {\lambda }=\frac {1}{5T}$ where $T$ is a r.v (the ﬁrst time to fail) which has the distribution $5\lambda e^{-5\lambda t}$, Hence we conclude that the distribution of $\hat {\lambda } \sim \frac {1}{5}\frac {1}{5\lambda e^{-5\lambda t}}$ But an exponential distribution is $\tau e^{-\tau t}$ , hence now we see that sampling distribution of $\hat {\lambda } \sim $multiple one over an exponential distribution with parameter $\left ( \tau =5\lambda \right ) $.

(When asked to ﬁnd distribution of some r.v., do we always have to express in terms of "known" distributions?)

___________________________________________________________________________________________

We need to ﬁnd the standard deviation of the sampling distribution of $\hat {\lambda }$ found above.

Since we found that $\hat {\lambda } \sim \frac {1}{\text {exponential distribution with parameter}\left ( \tau \right ) }$ and the variance of an exponential with parameter $\tau $ is $\frac {1}{\tau ^{2}}$, hence variance of $\hat {\lambda }=\tau ^{2}$

Hence standard error is the square root of this variance. Hence standard error of the MLE $\hat {\lambda }=$ $\tau =5\lambda $

First, I want to say that I am using the following deﬁntion of the Gamma function (using $\beta $ instead of $\lambda $) in the deﬁntion. Since The data given has units of time and are not rate (i.e. 1/time). So I am using this deﬁnition of Gamma PDF

Yes. The following shows the histogram of the data, and a plot of a Gamma distribution with the shape parameter $\alpha =1$ and scale parameter $\beta $ set to the average of the data.

Using method of moments. We need 2 equations since we have to estimate 2 parameters $\alpha ,\lambda .$ For Gamma

\begin{align*} \mu _{1} & =\frac {\alpha }{\lambda }\\ \mu _{2} & =Var\left ( X\right ) +\left ( E\left ( X\right ) \right ) ^{2}\\ & =\frac {\alpha }{\lambda ^{2}}+\left ( \frac {\alpha }{\lambda }\right ) ^{2}\\ & =\frac {\alpha }{\lambda ^{2}}\left ( \alpha +1\right ) \end{align*}

Now from the data itself, calculate the First and Second moments and equate to the above and solve for $\alpha ,\lambda \,$ and these will be our estimate. This little code does the above

\begin{align*} L\left ( \alpha ,\lambda \right ) & ={\displaystyle \prod \limits _{i}^{n}} \frac {\lambda ^{\alpha }}{\Gamma \left ( \alpha \right ) }X_{i}^{\alpha -1}e^{-\lambda X_{i}}\\ l\left ( \alpha ,\lambda \right ) & ={\displaystyle \sum \limits _{i}^{n}} \log \frac {\lambda ^{\alpha }}{\Gamma \left ( \alpha \right ) }X_{i}^{\alpha -1}e^{-\lambda X_{i}}\\ & ={\displaystyle \sum \limits _{i}^{n}} \log \frac {\lambda ^{\alpha }}{\Gamma \left ( \alpha \right ) }+\left ( \alpha -1\right ) {\displaystyle \sum \limits _{i}^{n}} \log X_{i}-\lambda {\displaystyle \sum \limits _{i}^{n}} X_{i}\\ & =n\alpha \log \lambda -n\log \Gamma \left ( \alpha \right ) +\left ( \alpha -1\right ) {\displaystyle \sum \limits _{i}^{n}} \log X_{i}-\lambda {\displaystyle \sum \limits _{i}^{n}} X_{i}\end{align*}

\begin{align*} \frac {\partial l\left ( \alpha ,\lambda \right ) }{\partial \alpha } & =n\log \lambda -n\ polyGamma(0,\alpha )+{\displaystyle \sum \limits _{i}^{n}} \log X_{i}\\ \frac {\partial l\left ( \alpha ,\lambda \right ) }{\partial \lambda } & =n\alpha \frac {1}{\lambda }-{\displaystyle \sum \limits _{i}^{n}} X_{i}\end{align*}

\begin{align*} 0 & =n\log \left ( \frac {\hat {\alpha }}{\bar {X}}\right ) -n\frac {\Gamma ^{\prime }\left ( \alpha \right ) }{\Gamma \left ( \alpha \right ) }+{\displaystyle \sum \limits _{i}^{n}} \log X_{i}\\ 0 & =n\log \left ( \hat {\alpha }\right ) -n\log \bar {X}-n\ polyGamma(0,\alpha )+{\displaystyle \sum \limits _{i}^{n}} \log X_{i}\end{align*}

And solve for $\hat {\alpha }$. Once we ﬁnd $\hat {\alpha }$ we then ﬁnd also $\hat {\lambda }=\frac {\hat {\alpha }}{\bar {X}}$

Now Fit this model again, and compare the MLE ﬁtting to the method of moments ﬁtting

This plot shows more closely the ﬁtting on top of each others. They are very close so hard to see the diﬀerence other than near the high frequency part.

Try for $n=500$ be the same size$.$ Use the method of moments parameters to generate an $n$ random variables from Gamma distribution. First time use the parameters estimated from the data as shown above.

Now, use the sample generated above to estimate the parameters from it again using also the method of moments. Use these parameters to generate another $n$ random variables. repeat this process for say $N=5000$ and ﬁnd the variances of the parameters $\alpha ,\lambda $, and hence we ﬁnd the standard error which is the square root of these variances.

(Last minute update), I am getting large result for standard error from the bootstrap method. I think I have something wrong. Here is the result I get and the code

4.7 Quiz 7

4.7.1 corrected problem 3

4.7.2 Graded