Multivariate Linear Regression Model

Definition:

• Linear regression but in multiple dimensions
• With $n$ independent observations on $Y$ and the associated values of $z_i$, the complete model becomes:
  • $Y_1={\beta }_0+{\beta }_1{z}_{11}+...+{\beta }_r{z}_{1r}+{ϵ}_1$
  • …
  • $Y_n={\beta }_0+{\beta }_n{z}_{n1}+...+{\beta }_r{z}_{nr}+{ϵ}_n$
• Error term, $ϵ$:
  • $E(e_j)=0$
  • $Var(e_j)={\sigma }^2$ (constant)
  • $Cov(e_j,e_k)=0$ for $j\ne k$
• Then we have $Y=Z\times \beta +ϵ$ where:
  • $Z$ is design matrix
  • $Cov(ϵ)=E(ϵ{ϵ}^{⊺})={\sigma }^{2}I$
• We introduce the artificial variable $z_{j0}=1$ so that ${\beta }_0+{\beta }_1{z}_{j1}+\cdots +{\beta }_r{z}_{jr}={\beta }_0{z}_{j0}+{\beta }_1{z}_{j1}+\cdots +{\beta }_r{z}_{jr}$

Least square estimation:

• We must determine the values for the regression coefficients $\beta$ and the error variance ${\sigma }^2$ consistent with the available data.
• Let $\mathbf{b}$ be trial values for $\mathbf{\beta }$. The method of least squares selects $b$ so as to minimize the sum of the squares of the differences:
  • $\begin{array}{rl}S(\mathbf{b})& =\sum _{j=1}^n{\left(y_j-b_0-b_1{z}_{j1}-\cdots -b_r{z}_{jr}\right)}^2\\ & =(y-zb{)}^T(y-zb)\end{array}$
• The coefficients $\mathbf{b}$ chosen by the least squares criterion are called least squares estimates of $\mathbf{\beta }$, denoted by $\hat{\mathbf{\beta }}$
• The deviations ${\widehat{ϵ}}_j=y_j-{\hat{\beta }}_0-{\hat{\beta }}_1{z}_{j1}-\cdots -{\hat{\beta }}_r{z}_{jr},j=1,\dots ,n$ are called residuals. The vector of residuals $\widehat{ϵ}=y-Z\hat{\beta }$ contains the information about the remaining unknown parameter ${\sigma }^2$
• Proposition:
  • Let $Z$ have full rank $r+1\le n$. The least squares estimate of $\beta$ is given by $\hat{\mathbf{\beta }}={\left({\mathbf{Z}}^T\mathbf{Z}\right)}^{-1}{\mathbf{Z}}^T\mathbf{y}$
  • Let $\hat{y}=Z\hat{\beta }=Hy$ denote the fitted values of $y$, where $\mathbf{H}=\mathbf{Z}{\left({\mathbf{Z}}^T\mathbf{Z}\right)}^{-1}{\mathbf{Z}}^T$ is called hat matrix.
  • Then the residuals $\widehat{ϵ}=y-\hat{y}=(I-H)y$ satisfy ${\mathbf{Z}}^T\widehat{\mathbf{ϵ}}=0$ and ${\hat{\mathbf{g}}}^T\widehat{\mathbf{ϵ}}=0$.
  • The residual sum of squares ${\sum }_{j=1}^n{\left(y_j-{\hat{\beta }}_0-{\hat{\beta }}_1{z}_{j1}-\cdots -{\hat{\beta }}_r{z}_{jr}\right)}^2={\widehat{\mathbf{ϵ}}}^T\widehat{\mathbf{ϵ}}={\mathbf{y}}^T\mathbf{y}-{\mathbf{y}}^{T}Z\hat{\mathbf{\beta }}$
• The coefficient of determination $R^2=\frac{{\sum }_{j=1}^n{\left({\hat{y}}_j-\bar{y}\right)}^2}{{\sum }_{j=1}^n{\left(y_j-\bar{y}\right)}^2}$ gives the proportion of the total variation in the $y_j$ ‘s explained by, or attributable to, the predictor variables $z_1,\dots ,z_r$.
  • Here $R^2$ equals 1 if the fitted equation passes through all the data points, so that ${\widehat{ϵ}}_j=0$ for all $j$.
  • In addition, $R^2=0$ if ${\hat{\beta }}_0=\bar{y}$ and ${\hat{\beta }}_1=\cdots ={\hat{\beta }}_r=0$. The predictor variables $z_1,\dots ,z_r$ have no influence on the response.
• The least squares estimator $\hat{\beta }={\left(Z^{T}Z\right)}^{-1}{Z}^{T}Y$ has $E(\hat{\mathbf{\beta }})=\mathbf{\beta }\text{ and }\mathrm{Cov}(\hat{\mathbf{\beta }})={\sigma }^2{\left({\mathbf{Z}}^{\top }\mathbf{Z}\right)}^{-1}$
  • The residuals $\widehat{\mathbf{ϵ}}$ have the properties $E(\widehat{\mathbf{ϵ}})=0\text{ and }\mathrm{Cov}(\hat{\mathbf{\epsilon }})={\sigma }^2(I-H)$
  • Also, $E\left({\widehat{\mathbf{ϵ}}}^T\widehat{\mathbf{ϵ}}\right)=(n-r-1){\sigma }^2$. Defining $s^2=\frac{{\widehat{\mathbf{ϵ}}}^T\widehat{\mathbf{ϵ}}}{n-r-1}=\frac{{\mathbf{Y}}^T(\mathbf{I}-\mathbf{H})\mathbf{Y}}{n-r-1}$
  • We have $E\left(s^2\right)={\sigma }^2$
  • Moreover, $\hat{\mathbf{\beta }}$ and $\widehat{\mathbf{ϵ}}$ are uncorrelated.

Inferences about the regression model:

• Let $\mathbf{Y}=\mathbf{Z}\mathbf{\beta }+\mathbf{ϵ}$ where $\mathbf{Z}$ has full rank $r+1$ and $\mathbf{ϵ}$ is distributed as $N_n\left(0,{\sigma }^2І\right)$. Then the maximum likelihood estimator of $\mathbf{\beta }$ is the same as the least squares estimator $\hat{\mathbf{\beta }}$.
  • Moreover, $\hat{\mathbf{\beta }}={\left({\mathbf{Z}}^{T}Z\right)}^{-1}{Z}^{T}Y\sim N_{r+1}\left(\beta ,{\sigma }^2{\left(Z^{T}Z\right)}^{-1}\right)$ and is distributed independently of the residuals $\widehat{ϵ}=Y-Z\hat{\beta }$.
  • Further, $n{\hat{\sigma }}^2={\widehat{\mathbf{ϵ}}}^T\widehat{\mathbf{ϵ}}\text{ is distributed as }{\sigma }^2{\chi }_{n-r-1}^2$ where ${\hat{\sigma }}^2$ is the maximum likelihood estimator of ${\sigma }^2$
• Let $Y=Z\beta +ϵ$, where $Z$ has full rank $r+1$ and $\mathbf{ϵ}$ is $N_n\left(0,{\sigma }^{2}I\right)$. Then a $100(1-\alpha )$ confidence region for $\beta$ is given by $(\mathbf{\beta }-\hat{\mathbf{\beta }}{)}^T{\mathbf{Z}}^{T}Z(\mathbf{\beta }-\hat{\mathbf{\beta }})\le (r+1)s^2{F}_{r+1,n-r-1}(\alpha )$
  • Also, simultaneous $100(1-\alpha )\%$ confidence intervals for the ${\beta }_i$ are given by ${\hat{\beta }}_i\mp \sqrt{\widehat{\mathrm{Var}}\left({\hat{\beta }}_i\right)}\sqrt{(r+1)F_{r+1,n-r-1}(\alpha )},i=0,\dots ,r$ where $\widehat{\mathrm{Var}}\left({\hat{\beta }}_i\right)$ is the diagonal element of $s^2{\left(Z^{T}Z\right)}^{-1}$ corresponding to ${\hat{\beta }}_i$.
• The confidence ellipsoid is centered at the maximum likelihood estimate $\hat{\mathbf{\beta }}$ and its orientation and size are determined by the eigenvalues and eigenvectors of $Z^{T}Z$.
  • If an eigenvalue is nearly zero, the confidence ellipsoid will be very long in the direction of the corresponding eigenvector.
  • Practitioners often use the intervals $\hat{\beta }\mp t_{n-r-1}\left(\frac{\alpha }{2}\right)\sqrt{\widehat{\mathrm{Var}}\left({\hat{\beta }}_i\right)}$ when searching for important predictor variables.

Likelihood ratio tests for regression parameters:

• Part of regression analysis is concerned with assessing the effects of particular predictor variables on the response variable. One null hypothesis of interest states that certain of the $z_j$‘s do not influence the response $Y$.
• These predictors will be labeled $z_{q+1},\dots ,z_r$. The statement that $z_{q+1},\dots ,z_r$ do not influence $Y$ translates into the statistical hypothesis $H_0:{\beta }_{q+1}=\cdots ={\beta }_r=0\text{ or }{H}_0:{\beta }_{(2)}=0$ where ${\mathbf{\beta }}_{(2)}^T=\left[{\beta }_{q+1},\dots ,{\beta }_r\right]$.
• Setting $Z=\left[\begin{array}{cc}{Z}_1& Z_2\\ n\times (q+1)& n\times (r-q)\end{array}\right]\beta =\left[\begin{array}{c}{\beta }_{(1)}\\ (q+1)\times 1\\ {\beta }_{(2)}\\ (r-q)\times 1\end{array}\right]$
  • we can express the general linear model as $Y=Z\beta +ϵ=Z_1{\beta }_{(1)}+Z_2{\beta }_{(2)}+ϵ$
• Define extra sum of squares $S{S}_{\text{res }}\left(Z_1\right)-S{S}_{\text{res }}(Z)$ to be ${\left(\mathbf{y}-Z_1{\hat{\mathbf{\beta }}}_{(1)}\right)}^T\left(\mathbf{y}-Z_1{\hat{\mathbf{\beta }}}_{(1)}\right)-(y-z\hat{\mathbf{\beta }}{)}^T(\mathbf{y}-\mathbf{z}\hat{\mathbf{\beta }})$
  • where ${\hat{\mathbf{\beta }}}_{(1)}={\left(Z_1^T{Z}_1\right)}^{-1}{Z}_1^{T}y$
• Let $Z$ have full rank $r+1$ and $\mathbf{ϵ}$ be distributed as $N_n\left(0,{\sigma }^2\mathbf{I}\right)$ The likelihood ratio test rejects $H_0$ if $\frac{S{S}_{\mathrm{res}}\left(Z_1\right)-S{S}_{\mathrm{res}}(Z)}{s^2(r-q)}>F_{r-q,n-r-1}(\alpha )$

Inferences from the estimated regression function

• Once an investigator is satisfied with the fitted regression model, it can be used to solve two prediction problems.
• Let $z_0^T=\left[1,z_{01},\dots ,z_{0r}\right]$ be selected values for the predictor variables.
• Let $Y_0$ denote the value of the response when the predictor variables have values $z_0^T$. According to the classical linear regression model, $E\left(Y_0\mid z_0\right)={\beta }_0+{\beta }_1{z}_{01}+\cdots +{\beta }_r{z}_{0r}=z_0^T\beta$
• Its least squares estimate is $z_0^T\hat{\mathbf{\beta }}$
• $z_0^T\hat{\mathbf{\beta }}$ is the unbiased linear estimator of $E\left(Y_0\mid z_0\right)$ with minimum variance $\mathrm{Var}\left(z_0^T\hat{\mathbf{\beta }}\right)=z_0^T{\left(Z^{T}Z\right)}^{-1}{z}_0{\sigma }^2$
  • If the errors $ϵ$ are normally distributed, then a $100(1-\alpha )\%$ confidence interval for $E\left(Y_0\mid z_0\right)=z_0^T\beta$ is $z_0^T\hat{\mathbf{\beta }}\mp t_{n-r-1}\left(\frac{\alpha }{2}\right)\sqrt{\left(z_0^T{\left(Z^{T}Z\right)}^{-1}{z}_0\right)s^2}$
• Prediction of a new observation, such as $Y_0$ at $z_0$, is more uncertain than estimating the expected value of $Y_0$.
  • $Y_0=z_0^T\beta +{ϵ}_0$where ${ϵ}_0\sim N\left(0,{\sigma }^2\right)$ and is independent of $ϵ$ and, hence, of $\hat{\mathbf{\beta }}$ and $s^2$.
• A new observation $Y_0$ has the unbiased predictor $z_0^T\hat{\mathbf{\beta }}={\hat{\beta }}_0+{\hat{\beta }}_1{z}_{01}+\cdots +{\hat{\beta }}_r{z}_{0r}$
  • The variance of the forecast error $Y_0-z_0^T\hat{\mathbf{\beta }}$ is $\mathrm{Var}\left(Y_0-z_0^T\hat{\mathbf{\beta }}\right)={\sigma }^2\left(1+z_0^T{\left(Z^{T}Z\right)}^{-1}{z}_0\right)$
  • When the errors $ϵ$ have a normal distribution, a $100(1-\alpha )\%$ prediction interval for $Y_0$ is given by $z_0^T\hat{\mathbf{\beta }}\mp t_{n-r-1}\left(\frac{\alpha }{2}\right)\sqrt{s^2\left(1+z_0^T{\left(Z^{T}Z\right)}^{-1}{z}_0\right)}$
• The prediction interval for $Y_0$ is wider than the confidence interval for estimating the value of the regression function $E\left(Y_0\mid z_0\right)$.
  • The additional uncertainty in forecasting $Y_0$, which is represented by the extra term $s^2$ in the expression $s^2\left(1+z_0^T{\left(Z^{T}Z\right)}^{-1}{z}_0\right)$, comes from the presence of the unknown error term ${ϵ}_0$.