π Lecture 15 β Multicollinearity
π Overview: This lecture explores the concept of multicollinearity in regression analysis, distinguishing between perfect and imperfect forms. It explains the sources, theoretical and practical consequences, and methods for detecting this common data problem, which is crucial for accurate model interpretation and valid inference.
ποΈ Topics Covered
The lecture begins by defining perfect and near/imperfect multicollinearity, illustrating perfect multicollinearity with a mathematical example showing parameter non-identifiability. It then presents an empirical example contrasting perfect and imperfect multicollinearity using Excel output. Sources of multicollinearity are listed, followed by theoretical consequences (unbiasedness, data deficiency) and practical consequences (large standard errors, low t-ratios, high RΒ²). The lecture concludes with detection methods: symptoms, correlation matrix, and Variance Inflation Factor (VIF)/Tolerance, including a decision table for severity.
π Lecture Summary
Perfect Multicollinearity: Definition
Perfect Multicollinearity is defined as an exact or perfect linear relationship between a pair or more of the explanatory variables. In a K-variable linear regression model Y = Ξ²βXβ + Ξ²βXβ + Ξ²βXβ + Ξ²βXβ + ... + Ξ²βXβ + eα΅’ (where Xβ = 1 for all observations to allow the intercept term), an exact linear relationship exists if Ξ»βXβ + Ξ»βXβ + Ξ»βXβ + ... + Ξ»βXβ = 0, where Ξ»α΅’ cannot all be zero at the same time.
π Definition β Perfect Multicollinearity: An exact linear relationship between two or more explanatory variables, where one variable can be expressed as a perfect linear combination of the others.
Near or Imperfect Multicollinearity: Definition
In practice, we rarely observe perfect multicollinearity but rather a degree of multicollinearity. For the model Y = Ξ²βXβ + Ξ²βXβ + Ξ²βXβ + Ξ²βXβ + ... + Ξ²βXβ + eα΅’, a near or imperfect linear relationship exists if Ξ»βXβ + Ξ»βXβ + Ξ»βXβ + ... + Ξ»βXβ + vα΅’ = 0, where vα΅’ is a stochastic error term.
π Definition β Near/Imperfect Multicollinearity: A situation where explanatory variables are highly, but not perfectly, correlated, meaning one variable is approximately a linear combination of others plus a random disturbance.
Understanding Perfect Multicollinearity
For perfect multicollinearity, if Ξ»βXβ + Ξ»βXβ + Ξ»βXβ + ... + Ξ»βXβ = 0 and all Ξ»α΅’ except Ξ»β and Ξ»β are zero, then Xβ = (Ξ»β/Ξ»β)Xβ (e.g., Xβ = 5Xβ). Both variables can be expressed as linear combinations of each other. Substituting into the regression: Y = Ξ²β + Ξ²βXβ + Ξ²βXβ + eα΅’ becomes Y = Ξ²β + Ξ²β(5Xβ) + Ξ²βXβ + eα΅’ = Ξ²β + (5Ξ²β + Ξ²β)Xβ + eα΅’ = Ξ²β + Ξ²βXβ + eα΅’. This is a simple regression where Y depends only on Xβ. Hence, we cannot estimate all the parameters.
π Formula: Xβ = (Ξ»β/Ξ»β)Xβ β One explanatory variable is a perfect multiple of another.
π Example: With Xβ = 5Xβ and the model Y = Ξ²β + Ξ²βXβ + Ξ²βXβ + eα΅’, substituting gives Y = Ξ²β + (5Ξ²β + Ξ²β)Xβ + eα΅’ = Ξ²β + Ξ²βXβ + eα΅’. Only the combined coefficient Ξ²β can be estimated, not Ξ²β or Ξ²β individually. Software may drop one variable (e.g., Xβ) and show a zero coefficient, effectively estimating a simple regression of Y on Xβ.
π‘ Why this matters: Perfect multicollinearity makes it impossible to estimate unique coefficients for the correlated variables, rendering the model unidentified.
An example dataset is provided with 10 observations for Y, Xβ, and Xβ where Xβ = 5Xβ. The Excel LINEST output shows coefficients for Xβ (0) and Xβ (0.8826), an intercept (-31.5), RΒ² (0.974), standard errors, and t-statistics (17.17 for Xβ). The zero coefficient for Xβ confirms it was dropped due to perfect collinearity.
The data is then slightly changed to create imperfect multicollinearity (e.g., Y: 10, Xβ: 10, Xβ: 52). The new LINEST output estimates all parameters: Xβ coefficient = 1.49 (se=0.86, t=1.74), Xβ coefficient = -2.93 (se=4.24, t=-0.69), intercept = -33.31 (se=4.85, t=-6.86), RΒ² = 0.98. Note that standard errors are large, t-statistics are low, coefficients may not seem significant, but the F-test shows a good fit.
Sources of Multicollinearity
- The data collection method: Sampling over a limited range or subgroups of a population can create correlation that only exists in that subgroup.
- Natural Constraints on the model or population: For example, if we regress GDP on exports and imports, high imports normally imply high exports as well; in a sample survey, both income and status of house are included as explanatory variables; model specification, such as adding polynomial terms, especially when X has a small range.
- Over-determined model: A large number of explanatory variables with very low degrees of freedom.
- Common Time Series Trend: The explanatory variables may share a common trend over time.
Theoretical consequences of Multicollinearity
- OLS estimators remain unbiased: There is no violation of the BLUE property; estimators are still unbiased and best linear unbiased.
- Interpretation of coefficients is not independent: A simple ceteris paribus interpretation does not seem to be valid.
- Multicollinearity is a data deficiency problem: Sometimes it is difficult to increase the sample size due to cost and time constraints.
- Multicollinearity is a sample phenomenon: This is especially true with non-experimental data that occurs naturally, like GDP and its determinants.
- We need larger samples: Larger samples are required than without multicollinearity.
Practical consequences of Multicollinearity
- Large variances and standard errors of coefficients: This occurs when the correlation between pairs of explanatory variables is high, leading to a fall in the precision of estimators.
- Wider confidence intervals: This is a direct consequence of larger standard errors.
- Insignificant t-ratios: As a consequence of larger standard errors, the null hypothesis
Hβ: Ξ²α΅’ = 0tends to be accepted. - High RΒ² but low t: This seems to be a contradictory result.
- Estimators and standard errors are very sensitive to changes in data: Model estimates are unstable.
- Wrong signs of coefficients: The estimated signs may not align with economic and finance theory.
- Difficulty in assessing the individual contribution of regressors: Due to correlated regressors, it is hard to isolate each variable's unique contribution.
Detecting Multicollinearity
Remember that multicollinearity is a question of degree, not just presence; it is a sample phenomenon; and it is a data deficiency problem. There are three detection methods:
- Look at the symptoms/indicators: High RΒ² but low t-statistics, large standard errors, and individual variables not being significant even though the model seems to be a good fit.
- Look at the Correlation Matrix: High pairwise correlations between the explanatory variables (e.g., greater than 0.8). A sample correlation matrix is provided for data containing Y, Xβ, Xβ, and Xβ.
- Calculate VIF or Tolerance:
- Method: Run auxiliary regressions where each explanatory variable is regressed on all other explanatory variables, and find the coefficient of determination (
RΒ²). - You can test the significance of
RΒ²using an F-test or goodness-of-fit test. - Estimate Tolerance or VIF.
- Method: Run auxiliary regressions where each explanatory variable is regressed on all other explanatory variables, and find the coefficient of determination (
π Formulas:
Tolerance = 1 - RΒ²VIF = 1 / (1 - RΒ²)β Variance Inflation Factor measures how much the variance of the coefficient is inflated due to collinearity. π Key Definitions:- Auxiliary Regression: A regression of one explanatory variable on all other explanatory variables to calculate
RΒ²for VIF. - Tolerance:
1 - RΒ²from the auxiliary regression; low values indicate high multicollinearity. - VIF (Variance Inflation Factor):
1 / Tolerance; a measure of how much the variance of an estimated coefficient is inflated due to collinearity.
Decision Table for Degree of Multicollinearity
| Severity | Tolerance | VIF |
|---|---|---|
| Problematic Multicollinearity | β€ 0.1 | β₯ 10 |
| Mild Multicollinearity | 0.1 < Tolerance < 0.2 | 5 < VIF < 10 |
| Nearly No Multicollinearity | β₯ 0.2 | β€ 5 |
Important Points to note:
- VIF or Tolerance for a variable is calculated by the
RΒ²of the auxiliary regression of that variable regressed on all other explanatory variables. - One single variable is not responsible for multicollinearity, so we may estimate a number of auxiliary regressions equal to one less than the number of explanatory variables.
β Key Takeaways
Multicollinearity is a data problem of degree, not just presence, where OLS estimators remain unbiased but lose precision, leading to large standard errors, low t-ratios, and potentially unstable coefficient estimates. Perfect multicollinearity is a crippling exact linear relationship that prevents unique parameter estimation, whereas imperfect multicollinearity is common and manifests when a model has high RΒ² but insignificant individual t-tests. Detection relies on three methods: observing symptoms (high RΒ², low t), examining pairwise correlations in a matrix, and using auxiliary regressions to compute VIF values, with a VIF above 10 indicating a problematic level of multicollinearity.
π§ Quick Revision Questions
- What is the key difference between perfect and imperfect (near) multicollinearity?
- Why can OLS estimators remain unbiased under multicollinearity, yet their standard errors become large?
- List and explain the three main methods for detecting multicollinearity in a regression model.
- How is the Variance Inflation Factor (VIF) calculated, and what does a VIF value of 10 or more indicate?
- According to the lecture, what is the primary practical consequence of high multicollinearity on the t-ratios of the estimated coefficients?
π Lecture 16 β Multicollinearity: Remedial Measures
π Overview: This lecture presents practical solutions for dealing with multicollinearity in regression models. It covers several remedial measures ranging from doing nothing to dropping variables or transforming them, illustrated with detailed business and economics examples using real datasets. Understanding these remedies is crucial for obtaining reliable coefficient estimates when predictors are highly correlated.
ποΈ Topics Covered
The lecture covers seven remedial measures for multicollinearity: doing nothing, dropping one variable, transforming variables, getting additional data, combining cross-section and time series, using panel data, and advanced methods like ridge regression. It then presents two detailed business examples: first using data on quantity demanded with prices and income, and second using Pakistani GDP data with population, investment, and trade variables. Each example demonstrates the diagnostic process using VIF, F-statistics, and correlation matrices, followed by application of remedial measures.
π Lecture Summary
1. Do nothing
If multicollinearity is mild or if the purpose is only forecasting, you may choose to do nothing. If data deficiency is the problem and we have no choice over data, it is better to try to increase the data by extending the sample if possible. Multicollinearity is not a problem if theory permits us to estimate the missing coefficient β e.g., in Cobb-Douglas production function, if we assume constant returns to scale, either of alpha or beta can be estimated if one is estimated by regression.
2. Drop one of the variables
Drop the variable that is less significant or drop the one with larger VIF. But this may lead to wrong model specification or may go against theoretical considerations (e.g., dropping price of substitute in demand function). An example of dropping variables is of import and export in GDP equation.
3. Transform the variable
Combine the variables β e.g., just add exports and imports to get a new variable labeled as openness. Another option is to convert the variables (import = f(GNP, CPI) can divide by CPI to get real imports = f(real GNP)); but error term may become heteroskedastic. Another way is to use first difference form (loss of one observation):
[ Y_t - Y_{t-1} = \beta_2(X_{2t} - X_{2,t-1}) + \beta_3(X_{3t} - X_{3,t-1}) + v_t ]
This may not be appropriate in cross sectional data β it has no sense.
Other options include: 4. Get additional data and increase the sample size 5. Combine cross section and time series (pool) 6. Use of panel data 7. Use ridge regression, factor analysis, etc.
Multicollinearity: Examples from Business and Economics
Consider the data given in MC.xlsx containing quantity demanded, prices, monthly income in thousands, and prices of two different substitutes. First example has 50 observations. Let us consider a small example with 10 observations:
| Y | Xβ | Xβ | Xβ |
|---|---|---|---|
| 10 | 10 | 52 | 26 |
| 20 | 11 | 54 | 23 |
| 30 | 16 | 77 | 23 |
| 40 | 15 | 78 | 25 |
| 50 | 19 | 93 | 22 |
| 60 | 21 | 104 | 21 |
| 70 | 22 | 109 | 19 |
| 80 | 24 | 119 | 18 |
| 90 | 27 | 136 | 19 |
| 100 | 31 | 153 | 17 |
Auxiliary Regressions produced using Microsoft Excel:
π Definition β Auxiliary Regression: A regression of one explanatory variable on the other explanatory variables to detect multicollinearity β a high RΒ² indicates multicollinearity exists.
Regression: Xβ on Xβ & Xβ
- First two variables show MC
- Look at F
- Look at T/VIF
[ X_1 = 4.977 + 0.188X_2 - 0.176X_3 ] RΒ² = 0.998 F = 1848.5 Tolerance = 0.002 VIF = 500
Regression: Xβ on Xβ & Xβ [ X_2 = -22.11 + 5.24X_1 + 0.794X_3 ] RΒ² = 0.9978 F = 1608.6 Tolerance = 0.0022 VIF = 454.5455
Regression: Xβ on Xβ & Xβ [ X_3 = 28.97 - 2.03X_1 + 0.3296X_2 ] RΒ² = 0.89 F = 28.43 Tolerance = 0.11 VIF = 9.090909
π‘ Why this matters: VIF values above 10 (or tolerance below 0.1) indicate serious multicollinearity β Xβ and Xβ show extremely high VIF values of 500 and 454.5.
Multicollinearity: Example-1 from Business and Economics
Using MC.xlsx, first look at the correlation matrix:
| Q | Price | income (000) | price of sub.-1 | price of sub.-2 | |
|---|---|---|---|---|---|
| Q | 1 | ||||
| Price | -0.94144 | 1 | |||
| income (000) | -0.04567 | 0.309407 | 1 | ||
| price of sub.-1 | 0.071323 | 0.234771 | 0.793596 | 1 | |
| price of sub.-2 | 0.017076 | 0.274298 | 0.79072 | 0.960501 | 1 |
Model 1 (N=10) R Square = 0.9106, F = 12.73**
| Coefficients | VIF | Comment | |
|---|---|---|---|
| Intercept | 977.7725*** | ||
| Price | -0.2560*** | 2.5 | No MC |
| Income | 2.33** | 5.12 | No MC |
| Price of Substitute-1 | -0.2384 | 33.04 | MC, wrong sign, insignificant |
| Price of Substitute-2 | 0.6310 | 22.75 | MC, insignificant |
Model 2 (N=50) R Square = 0.978, F = 508.41*
| Coefficients | VIF | Comment | |
|---|---|---|---|
| Intercept | 993.6106* | ||
| Price | -0.499* | 1.13 | No MC |
| Income | 0.498*** | 2.88 | No MC |
| Price of Substitute-1 | 0.1924* | 13.74 | MC |
| Price of Substitute-2 | -0.0912 | 13.6 | MC, wrong sign |
Note: In the second model VIF has decreased but still greater than 10.
Model 3 (N=50) β Dropping Substitute-2: R Square = 0.978, F = 687.98*
| Coefficients | VIF | Comment | |
|---|---|---|---|
| Intercept | 985.77* | ||
| Price | -0.5* | 1.13 | No MC |
| Income | 0.4768*** | 2.88 | No MC |
| Price of Substitute-1 | 0.1663* | 13.6 | No MC |
Multicollinearity: Example-2 from Business and Economics
Data on GDP, Population, GFCF, Exports, Imports (2000-2012 Pakistan) downloaded from WDI.
Model with all variables: Dependent Variable: GDP R-square = 0.9927, F = 271.4
| Explanatory Variables | Significance | VIF |
|---|---|---|
| Population | 1367.5* | 4.61 |
| GFCF | 0.196 | 7.45 |
| Exports | -0.247 | 10.28 |
| Imports | 0.879 | 14.36 |
| Openness | ||
| constant | large value |
Result: No significance, high VIF, wrong signs
After dropping imports: Dependent Variable: GDP R-square = 0.9897, F = 289.17
| Explanatory Variables | Significance | VIF |
|---|---|---|
| Population | 1307.9* | 4.53 |
| GFCF | 1.118** | 4.15 |
| Exports | 0.254 | 2.82 |
| Imports | ||
| Openness | ||
| constant | large value* |
New Results show: better significance, low VIF, signs correct now
Another option β Transform variables: Generate a new variable, volume of trade as a proxy for openness, defined as sum of imports and exports.
Dependent Variable: GDP R-square = 0.7947, F = 19.36
| Explanatory Variables | Significance | VIF |
|---|---|---|
| GFCF | 1.06 | 5.36 |
| Openness | 1.809** | 5.36 |
| constant | large value |
Results: openness is significant, low VIF indicates no multicollinearity, signs of coefficients are correct now β however we may need other variables.
β Key Takeaways
The most critical things to remember are: first, multicollinearity can be addressed through seven possible remedial measures β doing nothing (if mild or for forecasting), dropping variables, transforming them, increasing sample size, pooling data, using panel data, or advanced methods like ridge regression. Second, auxiliary regressions produce high RΒ² values and VIF exceeding 10 indicate serious multicollinearity β in the first example, the price of substitute variables had VIF values of 33 and 22.75, dropping one variable to VIF 13.6 (still above 10). Third, increasing sample size from 10 to 50 observations reduced VIF from 33.04 to 13.74 for Substitute-1. Fourth, in the Pakistan GDP example, dropping imports (VIF=14.36) improved all other variables' significance and signs. Fifth, transforming variables by creating an openness measure (exports + imports) successfully eliminated multicollinearity but reduced R-square from 0.99 to 0.79.
π§ Quick Revision Questions
- When is it appropriate to "do nothing" about multicollinearity?
- What is the formula for first differencing to transform variables, and what is its limitation for cross-sectional data?
- In Example-1, what happened to the VIF values when sample size increased from 10 to 50 observations?
- In the Pakistan GDP example, what was the VIF for imports before dropping it, and what happened to the significance of exports after dropping imports?
- How does creating an "openness" variable (sum of exports and imports) affect the regression results compared to including both separately?
π Lecture 18 β Detection of Heteroskedasticity: Formal Tests
π Overview: This lecture covers formal statistical tests for detecting heteroskedasticity in regression models. It explains multiple testing procedures including the Goldfeld-Quandt test, Park LM test, Harvey-Godfrey test, Glejser test, Breusch-Pagan test, and the White test, detailing their steps, assumptions, and limitations.
ποΈ Topics Covered
The lecture begins with the Goldfeld-Quandt test for heteroskedasticity detection using ordered data and subsample regressions, followed by its example and drawbacks. It then covers the Park LM test using logarithmic auxiliary regressions, the Harvey-Godfrey test using level variables, the Glejser test with multiple functional forms, and the Breusch-Pagan test with squared residuals. The lecture concludes with the White test, which includes squared terms and cross-products, and discusses the sensitivity of LM tests to normality violations.
π Lecture Summary
The Goldfeld-Quandt Test: Stephen Goldfeld, Richard Quandt
The following steps are performed for this test.
Step 1. Arrange the data from small to large values of the explanatory variable Xβ±Ό (the one we suspect responsible for HSK).
Step 2. Omit the middle C observations (C = roughly 20% observations).
Step 3. Run two separate regressions, one for small values of Xβ±Ό and one for large values of Xβ±Ό, omitting C middle observations and record the residual sum of squares RSS for each regression: RSSβ for large values of Xβ±Ό and RSSβ for small values of Xβ±Ό.
Step 4. Calculate the ratio: F = RSSβ / RSSβ
Degrees of Freedom = (NβC)/2 β K both in the numerator and the denominator, where N = total number of observations, C is the number of omitted observations, and K is the Number of explanatory variables + one.
Step 5. Test if F > F_Ξ±, (NβC)/2βK, (NβC)/2βK to reject homoskedasticity (Hβ).
π Definition β Goldfeld-Quandt Test: A formal test for heteroskedasticity that compares the residual variance from two separate regressions on ordered data subsamples. π Formula: F = RSS_high / RSS_low β Tests if variance is larger in one subsample π Example: Hβ: Homoskedasticity, Hβ: Heteroskedasticity, Ξ± = 0.05 or 0.01. Test statistic F = RSS_high / RSS_low. If you think var(e) is increasing function of X. Region of Rejection: F > F_Ξ±, (NβC)/2βK, (NβC)/2βK. RSS = Residual Sum of Squares, N = Number of observations, C = central observations excluded, K = Number of parameters estimated = Number of explanatory variables plus one.
π‘ Why this matters: The Goldfeld-Quandt test is intuitive but limited to single-variable heteroskedasticity detection.
The Goldfeld-Quandt Test: Drawbacks
It cannot handle situations where several variables jointly cause Heteroskedasticity. There is no fix rule to know how many middle observations should be excluded. It is also possible that the difference in variance of errors may be observed in subsamples with different number of observations. The middle C observations are lost. It accounts only for linear relationship of independent variable and the variance of errors.
The Park LM Test
The following steps are performed in this test.
Step 1. Run the required regression e.g. Y = Ξ²β + Ξ²βXβ + Ξ²βXβ + ... + Ξ²βXβ + e, obtain residuals e, compute ln eΒ².
Step 2. Run the auxiliary regression: ln eΒ² = Ξ±β + Ξ±βlnXβ + Ξ±βlnXβ + ... + Ξ±βlnXβ + u
Step 3. Compute LM = NΒ·RΒ² (N and RΒ² are from the auxiliary regression).
Step 4. If LM > ΟΒ²_βββ then reject Null hypothesis and conclude that there is significant evidence of Heteroskedasticity.
π Definition β Park LM Test: A test for heteroskedasticity using logarithmic auxiliary regression of squared residuals on log-transformed independent variables.
Harvey-Godfrey Test
The following steps are performed in this test.
Step 1. Run the required regression e.g. Y = Ξ²β + Ξ²βXβ + Ξ²βXβ + ... + Ξ²βXβ + e, obtain residuals e, compute ln eΒ².
Step 2. Run the auxiliary regression [assume ΟΒ² = exp(Ξ±β + Ξ±βXβ)]: ln eΒ² = Ξ±β + Ξ±βXβ + Ξ±βXβ + ... + Ξ±βXβ + u
Step 3. Compute LM = NΒ·RΒ² (N and RΒ² are from the auxiliary regression).
Step 4. If LM > ΟΒ²_βββ then reject Null hypothesis and conclude that there is significant evidence of Heteroskedasticity.
π Definition β Harvey-Godfrey Test: A test for heteroskedasticity using logarithmic auxiliary regression on the original (untransformed) independent variables.
Glejser Test
The following steps are performed in this test.
Step 1. Run the required regression e.g. Y = Ξ²β + Ξ²βXβ + e, obtain residuals e, compute ln eΒ².
Step 2. Run the auxiliary regressions [assume ΟΒ² = exp(Ξ±β + Ξ±βXβ)]: |eα΅’| = Ξ±β + Ξ±βXβ + uα΅’ |eα΅’| = Ξ±β + Ξ±ββXβ + uα΅’ |eα΅’| = Ξ±β + Ξ±β(1/Xβ) + uα΅’
Step 3. Compute LM = NΒ·RΒ² (N and RΒ² are from the auxiliary regression).
Step 4. If LM > ΟΒ²_βββ then reject Null hypothesis and conclude that there is significant evidence of Heteroskedasticity.
π Definition β Glejser Test: A test for heteroskedasticity that uses the absolute value of residuals and tests multiple functional forms of the relationship with independent variables.
Breusch-Pagan Test
The following steps are performed in this test.
Step 1. Run the required regression e.g. Y = Ξ²β + Ξ²βXβ + Ξ²βXβ + ... + Ξ²βXβ + e, obtain residuals e, compute ln eΒ².
Step 2. Run the auxiliary regression [assume ΟΒ² = exp(Ξ±β + Ξ±βXβ)]: eΒ² = Ξ±β + Ξ±βXβ + Ξ±βXβ + ... + Ξ±βXβ + u
Step 3. Compute LM = NΒ·RΒ² (N and RΒ² are from the auxiliary regression).
Step 4. If LM > ΟΒ²_βββ then reject Null hypothesis and conclude that there is significant evidence of Heteroskedasticity.
OR test F-statistic for the above regression in step 2 (Goodness of fit).
π Definition β Breusch-Pagan Test: A test for heteroskedasticity that regresses squared residuals on the original independent variables in linear form.
Problems with Breusch-Pagan and others
Specification of model for variance dependence is needed e.g. Breusch Pagan assume linear relation. If the errors are not normally distributed, then these tests may not be valid. Breusch Pagan Test has been shown to be sensitive to any violation of the normality assumption. Three other popular LM tests: the Glejser test; the Harvey-Godfrey test, and the Park test, are also sensitive to such violations.
The White Test: Most popular
The following steps are performed in this test.
Step 1. Run the required regression e.g. Y = Ξ²β + Ξ²βXβ + Ξ²βXβ + e, obtain residuals e, compute ΓͺΒ².
Step 2. Run the auxiliary regression: eΒ² = Ξ±β + Ξ±βXβ + Ξ±βXβ + Ξ±βXβΒ² + Ξ±βXβΒ² + Ξ±β XβXβ + u (Include all square and product terms)
OR Run the regression: eΒ² = Ξ³β + Ξ³βΕΆ + Ξ³βΕΆΒ² + v
Step 3. Compute LM = NΒ·RΒ² (N and RΒ² are from the auxiliary regression) OR Compute the F-Statistic.
Step 4. If LM > ΟΒ²_βββ then reject Null hypothesis and conclude that there is significant evidence of Heteroskedasticity OR test F-statistic for the above regression in step 2 (Goodness of fit).
π Definition β White Test: The most popular test for heteroskedasticity that includes squared terms and cross-products of independent variables in the auxiliary regression, without requiring a specific functional form for the variance.
β Key Takeaways
The Goldfeld-Quandt test requires ordering data by the suspected variable and comparing RSS from two subsamples, but it cannot handle multiple heteroskedasticity sources and loses middle observations. The Park, Harvey-Godfrey, Glejser, and Breusch-Pagan tests are LM tests using auxiliary regressions with NΒ·RΒ² statistic, but they require specifying the variance model and are sensitive to non-normality of errors. The White test is the most popular because it does not require specifying a particular functional form for heteroskedasticity and includes squared and cross-product terms in the auxiliary regression. All LM tests compare the test statistic to a chi-square distribution with (Kβ1) degrees of freedom. The Breusch-Pagan test assumes a linear relationship between squared residuals and independent variables, making it vulnerable to misspecification.
π§ Quick Revision Questions
- What are the five steps of the Goldfeld-Quandt test and how is the F-statistic calculated?
- What is the main drawback of the Goldfeld-Quandt test regarding multiple explanatory variables?
- How does the White test differ from other LM tests in terms of the auxiliary regression specification?
- Why are the Breusch-Pagan, Glejser, Harvey-Godfrey, and Park tests sensitive to violations of normality?
- What is the critical value and degrees of freedom used for all LM-based heteroskedasticity tests?
π Lecture 19 β Examples of detecting Heteroskedasticity
π Overview: This lecture demonstrates how to apply various formal tests for detecting heteroskedasticity in regression models using real data examples. It also covers methods for handling heteroskedasticity when detected, including model transformation, White standard errors, and Weighted Least Squares (WLS). Understanding these techniques is crucial for ensuring valid statistical inference in econometric analysis.
ποΈ Topics Covered
The lecture begins with the general procedure for all heteroskedasticity tests including null and alternative hypotheses, significance levels, test statistics (LM and F), and rejection regions. It then works through detailed examples of five different tests: the Goldfeld-Quandt test (revision), the Park LM test, the Harvey-Godfrey test, the Glesjer test, and the Breusch Pagan test, followed by the White test with two independent variables. The second half covers three methods for handling heteroskedasticity: changing model specification, using Huber/White standard errors, and applying GLS/WLS with known and unknown error variance, including transformation examples and a worked example.
π Lecture Summary
General Procedure for all the tests
The general procedure for testing heteroskedasticity begins with setting up the null hypothesis (H_0): Homoskedasticity (constant error variance) against the alternative hypothesis (H_1): Heteroskedasticity (non-constant error variance), typically using a significance level (\alpha = 0.05) or (\alpha = 0.01). The test statistic can be either the Lagrange Multiplier (LM) statistic calculated as (LM = N \cdot R^2), or the F-statistic calculated as (F = \frac{R^2/(1-R^2)}{(N-K)/(K-1)}). For the Goldfeld-Quandt test, the F-statistic is calculated as (F = \frac{RSS_{high}}{RSS_{low}}). The region of rejection for the LM test is when (LM > \chi^2_{\alpha, K-1}), and for the F-test when (F > F_{\alpha, K-1, N-K}) (or for Goldfeld-Quandt, (F > F_{\alpha, \frac{N-C}{2}-K, \frac{N-C}{2}-K})). All tests follow the same decision rule: if the test statistic exceeds the critical value, reject the null hypothesis and conclude that heteroskedasticity is present.
π Definition β Homoskedasticity: The assumption that the variance of the error terms is constant across all observations, i.e., (Var(e_i) = \sigma^2) for all (i).
π Definition β Heteroskedasticity: A violation of the classical linear regression assumption where the variance of the error terms is not constant across observations.
π Formula: LM Test Statistic: (LM = N \cdot R^2) β The sample size multiplied by the R-squared from the auxiliary regression. π Formula: F-Test Statistic: (F = \frac{R^2/(1-R^2)}{(N-K)/(K-1)}) β The ratio of explained to unexplained variation, adjusted for degrees of freedom.
The Goldfeld - Quandt Test: Example Revision
The Goldfeld-Quandt test is revisited as a revision example. This test involves ordering observations by the variable suspected to cause heteroskedasticity, omitting some central observations (C), and then running separate regressions on the low-variance and high-variance subsamples. The test statistic is the ratio of the Residual Sum of Squares (RSS) from the high-variance group to the low-variance group: (F = \frac{RSS_{high}}{RSS_{low}}). If this ratio exceeds the critical F-value, heteroskedasticity is present.
Example: The Park LM Test (ln(square of errors) and ln(income))
The Park LM test uses the natural logarithm of the squared errors as the dependent variable in an auxiliary regression. First, run the original regression of Saving on Income. Then generate the squared errors and take their natural logarithm, and also take the natural logarithm of the income variable. Run the auxiliary regression: (ln(e^2) = \alpha_0 + \alpha_1 ln(Income) + u). From the auxiliary regression, (R^2 = 0.0634). The LM statistic is (LM = N \cdot R^2 = 13(0.0634) = 0.8243). The critical Chi-square value at 5% with 1 degree of freedom is (CHISQ.INV.RT(0.05,1) = 3.84146). Since (LM (0.8243) < \chi^2_{0.05,1} (3.8415)), we cannot reject (H_0) and conclude that errors are not heteroskedastic. The F-test confirms this: (F = \frac{R^2/(1-R^2)}{(N-K)/(K-1)} = 0.7446), and (F_{0.05,1,11} = 4.8443), so (F < F_{critical}), again we cannot reject homoskedasticity.
π Example: Park LM Test Data β For 13 observations of Savings and Income, the squared errors range from 1,761.62 to 1,418,726.13, and the ln(eΒ²) values range from 7.4740 to 14.1653. The auxiliary regression of ln(eΒ²) on ln(Income) yields RΒ² = 0.0634, indicating that only 6.34% of the variation in ln(eΒ²) is explained by ln(Income). Both the LM test (0.8243 < 3.8415) and F-test (0.7446 < 4.8443) fail to reject homoskedasticity.
Example: Harvey-Godfrey Test for Heteroskedasticity
The Harvey-Godfrey test differs from the Park test in that it uses the original Income variable (not logged) as the explanatory variable in the auxiliary regression. After running the original regression of Savings on Income, find the errors and then (ln(square \ of \ errors)). Run the auxiliary regression: (ln(e^2) = \alpha_0 + \alpha_1 Income + u). From this auxiliary regression, (R^2 = 0.087076). The LM statistic is (LM = N \cdot R^2 = 13(0.087076) = 1.13199). The critical Chi-square value is 3.84146. Since (LM (1.132) < \chi^2_{0.05,1} (3.8415)), we cannot reject (H_0) and conclude that errors are not heteroskedastic using the LM test. Interestingly, the F-test gives (F = 1.0492) and (F_{critical} = 4.8443), so (F < F_{critical}), also indicating no heteroskedasticity. In this example, both tests agree that heteroskedasticity is not present, though the lecture text notes that sometimes the LM and F-tests can give conflicting conclusions.
π Example: Harvey-Godfrey Test β The auxiliary regression of ln(squared errors) on Income (not logged) gives RΒ² = 0.0871. LM = 13 Γ 0.0871 = 1.132, which is less than the critical value of 3.8415, so we cannot reject the null hypothesis of homoskedasticity. The F-statistic (1.049) is also less than the critical value (4.844), confirming the conclusion.
Example: Glesjer's Test for Heteroskedasticity
The Glesjer test is more flexible because it allows for different functional forms of the relationship between the absolute errors and the independent variable. After running the original regression of Savings on Income, find the errors, then create variables (1/X) and (\sqrt{X}). Run three different auxiliary regressions:
- Regression 1: (|e_i| = \alpha_0 + \alpha_1 X + u_i) β RΒ² = 0.11666, LM = 1.51657
- Regression 2: (|e_i| = \alpha_0 + \alpha_1 \sqrt{X} + u_i) β RΒ² = 0.10218, LM = 1.3284
- Regression 3: (|e_i| = \alpha_0 + \alpha_1 (1/X) + u_i) β RΒ² = 0.04528, LM = 0.5887
For all three regressions, the LM statistic is less than the critical value of 3.84146, so we have strong evidence not to reject (H_0) in all three cases. This means that regardless of which functional form we assume for the relationship between errors and X, we conclude that errors are not heteroskedastic.
Example: Breusch Pagan Test for Heteroskedasticity
The Breusch Pagan test uses the square of the errors (not logged) as the dependent variable in the auxiliary regression on the original independent variable(s). After running the original regression of Savings on Income, find the errors, then square them. Run the auxiliary regression: (e^2 = \alpha_0 + \alpha_1 Income + u). From this auxiliary regression, (R^2 = 0.143182). The LM statistic is (LM = N \cdot R^2 = 13(0.143182) = 1.86136). The critical Chi-square value is 3.84146. Since (LM (1.861) < \chi^2_{0.05,1} (3.8415)), we cannot reject (H_0) and conclude that errors are not heteroskedastic. The F-statistic is (F = 1.8382) with a critical value of (F_{0.05,1,11} = 4.8443). Since (F < F_{critical}), the F-test also indicates no heteroskedasticity.
Example: White Test for Heteroskedasticity - version 1
The White test is the most general test because it uses the squared errors as the dependent variable and regresses them on the original independent variables, their squares, and their cross-products. This test requires at least two independent variables. First, run a regression of Savings on Income and interest rate, find the errors, then square them. Then run the auxiliary regression: (e^2 = \alpha_0 + \alpha_1 X_1 + \alpha_2 X_2 + \alpha_3 X_1 \cdot X_2 + \alpha_4 X_1^2 + \alpha_5 X_2^2 + u). From this auxiliary regression, (R^2 = 0.5021). The LM statistic is (LM = N \cdot R^2 = 13(0.5021) = 6.5273). The critical Chi-square value with 5 degrees of freedom is (CHISQ.INV.RT(0.05,5) = 11.0705). Since (LM (6.5273) < \chi^2_{0.05,5} (11.0705)), we cannot reject (H_0) and conclude that errors are not heteroskedastic. The F-test gives (F = 0.4706) with critical value (F_{0.05,4,7} = 4.1203), also indicating no heteroskedasticity.
π‘ Why this matters: The White test is particularly powerful because it can detect heteroskedasticity that depends on the squares and interactions of variables, not just their levels. However, it uses many degrees of freedom, which can be a problem with small samples.
Handling Heteroskedasticity
When heteroskedasticity is detected, three main methods can be used to address it:
Method 1: Change the model specification / transform variables. The relationship between variables may not be linear, so consider transforming the model. Some important variable may be missing, so find and use them. If there are subgroup differences, use dummy variables. If possible, use panel data techniques.
Method 2: Use Huber / White Standard Errors. This type of standard error is also called Heteroskedasticity consistent standard errors. With heteroskedasticity, the standard errors are incorrect, so t-statistics would be incorrect (another formula is required for SE). White's idea: Use simple OLS and correct the standard errors. For a simple regression line, the White estimator of error variance is:
- (White \ Variance(\hat{\beta}_1) = \frac{\sum (X_i - \bar{X})^2 \hat{\sigma}_i^2}{(\sum (X_i - \bar{X})^2)^2})
- Estimate (\hat{\sigma}_i^2 = e_i^2) (using the squared residual for each observation as the estimate of its variance)
Method 3: GLS / WLS (Generalized/Weighted Least Squares). This method has two cases: one where the variance of errors is known, and another where the variance of errors is not known.
π Definition β GLS/WLS (variance of errors known): Consider the model (Y_i = \beta_0 + \beta_1 X_i + e_i). Suppose we know (\sigma_i), i.e., the standard deviation of the error for each observation. Divide all variables by this error standard deviation: (\frac{Y_i}{\sigma_i} = \frac{\beta_0}{\sigma_i} + \frac{\beta_1 X_i}{\sigma_i} + \frac{e_i}{\sigma_i}). The new model's variance is: (Var(\frac{e_i}{\sigma_i}) = \frac{1}{\sigma_i^2} Var(e_i) = \frac{\sigma_i^2}{\sigma_i^2} = 1). This transformed model has constant variance (homoskedastic) errors.
π Definition β GLS/WLS (variance of errors NOT known): Make assumptions about the variance structure. Always transform the model by dividing by the standard deviation of errors based on the assumed relationship between variance and X.
π Definition β What does WLS do? OLS minimizes the sum of squared errors and gives equal weight (importance) to all observations. WLS (Weighted Least Squares) gives observations with larger error variance less weight. WLS minimizes a weighted sum of square of errors, e.g., minimizes (\sum \frac{e_i^2}{Var(e_i)}).
π Formula: Transformation when (Var(e_i) = \sigma^2 X_i) (variance increases linearly with X): (\frac{Y_i}{\sqrt{X_i}} = \frac{\beta_0}{\sqrt{X_i}} + \beta_1 \frac{X_i}{\sqrt{X_i}} + \frac{e_i}{\sqrt{X_i}})
π Formula: Transformation when (Var(e_i) = \sigma^2 X_i^2) (variance is related to square of X): (\frac{Y_i}{X_i} = \frac{\beta_0}{X_i} + \beta_1 + \frac{e_i}{X_i}) Note: In this transformed model, the intercept becomes (\beta_1) and the slope becomes (\beta_0).
Example: Handling Heteroskedasticity
Consider the model (Y_i = \beta_0 + \beta_1 X_i + e_i) with 20 observations. The Breusch Pagan test gives (R^2 = 0.22837), (LM = N \cdot R^2 = 4.5673), and (F = 5.3271). Both F and LM indicate the presence of heteroskedasticity. The White test gives (R^2 = 0.34954), (LM = 6.9907), and (F = 4.5676). Both F and LM indicate the presence of heteroskedasticity in a non-linear fashion.
Assume the relationship (Var(e_i) = \sigma^2 X_i^2). Transform the model: (\frac{Y_i}{X_i} = \frac{\beta_0}{X_i} + \beta_1 + \frac{e_i}{X_i}). This can be rewritten as: (\frac{Y_i}{X_i} = \beta_1 + \beta_0 \frac{1}{X_i} + \frac{e_i}{X_i}). Note that the intercept is (\beta_1) (original slope) and the slope is (\beta_0) (original intercept). Estimating this transformed model gives: (\frac{Y_i}{X_i} = 11.51 + 25.61 (\frac{1}{X_i})).
Testing the new transformed regression for heteroskedasticity: using the White test on the transformed model, (R^2 = 0.03892), (LM = 0.7783), and (F = 0.3442). The critical values are (\chi^2_{0.05,2} = 5.9915) and (F_{0.05,2,17} = 3.5915). Since both test statistics are far below their critical values, we conclude that heteroskedasticity does not exist in the transformed model.
π Example: The original model (Y = \beta_0 + \beta_1 X + e) showed heteroskedasticity in both the Breusch Pagan test (LM = 4.57 > 3.84?) and White test (LM = 6.99 > 5.99?). After assuming (Var(e_i) = \sigma^2 X_i^2) and transforming by dividing by (X_i), the new model (\frac{Y_i}{X_i} = \beta_1 + \beta_0 \frac{1}{X_i} + \frac{e_i}{X_i}) yields White test results of LM = 0.78 (far below 5.99) and F = 0.34 (far below 3.59), indicating successful removal of heteroskedasticity.
β Key Takeaways
This lecture demonstrates the practical application of five different heteroskedasticity detection tests (Park, Harvey-Godfrey, Glesjer, Breusch Pagan, and White) using the same data, showing that different tests can give different conclusions. The general procedure for all tests involves setting up (H_0): Homoskedasticity, calculating either the LM statistic ((N \cdot R^2)) or F-statistic from an auxiliary regression, and comparing to the appropriate critical value. When heteroskedasticity is detected, three remedies are available: respectifying the model, using White (heteroskedasticity-consistent) standard errors, or applying Weighted Least Squares by transforming the model based on assumptions about how the error variance relates to the independent variables. The key practical skill is the transformation example where (Var(e_i) = \sigma^2 X_i^2) leads to dividing all variables by (X_i), which successfully removes heteroskedasticity as confirmed by subsequent testing.
π§ Quick Revision Questions
- What is the general formula for the LM test statistic in all heteroskedasticity tests, and how is it used to decide whether to reject the null hypothesis?
- How does the Park LM test differ from the Harvey-Godfrey test in terms of the auxiliary regression specification?
- In the Glesjer test, three different auxiliary regressions were run. What were the three functional forms tested, and why is this flexibility useful?
- When using the White test with two independent variables, what additional terms are included in the auxiliary regression beyond the original variables?
- In the handling heteroskedasticity example, after transforming the model by dividing by (X_i) (assuming (Var(e_i) = \sigma^2 X_i^2)), what happens to the interpretation of the intercept and slope coefficients in the transformed regression?