π Lecture 09 β Multiple Regressions
π Overview: This lecture introduces multiple regression analysis, which extends simple regression to include two or more independent variables to explain variation in a dependent variable. It covers the mathematical formulation, estimation using Ordinary Least Squares (OLS), derivation of parameters in deviation form, and the manual procedure for solving multiple regression equations, along with diagnostic tests for significance.
ποΈ Topics Covered
The lecture begins with the general idea of multiple regression and examples. It then presents the multiple linear regression equation for two independent variables and the concept of fitting a line in 3-dimensional space. The least square estimation method is detailed, including the derivation of normal equations. The deviation form is introduced to simplify calculations, and formulas for estimating coefficients Ξ²β, Ξ²β, and Ξ²β are derived. A manual procedure is outlined, along with a method to convert summations from normal to deviation form. An example is solved manually and verified using the LINEST function in Excel. Finally, diagnostic tests including t-tests for individual significance and the F-test for overall model significance are discussed.
π Lecture Summary
General Idea
Multiple Regression considers the relation between one dependent variable and two or more independent variables. This allows us to examine the impact of one independent variable on the dependent variable while holding other independent variables constant. For example, earnings may depend on both education and experience, or quantity demanded may depend on price, income, and prices of substitutes.
Multiple Regression Equation
The multiple regression equation with two independent variables is written as: να΅’ = Ξ²β + Ξ²βνβα΅’ + Ξ²βνβα΅’ + eα΅’ While a simple regression fits a line in 2-dimensional space, a multiple regression with two independent variables fits a plane in 3-dimensional space.
In multiple regression, we can:
- Use several variables simultaneously to explain variation in a continuous dependent variable.
- Isolate the unique effect of one variable on the dependent variable while accounting for other variables.
- Write a mathematical equation that shows the overall effects of several variables together and the unique effect of each.
Estimating a Multiple Regression Line: Least Square Estimation
We use the Ordinary Least Squares (OLS) method to estimate the parameters Ξ²β, Ξ²β, and Ξ²β by minimizing the sum of squared residuals. The minimization problem is: Min β eα΅’Β² = β(Y β ΕΆ)Β² = β(Y β Ξ²β β Ξ²βXβ β Ξ²βXβ)Β²
Taking first derivatives with respect to Ξ²β, Ξ²β, and Ξ²β and setting them to zero yields the three normal equations: β Y = nΞ²β + Ξ²β β Xβ + Ξ²β β Xβ β XβY = Ξ²β β Xβ + Ξ²β β XβΒ² + Ξ²β β XβXβ β XβY = Ξ²β β Xβ + Ξ²β β XβXβ + Ξ²β β XβΒ²
The Deviation Form in Regression
To simplify calculations, we can use the deviation form, where variables are expressed as deviations from their means. Small letters indicate a variable in deviation form. xβα΅’ = Xβα΅’ β XΜβ xβα΅’ = Xβα΅’ β XΜβ yα΅’ = Yα΅’ β Θ²
The regression equation in deviation form becomes: y = Ξ²βxβ + Ξ²βxβ
Minimizing the sum of squared residuals in deviation form results in two normal equations: β xβy = Ξ²β β xβΒ² + Ξ²β β xβxβ β xβy = Ξ²β β xβxβ + Ξ²β β xβΒ²
π Definition β Deviation Form: A method of expressing variables as the difference between each observation and its arithmetic mean, which simplifies regression calculations.
Deriving Expression for Ξ² Coefficients
Solving the two normal equations in deviation form provides formulas for Ξ²β and Ξ²β.
π Formula: Ξ²β = [β xβy β xβΒ² β β xβy β xβxβ] / [β xβΒ² β xβΒ² β (β xβxβ)Β²]
π Formula: Ξ²β = [β xβy β xβΒ² β β xβy β xβxβ] / [β xβΒ² β xβΒ² β (β xβxβ)Β²]
π Formula: Ξ²β = Θ² β Ξ²βXΜβ β Ξ²βXΜβ
Manual Procedure to Solve for Multiple Regression
The manual procedure uses the formulas in deviation form. The necessary summations in deviation form can be calculated directly from the original data using conversion formulas.
π Formula: β xβy = β XβY β (β Xβ β Y) / n π Formula: β xβy = β XβY β (β Xβ β Y) / n π Formula: β xβxβ = β XβXβ β (β Xβ β Xβ) / n π Formula: β xβΒ² = β XβΒ² β (β Xβ)Β² / n π Formula: β xβΒ² = β XβΒ² β (β Xβ)Β² / n
π Example: Using the provided dataset, the following deviation form summations were calculated: β xβy = 5513.5, β xβy = -1115, β xβxβ = -1665, β xβΒ² = 6780.1, β xβΒ² = 330. Substituting into the formulas gives Ξ²β = -0.0372 and Ξ²β = -5.1713. Then Ξ²β = Θ² β Ξ²βXΜβ β Ξ²βXΜβ = 264.67. The estimated regression line is: ΕΆ = 264.67 β 0.0372Xβ β 5.1713Xβ.
π‘ Why this matters: The manual calculation demonstrates the direct application of OLS formulas, building a deeper understanding before relying on software. The example shows how to interpret the coefficients: holding Xβ constant, a one-unit increase in Xβ is associated with a decrease of 5.1713 units in Y.
Using LINEST in Multiple Regression
The LINEST function in Excel can be used for multiple regression by selecting an array with 5 rows and as many columns as parameters (3 in this case). The function is entered as an array formula (Ctrl + Shift + Enter). The output provides coefficients, standard errors, R-squared, F-statistic, and the regression and residual sums of squares.
Simple vs Multiple Regression
A key difference is degrees of freedom: with n observations and k independent variables, degrees of freedom = n β k. In the example with 10 observations and 2 independent variables, the degree of freedom is 7. The coefficient of determination (RΒ²) in multiple regression is not simply the square of a single correlation coefficient.
Diagnostic Tests for Regression Analysis
We perform t-tests for individual slope coefficients and an F-test for overall model significance.
The t-statistic for a coefficient Ξ²β±Ό is: t = Ξ²β±Ό / se(Ξ²β±Ό)
π Example: For Ξ²β, t = -0.0372 / 0.02892 = -1.287. The critical value tβ.βββ ,β = 2.36. Since |t| < t_critical, Ξ²β is not significant, meaning Xβ does not have a significant impact on Y. π Example: For Ξ²β, t = -5.1713 / 0.1311 = -39.45. Since |t| > t_critical, Ξ²β is significant, meaning Xβ has a significant impact on Y.
The F-statistic tests the overall significance of the regression. Its formula in terms of RΒ² is: F = [RΒ² / (k β 1)] / [(1 β RΒ²) / (n β k)]
π Example: RΒ² = 0.9979, so F = 1667.53. The critical value Fβ.ββ ,β,β = 4.74. Since F > F_critical, the overall model fit is good.
β Key Takeaways
Multiple regression extends simple regression to analyze the unique effect of multiple independent variables on a dependent variable. The OLS method minimizes the sum of squared residuals, leading to normal equations that can be solved using deviation forms to derive formulas for Ξ² coefficients. The manual estimation procedure, though time-consuming, is essential for understanding the mechanics behind the analysis. The t-test is used to assess the significance of individual independent variables, while the F-test evaluates the overall model fit, both of which are critical for validating the regression results.
π§ Quick Revision Questions
- What is the main difference between simple and multiple regression?
- Why is the deviation form used in estimating multiple regression parameters?
- How many normal equations are obtained when minimizing the sum of squared residuals for a multiple regression with two independent variables?
- What does it mean if a t-test for a coefficient Ξ²β±Ό shows that |t| > t_critical?
- What does the F-test in multiple regression assess?
π Lecture 10 β Linear Regression Estimation with K-independent variables
π Overview: This lecture extends simple linear regression to include multiple (K) independent variables, explaining how estimation and interpretation change as more regressors are added. It provides a practical, step-by-step guide to using Microsoft Excel's Analysis ToolPak for multiple regression analysis, including installation, data setup, and interpretation of outputs. The lecture emphasizes both the computational convenience and the limitations of Excel for econometric work.
ποΈ Topics Covered
The lecture begins by defining the linear regression model with K independent variables and the normal equations derived from minimizing sum of squared residuals. It then discusses how adding independent variables changes key statistics like degrees of freedom, standard errors, and R-squared. A major portion is dedicated to installing, setting up, and using the Data Analysis ToolPak in various Excel versions, followed by a detailed walkthrough of performing regression with this add-in. The lecture concludes with interpreting regression output (coefficients, p-values, F-statistic, adjusted R-squared), practicing on real World Bank data for Pakistan, and outlining common problems in multiple regression such as multicollinearity and heteroskedasticity.
π Lecture Summary
Linear Regression Model with K Independent Variables
The general multiple linear regression model is defined as: Yα΅’ = Ξ²β + Ξ²βXβα΅’ + Ξ²βXβα΅’ + Ξ²βXβα΅’ + ... + Ξ²βXβα΅’ + eα΅’, where 1 < K < M and M is a finite number. Minimizing the sum of squared residuals yields K+1 normal equations that must be solved simultaneously, typically using matrix algebra in software like Microsoft Excel (via LINEST or Data Analysis Add-in).
π Definition β Degrees of Freedom: The degrees of freedom for a multiple regression is calculated as (N - K), where N is the number of observations and K is the number of independent variables. This decreases as more variables are added.
Adding independent variables changes several aspects of regression: degrees of freedom (N-k) decreases; potential problems like multicollinearity may arise; the formula for standard errors of coefficients changes; a larger number of observations may be needed; the coefficient of determination (RΒ²) increases artificially; and regression coefficient values may change when a new variable is added.
Analysis ToolPak: DATA ANALYSIS Add-In in Microsoft Excel
The Analysis ToolPak is a Microsoft Excel add-in that uses the LINEST function to perform regressions and produce basic regression statistics in tabular form. It has a limit of maximum 16 independent variables. The tool typically comes with Excel but must be added (installed) from within the programβusually without needing a CD unless it is not already installed.
Installation steps vary by Excel version:
- Excel 2003 and earlier: Tools Menu β Add-ins
- Excel 2007: Office Button β Excel Options β Add-ins β Manage Excel Add-ins β GO
- Excel 2010 (demonstrated in lecture): FILE β Options β Add-Ins β Click on Analysis ToolPak (not VBA) β GO β Check box β OK β If prompted, click YES to install
Once installed, verify by checking if "Data Analysis" appears under the DATA ribbon.
Regression with The Analysis ToolPak
To perform regression, first enter data (10 observations each for Y and X variables) in an Excel sheet. Navigate to the DATA ribbon, click Data Analysis (rightmost item), scroll to Regression, and click OK. In the regression dialog box, set Input Y Range by selecting the Y values column, set Input X Range by selecting all X variable columns (they must be adjoining), check New Worksheet, and click OK. Results display in a new worksheet; resize columns for better readability.
π Example: Using sample data with Y and three X variables (Xβ, Xβ, Xβ), the regression output table shows coefficients, standard errors, t-statistics, and p-values for each variable.
Understanding the Regression Results
The regression output table contains three main parts. The third part displays coefficients, standard errors, t-statistics, and p-values. P-values show the level of significance at which, based on the t-statistic, the null hypothesis (Hβ: coefficient is not significant) can be rejected. In the example, the coefficient of Xβ has a p-value of 0.00001003 (below 1%), making it statistically significant at the 1% level. The intercept also shows significance with p-value below 1%, while other coefficients are not significant.
π Definition β Adjusted RΒ²: Adjusted RΒ² = RΒ² β [(1 β RΒ²) Γ (k β 1) / (N β k)]. It adjusts the coefficient of determination for degrees of freedom, providing a more accurate measure of goodness-of-fit when multiple independent variables are present, as RΒ² is overstated when more variables are added.
Multiple correlation equals the square root of RΒ² and is computable from partial correlation coefficients. The Coefficient of Determination (RΒ²) represents the explanatory power of the modelβin the example, it is 0.99 or 99%. The F-statistic and its significance (p-value β 0.0000000178, essentially zero) indicate that the model is a good fit at the 1% level of significance.
The p-value equals Pr{|t| > t-Stat} where t follows a t-distribution with (n β k) degrees of freedom and t-Stat is the computed t-statistic. It represents the level of significance at which the calculated t-statistic becomes larger than the table value.
π‘ Why this matters: Understanding p-values and F-statistics allows researchers to make objective decisions about which variables and models are statistically meaningful, rather than relying on arbitrary cutoffs.
LIMITATIONS of Working with Data Analysis Toolpak
Excel's Analysis ToolPak has several important limitations: it restricts independent variables to maximum 16; requires all independent variables to be in adjoining columns; its standard errors, t-statistics, and p-values assume errors are independent with constant variance; and it does not provide alternative models like robust standard errors. More powerful software such as STATA, EVIEWS, or SPSS may be needed for advanced econometric analysis.
Practicing Regression Estimation
The lecture provides a practical exercise using real economic data from the World Bank's World Development Indicators (http://data.worldbank.org/data-catalog/world-development-indicators). Use the Data Bank resource to select Pakistan as country; variables: GDP (constant 2005 US$), Labor force (total), Gross fixed capital formation (constant 2005 US$), and Exports of goods and services (constant 2005 US$); and time period: 1991 to 2012. Download in Excel format.
After downloading, reshape data by copying all data and using Paste Special β Transpose (Ctrl+V then check TRANSPOSE). Then use the Data Analysis ToolPak to run a regression with GDP as dependent variable and Labor Force, Stock of Capital (using GFCF as proxy), and Exports as independent variables. All independent variables must be in adjacent columns.
Possible Problems
Multiple regression models can suffer from various issues: the relationship may not be linear (wrong specification); important variables may be omitted; indicators may be inappropriate; there may be outliers (e.g., abnormal years); errors may not be normally distributed; and there may be econometric problems like multicollinearity, heteroskedasticity, or autocorrelation (to be discussed in later lectures).
β Key Takeaways
The multiple linear regression model extends simple regression by incorporating K independent variables, which changes key statistical properties: degrees of freedom decrease, RΒ² artificially inflates, and coefficients can shift when new variables are added. The Analysis ToolPak in Excel provides a practical, step-by-step method for estimating these models, from installation and data setup to interpreting output tables with coefficients, p-values, and F-statistics. However, Excel's limitations (max 16 variables, no robust errors) mean that serious econometric work may require specialized software. The adjusted RΒ² is a more reliable measure of goodness-of-fit than RΒ² when comparing models with different numbers of predictors.
π§ Quick Revision Questions
- What are the key statistical changes that occur when additional independent variables are added to a linear regression model?
- How do you install and access the Analysis ToolPak in Microsoft Excel 2010?
- What does the p-value represent in the regression output, and how is it used to determine statistical significance?
- Write the formula for Adjusted RΒ² and explain why it is preferred over RΒ² in multiple regression.
- List three limitations of using Excel's Data Analysis ToolPak for regression analysis.
π Lecture 11 β Transformation for Regression
π Overview: This lecture explains that "linear" in regression refers to linearity in parameters, not necessarily in variables. It covers various non-linear functional forms including quadratic, cubic, logarithmic, and exponential relationships, as well as dynamic models with lagged variables and interaction terms, which allow for more realistic modeling of economic relationships.
ποΈ Topics Covered
The lecture covers the concept that linearity is often a poor approximation of reality, and explores alternative functional forms including exponential and logarithmic functions, linear-log, log-linear, and log-log forms, polynomial functions of higher order, dynamic models with lagged independent variables, and interaction terms. Examples include marginal cost curves, total cost curves, diminishing returns, and the impact of investment changes over time.
π Lecture Summary
Linear in the βLinear Regressionβ means that the model is βlinear in parametersβ and not necessarily linear in variables. Linearity is a poor approximation of truth.
Linear regression models assume linearity in parameters, but relationships between variables themselves can be non-linear. Common non-linear relationships include quadratic (U-shaped curves like marginal cost), cubic (S-shaped curves like total cost with inflection points), logarithmic (diminishing returns), and exponential (growth/decay) forms. The scatter plot of data can help identify which type of relationship exists between variables.
π‘ Why this matters: Economic relationships are rarely perfectly linear, so knowing how to model non-linear patterns is essential for accurate econometric analysis.
Other Functional Forms
The scatter plot may indicate the type of relation. Different functional forms include:
- Linear: Simple straight-line relationship
- Exponential: Curved relationship showing rapid growth or decay
- Logarithmic: Curved relationship showing diminishing growth
Exponential and Logarithmic Function
The exponential functional form is (y = b^x) where (b > 0). It can be estimated as (\ln y = x \ln b). This transforms to (y^* = b^* x) where (b^* = \ln b) and (y^* = \ln y). This becomes a regression equation without an intercept (though (y = ab^x) would include an intercept). When estimated, we need to compute the inverse log to get the original parameter (b).
π Definition β Logarithmic function: Inversely related to exponential functions. From the exponential case, by definition of logarithm to the base (a), the logarithmic function can be written as (x = \log_a y).
π Example: Uses include modeling growth or decay like economic growth models, investment increasing by a constant percentage each time period, sales of a company increasing at a constant percentage each year, and models of the spread of an epidemic.
Logs of variables
Linear-log, log-linear and log-log forms are commonly used:
- Linear-log form: (y = a + b \ln x + e)
- Log-linear or semi-log model: (\ln y = a + bx + e)
- Log-log form: Refer to the Cobb-Douglas production function discussed earlier
The graph of (y) against (x) is curved, but the graph of (y) against (\ln x) is a straight line.
π Example: Short-run production β (Y) experiences diminishing marginal returns with respect to increases in (X). The log transformation bends a concave curve into a straight line.
π Definition β Log-linear model: A model where the dependent variable is in logarithmic form while the independent variable remains in its original units.
Polynomial functions of higher order
(y = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \beta_4 x^4 + ... + \beta_k x^k + e)
The model is linear in parameters but not in variables. The squared term captures curvature in the relationship.
π Example β Total Cost (TC) curve: (TC = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3) where (\beta_3 \neq 0) and (\beta_2 < 0). This cubic form captures the S-shaped total cost curve.
π Example β Marginal Cost (MC) curve: Quadratic form (y = \beta_0 + \beta_1 x + \beta_2 x^2) where (\beta_0, \beta_2 > 0) and (\beta_1 < 0). This U-shaped curve reflects diminishing then increasing marginal costs.
Dynamic Models: Models with lagged independent Variable
Dynamic models capture relationships where variables have effects that persist over time.
π Definition β Lagged variable: A variable from a previous time period used as an independent variable in the current period's model.
π Example 1 β Multiplier effect: A change in investment in period (t-2) impacts output in periods (t-2), (t-1), and (t): (\Delta Q = \beta_0 + \beta_1 \Delta I_t + \beta_2 \Delta I_{t-1} + \beta_3 \Delta I_{t-2} + e)
π Example 2 β Agricultural supply: Supply of an agricultural crop depends on the price from the previous time period, as farmers sow based on last year's observed price: (Q^s = a + b P_{t-1} + e)
π Example 3 β Autoregressive model: A variable may depend on its own previous value: (y_t = a + b y_{t-1} + e)
What is Interaction term?
An interaction term captures the situation where the impact of one independent variable on the dependent variable depends on another independent variable.
π Definition β Interaction term: A new variable computed as the product of two independent variables, represented as ((X_1 * X_2)).
π Example 1 β Education: Marks in econometrics ((M)) depend on IQ level ((I)) and hard work ((H) = time spent studying). The impact of hard work on marks may depend on IQ level (more intelligent students may need less hard work). This is captured by: (M = a + bI + cH + d(I * H) + e)
The derivative of (M) with respect to (H) is ((c + dI)), which depends on the value of (I).
π Example 2 β Health: Consider the impact of hypertension ((B) = blood pressure) and diabetes ((S) = average blood sugar) on heart health ((H)). Both are risk factors, but having both simultaneously may have a greater impact than the sum of individual effects: (H = a + bB + cS + d(B * S) + e)
The coefficient of the interaction term (d) captures the additional impact of having BOTH risk factors simultaneously.
β Key Takeaways
The most critical concept is that "linear" in linear regression refers to linearity in parameters, not variables, allowing for flexible modeling of non-linear relationships through transformations. Students must remember the four main functional forms (linear-log, log-linear, log-log, and polynomial) and when to use each based on scatter plot patterns. Dynamic models with lagged variables capture time-dependent effects like multiplier impacts and agricultural supply responses. Interaction terms model how the effect of one variable depends on another, such as how IQ moderates the impact of study time on exam performance. Proper interpretation of coefficients changes depending on the functional form used, and exponential models require inverse log transformation to recover original parameters.
π§ Quick Revision Questions
- What does "linear" mean in the context of linear regression models?
- How does the exponential functional form (y = b^x) get transformed for estimation using OLS?
- What is the difference between linear-log, log-linear, and log-log functional forms?
- Why would a cubic polynomial be used to model a total cost curve rather than a quadratic?
- What does the coefficient of an interaction term ((I * H)) represent in the model (M = a + bI + cH + d(I * H) + e)?
π Lecture 12 β Regression on standardized variables
π Overview: This lecture covers the concept of standardizing variables in regression analysis to eliminate the influence of measurement units, allowing for direct comparison of coefficient magnitudes. It also discusses the theoretical foundation of regression, summary statistics, correlation matrices, graphs, and best practices for presenting regression results in research papers.
ποΈ Topics Covered
The lecture begins with the standardization of variables and interpretation of beta coefficients (standardized coefficients), along with their advantages and disadvantages. It then covers the theory behind regression, summary statistics in Excel, correlation matrices, graphs before regression, how to present regression results in research papers, and strategies for keeping or dropping variables. An example of presented regression results is provided, followed by a review of regression-related functions in Microsoft Excel and a summary of steps before regression.
π Lecture Summary
Regression on standardized variables
The units in which the regressand and regressor are expressed affect the interpretation of coefficients. Standardizing variables solves this problem. A variable is standardized by subtracting its mean and dividing by the standard deviation.
π Definition β Standardized variable: (X_i^* = \frac{X_i - \bar{X}}{S_X}), (Y_i^* = \frac{Y_i - \bar{Y}}{S_Y})
The standardized variables have zero means and unit variances: (\bar{X}^* = \bar{Y}^* = 0) and (Var(X_i^) = Var(Y_i^) = 1). It does not matter in what unit the original variables are expressed.
Running the regression (Y_i^* = a + bX_i^* + e) gives standardized coefficients, often called BETA coefficients. The intercept term will always be zero because (\bar{X}^* = \bar{Y}^* = 0) (since (a = \bar{Y}^* - b\bar{X}^*)).
π Interpretation: If the (standardized) regressor increases by one standard deviation, on average, the (standardized) regressand increases by Ξ² standard deviation units.
Advantages: Standard coefficients ignore the independent variable's scale of units, making comparisons easy.
Disadvantages: Such standardization can be misleading. The meaning of a standard deviation may vary markedly between non-normal distributions (e.g., skewed distributions).
Theory behind regression
Regression makes sense only if there is a sound theory behind it. Ensure to include all necessary predictor variables depending on the problem statement, theory, and previous knowledge.
Some variables may measure the same thingsβeither keep one of them or combine them. Also consider possible interactions between variables.
π‘ Why this matters: Including irrelevant or redundant variables can distort regression results; theoretical grounding protects against this.
Summary Statistics
In Excel, after entering data, click Data Analysis in the Data Ribbon, select Descriptive Statistics, and click OK. Click in the Input Range, select your data including variable names by dragging your mouse. Check the option Labels in First Row, check Summary Statistics, and click OK. The output appears in a new worksheet.
Correlation Matrix
In Excel, after entering data, click Data Analysis in the Data Ribbon, select Correlation, and click OK. Click in the Input Range, select your data including variable names by dragging your mouse. Check the option Labels in First Row and click OK. The resulting matrix of correlations helps understand the relations between variables.
Graphs
Graphs before regression include Scatter Diagrams, Scatter Diagrams with line plots, Histograms, and Pie Charts. In the Insert Ribbon, click on the required chart, provide the required information, and get the chart.
How to present Regression Results in research papers
Every paper uses a slightly different strategy. Standard information to report in a regression table includes:
- Dependent variable
- Explanatory variables
- Number of observations, sample period, data labels, etc.
- Estimates of intercept and other coefficients
- Standard errors of estimate
- Significance of variables/coefficients
- R-squared and other required statistics
It is common to present more than one regression result. Variables can be added or dropped and regressions performed.
Keeping or Dropping Variables
Regression results by adding or including variables vary markedly between non-normal distributions (e.g., skewed). A good strategy:
- Keep significant predictors
- Keep insignificant predictors with the expected sign
- Drop insignificant predictors with unexpected sign
- For significant predictors with unexpected sign, keep after review and including/excluding other variables
Examples of presented results
The lecture shows a sample regression table with two models. For example, in both Model 1 and Model 2, the dependent variable is Car Price. Millage coefficients are 855.25* and 852.55*, Weight coefficients are 1256.9** and 1246.1**, Foreign coefficients are 950.5* and 855.25*, and Price of Oil appears only in Model 2 as -132.25. Number of observations is 2500 for both, R-Squared is 0.65 and 0.69 respectively. *, ** significance at 1% and 5% respectively.
Regression related function in Microsoft Excel: A Brief review
The available functions include:
- The LINEST (up to 50 independent variables)
- Data Analysis ToolPak (up to 16 independent variables)
- RSQ
- Basic formulas: INTERCEPT, SLOPE, TREND
Steps before Regression
- Theory
- Graphs
- Summary statistics
- Model specification
- Regression
- Estimation
- Change of scale if required
- Different functional forms
- Interaction
- Post Regression: Change of scale if required, Different functional forms, Interaction
β Key Takeaways
Standardization removes units of measurement so coefficients represent changes in standard deviation units, enabling direct comparison of variable importanceβbut this is only valid for normally distributed data. Regression must be grounded in economic theory, not just data mining. Summary statistics, correlation matrices, and graphs are essential pre-regression diagnostics in Excel. When presenting results, always report coefficients, standard errors, significance levels, R-squared, and sample size. A disciplined strategy for variable selection is to keep significant predictors and those with expected signs, while dropping insignificant predictors with unexpected signs.
π§ Quick Revision Questions
- How do you standardize a variable, and what are the properties of standardized variables?
- How is a standardized (beta) coefficient interpreted in a regression model?
- What is the main advantage and the main disadvantage of using standardized coefficients?
- List the standard information that must be reported in a regression results table.
- What is the recommended strategy for keeping or dropping variables based on their significance and sign?
π Lecture 13 β Qualitative Independent Variable / Dummy Variables
π Overview: This lecture introduces dummy variables β binary or polytomous variables used to encode qualitative information (e.g., gender, race) into regression models. It explains why ignoring such variables can cause bias or inefficiency, how to include them, interpret their coefficients, and handle interactions.
ποΈ Topics Covered
This lecture covers the definition and types of dummy variables, the consequences of ignoring qualitative variables (correlated or uncorrelated with other regressors), binary dummy variable regression with interpretation using the wage example, polytomous (multi-category) dummy variables with the dummy variable trap and reference category, and interaction effects between dummy variables and continuous predictors, including interaction with multiple categories.
π Lecture Summary
Qualitative Independent Variable / Dummy Variables
This section defines dummy variables (also called categorical, indicator, binary, or dichotomous variables) as variables that encode qualitative information into numerical form, most commonly as binary (0 or 1). They divide data into mutually exclusive categories. Examples include gender (0/1), yes/no, control vs. treatment group, race, and educational categories. The assignment of 0 and 1 is arbitrary β it does not matter which status gets which value.
π Definition β Dummy Variable: A variable that encodes a qualitative characteristic, typically taking the value 0 or 1 to indicate the absence or presence of a category.
What happens if we ignore Gender?
If we ignore a qualitative variable like gender, the consequences depend on its relationship with other independent variables:
- Scenario 1 (Education and Gender not correlated): The slope estimates for other variables remain correct, but the model will have larger errors (higher standard errors).
- Scenario 2 (Education and Gender correlated): The slope estimates become biased and errors become larger, violating the Gauss-Markov theorem.
Possible solutions include: (1) running separate regressions for each gender, but this does not allow testing gender differences; (2) using dummy variables to capture the influence of gender in a single regression.
Example of binary dummy variable
Consider hourly wages (W) as a function of years of education (E), years of experience (X), and gender (G), where G = 1 for Female and G = 0 for Male. The regression model is: $$W = \beta_0 + \beta_1 E + \beta_2 X + \beta_3 G + e$$
Using the data file dummy.xlsx, the estimated regression is:
$$\hat{W} = 77.76 + 8.29E + 28.52X - 26.6G$$
π Interpretation: The coefficient -26.6 means that female respondents' wages are, on average, 26.6 rupees less than males, holding education and experience constant. π Example calculation: For a female with E=14, X=5: $$\hat{W} = 77.76 + 8.29(14) + 28.52(5) - 26.6(1) = 309.82$$ For a male with identical characteristics: $$\hat{W} = 77.76 + 8.29(14) + 28.52(5) - 26.6(0) = 336.42$$ The difference (336.42 β 309.82 = 26.6) is the average gender wage gap for individuals with the same education and experience.
Exam Model Question
Using the LINEST function from dummy.xlsx, the regression output for $W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + e$ yields:
| Coefficient | G | X | E | Constant |
|---|---|---|---|---|
| Estimate | -26.869143 | 26.92131443 | 9.56630262 | 68.48962199 |
| Std. Error | 12.59410484 | 3.958664082 | 5.24813727 | 66.42244505 |
| RΒ² = 0.787428706 | SEE = 35.74053401 | |||
| F = 38.27780854 | df = 31 | |||
| SSR = 146686.584 | SSE = 39598.95891 |
π Interpretation: The coefficient of the dummy variable (G) is -26.869, meaning females earn on average 26.87 rupees less than the reference (males). To test significance, calculate t-statistic = coefficient/SE: $$t = \frac{-26.869}{12.594} \approx -2.13$$ Compare with critical t-value (e.g., at 5% significance with 31 df β 2.04). Since |β2.13| > 2.04, the dummy coefficient is statistically significant at the 5% level.
Polytomous qualitative variables
Polytomous variables are qualitative variables with more than two categories (e.g., race: White, Black, Asian). For k categories, we need k-1 binary dummy variables to avoid perfect multicollinearity (the dummy variable trap). The excluded category becomes the reference category.
For race with three categories:
- $D_1 = 1$ if Black, 0 otherwise
- $D_2 = 1$ if Asian, 0 otherwise
- $D_3 = 1$ if White, 0 otherwise (reference β not included in regression)
The regression model becomes: $$W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + \beta_4D_1 + \beta_5D_2 + e$$
π Interpretation by race:
- White (reference): $W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + e$
- Asian: $W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + \beta_5 + e$
- Black: $W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + \beta_4 + e$
From the estimated output (using dummy.xlsx with 35 observations, RΒ² = 0.85, F = 33.83):
| Variable | Coefficient | Std. Error | t-value |
|---|---|---|---|
| Constant | 74.84 | 60.11 | 1.25 |
| Education | 11.53** | 4.67 | 2.47 |
| Experience | 26.11* | 3.51 | 7.44 |
| Female | -31.49* | 11.10 | -2.84 |
| Black (ref: White) | -45.17* | 13.15 | -3.44 |
| Asian (ref: White) | -35.43** | 13.96 | -2.54 |
*, ** denote significance at 1% and 5% respectively. Black individuals earn on average 45.17 fewer rupees than White individuals, and Asian individuals earn 35.43 fewer rupees than White individuals.
Interaction with Dummy Variables
Interaction terms are used when the effect of one independent variable depends on the value of another independent variable. This is different from correlation (which is about variables being related to each other) β interaction is about dependence of effects. Interaction terms are created by multiplying the dummy variable with the continuous variable. This does not cause perfect collinearity because the product is a nonlinear function.
Example with gender and education interaction: $$W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + \beta_4(G \times E) + e$$
π For Men (G=0): $W = \beta_0 + \beta_1E + \beta_2X + e$ π For Women (G=1): $W = (\beta_0 + \beta_3) + (\beta_1 + \beta_4)E + \beta_2X + e$
The interaction coefficient $\beta_4$ captures how the effect of education on wages differs between men and women.
For multiple categorical variables (e.g., gender and race), include interaction terms with each dummy: $$W = \beta_0 + \beta_1E + \beta_2X + \beta_3G + \beta_4D_1 + \beta_5D_2 + \beta_6(G \times D_1) + \beta_7(G \times D_2) + e$$
π All possible group equations:
- White Men: $W = \beta_0 + \beta_1E + \beta_2X + e$
- Black Men: $W = (\beta_0 + \beta_4) + \beta_1E + \beta_2X + e$
- Asian Men: $W = (\beta_0 + \beta_5) + \beta_1E + \beta_2X + e$
- White Women: $W = (\beta_0 + \beta_3) + \beta_1E + \beta_2X + e$
- Black Women: $W = (\beta_0 + \beta_3 + \beta_4 + \beta_6) + \beta_1E + \beta_2X + e$
- Asian Women: $W = (\beta_0 + \beta_3 + \beta_5 + \beta_7) + \beta_1E + \beta_2X + e$
π‘ Why this matters: Interaction terms allow the model to capture differential effects β for example, whether the gender wage gap is different for different races, or whether the return to education varies by gender.
β Key Takeaways
- Dummy variables encode qualitative data as binary (0/1) and are used to include categorical predictors in regression. For a variable with k categories, include kβ1 dummies to avoid perfect multicollinearity (the dummy variable trap), with the omitted category serving as the reference group.
- Ignoring a relevant qualitative variable that is correlated with other regressors causes biased and inconsistent slope estimates; if uncorrelated, estimates remain unbiased but standard errors increase.
- The coefficient on a dummy variable measures the average difference in the dependent variable between that category and the reference category, holding other variables constant β this can be tested for statistical significance using t-tests.
- Interaction terms (products of a dummy with a continuous variable) allow the slope of the continuous variable to differ across categories, capturing heterogeneous effects without causing perfect multicollinearity.
- For polytomous variables with interaction across multiple categorical variables (e.g., race and gender), include all main effects plus all cross-product interaction terms β the resulting regression yields distinct intercepts for each subgroup combination.
π§ Quick Revision Questions
- Why do we include only kβ1 dummy variables for a qualitative variable with k categories? What happens if we include all k?
- In the wage regression $\hat{W} = 77.76 + 8.29E + 28.52X - 26.6G$, interpret the coefficient β26.6. Is it the gender gap for all individuals or only for those with identical E and X?
- How would you test whether the coefficient on a dummy variable (e.g., gender) is statistically significant at the 5% level, given its estimate and standard error?
- Explain the difference between correlation between independent variables and interaction between them. Give an example where interaction is needed but correlation is irrelevant.
- Write the regression equation for a model with education (E), experience (X), gender (G), race (with dummies D1 for Black, D2 for Asian, ref=White), and all genderΓrace interaction terms. Then write the specific equation for a Black female.
π Lecture 14 β Transforming Variables in Regression
π Overview: This lecture introduces the use of logarithms in regression analysis to handle skewed distributions and interpret percentage relationships. It also provides a comprehensive overview of missing data mechanisms, including their probability distributions, and various methods for dealing with missing data through deletion and imputation techniques.
ποΈ Topics Covered
The lecture begins with the reasons for using logarithms in regression, including handling positively skewed variables and linearizing relationships. It then presents an empirical example using wage data with tenure and race. The remainder of the lecture focuses on missing data: why data may be missing, the probability of missing data across groups, missing data mechanisms (MCAR, MAR, MNAR), and methods for dealing with missing data including listwise deletion, pairwise deletion, single imputation, and model-based imputation.
π Lecture Summary
Using logs in regression analysis
Logarithms are used in regression when a variable has a positively skewed distribution, when residuals are skewed, when changes in variables are related in percentage terms, or to linearize a relationship. Taking the natural log of a variable can make its distribution more symmetric.
Logs in regression
The lecture presents regression results using the nlsw88.dta dataset, modeling wages with tenure and a dummy variable for race.
π Definition β Log-level model: A regression where the dependent variable is in logarithmic form but the independent variables are in their original units. π Formula: ln(wage) = Ξ²0 + Ξ²1(tenure) + Ξ²2(race) + Ξ΅ β A one-unit increase in tenure is associated with a 100*Ξ²1 percent change in wage. π Example: From the table, the coefficient on tenure in the ln(wage) model is 0.0317. This means one additional year of tenure is associated with a 3.17% increase in wage (holding race constant). The R-squared increases from 0.039 to 0.1043 when using ln(wage) instead of wage, indicating a better fit.
Treatment of Missing Data
Data may be missing for various reasons including attrition (death, dropouts, migration), lack of legal availability, non-response or refusal by respondents, conditional questions in surveys, and data collection or encoding issues.
Probability of Missing Data
Certain groups are more likely to have missing values. Examples include businessmen compared to salaried individuals, rural areas compared to urban areas in developing countries, less educated individuals compared to educated ones, and developing countries compared to developed countries. Some variables are more prone to missingness, such as income (especially for high-income groups), area-specific variables, and variables related to taboos.
Missing Data Mechanism: probability distributions
-
MCAR: Missing Completely at Random The probability that Y is missing is independent of both Y and X. Pr(Y is missing | X, Y) = Pr(Y is missing)
-
MAR: Missing at Random The probability that Y is missing depends on X but not on Y itself. Pr(Y is missing | X, Y) = Pr(Y is missing | X) π Example: The probability of missing income depends on occupation, but within each occupation, the probability is not dependent on income level.
-
Ignorable MAR This is essentially MAR where the parameters governing the missing data are distinct from the parameters being estimated. The missing data mechanism is said to be ignorable.
-
MNAR: Missing Not at Random The probability that Y is missing depends on a variable that is itself missing. This is the most problematic case. π‘ Why this matters: With MNAR, the data contains no information about the missing values, and results become very sensitive to the choice of model. Special techniques like Heckman regression are needed.
Dealing with missing data: Deletion Methods
1. Listwise deletion (complete case analysis) Only rows with no missing values for any variable are analyzed. Software like Stata may do this automatically. π Example: If any variable in a row is missing, the entire row is excluded from the analysis.
- Advantages: Simple to implement.
- Disadvantages: Reduces statistical power (sample size decreases).
2. Pairwise deletion (available case analysis) Missing values are handled on an analysis-to-analysis basis, using all available data for each specific calculation. π Example: Different observations may be used for calculating a correlation than for a regression.
- Advantages: Uses all possible information for each case.
- Disadvantages: The sample differs each time, making comparisons across models difficult.
Imputation
1. Single Imputation
a) Substitute Mean / Median / Mode
- Method: Replace missing values with the mean, median, or mode of the observed values.
- Problems: Weaker covariance and reduced variability; standard errors are artificially lowered.
b) Dummy variable control
- Method: Create an indicator (dummy) variable for rows with missing values, fill in missing values using single imputation, then include the dummy in the regression.
- Problems: Estimates are biased and this method lacks a strong theoretical foundation.
c) Regression Imputation
- Method: Use regression to predict and substitute the missing values.
- Problems: Overestimates model fit and produces weaker variance.
Model Based Imputation
1. Maximum Likelihood
- Method: Identify the set of parameters that produces the maximum log likelihood β the value most likely to have resulted in the observed data.
- Uses full estimation and provides unbiased estimates.
- Problem: Standard errors are biased downwards.
2. Multiple Imputation
- Method: Data is filled in by applying specific regressions. This experiment is repeated n times, generating separate datasets. Regression is performed on each dataset, and then the results are pooled.
- Advantages: Good variability is preserved.
- Disadvantages: Errors are possible when specifying the regression models used for imputation.
β Key Takeaways
The most critical concepts from this lecture are: first, know when and why to use log transformations in regression, particularly for skewed distributions and percentage-change interpretations. Second, understand the three main missing data mechanisms (MCAR, MAR, MNAR) and be able to distinguish them based on whether missingness depends on Y or X. Third, recognize that listwise deletion reduces power while pairwise deletion uses changing samples. Fourth, know the difference between single imputation methods (mean substitution, dummy control, regression) and model-based methods (maximum likelihood, multiple imputation), including the key problems with each. Finally, be aware that MNAR is the most serious missing data problem and requires specialized estimation.
π§ Quick Revision Questions
- What are four reasons for using logarithms in regression analysis?
- In the log-level model where ln(wage) is regressed on tenure, how is the coefficient on tenure interpreted?
- What distinguishes MAR from MCAR in missing data mechanisms?
- What is the main disadvantage of listwise deletion when dealing with missing data?
- What is the key advantage of multiple imputation over single imputation methods?