📘 Lecture 5 — Multivariate Optimization
📖 Overview: This lecture extends optimization from single-variable to multi-variable functions, teaching how to find local minima, maxima, and saddle points for functions with two independent variables. It then transitions into a comprehensive review of probability theory, covering fundamental concepts and important probability distributions essential for econometric analysis.
🗂️ Topics Covered
The lecture begins with multivariate optimization, explaining local minimum and maximum conditions through first and second order derivatives, followed by saddle points and the third condition using the Hessian determinant to rule out inflection points. Two worked examples demonstrate maximization and minimization of economic functions. The second half provides a thorough review of probability, covering random experiments, sample spaces, events, random variables, probability distributions, and concludes with the four major continuous probability distributions: Normal, t, Chi-square, and F-distributions.
📝 Lecture Summary
Local Minimum
Consider the following diagram: Point 'O' is a local minimum FROM ALL DIRECTIONS. At point 'O', the derivative of z w.r.t. x OR w.r.t. y both are zero — the slope of the tangents parallel to x-axis and y-axis at point 'O' are both zero, i.e. Zₓ = 0 AND Zᵧ = 0. This is the First Order Condition for a local minimum.
🔑 Definition — Local Minimum: A point where the function value is lower than all neighboring points in every direction. 📐 Formula: First Order Condition: Zₓ = 0 AND Zᵧ = 0 → At the optimum point, both partial derivatives equal zero. 📌 Example: If z = x² + y², then Zₓ = 2x = 0 gives x = 0, and Zᵧ = 2y = 0 gives y = 0, so (0,0) is a candidate for local minimum.
Local Minimum: Second Order Condition
Now consider point 'O' again. When we move the tangents parallel to x-axis or y-axis, there is a positive change in the derivative (derivative of the derivative is positive): Zₓₓ > 0 AND Zᵧᵧ > 0. This gives the Second Order Condition for minimization — both second order derivatives must be positive.
🔑 Definition — Second Order Condition (Minimum): Both second-order partial derivatives must be positive at the critical point, confirming the function curves upward in all principal directions. 📐 Formula: Zₓₓ > 0 AND Zᵧᵧ > 0 → The function is convex in both x and y directions.
Local Maximum
Consider the following diagram: Point 'a' is a local maximum FROM ALL DIRECTIONS. At point 'a', derivative of z w.r.t. x OR w.r.t. y both are zero — the slope of the tangents parallel to x-axis or y-axis at point 'a' are both zero, i.e. Zₓ = 0 AND Zᵧ = 0. This is the First Order Condition for a local maximum.
🔑 Definition — Local Maximum: A point where the function value is higher than all neighboring points in every direction. 📐 Formula: First Order Condition: Zₓ = 0 AND Zᵧ = 0 → Same first-order condition as minimum, but second-order condition differs.
Local Maximum: Second Order Condition
Now consider point 'a' again. When we move the tangents parallel to x-axis or y-axis, there is a negative change in the derivative (derivative of the derivative is negative): Zₓₓ < 0 AND Zᵧᵧ < 0. This gives the Second Order Condition for maximization — both second order derivatives must be negative.
📐 Formula: Zₓₓ < 0 AND Zᵧᵧ < 0 → The function is concave in both x and y directions.
Saddle Point: Second order derivatives have different signs
Consider point 'O' in the following diagram: A tangent at this point has a zero slope (first derivative is zero, so the first condition is met). If we shift the tangent in the direction of the x-axis, the slope of the tangent increases, so this is a local minima form one direction (x-axis): Zₓₓ > 0. But if we shift the tangent at point 'O' in the direction of the y-axis, its slope will decrease: Zᵧᵧ < 0. This is a saddle point.
🔑 Definition — Saddle Point: A critical point where the first-order condition is met, but the second-order partial derivatives have opposite signs, making it a minimum in one direction and a maximum in another. 📐 Condition: Zₓₓ > 0 AND Zᵧᵧ < 0 (or vice versa) → different signs for the two second-order partials. 💡 Why this matters: Saddle points can mislead optimization algorithms — they satisfy the first-order condition but are not true optima.
A third condition: ruling out point of inflection
When evaluated at the critical point(s), the product of the second order partials must exceed the product of the cross partials. This condition rules out critical points that are neither points of maximum or minimum, but are points of inflection. A point of inflection is where certain conditions of optima are met, but the function is not actually a maximum or minimum.
📐 Formula: Zₓₓ · Zᵧᵧ > (Zₓᵧ)²
This can be written as a determinant: |Zₓₓ Zₓᵧ| |Zᵧₓ Zᵧᵧ| > 0
We call this a Hessian determinant or simply Hessian, which shows that Zₓₓ · Zᵧᵧ - (Zₓᵧ)² > 0, or Zₓₓ · Zᵧᵧ > (Zₓᵧ)².
🔑 Definition — Hessian Determinant: A determinant of second-order partial derivatives that tests whether a critical point is a true maximum, minimum, or inflection point.
Example: Maximization
Consider the following profit function where x and y are the levels of output: π = 80x − 2x² − xy − 3y² + 100y
Step 1: First Order Condition πₓ = 80 − 4x − y = 0 πᵧ = −x − 6y + 100 = 0
Solving simultaneously gives: x̄ = 16.52 and ȳ = 13.91
Step 2: Second Order Condition πₓₓ = −4 < 0 πᵧᵧ = −6 < 0 Both negative, confirming profit is maximized from the principal directions at the critical points.
Step 3: Third Condition (Hessian) πₓₓ · πᵧᵧ = (−4)(−6) = 24 AND (πₓᵧ)² = (−1)² = 1 Hence 24 > 1, so πₓₓ · πᵧᵧ > (πₓᵧ)²
The profit function is maximized from all directions at the critical point. Maximum profit can be found by substituting the critical points into the profit function.
Example: Minimization
Consider the following marginal cost function where x and y are the level of output: MC = 5x² − 8x − 2xy − 6y + 4y² + 100y
Step 1: First Order Condition MCₓ = 10x − 8 − 2y = 0 MCᵧ = −2x − 6 + 8y = 0
Solving simultaneously gives: x̄ = 1 and ȳ = 1
Step 2: Second Order Condition MCₓₓ = 10 > 0 MCᵧᵧ = 8 > 0 Both positive, confirming MC is minimized from the principal directions at the critical points.
Step 3: Third Condition (Hessian) MCₓₓ · MCᵧᵧ = (10)(8) = 80 AND (MCₓᵧ)² = (−2)² = 4 Hence 80 > 4, so MCₓₓ · MCᵧᵧ > (MCₓᵧ)²
The function is minimized from all directions at the critical point. Minimum MC can be found by substituting the critical points into the MC function.
Review of Probability
Probability: This is only a 'Review'
Random Experiment
Any process of observation or measurement that has more than one possible outcome and we are not certain about which outcome will materialize.
🔑 Definition — Random Experiment: An experiment where all possible outcomes are known, but the specific outcome of any single trial cannot be predicted with certainty. 📌 Examples: Tossing a coin, throwing a pair of dice, drawing a card from a deck of cards.
Sample Space/Population
The set of all possible outcomes of an experiment.
🔑 Definition — Sample Space: The complete set of all possible outcomes of a random experiment. 📌 Example: When you toss a coin, S = {H, T} 📌 Example: When you toss two coins, S = {HH, HT, TH, TT}
Sample Point
Each member of the sample space is a sample point.
Event
An event is a particular collection of outcomes — a subset of the sample space. 📌 Example: Event 'A' is occurrence of one head and one tail in the experiment of tossing two coins: A = {HT, TH}
Mutually Exclusive Events
Occurrence of one event prevents the occurrence of the other event at the same time. 📌 Example: When we toss two coins, occurrence of two heads means the other three outcomes cannot occur at the same time. 📌 Example: When we toss a single coin, occurrence of a head means that the tail did not occur and cannot occur at the same time.
Equally Likely Events
If one event is as likely to occur as the other. 📌 Example: Head and tail have the same possibility or chance of occurring.
Collectively Exhaustive Events
If they exhaust all possible outcomes of an experiment. 📌 Example: Event is the sample space. A = Occurrence of a head or tail while tossing a single coin.
Stochastic or Random Variable
A variable whose value is determined by the outcome of an experiment.
🔑 Definition — Random Variable: A variable that takes numerical values determined by the outcome of a random experiment. 📌 Example: Let X = Number of heads in an experiment of tossing two coins, then X can have values of 0, 1, or 2.
A random variable can be discrete (can take only whole numbers and finite values) or continuous (can take any values within an interval, either whole numbers or fractions, e.g., height of an individual).
Classical Definition of Probability
Probability of an event 'A' = P(A) = Number of favorable outcomes / Number of total outcomes
📌 Example: Total number of outcomes in tossing two coins is 4 {HT, HH, TH, TT}. Probability of getting exactly one head = 2/4 = 0.5
Probability Distribution
The possible values that a random variable can take with the number of occurrences (frequency) of those values.
📌 Example: Probability Distribution of discrete random variable: Let X = Number of heads in an experiment of tossing two coins. X can have values of 0, 1, or 2.
| X | P[X = xᵢ] |
|---|---|
| 0 | ¼ = 0.25 |
| 1 | 2/4 = 0.50 |
| 2 | ¼ = 0.25 |
| Total | 1 |
Probability Mass Function or simply Probability Function
f(X = xᵢ) = P(X = xᵢ) for i = 1, 2, … and 0 if X ≠ xᵢ
Properties: • 0 ≤ f(xᵢ) ≤ 1 • Σ f(xᵢ) = 1 for all x
Probability Density Function (PDF)
Probability Distribution of a continuous random variable, e.g., X = height of person measured in inches.
• X is a continuous random variable • Probability of continuous random variable is always computed for a range, not for a single value • PDF is: P(x₁ < X < x₂) = ∫ f(x) dx from x₁ to x₂ This calculates the probability as the area under a curve between a range (x₁ to x₂).
Cumulative Distribution Function (CDF)
F(X) = P[X ≤ x]
Important Probability Distributions
Some important probability distributions are Normal Distribution, t distribution, Chi-square distribution, and the F-distribution.
Normal Distribution
It is the most important probability distribution for a continuous random variable. It has a bell-shaped curve (highest point at mean value) where X ~ N(μₓ, σ²), −∞ < X < ∞.
Change in μ shifts the curve to right or left, while change in σ increases or decreases the spread of the curve.
The function: f(x) = (1/σ√(2π)) · e^(-½[(x−μ)/σ]²)
Mathematical constants: π = 3.14159, e = 2.71828
Properties (Empirical Rule): • 68% of data lies within μ ± 1σ • 95% of data lies within μ ± 2σ • 99.7% of data lies within μ ± 3σ
Standard Normal Distribution
If Z = (X − μ̄)/σ, then μ = 0 and σ = 1. The distribution of Z is a 'Standard' normal distribution: Z ~ N(0,1)
p(Z) = (1/√(2π)) · e^(-½[Z]²)
The probabilities or areas under the standard normal curve are already calculated and available in the shape of tables (the Z-table).
Student's t-distribution
The t-distribution is a probability distribution of a continuous random variable when the sample size is small and the population variance is not known. Its curve is symmetric and bell-shaped but flatter than the normal distribution. The mean is zero but the variance is larger (heavier tails) than the variance of the standard normal distribution (which is unity). It has only one parameter: the degree of freedom. As the degree of freedom (or the number of observations) increases, the distribution approaches the normal distribution.
Chi-Square (χ²) distribution
The square of a standard normal variable is distributed as a Chi-square probability distribution with one degree of freedom. Z² = χ²(1)
If we need the sampling distribution of the sample variance, we have the Chi-square distribution.
Properties: • Chi-square takes only positive values (zero to infinity) • It is skewed (depending on the d.f.) unlike the normal distribution • As the d.f. increases, the distribution approaches the normal distribution • Its mean is k (=d.f.) and variance is 2k (variance is twice the mean)
💡 Why this matters: Normal distribution and t-distributions are related to means, but the Chi-square and F-distributions are related to variances.
F-distribution
This is a variance ratio distribution — the ratio of sample variances of two independent samples. This is also equal to ratios of two Chi-squares. It has two parameters k₁ and k₂ (degrees of freedom in both samples, i.e., numerator and denominator of F = S₁²/S₂²).
Uses: • Testing equality of variances • Tests in Regression models like Goodness of fit test
Properties: • Skewed to the right between zero and infinity • Approaches normal distribution as d.f. increases
⭐ Key Takeaways
For multivariate optimization, always follow three steps: first-order condition (both partial derivatives = 0), second-order condition (both second partials positive for minima, negative for maxima), and the third condition using the Hessian determinant (product of second partials must exceed square of cross-partial) to rule out inflection points. For probability, understand that a random variable can be discrete (probability mass function) or continuous (probability density function with area under curve), and the four key distributions — Normal (bell-shaped, for means with known variance), t-distribution (for means with unknown variance, flatter tails), Chi-square (for variance, skewed, positive-only), and F-distribution (ratio of variances) — each have specific properties, parameters, and applications in econometric testing.
🧠 Quick Revision Questions
- What are the three conditions needed to confirm a true maximum for a function of two variables?
- How does a saddle point differ from a local minimum or maximum in terms of second-order partial derivatives?
- What is the Hessian determinant, and what does it test for in multivariate optimization?
- What is the difference between a probability mass function (PMF) and a probability density function (PDF)?
- For each of the four major distributions (Normal, t, Chi-square, F), state one specific use in econometrics and one key property.
📘 Lecture 06 — The Simple Regression Model
📖 Overview: This lecture introduces the fundamental concept of regression analysis, a statistical method to model the relationship between a dependent variable and one or more independent variables. It explains how to estimate this relationship using a simple linear model and the Ordinary Least Squares (OLS) technique, which is the foundation for understanding more complex econometric models. This lecture is crucial as it provides the core tools for quantifying how changes in one variable affect another, a key skill for business analysis.
🗂️ Topics Covered
This lecture covers the basic concept of regression, distinguishing it from correlation. It introduces the simple regression model, the use of scatter diagrams to identify linear relationships, and the process of estimating the model's parameters (intercept and slope) using the Ordinary Least Squares (OLS) method. The lecture also derives the "normal equations," provides a step-by-step example with calculations, and discusses the interpretation of the results, including trend values, residuals, and the key assumptions and properties (BLUE) of the OLS estimators.
📝 Lecture Summary
The Basic Concept
Regression is a statistical measure to determine the strength of the relationship between a dependent variable (explained variable, response variable, or regressand) and a list of independent variables (explanatory variables, predictor variables, or regressors). It is a process of estimating the relationship among variables, focusing on how and how much the dependent variable changes in response to changes in the independent variable(s). This differs from correlation, as the goal is to understand the causal or predictive nature of the relationship. For example, in a demand function, we want to know how quantity demanded changes due to changes in price. The dependent variable is denoted by Y and the regressor(s) by X.
Since the relationship is not exact (e.g., the same X can yield different Y values), regression analyzes how Y changes, on average, in response to changes in X. Before performing regression, we must have an idea about the nature of the functional relationship (e.g., linear, quadratic, exponential). This insight can be gained by examining a scatter diagram.
Scatter Diagram
A scatter diagram shows pairs of actual observations, typically plotting the dependent variable (Y) against an explanatory variable (X) to observe a pattern. For instance, the diagram below shows that expenditure on food is a direct (increasing) function of income levels. The plotted points do not lie exactly on a straight line but are scattered around a hypothetical straight line, suggesting a positive linear relationship.
Similarly, the second diagram shows that annual sales seem to be inversely related to the price of the commodity, as the dots of observations are scattered around a hypothetical straight line that is negatively sloped.
Remember that straight lines are shown by equations of the type Y = a + bX where a is the y-intercept (the point where the line intersects the Y-axis) and b is the slope of the line (the change in Y due to a one-unit change in X, or ΔY/ΔX). In simple regression, we try to estimate the best values of a and b using an appropriate technique, such as Ordinary Least Squares (OLS) .
Simple Regression Line by OLS
The relationship is captured by writing a simple regression equation: Y = a + bX + e, or Y = β₀ + β₁X + e.
- Residual (e) : Note that we have added e, which is called an error term or residual. We add this because the actual values do not exactly lie on a straight line but are scattered around it. To account for this difference, we capture it in the residual e. When we estimate the parameters ‘a’ and ‘b’, they do not provide exact estimates of the value of the dependent variable. The difference is called the error term or residual.
The equation with subscripts is: Yᵢ = β₀ + β₁Xᵢ + eᵢ. The subscript i indicates that the variable may have multiple observations. β₀ and β₁ are written instead of a and b to follow the tradition of regression analysis.
Regression Explained
The Population Regression Equation is an assumed equation that may have possibly been estimated from a population. We use samples to get the values of the parameters β₀ and β₁ as all the population may not be available or observed. The equation is Yᵢ = β₀ + β₁Xᵢ + eᵢ, where Yᵢ is the dependent variable, β₀ and β₁ are parameters, and X is the independent variable.
Regression Equation Estimation
Given Yᵢ = β₀ + β₁Xᵢ + eᵢ as the population regression equation, let a and b be the estimated values of β₀ and β₁, respectively. We estimate a and b from a sample. The estimated value of Y based on the estimated regression equation is Ŷ = a + bX, where Ŷ is the estimated value or ‘Trend Value’. Then, eᵢ = Y – Ŷ.
It is good to have low values of errors (residuals). Negative and positive errors cancel each other out, and we want to ‘magnify’ larger errors. Therefore, we focus on the ‘Square of Errors’ and try to minimize their sum. In least square estimation, we minimize the ‘Sum of Squared Residuals’ or ‘Sum of Square of Errors’.
We try to estimate the parameters a and b for which we have the minimum possible ‘Sum of Square of Residuals’. Finding the values of ‘a’ and ‘b’ in the regression equation is a minimization problem: Min Σ e².
Remember that:
- e = Y – Ŷ
- Ŷ = a + bX
- For ‘Optimization’ we take the first derivative and set it equal to zero.
- In this minimization problem, ‘a’ and ‘b’ are considered the unknowns.
The above minimization becomes: Min Σ e² = Σ (Y – Ŷ)² = Σ (Y – a – bX)². Let Z denote this expression, so we need to minimize Min Z = Σ (Y – a – bX)².
Deriving the Normal Equations
To find a and b that minimize Z, we use calculus. We partially differentiate Z with respect to ‘a’ and then with respect to ‘b’, and set each derivative to zero.
Partially differentiating Z w.r.t. ‘a’ and setting equal to zero: ∂Z/∂a = Σ 2(Y – a – bX)(-1) = 0 This simplifies to: Σ (Y – a – bX) = 0 Applying summation algebra: Σ Y – na – b Σ X = 0 This gives the first normal equation: Σ Y = na + b Σ X
Partially differentiating Z w.r.t. ‘b’ and setting equal to zero: ∂Z/∂b = Σ 2(Y – a – bX)(-X) = 0 This simplifies to: Σ (Y – a – bX)(X) = 0 Applying summation algebra: Σ XY – a Σ X – b Σ X² = 0 This gives the second normal equation: Σ XY = a Σ X + b Σ X²
In summary, to estimate a linear regression line Yᵢ = β₀ + β₁Xᵢ + eᵢ, where ‘a’ is a sample estimate of β₀ and ‘b’ is a sample estimate of β₁, we minimized the Sum of Squared Residuals, Min Z = Σ (Y – a – bX)². As a result, we got two normal equations. To find the values of ‘a’ and ‘b’, we need some observations of X and Y. We can solve these two equations simultaneously.
Finding the Values of Parameters Directly
Instead of solving two normal equations, we can derive direct expressions for a and b. Starting from the normal equations:
- Σ Y = na + b Σ X
- Σ XY = a Σ X + b Σ X²
From the first normal equation, dividing by n gives: ΣY/n = a + b(ΣX/n), which yields a = (ΣY/n) – b(ΣX/n) or a = Ȳ – bX̄, where Ȳ and X̄ are the means of Y and X.
Substituting this value of ‘a’ into the second normal equation and solving for b gives the formula for the slope coefficient: b = (n Σ XY – Σ X Σ Y) / (n Σ X² – (Σ X)²)
And a = Ȳ – bX̄
Example
Consider the example where X = Income (in thousand rupees) and Y = Expenditure on food items (thousand rupees).
| Observation # | X | Y | XY | X² |
|---|---|---|---|---|
| 1 | 25 | 20 | 500 | 625 |
| 2 | 30 | 24 | 720 | 900 |
| 3 | 35 | 32 | 1120 | 1225 |
| 4 | 40 | 33 | 1320 | 1600 |
| 5 | 45 | 36 | 1620 | 2025 |
| Totals | 175 | 145 | 5280 | 6375 |
The normal equations are: Σ Y = na + b Σ X and Σ XY = a Σ X + b Σ X². Substituting values gives: 145 = 5a + 175b and 5280 = 175a + 6375b.
Solving these simultaneously gives a = 0.3 and b = 0.82. The regression line can be written as Ŷ = 0.3 + 0.82 X.
Using the direct formula for b: b = (5 * 5280 – 175 * 145) / (5 * 6375 – 175²) b = (26400 – 25375) / (31875 – 30625) b = 1025 / 1250 = 0.82
The value of a is: a = Ȳ – bX̄ = (145/5) – 0.82 * (175/5) = 29 – 0.82 * 35 = 29 – 28.7 = 0.3
Interpretation: The value of a (0.3) is the Y-intercept. The value of b (0.82) is the slope of the line. This means that a one unit (1000 rupees) change in X (income) brings a 0.82 unit change in Y (expenditure on food), on the average. An increase in income by 1000 rupees may increase expenditure on food items by 820 rupees.
Trend Values and Errors
We can substitute the values of X in the estimated regression equation to find Trend Values (Ŷ).
| Observation # | X | Y | Trend Value (Ŷ) | Residual (e = Y – Ŷ) | e² |
|---|---|---|---|---|---|
| 1 | 25 | 20 | 20.8 | -0.8 | 0.64 |
| 2 | 30 | 24 | 24.9 | -0.9 | 0.81 |
| 3 | 35 | 32 | 29.0 | 3.0 | 9.00 |
| 4 | 40 | 33 | 33.1 | -0.1 | 0.01 |
| 5 | 45 | 36 | 37.2 | -1.2 | 1.44 |
| Totals | 175 | 145 | 145 | Zero | 11.9 |
The first trend value is computed as Ŷ = 0.3 + 0.82(25) = 20.8. If you change the values of a and b and compute new squares of errors, the new sum would be larger than 11.9 (the sum of least squares).
The Error Term
We assume that errors are normally distributed with zero mean and constant variance: e ~ N(0, σ²). As noticed during estimation, Σ eᵢ = 0 and you can verify that Σ eᵢXᵢ = 0. Also, as we got the regression equation by minimization process, Σ e² is minimum.
Nature of the Error Term:
- The error term may represent the influence of variables NOT included in the model (Missing Variables).
- Even if all variables are included, there will remain randomness in the error as human behavior is not perfectly predictable.
- The error term (e) may represent ‘Measurement Error’ when data is collected through rounding or observation in ranges.
Assumptions of OLS Estimators (Gauss-Markov assumptions)
- Linear in Parameters: The model is linear in its parameters.
- Random Sampling: A random sample of n observations is used.
- Sample variation in explanatory variable (Xᵢ): The Xᵢ values are not all the same.
- Zero conditional mean: The error e has an expected value of 0, given any values of the explanatory variable. E(e|X) = 0.
- Homoskedasticity: The error has the same variance given any value of the explanatory variable. Var(e|X) = σ².
BLUE: Best Linear Unbiased Estimators
Under the Gauss-Markov Assumptions, the OLS estimators are Best, Linear, and Unbiased.
- Linear: The model is linear in parameters (e.g., Y = a + bX is linear, Y = a + b²X is not, but Y = a + bX + cX² is fine). ‘Linear’ means the slope coefficient can be expressed as a linear function of Y.
- Unbiased: A parameter is unbiased if the average value of the estimator in repeated samples is equal to the true population parameter. E(bⱼ) = βⱼ.
- Best / Efficient: A parameter is best if its variance is less than any other estimator of the parameter. Var(b) ≤ Var(b̃), where b̃ is any other unbiased estimator of β₁.
🔑 Definition — Slope Coefficient (b): The slope coefficient (b) represents the estimated change in the dependent variable (Y) for a one-unit change in the independent variable (X), on average. 📐 Formula: b = (n Σ XY – Σ X Σ Y) / (n Σ X² – (Σ X)²) → This formula gives the sample estimate of the population slope parameter (β₁). 📌 Example: In the income-expenditure example, b = 0.82. This means that for every additional 1000 rupees in income, the expected expenditure on food increases by 820 rupees.
⭐ Key Takeaways
The core concept of this lecture is the simple linear regression model, which is used to quantify the relationship between a dependent and an independent variable. The OLS method is the standard technique for estimating the parameters (slope and intercept) of this model by minimizing the sum of squared residuals. You must be able to derive and solve the two normal equations (Σ Y = na + b Σ X and Σ XY = a Σ X + b Σ X²) to find the intercept (a) and slope (b). Finally, it is critical to understand the BLUE property, which states that under the Gauss-Markov assumptions, the OLS estimators are the Best (minimum variance), Linear, and Unbiased estimators of the regression parameters, making them the preferred choice.
🧠 Quick Revision Questions
- What is the key difference between regression and correlation?
- State the two 'normal equations' used to estimate the parameters (a and b) of a simple linear regression model.
- A regression analysis yields b = 0. Using all other data, would this be a useful model? Why or why not?
- What does the "zero conditional mean" assumption (E(e|X) = 0) imply for OLS estimation?
- Under the Gauss-Markov assumptions, what does the "Best" property in "BLUE" refer to?
📘 Lecture 07 — Estimation and Testing in Regression Analysis
📖 Overview: This lecture focuses on interpreting regression coefficients, measuring prediction accuracy, and testing the statistical significance of regression results. It explains standard error calculations, hypothesis testing for slope coefficients, and goodness-of-fit measures using R-squared and F-tests, with practical examples and Microsoft Excel applications.
🗂️ Topics Covered
The lecture covers interpretation of intercept (β₀) and slope (β₁) coefficients, calculation of standard error of estimate and standard error of 'b', the t-test for significance of the slope coefficient, the coefficient of determination (R²) for explanatory power, the F-test for overall model fit, and practical implementation using Excel formulas including LINEST and TREND functions.
📝 Lecture Summary
Example
Consider data where X = Income in thousand rupees and Y = expenditure on food items (thousand rupees):
| Sr. # | X | Y | XY | X² | Ŷ | e = Y - Ŷ | e² |
|---|---|---|---|---|---|---|---|
| 1 | 25 | 20 | 500 | 625 | 20.8 | -0.8 | 0.64 |
| 2 | 30 | 24 | 720 | 900 | 24.9 | -0.9 | 0.81 |
| 3 | 35 | 32 | 1120 | 1225 | 29 | 3.0 | 9 |
| 4 | 40 | 33 | 1320 | 1600 | 33.1 | -0.1 | 0.01 |
| 5 | 45 | 36 | 1620 | 2025 | 37.2 | -1.2 | 1.44 |
| Totals | 175 | 145 | 5280 | 6375 | 145 | Zero | 11.9 |
Calculations:
b = [n∑XY − (∑X)(∑Y)] / [n∑X² − (∑X)²] b = [5(5280) − (175)(145)] / [5(6375) − (175)²] = 0.82
a = Ȳ − bX̄ = (145/5) − 0.82(175/5) = 0.3
Interpretation of Regression Coefficients
The Intercept 'a' (usually also denoted by β₀) This is the y-intercept of the straight line and indicates the value of Y when X is zero; usually the base or the initial value. Examples: Autonomous Consumption in the Keynesian Consumption function, Autonomous Investment in the Investment function.
The Slope Coefficient 'b' (Usually denoted also by β₁) In linear regression lines, it shows the average unit change in the dependent variable due to one unit change in the independent variable. In a linear equation, it can also be called the derivative of the dependent variable with respect to the independent variable. The slope coefficient b is an unbiased estimate of the population regression coefficient β₁.
Standard Error of Estimate/Standard Error of Regression
The standard error of the estimate is a measure of the accuracy of predictions. It is the standard deviation of errors and defined as:
🔑 Definition — σ̂ₑ = √[∑eᵢ² / (N − k)]
Where N = Number of observations, k = number of restrictions imposed which is equal to number of parameters, N − k = degrees of freedom.
Standard Error of 'b' (for regression with one independent variable)
se(b) = σ̂ₑ / √[∑(X − X̄)²] Or equivalently: se(b) = σ̂ₑ / √[∑X² − (∑X)²/N]
In the example: σ̂ₑ = √(11.9 / (5−2)) = √(11.9/3) = 1.9916
se(b) = 1.9916 / √[6375 − (175)²/5] = 1.9916 / 15.81139 = 0.12596
Testing for the significance of slope coefficient: Individual variable significance test
The procedure is as follows:
- H₀: b = 0
- H₁: b ≠ 0
- α = 0.05 (or 0.01)
- Level of significance: Probability of type I error (rejecting a true hypothesis)
🔑 Test statistic: t = b / se(b)
Region of Rejection: |t| > t_α/2, n−k
As this is a two-tailed test, search for the value of t in the table of t-distribution corresponding to α/2 and n−k.
In this example: t = 0.82 / 0.12596 = 6.51
Looking into the table of t-distribution: t_0.025,3 = 3.182
As |t| > t_α/2, n−k: 6.51 > 3.182
So we reject H₀ and conclude that 'b' is significant and the variable X has a significant impact on the variable Y.
Reading the value of t-distribution in Microsoft Excel (2007 or later): Type in any cell and press ENTER: = T.INV.2T(0.05,3) The cell will display the value 3.182. This formula has two parameters: the first is the level of significance and the second is the d.f. (N − K).
💡 Why this matters: The t-test tells us whether the independent variable actually has a statistically significant effect on the dependent variable, or whether the relationship we observe could be due to random chance.
The Coefficient of Determination: Explanatory power of the model
The proportion of variation in Y that is explained by X:
R² = Explained Variation / Total Variation = ∑(Ŷ − Ȳ)² / ∑(Y − Ȳ)²
Since Total Variation = Explained Variation + Unexplained Variation:
R² = 1 − [Unexplained Variation / Total Variation] = 1 − [∑e² / ∑(Y − Ȳ)²]
Which can also be transformed as:
R² = 1 − [N∑e²] / [N∑Y² − (∑Y)²]
In the current example: R² = 1 − [5(11.9)] / [5(4385) − (145)²] = 0.934
Interpretation: The explanatory power of the model is 93.4%. Or: With the simple regression model, the variations in X can explain 93.4% of variation in Y.
Note: The coefficient of determination is equal to the square of the correlation coefficient only in case of the simple regression line with one independent variable.
The Goodness of Fit Test: Using the F-distribution
The procedure is as follows:
- H₀: The Fit is not good
- H₁: The Fit is good
- α = 0.05 (or 0.01)
- Test statistic: F = [R² / (1−R²)] × [(N−k) / (k−1)]
Region of Rejection: F > F_α, k−1, N−k
In the current example: F = [0.934 / (1−0.934)] × (3/1) = 42.4
Looking into the table of F-distribution: F_0.05, 1,3 = 10.13
As F > F_α, k−1, N−k: 42.4 > 10.13
So we reject H₀ and conclude that The Fit is Good.
Reading the value of F-distribution in Microsoft Excel (2007 or later): Type in any cell and press ENTER: = F.INV.RT(0.05,1,3) The cell will display the value 10.13. This formula has three parameters: the level of significance, k−1, and N−k.
Using basic Microsoft Excel Formulas: intercept and slope
To estimate Y = a + bX + e, find the values of parameter 'a' (intercept) and parameter 'b' (slope) in Microsoft Excel:
- Value of 'a': = intercept (Cell range of Values of Y, Cell range of Values of X)
- Value of 'b': = slope (Cell range of Values of Y, Cell range of Values of X)
The cells will display values 0.3 and 0.82.
Using basic Microsoft Excel Formulas: the LINEST formula
Syntax: LINEST(known_y's, [known_x's], [const], [stats])
Steps:
- Select cells D3 to E7 (2 columns and five rows to display results)
- Type = LINEST(B3:B7, C3:C7, TRUE, TRUE)
- Press and hold Ctrl + SHIFT and press ENTER (Ctrl + SHIFT + ENTER)
Results are displayed as:
- value of 'b' | value of 'a'
- standard Error of 'b' | standard error of 'a'
- R-squared | standard error of estimate
- F-statistic | N−k
- Regression Sum of Squares | Residual SS = Sum of square of errors
💡 Why this matters: The LINEST function provides a complete regression output in a single array, giving you all key statistics (coefficients, standard errors, R², F-statistic) needed for econometric analysis without manual calculations.
Understanding the TREND formula
You may compute all the trend values without calculating the regression coefficients by using this formula. Note the $ signs; they are inserted around the letter so that the reference does not change when you extend the formula to other cells by dragging. Drag the formula to get other trend values.
⭐ Key Takeaways
The slope coefficient (b=0.82) indicates that for every thousand rupee increase in income, food expenditure increases by 0.82 thousand rupees. The standard error of the estimate (σ̂ₑ=1.9916) measures the average prediction error. The t-test (t=6.51 > critical value 3.182) confirms that income has a statistically significant effect on food expenditure. The R² value of 0.934 indicates that income explains 93.4% of the variation in food expenditure, and the F-test (F=42.4 > critical value 10.13) confirms the overall model fit is good.
🧠 Quick Revision Questions
- How do you calculate the standard error of the estimate (σ̂ₑ), and what does it measure?
- What are the null and alternative hypotheses when testing the significance of the slope coefficient b?
- If R² = 0.934, what does this value tell you about the relationship between X and Y?
- How is the F-statistic calculated from R², and what is its rejection rule?
- How would you use the LINEST function in Excel to obtain regression statistics for a simple linear regression?
📘 Lecture 08 — Examples Using Microsoft Excel
📖 Overview: This lecture demonstrates the step-by-step procedure for performing Ordinary Least Squares (OLS) regression using Microsoft Excel formulas. It covers the manual calculation of all key regression statistics — coefficients, standard errors, t-statistics, R-squared, and F-statistic — and verifies these results using Excel’s LINEST function. The lecture also introduces unit of measurement changes and transformation of non-linear models for OLS estimation.
🗂️ Topics Covered
This lecture covers the manual calculation of regression parameters ‘a’ and ‘b’ using Excel formulas, followed by computation of the standard error of estimate and standard error of b. It then explains how to test individual variable significance using t-statistics and assess goodness of fit using the F-distribution and R-squared. The LINEST Excel formula is introduced to verify all results. The lecture concludes with change of unit of measurement, non-linear model transformations (including Cobb-Douglas and various functional forms), interpretation of coefficients across models, and the concept of outliers in regression.
📝 Lecture Summary
Model Exam Question
The lecture presents a model exam question requiring the student to calculate regression parameters ‘a’ and ‘b’ for the model Y = a + b X + e using learned formulas, compute standard error of estimate and standard error of b, calculate t-values for variable significance, compute the coefficient of determination, calculate the F-statistic for goodness of fit, and verify all results using the LINEST Excel formula. Students are instructed to interpret results at each step and use the equation editor for formulas.
Calculating Regression Coefficients ‘a’ and ‘b’
The data is entered in Excel with P as X and Q as Y. The slope coefficient b is calculated using the formula: 📐 Formula: b = (n ΣXY − ΣX ΣY) / (n ΣX² − (ΣX)²) → This formula computes the slope of the regression line by dividing the covariance of X and Y by the variance of X. Using the given data: b = (10 * C14 − B14 * A14) / (10 * D14 − (B14)²) = −0.84737
The intercept a is calculated using: 📐 Formula: a = Ȳ − b X̄ → The intercept is the mean of Y minus the slope times the mean of X. Using the data: a = (A14/10) − K6 * (B14/10) = 340.8812
Verification is done using Excel’s SLOPE and INTERCEPT functions: =SLOPE(A4:A13,B4:B13) for b =INTERCEPT(A4:A13,B4:B13) for a
The regression equation is: Ŷ = 340.8812 − 0.84737 X
🔑 Definition — Interpretation of ‘b’ (the slope coefficient): For every one unit change in X, there may be, on the average, 0.84737 unit change in Y in the inverse direction.
Standard Error of Estimate and Standard Error of b
The standard error of estimate (σ̂ₑ) measures the typical distance between observed values and the regression line. 📐 Formula: σ̂ₑ = √(Σeᵢ² / (N − k)) = 10.63916 (Where N = number of observations, k = number of parameters estimated)
The standard error of b (se(b)) measures the precision of the slope estimate. 📐 Formula: se(b) = σ̂ₑ / √(ΣX² − (ΣX)²/N) = 0.08668
Individual Variable Significance Test
This test determines whether the independent variable X has a statistically significant effect on Y.
Hypotheses: H₀: b = 0 (X has no effect on Y) H₁: b ≠ 0 (X has a significant effect on Y) α = 0.05
Test statistic: 📐 Formula: t = b / se(b) = −0.84737 / 0.08668 = −9.77614
Region of Rejection: |t| > t_(α/2, n−k) t_(0.025, 8) = 2.306
✅ Conclusion: Since the absolute value of t-statistic (9.77614) is greater than 2.306, we reject H₀ and conclude that ‘b’ is significant.
The Goodness of Fit Test: Using the F-distribution
This test evaluates whether the overall regression model provides a good fit to the data.
Hypotheses: H₀: The fit is not good H₁: The fit is good α = 0.05
Coefficient of Determination (R²): 📐 Formula: R² = 1 − (Σe² / (ΣY² − (ΣY)²/N)) = 0.9228 🔑 Definition — R-squared: The proportion of the total variation in Y that is explained by the regression model.
F-statistic: 📐 Formula: F = (R² / (k−1)) / ((1−R²) / (N−k)) = (0.9228 / 1) / ((1−0.9228) / 8) = 95.573
Region of Rejection: F > F_(α, k−1, N−k) = F > F_(0.05, 1, 8) = 5.3177
✅ Conclusion: Since 95.573 > 5.3177, we reject H₀ and conclude that the FIT IS GOOD.
Verifying Results with the LINEST Formula
Syntax: LINEST(known_y's, [known_x's], [const], [stats])
Procedure:
- Select cells D18:D22
- Type: =LINEST(A4:A13,B4:B13,TRUE,TRUE)
- Press Ctrl + SHIFT + ENTER (array formula)
Output layout:
| Row | Cell 1 | Cell 2 |
|---|---|---|
| 1 | value of 'b' | value of 'a' |
| 2 | standard Error of 'b' | standard error of 'a' |
| 3 | R-squared | standard error of estimate |
| 4 | F-statistic | N-k |
| 5 | Regression Sum of Squares | Residual SS = Σe² |
All values displayed by LINEST are identical to the manual calculations. The t-statistic can be calculated by dividing the value of ‘b’ by the cell containing the standard error of b.
Change of Unit of Measurement
Sometimes units are changed for larger or smaller values to make calculations easier and coefficients more meaningful.
- Examples: Population in millions, Prices in thousand rupees
🔑 Definition — Linear transformation: Dividing or multiplying all values of a variable by a constant factor. This preserves linear relationships between variables; therefore, correlation between x and y does not change.
Transformation of Models and Use of OLS
Some models are not linear but can be transformed to use OLS estimation.
🔑 Definition — Nonlinear transformation: A transformation that changes (increases or decreases) linear relationships between variables. Correlation between variables changes.
- Examples: Taking natural logs, squares, square roots, or reciprocals of a variable
Cobb-Douglas Production Function Example: Original: Q = A Lᵅ Kᵝ Reduced to one variable: Q = A Kᵝ Taking log on both sides: ln Q = ln A + β ln K Let Y = ln Q, α = ln A, and X = ln K, then: Y = α + βX (estimable by OLS)
Summary of Functional Forms:
| Method | Transformation | Regression Equation |
|---|---|---|
| Standard linear regression | Not required | y = b₀ + b₁x |
| Exponential model (log-linear) | Dependent variable = log(y) | log(y) = b₀ + b₁x |
| Logarithmic model (linear-log) | Independent variable = log(x) | y = b₀ + b₁log(x) |
| Double log functional form | Both variables = log | log(y) = b₀ + b₁log(x) |
| Cobb-Douglas Production Function | ln Y = ln A + α ln L + β ln K | ln Y = ln A + α ln L + β ln K |
Interpretation of Different Functional Forms:
| Model | Interpretation | Marginal Effect | Elasticity |
|---|---|---|---|
| Linear: Y = a + bX | One unit change in X → 'b' units change in Y | b | b (X/Y) |
| Double log: ln Y = a + b ln X | One % change in X → 'b' % change in Y | b (Y/X) | b |
| Level-Log: Y = a + b ln X | One % change in X → b/100 units change in Y | b/X | b/Y |
| Log-Level: ln Y = a + b X | One unit change in X → approximately (b×100)% change in Y | bY | bX |
💡 Why this matters: For interpretation, we assume that Gauss-Markov assumptions hold and parameters are significant. The marginal effect is the partial derivative of Y with respect to X, and elasticity may be computed at mean values of X and Y.
Outliers in Regression
🔑 Definition — Outliers: Points that diverge a lot from the data in general and may affect the slope of the regression equation or the predictive power of the model.
Types of Outliers:
- Extreme X values
- Extreme Y values
- Extreme X and Y values
- Distant point with normal X or Y value
🔑 Definition — Influential point: If the removal of an outlier changes the slope or coefficient of determination substantially, it may be called an influential point.
Two diagrams in the lecture illustrate outliers as dots that appear far from the main cluster of data points.
⭐ Key Takeaways
The OLS regression parameters must be computed systematically: first the slope ‘b’ using the covariance-variance ratio formula, then the intercept ‘a’ using the means of X and Y. The standard error of b and the t-statistic are essential for testing whether the independent variable is statistically significant. Goodness of fit is evaluated through R-squared (proportion of explained variation) and the F-statistic, which compares explained to unexplained variance; a significant F confirms the model fits well. The LINEST function in Excel provides a convenient verification tool, returning all key regression statistics in a compact array. Finally, understanding functional form transformations (log-log, level-log, log-level) is critical because they change how coefficients are interpreted — particularly the difference between marginal effects and elasticities — and being aware of outliers is essential for assessing the robustness of regression results.
🧠 Quick Revision Questions
- What formula is used to calculate the slope coefficient ‘b’ in OLS regression, and how is it computed using Excel cell references?
- How is the standard error of b calculated, and what does it measure about the regression coefficient?
- In the individual variable significance test, what is the test statistic formula and decision rule using the t-distribution?
- How is the F-statistic derived from R-squared, and what conclusion does an F-statistic of 95.573 with critical value 5.3177 lead to?
- For a log-log model where ln Y = a + b ln X, how would you interpret the coefficient ‘b’, and how does this differ from the interpretation of ‘b’ in a simple linear model?