π Lecture 23 β Estimating Non-Linear equation by OLS
π Overview: This lecture covers how to estimate non-linear relationships using Ordinary Least Squares (OLS) by introducing polynomial terms and logarithmic transformations. It explains the methodology for estimating quadratic, cubic, and Cobb-Douglas production functions, along with the interpretation of different functional forms and key diagnostic checks.
ποΈ Topics Covered
Estimating quadratic equations by OLS with U-shaped and inverted U-shaped curves; derivation of normal equations; practical example using Excel's LINEST function and centering X to reduce multicollinearity; estimation of cubic equations for functions like total cost; derivation of normal equations for four parameters; transformation and estimation of the Cobb-Douglas production function using logs; a comprehensive table of transformations and interpretation of different functional forms (linear, double-log, level-log, log-level); and a checklist of seven critical questions to evaluate OLS models.
π Lecture Summary
Estimating Quadratic Equation by OLS
Quadratic equations capture U-shaped (convex) or inverted U-shaped (concave) relationships between variables. To estimate such relationships, we include the square of the independent variable in the regression. For U-shaped (convex) curves, the coefficient of the quadratic term is positive. For inverted-U-shaped (concave) curves, the coefficient of the quadratic term is negative.
The quadratic equation (ignoring subscripts) is: ν = ν + ννΏ + ννΏΒ² + ν
To find the parameters a, b, and c, we minimize the sum of squared residuals: Min Z = Ξ£νΒ² = Ξ£(ν β νΜ)Β² = Ξ£(ν β ν β ννΏ β ννΏΒ²)Β²
Taking partial derivatives and setting them to zero yields three normal equations:
- Ξ£ν = νν + νΞ£ν + νΞ£νΒ²
- Ξ£νν = νΞ£ν + νΞ£νΒ² + νΞ£νΒ³
- Ξ£νΒ²ν = νΞ£νΒ² + νΞ£νΒ³ + νΞ£νβ΄
The practical approach uses Excel's LINEST function.
π Example I: Given data for Y and X (n=15), the LINEST output provides coefficients. The estimated regression line is: Y = 18.26 - 4.01934 X + 0.44 XΒ² The SSR from a linear equation is 930.5107.
Example I Modified: Centering X
When X is centered (x = X - XΜ), the correlation between x and xΒ² becomes zero, eliminating multicollinearity. This improves prediction accuracy. The estimated regression line is: Y = 14.51 - 3.082 x + 0.44 xΒ² The correlation between X and XΒ² is 0.97, but between x and xΒ² it is 0.
π Example II: Another dataset yields a concave relationship with a negative quadratic term. The LINEST output gives: Y = 48.97 + 5.099 X - 0.4955 XΒ² The SSR from a linear equation is 1308.171.
Estimating Cubic Equation by OLS
Cubic equations model relationships like the Total Cost function. For this shape, the coefficient of the quadratic term is expected to be negative and the coefficient of the cubic term should be non-zero.
The cubic equation is: ν = ν + ννΏ + ννΏΒ² + ν νΏΒ³ + ν
Minimizing the sum of squared residuals leads to four normal equations:
- Ξ£ν = νν + νΞ£ν + νΞ£νΒ² + νΞ£νΒ³
- Ξ£νν = νΞ£ν + νΞ£νΒ² + νΞ£νΒ³ + νΞ£νβ΄
- Ξ£νΒ²ν = νΞ£νΒ² + νΞ£νΒ³ + νΞ£νβ΄ + νΞ£νβ΅
- Ξ£νΒ³ν = νΞ£νΒ³ + νΞ£νβ΄ + νΞ£νβ΅ + νΞ£νβΆ
π Example: Using given Y and X (n=15), the LINEST output for a cubic function yields: Y = 1023.7 + 221.87 X - 1.306 XΒ² + 0.00000984 XΒ³ The SSR from a linear equation is 45562.33.
Estimating Cobb-Douglas Production Function
The Cobb-Douglas Production Function is: ν = ν΄ ν^Ξ± ν^Ξ²
By taking natural logs, it becomes linear in parameters: ln ν = ln ν΄ + Ξ± ln ν + Ξ² ln ν
Letting ν = ln ν, νΌ = ln ν΄, νΏ = ln ν, νΎ = ln ν: ν = νΌ + Ξ± νΏ + Ξ² νΎ
This linear form can be estimated by OLS.
π Example: Given output (Y), labor (L), and capital (K) data, the log-transformed variables produce OLS estimates. The estimated function is: ln Y = 0.283 + 0.41 ln L + 0.32 ln K
- Elasticity (labor) = 0.41 (a 1% change in labor leads to a 0.41% change in output)
- Elasticity (Capital) = 0.32
- Returns to scale = 0.73 (sum of elasticities; less than 1 indicates decreasing returns to scale)
Transformation of models and Use of OLS
| Method | Transformation | Regression equation |
|---|---|---|
| Standard linear | Not required | y = b + b x |
| Exponential model | Dependent variable = log(y) | log(y) = b + b x |
| Logarithmic model | Independent variable = log(x) | y = b + b log(x) |
| Double log | Dependent & Independent = log(y), log(x) | log(y) = b + b log(x) |
| Cobb Douglas | ln Y = ln A + Ξ± ln L + Ξ² ln K | ln Y = ln A + Ξ± ln L + Ξ² ln K |
Interpretation of different functional forms using OLS
| Model | Interpretation | Marginal Effect | Elasticity |
|---|---|---|---|
| Linear in variable Y = a + bX | One unit change in X causes 'b' units change in Y | b | b * (X/Y) |
| Double log form (log-log) | One percent change in X causes 'b' % change in Y | b * (Y/X) | b |
| Level-Log Y = a + b ln X | One percent change in X changes Y by b/100 units | b/X | b/Y |
| Log-Level form ln Y = a + b X | One unit change in X changes Y by (b*100)% | bY | bX |
For interpretation, we assume Gauss-Markov assumptions hold and parameters are significant. The marginal effect of X is the partial derivative of Y w.r.t. X. Marginal effect and elasticity (dy/dx * X/Y) may be computed at mean values of X and Y.
Problems with OLS
We must take care of:
- Outliers
- Non-linearity
- Wrong specification
- Missing Variables
- Multicollinearity
- Heteroskedasticity
- Autocorrelation
Taking care in use of OLS: 7 Questions
- Are the explanatory variables helping the model?
- Are the relationships what we expect?
- Are any of the explanatory variables redundant?
- Are the residuals normal?
- Have all important variables been included?
- How well the model explains the dependent variable?
- Are the results free from Multicollinearity, Heteroskedasticity, and Autocorrelation?
Remember:
- Explanatory variables can have categories.
- We can use dummy variables for categories.
- If the dependent variable is categorical, do not use simple regression.
- Take care of origin and scale for interpretable coefficients.
- Models not linear in parameters cannot be transformed for OLS.
π‘ Why this matters: Understanding how to estimate non-linear relationships allows you to model real-world economic phenomena like cost curves, production functions, and diminishing returns that cannot be captured by simple linear models.
β Key Takeaways
Estimating non-linear equations with OLS requires introducing polynomial terms (squares, cubes) or taking logarithmic transformations. For quadratic equations, the sign of the squared term determines whether the curve is U-shaped (positive) or inverted U-shaped (negative). Centering the X variable eliminates multicollinearity between X and XΒ². The Cobb-Douglas production function is linearized by taking natural logs of both sides, and the coefficients directly represent elasticities (labor elasticity and capital elasticity), with their sum indicating returns to scale. Different functional forms (linear, log-log, level-log, log-level) have distinct interpretations for marginal effects and elasticities. Finally, always evaluate a model using seven diagnostic questions about variable significance, expected signs, normality, specification, explanatory power, and freedom from multicollinearity, heteroskedasticity, and autocorrelation.
π§ Quick Revision Questions
- What is the sign of the coefficient of the quadratic term for a U-shaped (convex) curve?
- Write down the three normal equations needed to estimate a quadratic regression equation by OLS.
- What is the advantage of centering the independent variable X in a quadratic regression?
- For a Cobb-Douglas production function estimated as ln Y = 0.283 + 0.41 ln L + 0.32 ln K, what are the labor elasticity and returns to scale?
- If the model is ln Y = a + bX, how do you interpret the coefficient 'b'?
π Lecture 24 β Introduction to Stata- I
π Overview: This lecture introduces the Stata statistical software environment, its interface components, file types, and basic commands. It covers how to navigate Stata's graphic user interface and command window, open different types of data files, and use essential commands like help, describe, and list to explore datasets. This foundational knowledge is critical for conducting econometric analysis in Stata.
ποΈ Topics Covered
The lecture begins with an overview of the Stata environment showing five windows: output, command, variable, properties, and review. It then discusses file types used in Stata including .dta, .log, and .do files, and three types of data files. The graphic user interface and command window are explained with their advantages. The help command syntax and structure are covered in detail, followed by methods for opening files. Finally, the describe and list commands are explained with their three usage patterns and examples.
π Lecture Summary
The Stata Environment
The Stata interface consists of five visible windows. The main window (output window) displays all output. Below it is the command window where users type and execute commands. While Stata has a good graphic user interface, the command line allows for quick and efficient work. On the right side, the variable window displays all variables, and below it the properties window shows variable properties like names, storage types, and display format. The review window on the left displays all commands executed in the current session; clicking any command executes it again without retyping.
Several file types are used in Stata:
- Data files have extension .dta (can type without extension)
- Log files record sessions (.smcl or .log)
- Do files perform multiple operations at once (.do)
π Definition β .dta files: The data file format used by Stata for storing datasets.
Types of Data Files (.dta)
There are three types of data files:
β’ Sample data files shipped with Stata (opened by sysuse command)
β’ Files available on the web (using the webuse command)
β’ Files created and saved on hard disk (using use command)
Other data files (like MS Excel) can be imported into Stata.
π‘ Why this matters: Knowing which command to use for different data sources allows efficient access to datasets for analysis.
The Graphic User Interface
A strong graphic user interface (GUI) is available where operations can be performed by clicking on different icons. This is very user friendly and allows users to perform operations without typing commands.
The Command Window
The command window allows typing and executing different commands. All operations available through the GUI can be performed in the command window. This is undoubtedly a very quick way of using Stata.
Advantages of using the command window: β’ Typing commands is less time consuming β’ You remember the commands this way β’ It may be helpful if you need to program in Stata
Rules for commands:
β’ Use small letters; Stata is case sensitive
β’ Short names can be used (e.g. d / des instead of describe)
β’ Syntax can be viewed in the help window
β’ Underlined part in the syntax is the shortcut
β’ Commands have options (after typing comma ,)
β’ Conditions can be used with commands (e.g. if, in etc.)
The 'help' Command
The syntax for the help command is:
π Formula: help [command or topic] [,options]
The underlined part is the minimum that should be typed for help. For example, typing h describe opens a window showing help for 'describe'.
From the graphic user interface: Help > Stata Command... and write the command in the window that appears.
Typing 'help help' opens a help window explaining the help command itself. Typing 'help describe' opens the help window for describe. Almost all help windows have the same structure: they show the command, its shortcut, its syntax, description, and practical examples of use.
Opening Files in Stata
To open different types of files:
- Sample data files shipped with Stata: Type
sysuse filenamein the command window - Files on the web: Type
webuse filename - Files on hard disk: Use
use "path and file name"oruse "path and file name", clear
The ,clear option can be used to clear existing data from memory before opening a new file.
To open a file using the graphic user interface: File > Open (browse, find file, and open)
To know which files are already available with Stata, type sysuse dir. A list of files will appear in the output window. Any one of them can be opened. For example, to open the first file, type sysuse auto.
π Example: sysuse auto opens the auto dataset. A message appears in the output window, variables appear in the variable window, variable descriptions appear in the properties window, and all typed commands appear in the review window.
The describe Command
The describe command provides a description of data in the current file. It lists the variables, labels, formats, storage type, number of observations, and date the file was created.
It can be used mainly in three ways:
β’ describe: describes all variables in memory
β’ describe variable-names: describes only the specified variables
β’ describe variable1-variablen: describes all variables in the range
To use the graphic user interface: Data > Describe data > Describe data in memory or in a file. A window appears asking for variable names (use pull-down menu if file is open).
π Example: The command sequence sysuse auto then describe produces output showing variable names, data type, display type, and any labels. This gives the current file at a glance with all variables.
π Example: describe mpg price describes only the variables mpg and price.
π Example: describe price - length describes all variables from price through length (the range).
The list Command
The list command lists rows and columns of the data file. It provides the values of observations of data.
It can be used mainly in three ways (same pattern as describe):
β’ list: lists values of all variables in memory
β’ list variable-names: lists values of only the specified variables
β’ list variable1-variablen: lists values of all variables in the range
To use the graphic interface: Data > Describe data > List data. A window appears asking for variable names using the pull-down menu. Select variables and click OK.
Using list with in and if:
Conditions can be imposed for filtered data to be displayed (using if). Specific rows of data can be displayed (data in specific rows). Conditions also may be attached.
β Key Takeaways
The Stata environment has five key windows: output, command, variables, properties, and review. Data files (.dta) come in three typesβshipped with Stata (sysuse), web-based (webuse), and local files (use). The command window is faster than GUI, requires lowercase case-sensitive commands, and allows shortcuts (the underlined part of syntax). The help command shows syntax, description, and examples for any Stata command. The describe command reveals dataset structure (variables, types, labels, observations), while the list command displays actual data values, both supporting three usage patterns (all, specific variables, or ranges) with optional conditions using in and if.
π§ Quick Revision Questions
- What are the five windows in the Stata environment and what is the function of each?
- What are the three types of .dta files and which command opens each type?
- What is the syntax for the help command and how can you find the shortcut for any command?
- What information does the describe command provide and what are its three usage patterns?
- How does the list command differ from the describe command, and how can conditions like
inandifbe used with it?
π Lecture 25 β Introduction to Stata- II
π Overview: This lecture introduces additional Stata commands for summary statistics and frequency tables. It covers how to use
summarizewith options,tabulatefor one-way and two-way tables,tabstatfor customized summary statistics, andcorrelatefor correlation/covariance matrices β essential tools for exploratory data analysis in econometrics.
ποΈ Topics Covered
The lecture covers five main Stata commands: summarize with syntax and options, tabulate for one-way and two-way frequency tables, tab1 for multiple one-way tables, tabstat for flexible summary statistics with the by() and stats() options, and correlate for correlation and covariance matrices. Each command is demonstrated using the auto dataset with examples.
π Lecture Summary
The summarize Command
The summarize command provides summary statistics for numeric variables. Its syntax is:
summarize [varlist] [if] [in] [weight] [, options]
From the menu: Statistics > Summaries, tables, and tests > Summary and descriptive statistics > Summary statistics
If the file auto is open (sysuse auto), summarize gives summary statistics of all variables in the file. Like previous commands, it can be used standalone, with specific variables, or with a range of variables.
π EXAMPLES:
summarizeβ summary for all variablessummarize price mpg lengthβ for specific variablessummarize price-lengthβ for a range of variables
π‘ Why this matters: The ability to quickly generate descriptive statistics is fundamental for any data analysis project. The in and if qualifiers allow you to subset your data, which is essential when working with large datasets.
The tabulate Command
The tabulate command provides one-way or two-way frequency tables. Its syntax is:
tabulate varname [if] [in] [weight] [, tabulate1_options]
From the menu: Statistics > Summaries, tables, and tests > Frequency tables > One-way table
The command requires a variable name. It can be used with the summarize option to get summary statistics by categories:
tabulate foreign, summarize(mpg)
This provides summary statistics of mpg according to categories in foreign. All options in Stata must follow a comma.
π Definition β tabulate options: Options like summarize() are used after a comma to modify the tabulate command's output. The summarize() option produces summary statistics of a specified variable broken down by the categories of the tabulated variable.
Tabulating Multiple Variables β tab1
The tab1 command allows tabulating multiple variables in one command. For example, to get frequency tables for rep78 and foreign in the auto file, you would type:
tab1 rep78 foreign
This is more efficient than running separate tabulate commands for each variable.
Two Way Frequency Tables
Two-way tables can be created with tabulate when two variable names are specified. For this to work well, at least one variable should not have many categories. The example in the lecture produces a two-way table (the output image shows cross-tabulation of two variables).
π‘ Why this matters: Two-way tables are essential for exploring relationships between categorical variables, such as whether foreign cars have different repair records than domestic ones.
The tabstat Command
The tabstat command displays summary statistics for a series of numeric variables. It is a good alternative to summarize because it allows you to specify the list of statistics to be displayed. Its syntax is:
tabstat varlist [if] [in] [weight] [, options]
From the menu: Statistics > Summaries, tables, and tests > Other tables > Compact table of summary statistics
π EXAMPLES:
tabstat price mpg weightβ provides means (default) of the specified variablestabstat price mpg weight, stats(mean sd min max)β provides mean, standard deviation, minimum, and maximumtabstat price mpg, by(foreign)β provides statistics by categories of a variabletabstat price mpg, by(foreign) stats(mean sd n)β combines both options
π Definition β stats() option: An option in tabstat that specifies which summary statistics to display (e.g., mean, sd, min, max, n). Without it, only means are shown.
π‘ Why this matters: tabstat gives you precise control over the summary statistics you want to see, which is very useful when you need specific measures (like standard deviations or counts) for a research paper.
The correlate Command
The correlate command provides the correlation or covariance matrix of the specified variables. Its syntax is:
correlate [varlist] [if] [in] [weight] [, correlate_options]
From the menu: Statistics > Summaries, tables, and tests > Summary and descriptive statistics > Correlations and covariances
π EXAMPLES:
correlate price mpg weightβ displays the correlation matrix- To get summary statistics along with correlations, use
summarizefollowed bycorrelate:summarize price mpg weight correlate price mpg weight - Use the
covarianceoption to get the covariance matrix instead of correlations
π Definition β correlation matrix: A table showing correlation coefficients between variables, ranging from -1 to +1, indicating the strength and direction of linear relationships. The covariance matrix shows covariances instead.
β Key Takeaways
The most critical commands introduced are summarize, tabulate, tabstat, and correlate, each serving specific descriptive analysis purposes. summarize provides quick summary statistics of numeric variables, with options to use if, in, and variable ranges. tabulate creates one-way or two-way frequency tables and can include the summarize() option for grouped summary statistics, while tab1 allows multiple one-way tables in one command. tabstat is the most flexible command for summary statistics as it lets you specify exactly which statistics to display using the stats() option and can break results down by categories using by(). Finally, correlate generates correlation or covariance matrices, essential for understanding relationships between variables, with the covariance option providing the covariance matrix instead.
π§ Quick Revision Questions
- What is the syntax of the
summarizecommand, and how can you restrict it to only variablesprice,mpg, andlength? - What is the difference between
tabulateandtab1commands in Stata? - How would you produce the mean, standard deviation, and count of
priceandmpgbyforeignusingtabstat? - What does the
correlatecommand produce by default, and how can you change it to produce a covariance matrix? - In the command
tabulate foreign, summarize(mpg), what does thesummarize(mpg)option do?
π Lecture 26 β Introduction to Stata- III
π Overview: This lecture covers essential data management techniques in Stata, including how to import data from various file formats, create and modify variables, and manage dataset metadata. Understanding these skills is crucial for efficiently preparing and cleaning data for econometric analysis.
ποΈ Topics Covered
The lecture covers importing data from Excel and other file formats, entering data manually, rules for variable naming, creating and modifying variables using generate and replace commands, arithmetic and logical operators, mathematical functions, assigning variable and data labels, creating notes, dropping or keeping variables and observations, and using log files and do files to document and automate work.
π Lecture Summary
Data Management in Stata
This section introduces the core data management capabilities in Stata, which are essential for preparing datasets for analysis. It covers importing data from external sources, entering data manually, and understanding the conventions for naming variables.
Importing Data
Data can be imported into Stata from various types of files including Microsoft Excel files, raw data files, text files, csv files etc. Several commands are available for importing data: import excel is used for .xls or .xlsx files with the syntax import excel [using] filename [, import_excel_options]. Other commands include infix for fixed format files, infile for free format files, insheet to import ASCII text data, and input to enter data from the keyboard. To import an Excel file, you can use the menu path: File > Import > Excel spread sheet (or others), then browse for the file, select "import variables names as first row", and click OK.
Entering Data Data can be entered directly by clicking on the icon 'data editor (edit)' and then entering data as you would in Excel. Variable names can be changed within this editor.
Variable Names
Variable names in Stata must follow specific conventions: they can be up to 32 characters (12 displayed), can include 0 to 9, A to Z, and underscore (_), the first letter cannot be a number, and variable names are case sensitive. Valid examples include mpg, mpg2, mpg_domestic, and _2014. The Variable manager is used to change variable names, or you can use the rename command (rename oldname newname).
Creating Variables
Variables are created and/or modified using the generate and replace commands.
The generate command creates a new variable. The value of the new variable is given by the =exp. Its syntax is: generate [type] newvar[:lblname] =exp [if] [in]. The command can also be executed using the menu: Data > Create or change data > Create new variable, then fill in the required information.
π Definition β generate: Creates a new variable in the dataset.
π Example: generate income = 0 creates a new variable named income with all values set to 0.
π Example: gen income = salary + bonus creates a new variable that is the sum of two existing variables.
π Example: gen int agesquare = age^2 creates a new integer variable that is the square of the age variable.
π Example: gen lngdp = ln(gdp) creates a new variable containing the natural log of the gdp variable.
The replace command
The replace command is used along with generate to modify existing variables. It has the syntax: replace oldvar =exp [if] [in] [, nopromote]. Using the in qualifier is helpful for targeting specific observations. From the menu, you can click: Data > Create or change data > Change contents of variable, then fill in the variable name and use the if/in ribbon to type a condition.
π Example: replace income = 1 in 5 changes the income value for the 5th observation to 1.
π Example: replace income = 1000 if income == 0 replaces all income values that equal 0 with 1000.
π Example: replace highincome = 1 if income > 5000 sets highincome to 1 for all observations where income exceeds 5000.
π Example: gen age2 = 0 followed by replace age2 = age^2 creates a new variable then fills it with squared ages.
π Example: replace age2 = age^2 , nopromote replaces values without promoting the variable type.
Operators in stata
The following operators are commonly used in Stata:
- Arithmetic: + (add), - (subtract), * (multiply), / (divide), ^ (raise to a power), + (for strings)
- Logical: ! or ~ (not), | (or), & (and)
- Relational: == (equal), != or ~= (not equal), < (less than), > (greater than), <= (less than or equal to), >= (greater than or equal to)
Functions
Type help mathfun to see all functions. Common examples include:
abs(x)β Absolute value of xint(x)β Integer of xlog(x)β Logarithmsqrt(x)β Square root of xround(x)β Round x to the nearest integer
Assigning labels to variables
Variable labels are useful for understanding what variables contain. Labels provide data labels and variable labels, which are helpful in understanding the data. Use the label variable command for assigning labels to variables. To label the dataset as a whole: label data ["label"] (e.g., label data "1978 Automobile Data"). To create variable labels: label variable varname ["label"] (e.g., label variable make "Make and Model").
Creating Notes
The note command places a note in the data, which can be general or attached to a specific variable. To create notes for a variable: notes [evarname]: text. For example: note make: this is a note for variable make. To see the note, type note make. To create general notes: note: this version is from June 2014. To see general notes, type notes or notes _dta.
Dropping or Keeping Variables
The commands drop and keep are used to drop specific rows of data or entire variables.
- To drop variables:
drop varlist - To drop observations:
drop if exp - To drop a range of observations:
drop in range [if exp] - To keep variables:
keep varlist - To keep observations that satisfy a specified condition:
keep if exp - To keep a range of observations:
keep in range [if exp]
Using log files and do files
Log files are used to record your session, using .smcl, .log, or .txt extensions depending on needs. Do files are used to run several commands at a time or replicate results. To open log files, click the notepad-like icon on the menu. You can create files with .smcl or .log extensions (the latter can be opened in Notepad). Log files record all your activity; you can suspend them to stop recording and resume later, and finally close them to view later. Do files (.do extension) are text editor files that can execute several commands at a time. Click the corresponding icon to open a do file, type a list of commands, and execute them all at once.
π‘ Why this matters: Mastering log and do files ensures reproducibility of your analysis, allowing you to efficiently rerun complex analyses and share your work with others.
β Key Takeaways
Students must remember the commands for importing data (especially import excel), the rules for valid variable names, and how to use generate and replace to create and modify variables. The correct syntax for arithmetic, logical, and relational operators, along with common mathematical functions like log() and sqrt(), are essential for data transformation. Understanding how to assign labels and create notes improves data documentation, while the drop and keep commands are critical for subsetting data. Finally, using log files for recording sessions and do files for automating analysis is vital for replicable research.
π§ Quick Revision Questions
- What Stata command would you use to import an Excel file named "sales.xlsx"?
- Explain the difference between the
generateandreplacecommands, providing an example of each. - What is the purpose of the
label variablecommand, and how is it used? - How does the
dropcommand differ from thekeepcommand when used withifconditions? - What are the two types of files discussed for recording and replicating your Stata work, and what is the main purpose of each?
π Lecture 27 β Introduction to Stata- IV
π Overview: This lecture introduces students to Stataβs graphical capabilities, covering how to create various types of graphs including histograms, pie charts, scatter plots, and line plots. It emphasizes both menu-driven and command-line methods for producing publication-quality graphics, which is essential for visualizing economic data.
ποΈ Topics Covered
The lecture covers creating histograms for continuous and categorical variables with options like normal curves and grids; constructing pie charts with labels and titles; generating two-way graphs including scatter plots with fitted lines and by-group comparisons; and producing line plots and connected plots using built-in Stata datasets.
π Lecture Summary
Introduction to Stataβs Graph Capabilities
Stata has a very good capability to produce graphs with the following properties:
- Rich set of tools
- Publishable quality of graphs
- GUI / Command line both
- Graph Editor
- Graphs can be copied/saved
You can create graphs by clicking on: Graphics menu option.
Examples
Histograms
We can create Histograms for both continuous and categorical variables.
π Definition β Histogram: A graphical representation showing the distribution of numerical data by grouping data into bins (bars) whose height represents frequency or density.
Syntax: histogram varname [if] [in] [weight] [, [continuous_opts | discrete_opts] options]
From the menu: Graphics > Histogram
π‘ Why this matters: Histograms help identify the shape, center, and spread of a variableβs distribution, which is fundamental for exploratory data analysis.
EXAMPLES:
-
Basic histogram of price:
sysuse auto histogram price -
Histogram of length:
sysuse auto histogram length -
Histogram with normal curve:
sysuse auto histogram rep78, discrete normalThe option
discretetells Stata the variable is categorical, andnormaloverlays a normal distribution curve. -
Histogram with frequency labels, grids, and custom axis:
sysuse auto histogram mpg, discrete freq addlabels ylabel(,grid) xlabel(10(5)40)freqshows frequencies instead of densitiesaddlabelsadds value labels on top of barsylabel(,grid)adds horizontal grid lines on the y-axisxlabel(10(5)40)specifies x-axis labels from 10 to 40 in steps of 5
PIE Charts
π Definition β Pie Chart: A circular statistical graphic divided into slices to illustrate numerical proportion, where the arc length of each slice is proportional to the quantity it represents.
Syntax: graph pie varlist [if] [in] [weight] [, options]
Menu: Graph > Pie
EXAMPLES:
- Pie chart showing proportion of cars by repair record:
sysuse auto graph pie, over(rep78) plabel(_all name) title("Repair Record 1978")over(rep78)splits the pie by categories of rep78plabel(_all name)labels each slice with the category nametitle()adds a title to the graph
π Example: This produces a pie chart with slices for each repair record value (1, 2, 3, 4, 5), each labeled with its name.
-
Creating a pie chart from manually entered data:
graph pie Statistics Mathematics Economics Accounting LawThis uses variable names directly as categories.
-
Pie chart with all data and percentage labels:
graph pie _all , plabel(_all name) title("Distribution of marks")_alluses all numeric variables in the dataset to create the pie slices.
Two way graphs
When more than one variable is involved, we create twoway graphs like scatter plots.
Creation Methods:
- Menu: Graphics > Twoway graph
- Click on Create to define the plot type (e.g., scatter)
- Select the variables and click Accept
- Use ribbons to select titles for axes
Command syntax: twoway (scatter yvar xvar) [, options]
Examples of Scatter Plots (first type sysuse auto, clear to open the file auto.dta):
-
Basic scatter plot with axis and title labels:
twoway (scatter price length), ytitle(Price) xtitle(Length) title(Price and Length)This creates a scatter plot with Price on y-axis and Length on x-axis, with labeled axes and a title.
-
Scatter plot separated by car type (domestic vs. foreign):
twoway (scatter price length), by(foreign)The
by(foreign)option creates separate scatter plots for each category of the variable 'foreign'. -
Scatter plot separated by repair record:
twoway (scatter price length), by(rep78)Creates separate scatter plots for each value of rep78.
-
Scatter plot with fitted linear regression line:
twoway (scatter price length) (lfit price length)scatter price lengthcreates the data pointslfit price lengthadds a linear fit (regression) line
-
Scatter plot with fitted line, separated by car type:
twoway (scatter price length) (lfit price length), by(foreign)This produces two separate scatter plots (domestic and foreign), each with its own fitted regression line.
LINE plots
π Definition β Line Plot: A graph that connects data points with straight lines, typically used to display trends over time or ordered sequences.
Syntax: [twoway] line varlist [if] [in] [, options]
Usage: Using the menu, Line plots are created in the same way as scatter plots. Dataset: Open the file sp500 which is a file available with Stata.
EXAMPLES:
-
Line plot of opening price over date:
sysuse sp500 twoway (line open date)This plots the 'open' variable (opening price) against 'date', creating a time series line graph.
-
Connected scatter plot (points connected by lines):
sysuse sp500 twoway (connected close date)The
connectedoption shows individual data points connected by lines, useful for showing both individual values and trends.
β Key Takeaways
The most critical points to remember from this lecture are: Stata enables creation of publication-quality graphs through both menu-based (Graphics menu) and command-line methods. For histograms, use the histogram command with options like discrete, normal, freq, addlabels, and custom axis labels/ticks. For pie charts, use graph pie with over() for categorical breakdowns and plabel() for slice labeling. Two-way graphs (like scatter plots) use the twoway framework, allowing multiple plot types to be combined β e.g., twoway (scatter y x) (lfit y x) adds a linear fit line. The by() option creates separate graphs for each category of a grouping variable. Line plots use the line or connected commands and are ideal for time series data.
π§ Quick Revision Questions
- What command creates a histogram with a normal curve overlay, and which option enables this?
- How do you create a pie chart showing proportions of a categorical variable with labels on each slice?
- Write the Stata command to create a scatter plot of 'mpg' vs. 'weight' with a linear fit line.
- What does the
by(foreign)option do when added to a twoway graph command? - Which command and dataset would you use to create a line plot of stock opening prices over time?
π Lecture 29 β Introduction to Stata- VI
π Overview: This lecture provides a comprehensive step-by-step guide to performing regression analysis in Stata using the
regresscommand. It covers pre-regression data exploration, running the model, conducting post-regression diagnostics (normality, heteroskedasticity, multicollinearity, specification errors, autocorrelation), and applying remedies such as robust standard errors and the Prais-Winsten estimator.
ποΈ Topics Covered
The lecture begins with the syntax and menu-based approach for the regress command, followed by a step-by-step example using the auto.dta dataset, including scatter plots with fitted lines and a correlation matrix. It then covers running a multiple regression model, generating predicted values and residuals, checking residual normality, transforming the dependent variable using inverse and ladder/gladder commands, and finally performing post-regression tests for heteroskedasticity (IM test, hettest), multicollinearity (VIF), model specification (linktest, ovtest), and autocorrelation (Durbin-Watson test), concluding with remedies for heteroskedasticity and autocorrelation.
π Lecture Summary
Regression with Stata
Regression analysis in Stata is mainly done by the regress command. The syntax is: regress depvar [indepvars] [if] [in] [weight] [, options]. The menu can also be used via Statistics > Linear models and related > Linear regression, which opens a window to specify the dependent and independent variables.
STEP by STEP example of regression
First, open the file auto.dta using sysuse auto, clear. Summarize and analyze the data, then look at various two-way graphs. Use graph matrix price mpg length weight, half to see pairwise scatter plots. To get a scatter with a line fit, use: twoway (scatter price weight) (lfit price weight) and twoway (scatter price mpg) (lfit price mpg). These graphs help visualize relationships between variables.
Now perform a regression using the command: regress price rep78 weight foreign. The output shows standard regression results that have the usual interpretation (as done with Microsoft Excel). Another example: regress price mpg length foreign.
Post regression Analysis
Use the predict command to get trend values and residuals:
predict phatcreates the variablephat(predicted values).predict e, residualcreates the variablee(residuals).list price phat e in 1/20shows the first 20 observations.
Check if the residuals follow the assumption of normality using histogram e,normal. If residuals are not normally distributed, one reason could be that the dependent variable is not normally distributed. Check the distribution of price with histogram price, normal.
Transform the variable price
We can transform the variable price to achieve normality. The ladder and gladder commands help identify transformations. gladder price produces histograms for various transformations (log, inverse, square, etc.). The inverse and 1/square transformations show the smallest chi-square values.
Generate the inverse of price: gen priceinv=1/price. Then run regression: reg priceinv mpg length foreign. Generate new residuals: predict e1, residual. Check normality: histogram e1, normal. The residuals from regressing the inverse of price look somewhat normal.
π‘ Why this matters: Transforming the dependent variable can help meet the normality assumption of OLS regression, leading to valid inference.
Post regression tests
Testing for Heteroskedasticity:
- IM TEST (White Test): returns a test indicating presence of heteroskedasticity.
- hettest: also shows evidence of heteroskedasticity.
π Definition β Heteroskedasticity: When the variance of the error terms is not constant across observations, violating OLS assumption.
Testing for Multicollinearity:
- Use the VIF (variance inflation factor) command. If VIF values are less than 5, there is no multicollinearity detected.
π Definition β VIF: Measures how much the variance of an estimated regression coefficient increases if predictors are correlated. VIF < 5 generally indicates no serious multicollinearity.
Model Specification Tests:
- linktest: performs a model specification test. The null hypothesis is that the model is specified correctly. If the variable
_hatsqis significant, the model is not specified correctly. - ovtest: tests for missing variables. Significant result suggests there may be missing variables in the model.
Testing for Autocorrelation:
Autocorrelation is normally a time series phenomenon. Open a time series data file: webuse klein, clear, set time series: tsset yr, run regression: regress consump wagegovt, then perform Durbin-Watson test: estat dwatson. If the p-value for F is less than 5%, we reject H0 (no autocorrelation).
π Definition β Autocorrelation: Correlation of error terms across time periods, violating OLS assumption in time series data.
REMEDIES for Heteroskedasticity
White developed an estimator for standard errors that is robust to the presence of heteroskedasticity. Use the robust option with regression:
- Run
reg price mpg length foreign(using original price) β errors will be heteroskedastic. - Then run
reg price mpg length foreign, robustβ standard errors change, coefficients remain the same.
Remedy for Autocorrelation
Run the analysis with the Prais-Winsten command, specifying the Cochran-Orcutt option. After iterations, the Durbin-Watson statistic improves. It is better to use models other than least squares (AR1, ARIMA, lagged variable, GLS) for autocorrelated data.
π Definition β Prais-Winsten estimator: A method for estimating regression models with AR(1) autocorrelated errors, transforming the data to remove autocorrelation.
β Key Takeaways
The lecture shows the complete workflow for OLS regression in Stata: data exploration, running regress, generating predicted values and residuals, and diagnosing five core regression assumptions. You must remember that the robust option fixes standard errors for heteroskedasticity (VIF < 5 indicates no multicollinearity), the linktest and ovtest check for specification errors, and the Durbin-Watson test (via estat dwatson) detects autocorrelation, which can be remedied using the Prais-Winsten command. Transforming the dependent variable (e.g., inverse) can restore normality in residuals, and post-estimation commands like predict and histogram are essential for validation.
π§ Quick Revision Questions
- What is the Stata syntax for running a linear regression of
priceonmpgandforeign? - How do you generate predicted values and residuals after a regression in Stata?
- What does the IM test (White test) check for, and how do you conduct it in Stata?
- What is the rule of thumb for detecting multicollinearity using VIF values?
- What is the Stata command to perform the Durbin-Watson test for autocorrelation, and how do you remedy autocorrelation if detected?