Term 3 · Module 3 of 4

Multiple Linear Regression I

Business Statistics for Entrepreneurs III

Practise this module in Claude, ChatGPT or Gemini

Set it up free
Mind map

7.1.1 Multiple Linear Regression: Introduction

Multiple linear regression (MLR) explains or predicts one dependent variable using two or more independent variables. It generalizes simple linear regression by letting several factors contribute to the expected value of YY. The residual part captures variation the predictors do not explain.

For pp predictors, the population model and its mean response are:

Y=β0+β1X1+⋯+βpXp+εY=\beta_0+\beta_1X_1+\cdots+\beta_pX_p+\varepsilon

E(Y∣X1,…,Xp)=β0+β1X1+⋯+βpXpE(Y\mid X_1,\ldots,X_p)=\beta_0+\beta_1X_1+\cdots+\beta_pX_p

Here β0,…,βp\beta_0,\ldots,\beta_p are unknown population parameters and ε\varepsilon is the random error. The model assumes E(ε)=0E(\varepsilon)=0, which makes the conditional mean of YY the linear combination above. From sample/training data, estimate the parameters with b0,…,bpb_0,\ldots,b_p and obtain a fitted response:

Y^=b0+b1X1+⋯+bpXp\hat{Y}=b_0+b_1X_1+\cdots+b_pX_p

Each slope is a partial regression coefficient: bib_i estimates the change in the response associated with a one-unit increase in XiX_i, holding the other included predictors constant. This qualification distinguishes a multiple-regression coefficient from the slope in a one-predictor model. The relationship is association; regression alone does not establish causation.

A practical model-building cycle

Model development is iterative: diagnostics or validation may send the analyst back to an earlier choice.

  1. Collect relevant data from suitable primary sources (such as surveys/interviews) or secondary sources (such as company databases or census data).
  2. Preprocess and assess completeness and correctness. Missing observations and qualitative/categorical variables may require attention or transformation.
  3. Use descriptive statistics and visualizations (including scatterplots and boxplots) to inspect distributions and correlations, anticipate multicollinearity or overfitting, and identify proxy variables when a variable cannot be used for ethical or regulatory reasons.
  4. Choose predictors with the problem in mind. Seek independent, controllable decision variables where relevant, and performance that is consistent across training and validation data. For example, store price can be changed by a manager; weather generally cannot.
  5. Split the data into training and validation sets; use multiple subsets when appropriate to assess performance repeatedly.
  6. Specify a functional form. Start with a linear form; transformations such as X2X^2 can represent curvature while remaining linear in the regression coefficients.
  7. Estimate parameters, commonly with ordinary least squares (OLS).
  8. Diagnose the model: assess predictor significance and assumptions, including residual normality, multicollinearity, and constant error variance (heteroscedasticity checks).
  9. Validate on held-out data using measures such as R2R^2, adjusted R2R^2, mean absolute percentage error (MAPE), and root mean square error (RMSE). Prefer consistent performance (low bias and variance) across training and validation data.
  10. Deploy the model into an actionable implementation plan and monitor performance; revise it if conditions or performance change.

Key takeaways

  • MLR uses one response and p≥2p\ge2 predictors; its mean response is linear in the predictors.
  • Interpret each slope with the other included predictors held constant.
  • Predictor choice, diagnostics, validation, deployment, and monitoring form an iterative process.
  • Statistical prediction supports decisions but does not by itself show causation.

7.1.2 Multiple Regression by Ordinary Least Squares

OLS chooses coefficients to make fitted values collectively as close as possible to observed responses, with large errors penalized more heavily because they are squared. For observation ii, the residual is ei=yi−y^ie_i=y_i-\hat y_i. OLS minimizes the sum of squared errors:

SSE⁡=∑i=1n(yi−y^i)2,y^i=b0+b1xi1+⋯+bpxip\operatorname{SSE}=\sum_{i=1}^{n}(y_i-\hat y_i)^2,\qquad \hat y_i=b_0+b_1x_{i1}+\cdots+b_px_{ip}

With multiple predictors, coefficient formulas involve matrix algebra; the emphasis here is on interpreting computed output. OLS is described as providing the best linear unbiased estimator under the relevant regression assumptions.

Hanumantha’s credit-card spending example

For 55 customers, monthly credit-card spend is YY, annual household income in lakhs is X1X_1, and household size is X2X_2. The one-predictor fits were:

Model Fitted equation Reported result
Spend on annual income Y^=1047.37+85.27X1\hat Y=1047.37+85.27X_1 F=98.24F=98.24, reported pp approximately 0; R2=0.65R^2=0.65 in the fit comparison
Spend on household size Y^=2083.67+520.13X1\hat Y=2083.67+520.13X_1 F=34.76F=34.76, reported pp approximately 0; significant slope (reported t=5.89t=5.89)
Spend on income and household size Y^=355.68+71.16X1+339.9X2\hat Y=355.68+71.16X_1+339.9X_2 R2=0.80R^2=0.80

The two-predictor model explains more observed variation than the income-only model. Its income coefficient is smaller than the simple-regression slope because it now describes income’s association at a fixed household size. Specifically, an additional lakh in annual income is associated with an estimated 71.16 increase in monthly spend, holding household size constant. One additional household member is associated with 339.9 more monthly spend, holding annual income constant.

Key takeaways

  • OLS minimizes squared residuals over observations.
  • Multiple-regression coefficients are estimated from the same sample jointly.
  • Adding household size changes the interpretation and estimate of the income slope.
  • The reported fitted values are sample estimates, not known population parameters.

7.1.3 Multiple Coefficient of Determination

The multiple coefficient of determination, still denoted R2R^2, is the fraction of the sample variation in YY explained by the fitted regression. The total variation splits into explained and unexplained parts:

SST=∑i=1n(yi−yˉ)2,SSR=∑i=1n(y^i−yˉ)2,SSE=∑i=1n(yi−y^i)2SST=\sum_{i=1}^{n}(y_i-\bar y)^2,\quad SSR=\sum_{i=1}^{n}(\hat y_i-\bar y)^2,\quad SSE=\sum_{i=1}^{n}(y_i-\hat y_i)^2

SST=SSR+SSE,R2=SSRSST=1−SSESSTSST=SSR+SSE,\qquad R^2=\frac{SSR}{SST}=1-\frac{SSE}{SST}

Thus R2=0.80R^2=0.80 means the fitted equation accounts for 80% of the observed variation in the response in that dataset; it does not mean 80% of individual outcomes are predicted exactly. Since adding predictors cannot increase the OLS training SSE, ordinary R2R^2 generally rises or stays unchanged when predictors are added, even if a new variable adds little useful information.

Adjusted R2R^2 accounts for sample size nn and predictor count pp:

Radj2=1−(1−R2)n−1n−p−1R^2_{\mathrm{adj}}=1-(1-R^2)\frac{n-1}{n-p-1}

For the 55-customer example, the income-only model (R2=0.65R^2=0.65, p=1p=1) gives:

Radj2=1−(1−0.65)55−155−1−1=1−0.355453≈0.64R^2_{\mathrm{adj}}=1-(1-0.65)\frac{55-1}{55-1-1}=1-0.35\frac{54}{53}\approx0.64

With income and household size (R2=0.80R^2=0.80, p=2p=2):

Radj2=1−(1−0.80)5452≈0.793R^2_{\mathrm{adj}}=1-(1-0.80)\frac{54}{52}\approx0.793

The adjusted value is slightly below ordinary R2R^2 here. With too many predictors for the number of observations, the adjustment can become negative; the treatment described is to set adjusted R2R^2 to zero in that situation.

Key takeaways

  • R2=SSR/SSTR^2=SSR/SST measures explained sample variation; SST=SSR+SSESST=SSR+SSE.
  • Adding predictors generally raises ordinary R2R^2, so it alone does not guard against unhelpful complexity.
  • Adjusted R2R^2 penalizes predictor count relative to sample size.
  • Compare fit measures alongside diagnostics and validation performance.

7.1.4–7.1.6 Significance Tests for a Multiple Regression

A high R2R^2 does not by itself establish that the assumed relationships are statistically supported. Significance tests rely on assumptions about the error term, and answer two different questions: whether the predictors jointly relate to YY, and whether each predictor contributes conditional on the others.

Error assumptions and response surface

The regression model assumes:

  • E(ε∣X1,…,Xp)=0E(\varepsilon\mid X_1,\ldots,X_p)=0.
  • The error variance is constant, Var⁡(ε∣X1,…,Xp)=σ2\operatorname{Var}(\varepsilon\mid X_1,\ldots,X_p)=\sigma^2 (homoscedasticity).
  • Errors across observations are independent.
  • Errors are normally distributed at each predictor setting: ε∣X1,…,Xp∼N(0,σ2)\varepsilon\mid X_1,\ldots,X_p\sim N(0,\sigma^2).

Since YY is a linear function of the error for given predictor values, it is normally distributed under these assumptions. With two predictors, the conditional mean equation forms a plane in three-dimensional (X1,X2,Y)(X_1,X_2,Y) space; the error is the vertical difference between an observed YY and its expected value. This surface is also called a response surface.

Overall FF test

The overall test asks whether the entire predictor set has a linear relationship with the response:

H0:β1=β2=⋯=βp=0H_0:\beta_1=\beta_2=\cdots=\beta_p=0

Ha:at least one βj≠0H_a:\text{at least one }\beta_j\ne0

With nn observations and pp predictors, regression and error mean squares and the test statistic are:

MSR=SSRp,MSE=SSEn−p−1,F=MSRMSEMSR=\frac{SSR}{p},\qquad MSE=\frac{SSE}{n-p-1},\qquad F=\frac{MSR}{MSE}

Under H0H_0 and the assumptions, FF follows an FF distribution with pp numerator and n−p−1n-p-1 denominator degrees of freedom. Reject H0H_0 when the pp-value is at most the chosen significance level α\alpha. Rejection means evidence that at least one slope is nonzero; it does not assert that every predictor is significant.

For Hanumantha’s two-predictor model, MSR=45,734,426MSR=45{,}734{,}426, MSE=437,098MSE=437{,}098, and F=104.63F=104.63 with 22 and 5252 degrees of freedom; the reported pp-value is close to zero. Thus the overall relationship is significant. The residual standard error estimate is:

s=MSE=437,098=661.13s=\sqrt{MSE}=\sqrt{437{,}098}=661.13

Individual tt tests

After assessing overall significance, test each slope separately, conditional on the other predictors:

H0:βi=0,Ha:βi≠0,t=biSE(bi)H_0:\beta_i=0,\qquad H_a:\beta_i\ne0,\qquad t=\frac{b_i}{SE(b_i)}

The test has n−p−1n-p-1 degrees of freedom; reject when its pp-value is at most α\alpha. In Hanumantha’s model, income has b1=71.17b_1=71.17, SE(b1)=6.92SE(b_1)=6.92, so t=71.17/6.92=10.28t=71.17/6.92=10.28 (5252 df). Household size has b2=339.92b_2=339.92, SE(b2)=54.05SE(b_2)=54.05, so t=339.92/54.05=6.29t=339.92/54.05=6.29 (5252 df). Both reported pp-values are close to zero, so both slopes are significant in this model.

Test Question Null hypothesis
Overall FF Does the predictor set have a significant linear relationship with YY? All slopes are zero
Individual tt Does predictor XiX_i contribute, given the other included predictors? Its slope is zero

Key takeaways

  • The FF test evaluates the slopes jointly; each tt test evaluates one slope conditionally.
  • Use pp numerator and n−p−1n-p-1 denominator degrees of freedom for the overall FF test.
  • The regression standard error is s=MSEs=\sqrt{MSE}.
  • Significance claims depend on the error assumptions and the chosen α\alpha.

7.1.7 Multicollinearity

Multicollinearity is correlation among predictors. “Independent variable” means a predictor in the regression; it does not mean that predictors are statistically independent. Some correlation is common, but severe overlap makes it hard to isolate each predictor’s separate contribution.

In Hanumantha’s original model, annual income and household size had sample correlation 0.320.32; both slopes were significant. A hypothetical replacement of household size with monthly income produced a correlation of 0.830.83 between monthly and annual income. The resulting model had R2=0.66R^2=0.66 and a significant overall FF test, but the monthly-income tt statistic was only 0.180.18, while the annual-income result was significant. Given annual income already in the model, monthly income added little distinct information. Its estimated slope was −202.85-202.85, opposite the expected positive direction, while annual income’s estimate was 102.74102.74—a warning that individual coefficients are unstable or difficult to interpret under severe multicollinearity.

A sample correlation above 0.70.7 or below −0.7-0.7 between two predictors is offered as a rule of thumb warning, not a definitive test. Multicollinearity may appear as a strong overall FF result alongside weak individual tt results, unexpected coefficient signs, or difficulty assigning separate effects. If feasible, avoid including highly correlated predictors together; when that is not feasible, interpret the individual estimates cautiously. Regression diagnostics guide predictor selection rather than mechanically proving that a particular variable is irrelevant.

In the property example, area and bedrooms correlated at 0.5420.542, while area and bathrooms correlated at 0.6590.659. Though both are below the stated rule-of-thumb threshold, adding either variable did not yield a significant individual slope in the reported fits. Area remained significant; bedrooms had p=0.36p=0.36 and bathrooms p=0.48p=0.48. This illustrates why correlation thresholds are only a warning and model diagnostics still matter.

Key takeaways

  • Predictor correlation can make individual effects hard to separate.
  • Overall significance does not guarantee every predictor has a significant conditional contribution.
  • Unexpected signs and large individual pp-values can signal overlap, but require investigation.
  • The ±0.7\pm0.7 correlation threshold is a heuristic, not a universal cutoff.

7.1.7–7.1.9 Residual Analysis, Outliers, and Influence

A residual ei=yi−y^ie_i=y_i-\hat y_i is the observed response minus its fitted value. Residuals are observable clues about the unobserved errors, so plots help assess whether the model’s functional form and constant-variance assumptions are plausible.

Residual plots

Inspect residuals against each predictor and, especially when there are several predictors, against fitted values y^\hat y. A suitable pattern is a roughly horizontal band centered at zero. A funnel suggests changing variance (heteroscedasticity); a curved or U-shaped pattern suggests the chosen functional form may not represent the relationship adequately.

For Hanumantha’s spending data, residual plots against income, household size, and fitted spend looked broadly like bands without a clear funnel or curve. The fitted-value plot is especially useful in multiple regression and is often examined first; predictor-specific plots can then add detail. The analysis tool’s plots are not all automatic, so a fitted-value plot may need to be made from the output.

Standardized residuals put residuals on a comparable scale. Under normally distributed errors, roughly 95% should lie between −2-2 and +2+2. Values outside this range are potential outliers, not automatic grounds for deletion. In Hanumantha’s data most standardized residuals were within this range, with a few beyond it.

Responding to unusual observations

An outlier does not fit the pattern of the remaining data. It may be a recording error (correct it), evidence the model is inadequate (consider another form), or a valid unusual observation that should remain. An influential observation materially changes the fitted slope or other regression parameters. It may have an unusual response, an extreme predictor value far from its mean, or both.

Check collection and recording first. Correct errors and refit. If the point is valid, it may help reveal the relationship; consider gathering observations at intermediate predictor values to understand it better. Do not remove an outlier or influential point without clear evidence it does not belong to the population being studied.

Key takeaways

  • Residuals should show no systematic pattern; funnels and curves flag different model concerns.
  • Standardized residuals beyond ±2\pm2 warrant examination, not automatic deletion.
  • Influence concerns the effect of an observation on the fitted equation; it is distinct from being an outlier.
  • Verify data, model form, and population relevance before changing or removing a point.

7.2.1–7.2.2 Recap: Hanumantha’s Model

The income-and-household-size model for monthly credit-card spend was:

Y^=355.7+71.17X1+339.92X2\hat Y=355.7+71.17X_1+339.92X_2

It had R2=0.80R^2=0.80, adjusted R2=0.79R^2=0.79, and standard error s=661.1s=661.1. Its overall statistic was F=104.6F=104.6 and both slope tests had reported pp-values close to zero. There was no evident multicollinearity concern for these two predictor choices. Residual plots showed no clear funnel or curved pattern; most standardized residuals were between −2-2 and +2+2, with a few slightly beyond.

The broader lesson is to use the fitted equation, R2R^2, adjusted R2R^2, FF and tt tests, predictor diagnostics, and residual analysis together. Building and improving the model is iterative.

Key takeaways

  • The two-predictor model improved fit over income alone in this dataset.
  • Both included predictors were individually significant in the reported analysis.
  • Residual checks were broadly reassuring, with a few potential outliers to inspect.

7.3.1 Applications: Household Debt with Income and Payments

A Bengaluru household survey of about 500 observations was used to examine debt YY with first income X1X_1 and monthly payment X2X_2. The two-predictor model had R2=0.45R^2=0.45 (adjusted R2≈0.448R^2\approx0.448) and standard error 25,64425{,}644. The one-predictor models had R2=0.31R^2=0.31 for first income and 0.370.37 for monthly payments, with standard errors 28,61928{,}619 and 27,51827{,}518, respectively.

Reported ANOVA values were SSR=2.68×1011SSR=2.68\times10^{11}, SSE=3.27×1011SSE=3.27\times10^{11}, and SST=5.94×1011SST=5.94\times10^{11}. With two predictors and 497497 error degrees of freedom, MSR=1.34×1011MSR=1.34\times10^{11}, MSE=65,762,405MSE=65{,}762{,}405, and F=203F=203 (reported pp close to zero). The fitted equation was:

Y^=13,081+0.025X1+2.09X2\hat Y=13{,}081+0.025X_1+2.09X_2

Holding monthly payment constant, each rupee of first income was associated with an estimated 0.0250.025 increase in debt; holding first income constant, each rupee of monthly payment was associated with an estimated 2.092.09 increase. Both slope tests were significant: first income had SE=0.003SE=0.003, t=8.74t=8.74; monthly payment had SE=0.188SE=0.188, t=11.1t=11.1 (each with 497497 df; reported pp very small). Residual plots were oval/random-looking without a clear funnel or curve; a few standardized residuals lay outside ±2\pm2.

Key takeaways

  • First income and monthly payment together explained 45% of observed debt variation.
  • Both slopes were significant conditional on the other predictor.
  • A fitted association between utility/payment and debt is not evidence of causation.

7.3.2 Applications: Household Debt with Payments and Utilities

Replacing first income with utilities improved the household-debt fit. For debt YY on monthly payment X1X_1 and utilities X2X_2, reported R2=0.67R^2=0.67, adjusted R2R^2 slightly lower, and standard error 19,83919{,}839. The comparison model using first income and monthly payment had standard error 25,64425{,}644. The total variation was unchanged (SST=5.94×1011SST=5.94\times10^{11}), while this model had SSR=3.99×1011SSR=3.99\times10^{11} and SSE=1.96×1011SSE=1.96\times10^{11}.

The fitted equation and overall test were:

Y^=−140,771+1.44X1+168.91X2\hat Y=-140{,}771+1.44X_1+168.91X_2

MSR=1.99×1011,MSE=393,601,978,F=506.9MSR=1.99\times10^{11},\quad MSE=393{,}601{,}978,\quad F=506.9

The overall pp-value was reported close to zero. Payment’s slope had SE=0.143SE=0.143, t=10.02t=10.02; utilities’ slope had SE=7.87SE=7.87, t=21.47t=21.47; both tests were significant (497497 df, reported pp close to zero). Holding the other included variable constant, each rupee of payment was associated with 1.441.44 more debt and each rupee of utilities with 168.91168.91 more debt. Utilities’ coefficient is an association; the model does not establish that utility spending causes debt. Residuals looked broadly random around zero, although a few standardized values were outside ±2\pm2.

Key takeaways

  • The payments-and-utilities model fit better than the income-and-payments model in this sample.
  • Both predictors were individually significant in the reported model.
  • Better prediction or fit does not establish a causal mechanism.

7.3.3 Applications: Household Debt with Three Predictors

Using first income X1X_1, monthly payment X2X_2, and utilities X3X_3 produced the best reported household-debt fit among the compared models: R2=0.708R^2=0.708 (about 0.700.70 adjusted) and standard error 18,69918{,}699. Here SSR=4.21×1011SSR=4.21\times10^{11}, SSE=1.73×1011SSE=1.73\times10^{11}, and SST=5.94×1011SST=5.94\times10^{11}. With 33 regression and 496496 error degrees of freedom, MSR=1.4×1011MSR=1.4\times10^{11}, MSE=349,653,451MSE=349{,}653{,}451, and F=401.6F=401.6; the reported overall pp-value was close to zero.

Y^=−140,378+0.017X1+0.96X2+157.95X3\hat Y=-140{,}378+0.017X_1+0.96X_2+157.95X_3

All three reported individual tests were significant: first income t≈7.96t\approx7.96, monthly payment t=6.5t=6.5, utilities t=20.95t=20.95, each with 496496 degrees of freedom and reported pp close to zero. Interpret each slope holding the other two predictors fixed. Residual plots against the three predictors and fitted debt looked random around zero. A few standardized residuals exceeded +3+3 or fell below −3-3, so those observations merit review as possible outliers.

Key takeaways

  • The three-predictor model explained about 71% of debt variation in this sample.
  • Fit improved and standard error fell relative to the two-predictor alternatives reported.
  • A few residuals beyond ±3\pm3 are flags for examination, not automatic exclusions.

7.3.4–7.3.5 Applications: Property Prices and Predictor Choice

For 30 flats in Jayalakshmi Puram, Mysuru, selling price (in lakhs) was first regressed on area in square feet. The fitted line and R2R^2 were:

Y^=28.37+0.0968X,R2=0.44\hat Y=28.37+0.0968X,\qquad R^2=0.44

The slope is about 0.090.09 lakhs per additional square foot (the reported approximate interpretation is ₹90,000), but this one-predictor association may partly reflect omitted property characteristics. The model’s standard error was 70.5770.57; SSR=111,539SSR=111{,}539, SSE=139,462SSE=139{,}462, SST=251,001SST=251{,}001, F=22.39F=22.39, and the reported overall and slope-test pp-values were close to zero. Some standardized residuals were above +3+3 or below −3-3; these are possible outliers to investigate.

The dataset also included bedrooms, bathrooms, and qualitative indicators (premium locality, gated complex, amenities). Values coded 00 and 11 do not make those indicators quantitative measurements: they represent unordered categories. The models below use the quantitative bedroom and bathroom counts.

Predictors R2R^2 Adjusted R2R^2 Standard error Overall result
Area + bedrooms 0.460.46 (reported as 0.5464 once, inconsistent with the accompanying 46% and sum-of-squares ratio) About 0.420.42 70.7470.74 F=11.58F=11.58, p=0.0002p=0.0002
Area + bathrooms 0.450.45 About 0.410.41 71.1871.18 F=11.26F=11.26, p=0.0003p=0.0003

For area and bedrooms, the fitted equation was:

Y^=−29.19+0.085(area)+23.46(bedrooms)\hat Y=-29.19+0.085(\text{area})+23.46(\text{bedrooms})

The area slope was significant (t=3.46t=3.46, p=0.002p=0.002), but the bedrooms slope was not (t=0.932t=0.932, p=0.36p=0.36). The reported predictor correlation was 0.5420.542. For area and bathrooms, the fitted equation was:

Y^=10.61+0.084(area)+17.80(bathrooms)\hat Y=10.61+0.084(\text{area})+17.80(\text{bathrooms})

Area remained significant (p=0.005p=0.005); bathrooms did not (t=0.723t=0.723, p=0.48p=0.48). Area and bathrooms correlated at 0.6590.659. Both two-predictor models had significant overall FF tests, yet the extra count variable added little significant information conditional on area. This is compatible with an overall significant model and a nonsignificant individual predictor. Residual plots showed no clear funnel or curve; the bedroom model had two standardized residuals above 22.

The source’s reported area-plus-bedrooms R2R^2 value “0.5464” conflicts with its stated 46%, its SSR/SST calculation (115,892/251,001≈0.46115{,}892/251{,}001\approx0.46), and its later statement. The consistent value is approximately 0.460.46; retain the discrepancy in mind when reading the output.

Key takeaways

  • Area alone had a significant relationship with price but explained only 44% of observed variation.
  • Additional predictors can raise overall fit without having individually significant conditional slopes.
  • A numeric category code remains qualitative when it only labels a category.
  • The reported bedroom-model R2R^2 contains an internal inconsistency; the sums of squares support approximately 0.460.46.

7.4.1 Intercept, Extrapolation, and Omitted Variable Bias

Intercept and prediction range

The intercept β0\beta_0 is the expected response when every predictor equals zero. Computer output often tests H0:β0=0H_0:\beta_0=0, but that test is usually not the main question of interest; inference on the intercept is connected to inference for fitted values. If zero is outside the observed predictor ranges, the intercept describes an extrapolated setting and may have little practical meaning. More broadly, predictions outside the training data’s predictor ranges are extrapolations and are risky.

Omitted variable bias

Omitted variable bias (OVB) occurs when excluding a relevant predictor distorts the coefficient of an included predictor: the included variable can partly absorb the omitted variable’s association with the response. The lecture’s stated conditions that can permit OVB are: an omitted variable is correlated with the dependent variable, and/or it is correlated with an included predictor. These correlations make the simple regression coefficient differ from the included variable’s true separate contribution. This module identifies the issue conceptually; it does not derive a numerical bias formula.

In the household-debt example, first-income-only regression had R2=0.31R^2=0.31, while the model with first income, monthly payment, and utilities had R2=0.71R^2=0.71. Debt’s sample correlations with first income, monthly payment, and utilities were 0.560.56, 0.610.61, and 0.780.78, respectively. First income correlated with monthly payment at 0.550.55 and utilities at 0.390.39; monthly payment and utilities correlated at 0.490.49. Those relationships make omitted-variable bias plausible when relevant variables are left out.

Possible warning signs when needed variables are added include a substantial increase in R2R^2, a decrease in model standard error, and changes in coefficient standard errors or pp-values. These are indicators for comparing model specifications, not a complete formal test of OVB.

Key takeaways

  • The intercept’s zero-predictor interpretation can be extrapolative and unimportant in context.
  • Avoid using the fitted equation beyond the predictor ranges represented in training data.
  • Omitted relevant variables can distort included slopes when the stated correlation conditions hold.
  • Compare specifications and diagnostics; a higher R2R^2 alone does not prove causality or eliminate bias.

7.4.2 Module Synthesis

MLR estimates one response from several predictors. OLS chooses coefficients by minimizing SSE; R2R^2 and adjusted R2R^2 summarize fit. The FF test evaluates the predictors jointly, while tt tests evaluate conditional individual slopes. Those inferences depend on error assumptions, which residual plots help assess. Multicollinearity complicates separate slope interpretation; outliers and influential cases require careful review. Practical model building cycles through predictor choice, estimation, diagnostics, validation, and monitoring. Quantitative examples here show that adding relevant predictors can improve fit, while correlated or redundant predictors may contribute little. Qualitative variables require an appropriate representation and are taken up separately.

Key takeaways

  • Read fit, overall significance, individual significance, and residual diagnostics together.
  • Interpret coefficients conditionally and avoid causal claims from regression association alone.
  • Investigate correlated predictors and unusual observations before settling on a model.
  • Validate model performance on data not used to estimate its coefficients.
Revise before your examAll 2 exam tips from this module in one list
  1. In multiple regression, “holding the other predictors constant” is part of the slope’s meaning, not an optional caveat.
  2. A significant FF test can coexist with nonsignificant individual tt tests, especially when predictors overlap in the information they carry.

Test yourself on this module

Questions from these notes, one at a time, with feedback. Your weak topics are saved.

Set it up free Takes about a minute. Free plans work.