7.1.1 Multiple Linear Regression: Introduction
Multiple linear regression (MLR) explains or predicts one dependent variable using two or more independent variables. It generalizes simple linear regression by letting several factors contribute to the expected value of . The residual part captures variation the predictors do not explain.
For predictors, the population model and its mean response are:
Here are unknown population parameters and is the random error. The model assumes , which makes the conditional mean of the linear combination above. From sample/training data, estimate the parameters with and obtain a fitted response:
Each slope is a partial regression coefficient: estimates the change in the response associated with a one-unit increase in , holding the other included predictors constant. This qualification distinguishes a multiple-regression coefficient from the slope in a one-predictor model. The relationship is association; regression alone does not establish causation.
A practical model-building cycle
Model development is iterative: diagnostics or validation may send the analyst back to an earlier choice.
- Collect relevant data from suitable primary sources (such as surveys/interviews) or secondary sources (such as company databases or census data).
- Preprocess and assess completeness and correctness. Missing observations and qualitative/categorical variables may require attention or transformation.
- Use descriptive statistics and visualizations (including scatterplots and boxplots) to inspect distributions and correlations, anticipate multicollinearity or overfitting, and identify proxy variables when a variable cannot be used for ethical or regulatory reasons.
- Choose predictors with the problem in mind. Seek independent, controllable decision variables where relevant, and performance that is consistent across training and validation data. For example, store price can be changed by a manager; weather generally cannot.
- Split the data into training and validation sets; use multiple subsets when appropriate to assess performance repeatedly.
- Specify a functional form. Start with a linear form; transformations such as can represent curvature while remaining linear in the regression coefficients.
- Estimate parameters, commonly with ordinary least squares (OLS).
- Diagnose the model: assess predictor significance and assumptions, including residual normality, multicollinearity, and constant error variance (heteroscedasticity checks).
- Validate on held-out data using measures such as , adjusted , mean absolute percentage error (MAPE), and root mean square error (RMSE). Prefer consistent performance (low bias and variance) across training and validation data.
- Deploy the model into an actionable implementation plan and monitor performance; revise it if conditions or performance change.
Key takeaways
- MLR uses one response and predictors; its mean response is linear in the predictors.
- Interpret each slope with the other included predictors held constant.
- Predictor choice, diagnostics, validation, deployment, and monitoring form an iterative process.
- Statistical prediction supports decisions but does not by itself show causation.
7.1.2 Multiple Regression by Ordinary Least Squares
OLS chooses coefficients to make fitted values collectively as close as possible to observed responses, with large errors penalized more heavily because they are squared. For observation , the residual is . OLS minimizes the sum of squared errors:
With multiple predictors, coefficient formulas involve matrix algebra; the emphasis here is on interpreting computed output. OLS is described as providing the best linear unbiased estimator under the relevant regression assumptions.
Hanumantha’s credit-card spending example
For 55 customers, monthly credit-card spend is , annual household income in lakhs is , and household size is . The one-predictor fits were:
| Model | Fitted equation | Reported result |
|---|---|---|
| Spend on annual income | , reported approximately 0; in the fit comparison | |
| Spend on household size | , reported approximately 0; significant slope (reported ) | |
| Spend on income and household size |
The two-predictor model explains more observed variation than the income-only model. Its income coefficient is smaller than the simple-regression slope because it now describes income’s association at a fixed household size. Specifically, an additional lakh in annual income is associated with an estimated 71.16 increase in monthly spend, holding household size constant. One additional household member is associated with 339.9 more monthly spend, holding annual income constant.
Key takeaways
- OLS minimizes squared residuals over observations.
- Multiple-regression coefficients are estimated from the same sample jointly.
- Adding household size changes the interpretation and estimate of the income slope.
- The reported fitted values are sample estimates, not known population parameters.
7.1.3 Multiple Coefficient of Determination
The multiple coefficient of determination, still denoted , is the fraction of the sample variation in explained by the fitted regression. The total variation splits into explained and unexplained parts:
Thus means the fitted equation accounts for 80% of the observed variation in the response in that dataset; it does not mean 80% of individual outcomes are predicted exactly. Since adding predictors cannot increase the OLS training SSE, ordinary generally rises or stays unchanged when predictors are added, even if a new variable adds little useful information.
Adjusted accounts for sample size and predictor count :
For the 55-customer example, the income-only model (, ) gives:
With income and household size (, ):
The adjusted value is slightly below ordinary here. With too many predictors for the number of observations, the adjustment can become negative; the treatment described is to set adjusted to zero in that situation.
Key takeaways
- measures explained sample variation; .
- Adding predictors generally raises ordinary , so it alone does not guard against unhelpful complexity.
- Adjusted penalizes predictor count relative to sample size.
- Compare fit measures alongside diagnostics and validation performance.
7.1.4–7.1.6 Significance Tests for a Multiple Regression
A high does not by itself establish that the assumed relationships are statistically supported. Significance tests rely on assumptions about the error term, and answer two different questions: whether the predictors jointly relate to , and whether each predictor contributes conditional on the others.
Error assumptions and response surface
The regression model assumes:
- .
- The error variance is constant, (homoscedasticity).
- Errors across observations are independent.
- Errors are normally distributed at each predictor setting: .
Since is a linear function of the error for given predictor values, it is normally distributed under these assumptions. With two predictors, the conditional mean equation forms a plane in three-dimensional space; the error is the vertical difference between an observed and its expected value. This surface is also called a response surface.
Overall test
The overall test asks whether the entire predictor set has a linear relationship with the response:
With observations and predictors, regression and error mean squares and the test statistic are:
Under and the assumptions, follows an distribution with numerator and denominator degrees of freedom. Reject when the -value is at most the chosen significance level . Rejection means evidence that at least one slope is nonzero; it does not assert that every predictor is significant.
For Hanumantha’s two-predictor model, , , and with and degrees of freedom; the reported -value is close to zero. Thus the overall relationship is significant. The residual standard error estimate is:
Individual tests
After assessing overall significance, test each slope separately, conditional on the other predictors:
The test has degrees of freedom; reject when its -value is at most . In Hanumantha’s model, income has , , so ( df). Household size has , , so ( df). Both reported -values are close to zero, so both slopes are significant in this model.
| Test | Question | Null hypothesis |
|---|---|---|
| Overall | Does the predictor set have a significant linear relationship with ? | All slopes are zero |
| Individual | Does predictor contribute, given the other included predictors? | Its slope is zero |
Key takeaways
- The test evaluates the slopes jointly; each test evaluates one slope conditionally.
- Use numerator and denominator degrees of freedom for the overall test.
- The regression standard error is .
- Significance claims depend on the error assumptions and the chosen .
7.1.7 Multicollinearity
Multicollinearity is correlation among predictors. “Independent variable” means a predictor in the regression; it does not mean that predictors are statistically independent. Some correlation is common, but severe overlap makes it hard to isolate each predictor’s separate contribution.
In Hanumantha’s original model, annual income and household size had sample correlation ; both slopes were significant. A hypothetical replacement of household size with monthly income produced a correlation of between monthly and annual income. The resulting model had and a significant overall test, but the monthly-income statistic was only , while the annual-income result was significant. Given annual income already in the model, monthly income added little distinct information. Its estimated slope was , opposite the expected positive direction, while annual income’s estimate was —a warning that individual coefficients are unstable or difficult to interpret under severe multicollinearity.
A sample correlation above or below between two predictors is offered as a rule of thumb warning, not a definitive test. Multicollinearity may appear as a strong overall result alongside weak individual results, unexpected coefficient signs, or difficulty assigning separate effects. If feasible, avoid including highly correlated predictors together; when that is not feasible, interpret the individual estimates cautiously. Regression diagnostics guide predictor selection rather than mechanically proving that a particular variable is irrelevant.
In the property example, area and bedrooms correlated at , while area and bathrooms correlated at . Though both are below the stated rule-of-thumb threshold, adding either variable did not yield a significant individual slope in the reported fits. Area remained significant; bedrooms had and bathrooms . This illustrates why correlation thresholds are only a warning and model diagnostics still matter.
Key takeaways
- Predictor correlation can make individual effects hard to separate.
- Overall significance does not guarantee every predictor has a significant conditional contribution.
- Unexpected signs and large individual -values can signal overlap, but require investigation.
- The correlation threshold is a heuristic, not a universal cutoff.
7.1.7–7.1.9 Residual Analysis, Outliers, and Influence
A residual is the observed response minus its fitted value. Residuals are observable clues about the unobserved errors, so plots help assess whether the model’s functional form and constant-variance assumptions are plausible.
Residual plots
Inspect residuals against each predictor and, especially when there are several predictors, against fitted values . A suitable pattern is a roughly horizontal band centered at zero. A funnel suggests changing variance (heteroscedasticity); a curved or U-shaped pattern suggests the chosen functional form may not represent the relationship adequately.
For Hanumantha’s spending data, residual plots against income, household size, and fitted spend looked broadly like bands without a clear funnel or curve. The fitted-value plot is especially useful in multiple regression and is often examined first; predictor-specific plots can then add detail. The analysis tool’s plots are not all automatic, so a fitted-value plot may need to be made from the output.
Standardized residuals put residuals on a comparable scale. Under normally distributed errors, roughly 95% should lie between and . Values outside this range are potential outliers, not automatic grounds for deletion. In Hanumantha’s data most standardized residuals were within this range, with a few beyond it.
Responding to unusual observations
An outlier does not fit the pattern of the remaining data. It may be a recording error (correct it), evidence the model is inadequate (consider another form), or a valid unusual observation that should remain. An influential observation materially changes the fitted slope or other regression parameters. It may have an unusual response, an extreme predictor value far from its mean, or both.
Check collection and recording first. Correct errors and refit. If the point is valid, it may help reveal the relationship; consider gathering observations at intermediate predictor values to understand it better. Do not remove an outlier or influential point without clear evidence it does not belong to the population being studied.
Key takeaways
- Residuals should show no systematic pattern; funnels and curves flag different model concerns.
- Standardized residuals beyond warrant examination, not automatic deletion.
- Influence concerns the effect of an observation on the fitted equation; it is distinct from being an outlier.
- Verify data, model form, and population relevance before changing or removing a point.
7.2.1–7.2.2 Recap: Hanumantha’s Model
The income-and-household-size model for monthly credit-card spend was:
It had , adjusted , and standard error . Its overall statistic was and both slope tests had reported -values close to zero. There was no evident multicollinearity concern for these two predictor choices. Residual plots showed no clear funnel or curved pattern; most standardized residuals were between and , with a few slightly beyond.
The broader lesson is to use the fitted equation, , adjusted , and tests, predictor diagnostics, and residual analysis together. Building and improving the model is iterative.
Key takeaways
- The two-predictor model improved fit over income alone in this dataset.
- Both included predictors were individually significant in the reported analysis.
- Residual checks were broadly reassuring, with a few potential outliers to inspect.
7.3.1 Applications: Household Debt with Income and Payments
A Bengaluru household survey of about 500 observations was used to examine debt with first income and monthly payment . The two-predictor model had (adjusted ) and standard error . The one-predictor models had for first income and for monthly payments, with standard errors and , respectively.
Reported ANOVA values were , , and . With two predictors and error degrees of freedom, , , and (reported close to zero). The fitted equation was:
Holding monthly payment constant, each rupee of first income was associated with an estimated increase in debt; holding first income constant, each rupee of monthly payment was associated with an estimated increase. Both slope tests were significant: first income had , ; monthly payment had , (each with df; reported very small). Residual plots were oval/random-looking without a clear funnel or curve; a few standardized residuals lay outside .
Key takeaways
- First income and monthly payment together explained 45% of observed debt variation.
- Both slopes were significant conditional on the other predictor.
- A fitted association between utility/payment and debt is not evidence of causation.
7.3.2 Applications: Household Debt with Payments and Utilities
Replacing first income with utilities improved the household-debt fit. For debt on monthly payment and utilities , reported , adjusted slightly lower, and standard error . The comparison model using first income and monthly payment had standard error . The total variation was unchanged (), while this model had and .
The fitted equation and overall test were:
The overall -value was reported close to zero. Payment’s slope had , ; utilities’ slope had , ; both tests were significant ( df, reported close to zero). Holding the other included variable constant, each rupee of payment was associated with more debt and each rupee of utilities with more debt. Utilities’ coefficient is an association; the model does not establish that utility spending causes debt. Residuals looked broadly random around zero, although a few standardized values were outside .
Key takeaways
- The payments-and-utilities model fit better than the income-and-payments model in this sample.
- Both predictors were individually significant in the reported model.
- Better prediction or fit does not establish a causal mechanism.
7.3.3 Applications: Household Debt with Three Predictors
Using first income , monthly payment , and utilities produced the best reported household-debt fit among the compared models: (about adjusted) and standard error . Here , , and . With regression and error degrees of freedom, , , and ; the reported overall -value was close to zero.
All three reported individual tests were significant: first income , monthly payment , utilities , each with degrees of freedom and reported close to zero. Interpret each slope holding the other two predictors fixed. Residual plots against the three predictors and fitted debt looked random around zero. A few standardized residuals exceeded or fell below , so those observations merit review as possible outliers.
Key takeaways
- The three-predictor model explained about 71% of debt variation in this sample.
- Fit improved and standard error fell relative to the two-predictor alternatives reported.
- A few residuals beyond are flags for examination, not automatic exclusions.
7.3.4–7.3.5 Applications: Property Prices and Predictor Choice
For 30 flats in Jayalakshmi Puram, Mysuru, selling price (in lakhs) was first regressed on area in square feet. The fitted line and were:
The slope is about lakhs per additional square foot (the reported approximate interpretation is ₹90,000), but this one-predictor association may partly reflect omitted property characteristics. The model’s standard error was ; , , , , and the reported overall and slope-test -values were close to zero. Some standardized residuals were above or below ; these are possible outliers to investigate.
The dataset also included bedrooms, bathrooms, and qualitative indicators (premium locality, gated complex, amenities). Values coded and do not make those indicators quantitative measurements: they represent unordered categories. The models below use the quantitative bedroom and bathroom counts.
| Predictors | Adjusted | Standard error | Overall result | |
|---|---|---|---|---|
| Area + bedrooms | (reported as 0.5464 once, inconsistent with the accompanying 46% and sum-of-squares ratio) | About | , | |
| Area + bathrooms | About | , |
For area and bedrooms, the fitted equation was:
The area slope was significant (, ), but the bedrooms slope was not (, ). The reported predictor correlation was . For area and bathrooms, the fitted equation was:
Area remained significant (); bathrooms did not (, ). Area and bathrooms correlated at . Both two-predictor models had significant overall tests, yet the extra count variable added little significant information conditional on area. This is compatible with an overall significant model and a nonsignificant individual predictor. Residual plots showed no clear funnel or curve; the bedroom model had two standardized residuals above .
The source’s reported area-plus-bedrooms value “0.5464” conflicts with its stated 46%, its SSR/SST calculation (), and its later statement. The consistent value is approximately ; retain the discrepancy in mind when reading the output.
Key takeaways
- Area alone had a significant relationship with price but explained only 44% of observed variation.
- Additional predictors can raise overall fit without having individually significant conditional slopes.
- A numeric category code remains qualitative when it only labels a category.
- The reported bedroom-model contains an internal inconsistency; the sums of squares support approximately .
7.4.1 Intercept, Extrapolation, and Omitted Variable Bias
Intercept and prediction range
The intercept is the expected response when every predictor equals zero. Computer output often tests , but that test is usually not the main question of interest; inference on the intercept is connected to inference for fitted values. If zero is outside the observed predictor ranges, the intercept describes an extrapolated setting and may have little practical meaning. More broadly, predictions outside the training data’s predictor ranges are extrapolations and are risky.
Omitted variable bias
Omitted variable bias (OVB) occurs when excluding a relevant predictor distorts the coefficient of an included predictor: the included variable can partly absorb the omitted variable’s association with the response. The lecture’s stated conditions that can permit OVB are: an omitted variable is correlated with the dependent variable, and/or it is correlated with an included predictor. These correlations make the simple regression coefficient differ from the included variable’s true separate contribution. This module identifies the issue conceptually; it does not derive a numerical bias formula.
In the household-debt example, first-income-only regression had , while the model with first income, monthly payment, and utilities had . Debt’s sample correlations with first income, monthly payment, and utilities were , , and , respectively. First income correlated with monthly payment at and utilities at ; monthly payment and utilities correlated at . Those relationships make omitted-variable bias plausible when relevant variables are left out.
Possible warning signs when needed variables are added include a substantial increase in , a decrease in model standard error, and changes in coefficient standard errors or -values. These are indicators for comparing model specifications, not a complete formal test of OVB.
Key takeaways
- The intercept’s zero-predictor interpretation can be extrapolative and unimportant in context.
- Avoid using the fitted equation beyond the predictor ranges represented in training data.
- Omitted relevant variables can distort included slopes when the stated correlation conditions hold.
- Compare specifications and diagnostics; a higher alone does not prove causality or eliminate bias.
7.4.2 Module Synthesis
MLR estimates one response from several predictors. OLS chooses coefficients by minimizing SSE; and adjusted summarize fit. The test evaluates the predictors jointly, while tests evaluate conditional individual slopes. Those inferences depend on error assumptions, which residual plots help assess. Multicollinearity complicates separate slope interpretation; outliers and influential cases require careful review. Practical model building cycles through predictor choice, estimation, diagnostics, validation, and monitoring. Quantitative examples here show that adding relevant predictors can improve fit, while correlated or redundant predictors may contribute little. Qualitative variables require an appropriate representation and are taken up separately.
Key takeaways
- Read fit, overall significance, individual significance, and residual diagnostics together.
- Interpret coefficients conditionally and avoid causal claims from regression association alone.
- Investigate correlated predictors and unusual observations before settling on a model.
- Validate model performance on data not used to estimate its coefficients.
Revise before your examAll 2 exam tips from this module in one list
- In multiple regression, “holding the other predictors constant” is part of the slope’s meaning, not an optional caveat.
- A significant test can coexist with nonsignificant individual tests, especially when predictors overlap in the information they carry.
Test yourself on this module
Questions from these notes, one at a time, with feedback. Your weak topics are saved.