6.2.1 Linear Regression: Purpose, Language, and Model
Regression predicts a continuous numerical outcome from one or more explanatory variables. It is supervised learning: the training observations include both the predictor and the outcome. In simple linear regression (SLR), one predictor is used to explain or predict one response through a relationship linear in its unknown coefficients.
Galton studied heights in 905 adult children from 205 families, converting female heights to male equivalents by multiplying by . Children of above-average mid-parent height tended to be shorter than their parents, while children of below-average mid-parent height tended to be taller. He called this movement toward the offspring population mean regression.
| Term | Meaning |
|---|---|
| Dependent variable | Outcome, response, or quantity being predicted |
| Independent variable | Predictor, explanatory variable, or feature |
| Error | Random unexplained part of observation |
| Regression coefficients | Population intercept and slope |
The population model is
The systematic component is the conditional expected response at ; captures what the predictor and line do not explain. Examples include sales from advertising, customer ratings from wait times, treatment cost from patient weight, and maintenance cost from bus age. With two or more predictors the model is multiple regression.
“Linear” refers to the coefficients, not necessarily to the shape in . Both and are linear in the parameters. A form such as is nonlinear in a coefficient. Transformations can sometimes turn a nonlinear-looking relationship into a model linear in its parameters.
Key takeaways
- SLR has one predictor and one continuous response.
- The error term separates the fitted conditional mean from an individual observed outcome.
- Linearity is about unknown coefficients; a transformed predictor can still yield linear regression.
Building a regression model
The workflow moves from data quality to a decision-ready model: collect primary (such as surveys/interviews) or secondary data; preprocess it (derive comparable features, address missing values, and investigate outliers); split observations into training and validation sets (the lecture suggests about 75% training); inspect training-set statistics and plots; choose a functional form; estimate coefficients, commonly by OLS; diagnose assumptions and form; evaluate on validation data; and deploy only when performance is acceptable. Training performance alone can mislead: overfitting occurs when a model performs well on training data but poorly on validation data. The goal is consistent performance, with low bias and low variance.
Key takeaways
- Data collection, preprocessing, exploration, form choice, estimation, diagnostics, validation, and deployment are connected steps.
- Keep validation data separate from model fitting to check generalization.
- Overfitting is a gap between strong training results and poor validation results.
6.2.2 Population Relationship, Estimates, and Prediction
For customers with the same income , monthly spending varies. The model describes the mean of that conditional distribution:
The population parameters are unknown, so sample data estimate them as and . The estimated regression equation is
Here is the estimated intercept and the estimated slope. For a given , estimates the mean response and can also serve as a prediction for an individual response. Predictions should remain within the observed predictor range; the fitted pattern alone does not justify extrapolation beyond it.
For Hanumantha’s Canara Bank sample, lakhs, , , and . Thus:
Each additional lakh of annual income is associated with an estimated ₹85.27 increase in monthly card spend. At lakhs:
This is an estimate based on the fitted sample relationship, not a guarantee for a customer.
Key takeaways
- The population line uses unknown parameters; the sample fitted line uses estimates.
- estimates the conditional mean and supplies a point prediction.
- Interpret slope in the units of per one unit of ; avoid unsupported extrapolation.
6.3.1 Ordinary Least Squares
Ordinary least squares (OLS) chooses the line whose predictions have the smallest sum of squared vertical errors. Squaring prevents positive and negative errors from cancelling and penalizes larger misses more.
Minimizing this quantity gives:
The calculation follows a clear sequence: compute ; for each row compute and ; sum their products for the numerator and sum squared deviations for the denominator; calculate , then . The fitted line passes through .
For the 55-customer income example, the resulting estimates are and . A scatterplot with on the horizontal axis and on the vertical axis is a useful first check for an approximately linear trend. Excel can show a linear trendline, its equation, and ; its regression tool also produces the broader diagnostics described below.
Key takeaways
- OLS minimizes squared residuals, not the sum of residuals or absolute residuals.
- The slope is centered cross-deviation divided by predictor variation.
- Inspect the data pattern before treating a straight line as an adequate form.
6.3.2 Explained Variation and the Coefficient of Determination
For observation , the residual is actual minus fitted response, . Three sums of squares partition the response variation for an OLS line with an intercept:
SST is total variation around the sample mean; SSR is variation explained by fitted values; SSE is leftover variation around the line. A perfect fit has ; a line explaining none of the total variation has .
The coefficient of determination is
In this setting it ranges from to . It is the proportion of sample variation in accounted for by the fitted line, not the proportion of individual outcomes predicted exactly, and not evidence that causes .
For Hanumantha’s sample, the observation with income ₹22 lakhs has actual spend ₹2,429 and fitted spend ₹2,923, so . Across all observations, , , and (rounding accounts for a one-unit difference in the sum). Thus : about 65% of the observed sample variation in monthly spend is explained by the fitted income line.
Key takeaways
- Residuals are vertical differences between observed and fitted values.
- Total variation divides into explained and unexplained sums of squares.
- describes in-sample fit; it does not establish causation or guarantee predictive accuracy on new data.
6.3.3 Correlation and
In simple linear regression with an intercept, the sample correlation between and is the signed square root of :
The sign comes from the slope because squaring removes direction. For Hanumantha’s and positive slope, . Correlation ranges from to : the endpoints indicate perfect negative or positive linear association, while values near zero indicate little linear association. Correlation summarizes direction and strength of linear association; summarizes the share of response variation explained by a fitted model. Unlike the two-variable correlation coefficient, also applies in nonlinear and multiple-predictor settings. Neither proves a causal mechanism.
Key takeaways
- In SLR with an intercept, and the slope gives the sign.
- Correlation and fit are descriptive; they do not, by themselves, identify cause.
- Distinguish a plausible explanatory mechanism from an unexplained association.
6.3.4 Error-Term Assumptions
Significance tests rely on assumptions about the population error :
| Assumption | Meaning |
|---|---|
| Zero conditional mean | , so |
| Constant variance (homoscedasticity) | for every |
| Independence | Errors for different observations are independent |
| Normality | Errors are normally distributed at each |
Together, the errors have mean zero and common variance ; because is a linear function of the error, conditional responses are normally distributed around a mean that changes with and with the same variance. If , the conditional mean does not vary linearly with ; if , a linear mean relationship exists.
Key takeaways
- The assumptions underpin the usual and inference for regression.
- Constant variance means the vertical spread around the line does not systematically change with .
- A significant slope is a statement about a linear association in the population under the model assumptions.
6.3.5 Testing Whether the Slope Is Significant
The central test is against . Under the null, the mean response does not change linearly with . A small -value is evidence against that null, conditional on the model assumptions.
Estimate error variance using
There are error degrees of freedom because the intercept and slope were estimated. The slope standard error and statistic under the null are
Use a two-sided distribution with degrees of freedom to obtain the -value. For the 55-customer example, , so and . With , . Then (53 degrees of freedom; reported ).
The equivalent SLR overall test compares explained and residual mean squares:
Its degrees of freedom are and . In simple regression, for the slope test, so both tests give the same significance conclusion. Significance does not mean the relationship is large, useful, causal, or correctly specified.
Key takeaways
- Test the population slope, not whether the fitted sample slope happens to be zero.
- estimates error variance; the slope test uses its square root in the standard error.
- In SLR, the one-slope test and overall test are equivalent.
6.3.6 Reading Regression Summary Output
An analysis tool’s summary brings the preceding calculations together:
| Output | Interpretation |
|---|---|
| Fraction of sample response variation explained by the fitted line | |
| Standard error / | , estimated residual standard deviation |
| ANOVA: SSR, SSE | Explained and residual sums of squares |
| MSR, MSE, , Significance | Overall slope test; df and |
| Coefficients | (intercept) and (slope) |
| Slope standard error, , | Test of |
For the 55-customer example, the tool reports , , , , and with slope standard error about and . The tool is useful because it automates arithmetic; understanding each quantity makes its output interpretable.
Key takeaways
- Check that selected and columns correspond row by row.
- ANOVA’s residual sum of squares is SSE; “Significance F” is the overall test -value.
- Coefficient output and ANOVA should tell consistent stories about the slope.
6.3.7 Residual Analysis
Residuals are observable clues about errors that cannot be directly observed. Check plots of residuals against , fitted values , and standardized residuals. A residual plot should look roughly like a horizontal band around zero if the line and constant-variance assumption are reasonable.
| Pattern | Possible implication |
|---|---|
| Roughly horizontal band | No obvious nonlinearity or changing spread |
| Funnel, widening with | Nonconstant error variance; homoscedasticity may fail |
| Curved / U-shaped pattern | Straight-line mean form may be inadequate; consider a transformed or richer model |
For simple regression, residuals against and fitted show the same pattern because fitted values are a linear function of . The fitted-value plot becomes especially useful with multiple predictors. Standardized residuals are residuals divided by their estimated standard deviation. If normality is plausible, roughly 95% should lie between and ; this is a diagnostic clue, not a proof of normality.
Key takeaways
- Look for structure, changing spread, and unusually large standardized residuals.
- A visually random band supports—but cannot prove—the assumptions.
- Residual diagnostics qualify the significance conclusions drawn from model output.
6.3.8 Outliers and Influential Observations
An outlier departs substantially from the trend of the remaining observations. A large absolute standardized residual flags a candidate; the lecture uses values outside about as a warning to inspect. Do not automatically delete it. Check for recording or measurement error and correct errors; if valid, it may be a chance extreme or reveal that the model is inadequate.
An influential observation can materially change the fitted line, especially its slope, if removed. It may combine an unusual with an extreme , or be far out in even if close to the trend. Inspect the data source and validity. If valid, examine its context and, where possible, collect additional observations at intermediate values to clarify the relationship.
Key takeaways
- An outlier concerns unusual response relative to the pattern; influence concerns impact on the fitted model.
- Verify observations before correcting, retaining, or changing the model.
- Extreme predictor values can be influential even when their residual is not especially large.
6.3.9 Putting Inference and Diagnostics Together
The workflow connects fitting, fit assessment, inference, and assumption checking:
For Hanumantha, the fitted equation is , , correlation is about , and . The slope tests are highly significant (, ), while residual plots show no clear funnel or curved pattern; most standardized residuals lie within , with two slightly beyond. Statistical evidence, fit, and diagnostics answer different questions and should be read together.
Key takeaways
- A significant slope does not replace checking fit and assumptions.
- Use for explained sample variation, tests for evidence of a slope, and residuals for model diagnostics.
- Draw population conclusions only with the model assumptions and sampling context in view.
6.4.1 Application: Household Size and Monthly Card Spend
The first Excel application uses household size to predict Hanumantha’s customers’ monthly card spend. The scatterplot suggests a positive but noisy linear trend. The fitted equation is , with (about 0.39); household size explains about 39% of sample variation. Correlation is positive, . The standard error is , so . ANOVA gives , , error df , and ; the -value is close to zero. The slope standard error is , so , also with a very small -value.
Three standardized residuals exceed and sit above the line. Residuals against household size show neither a clear funnel nor curvature, but residuals against predicted spend show a trend, suggesting other influences such as annual income. Household size is statistically related to spend, but income explains variation better in its separate model.
Key takeaways
- A visible positive trend can coexist with substantial scatter and modest .
- A single predictor rarely captures every determinant of spending.
- Interpret model usefulness separately from whether a slope is statistically significant.
6.4.2 Application: Monthly Spend and Annual Income
For 55 customers, the descriptive and OLS calculations give lakhs, , and . Income and spend have a positive linear association. For annual income ₹35 lakhs, predicted monthly spend is approximately ₹4,032. The fitted equation is intended for the income range represented in the data.
The fit and inferential output are , correlation , , , , , , and slope with close to zero. Residual plots broadly resemble a band; most standardized residuals are within , with two slightly outside. The relationship is statistically significant and explains a meaningful share, but not all, of variation.
Key takeaways
- Read coefficient size in business units: ₹85.27 more monthly spend per ₹1 lakh of annual income.
- The test supports a nonzero population slope under the model assumptions.
- The remaining 35% of variation and unusual residuals remain relevant to decisions.
Additional Worked Example: Season Pass Holders and PIN-Code Population
Arun relates the number of season pass holders () to PIN-code population (). The estimated line is . Thus an additional 1,000 residents corresponds to about nine additional season pass holders. ANOVA gives , , , and error degrees of freedom :
The -test -value is close to zero, so reject . The slope standard error is , giving (the transcript reports , reflecting rounded displayed inputs); the -value is very small and leads to the same conclusion. One standardized residual is while the others are within , flagging a PIN code well above the line. Apart from that observation, residuals against population look random. Population is related to season-pass counts, but demographics (families and children) and income may also matter. The relationship can help target marketing; it does not alone explain local differences.
Key takeaways
- The fitted slope translates population differences into expected pass-holder differences.
- Both the overall and slope tests indicate a significant linear association.
- Investigate the high residual and collect other plausible predictors before over-interpreting the line.
Additional Worked Example: Bus Age and Annual Maintenance Cost
For 30 BMTC buses aged 1–5 years, mean age is 3 years (median 2.5); mean annual maintenance cost is ₹6.1 lakhs with standard deviation about ₹2 lakhs and observed costs up to ₹9.5 lakhs. The scatterplot suggests a linear increase. The estimated regression line is
where is annual maintenance cost in lakhs and is age in years. Each additional year is associated with about ₹1.32 lakhs higher annual maintenance cost. Though the intercept is 2.2, it must not be read as the cost of a new bus: age zero is outside the observed range, so that is extrapolation. The line is for ages 1 through 5.
The fit is , hence positive correlation . Standard error , so . ANOVA values are , , , with regression df and error df :
The -test -value is very small. The slope standard error is , giving and a very small -value; reject . All standardized residuals are between and , and the residual plot against age looks random. The line can help quantify maintenance costs for fleet budgeting and replacement planning within the sampled age range.
Key takeaways
- Strong fit and significant slope support an age–maintenance association in this sample.
- A plausible coefficient interpretation does not make extrapolation valid.
- Residual checks and the observed predictor range bound how the model should be used.
6.4.2 Application: Computer Price and Customer Satisfaction
In a 36-response computer satisfaction survey, price (in thousands of rupees) is the predictor and score the response. The trendline is
Price explains 36% of score variation. The residual standard error is . The output reports , in one utterance but then uses in the calculation; , while the reported corresponds to about . The reported and slope standard error give (reported ); both tests have very small -values. The positive slope means a ₹1,000 increase in price is associated with a point higher score in these data, even though intuition suggested the opposite. Regression describes the observed sample pattern; unexpected direction merits investigation.
Two standardized residuals are below (about and ), where the model overpredicts scores. Residual plots show no clear funnel or curvature against price, but a trend against score suggests other factors influence satisfaction.
Key takeaways
- A statistically significant direction can conflict with prior expectations; the fitted model follows the data.
- leaves most score variation unexplained by price alone.
- The spoken ANOVA sums of squares are inconsistent; the notes preserve that ambiguity.
6.4.3 Application: Household Debt and Income and Monthly Payment
The household survey has about 500 Bengaluru observations. The analysis repeatedly keeps one predictor at a time, so each result is a separate SLR rather than a controlled “effect” holding other variables constant.
Debt and first income
With household debt as and first income as , the trendline is and , or correlation about . The standard error is ₹28,619. ANOVA reports and , with error degrees of freedom; and is close to zero. The coefficient output gives slope about (one spoken calculation says , inconsistent with the equation and reported ; the equation’s is the coherent displayed estimate). The association is significant but income alone explains only 31% of debt variation. Residuals include values beyond and an oval-like pattern, leaving scope for other predictors.
Debt and monthly payment
With debt as and monthly payment as , the trendline is , with and positive correlation about . Standard error is ₹27,518. ANOVA reports and ; with 498 error degrees of freedom, and . The slope standard error is , so and the -value is close to zero. Several standardized residuals exceed and plots suggest remaining structure. Monthly payment alone explains 37% of debt variation. Since debt can depend on several factors, the slope is an unadjusted association; multiple regression is needed to study several predictors together.
Key takeaways
- Statistical significance can coexist with modest explanatory power and residual concerns.
- One-predictor slopes can reflect omitted factors and should not be called isolated causal effects.
- Unexpected signs, outliers, and residual patterns are prompts to investigate and improve the model.
Revise before your examAll 1 exam tip from this module in one list
- A fitted line or high training is not enough for deployment. Check assumptions and validation performance.
Test yourself on this module
Questions from these notes, one at a time, with feedback. Your weak topics are saved.