Categorical Predictors in Multiple Linear Regression
Regression can use qualitative information such as brand, location, ownership, or smoking history alongside numerical measurements. A categorical predictor lets the model represent group differences while retaining quantitative predictors such as price or age.
For a response , quantitative predictor , and binary indicator ,
The fitted coefficients are obtained by least squares as in ordinary multiple regression. Model assessment also follows the same pattern: use and adjusted , the overall -test, individual -tests, and residual diagnostics. The indicator simply adds a predictor column to the analysis.
Categorical Predictors in Multiple Linear Regression
Binary indicators and a computer-brand example
An indicator (dummy) variable codes group membership as for the named group and for its comparison group. In a customer-satisfaction data set with 36 respondents, is satisfaction score, is computer price in lakhs, and for Lenovo and for Dell.
The price-only model had , , and a price coefficient , with p-values close to zero. Adding brand yielded , standard error , , , and unchanged . The overall -statistic was with p-value close to zero, rejecting the joint null that both slope coefficients are zero.
The fitted coefficients were , , and . For price, (t=b_1/SE=1.29/0.283\approx4.58), with 33 degrees of freedom and p-value close to zero. For brand, (t=b_2/SE=3.93/1.63\approx2.42), also with 33 degrees of freedom; the source reports its p-value as close to zero. Thus both predictors were treated as significant in this analysis.
Reading coefficients as group comparisons
The indicator coefficient compares the named group with the reference group at the same values of the other predictors. For Dell () and Lenovo (), the population mean equations are:
The two groups have a common slope , while their intercepts differ by . A positive means a higher mean response for Lenovo than Dell at a fixed price; a negative value means lower; zero means no modeled difference.
Using the estimates, the predicted equations are approximately:
Thus the estimated mean Lenovo satisfaction score is points above Dell at the same price. (The intercept is also rounded to .) Residuals should be inspected against each predictor and fitted values. Binary predictors naturally produce two vertical clusters; look for positive and negative residuals without a systematic pattern. Standardized residuals beyond roughly merit attention, not automatic deletion.
More than two categories: indicators
For a categorical variable with levels, use binary indicators and leave one level as the reference (baseline) category. Coding every level with its own indicator alongside an intercept creates redundancy; coding a category as incorrectly imposes equal numerical spacing and a linear trend across labels.
For brands Lenovo, Dell, and Asus, choose Asus as baseline and define for Lenovo and for Dell (each zero otherwise):
| Brand | Lenovo | Dell | Expected score |
|---|---|---|---|
| Asus (baseline) | 0 | 0 | |
| Lenovo | 1 | 0 | |
| Dell | 0 | 1 |
Here is the expected response for Asus, is Lenovo minus Asus, and is Dell minus Asus. The baseline choice is arbitrary: choosing Lenovo instead changes the coefficient comparisons and intercept, but not the fitted group means or model predictions.
Key takeaways
- Code a two-group category with one (0/1) indicator; its coefficient is an adjusted group difference.
- With (k) categories, use indicators and interpret each against the chosen baseline.
- Categorical predictors enter the same regression workflow as quantitative predictors.
- Check individual coefficient evidence and residual patterns; a higher alone does not establish a useful predictor.
Worked Example: Household Debt and Categorical Predictors
The household-survey response is debt; existing quantitative predictors are first income , monthly payment , and utilities . Their initial fitted coefficients were , , , , with . All three slopes were reported significant.
Worked Example: Household Debt and Categorical Predictors
Adding home ownership
Let for owning and for renting. The revised model had (a small increase) and standard error (18{,}594), down from (18{,}699). Its overall -test p-value was very small. Estimates were , , , , and . Holding the other predictors fixed, owners were predicted to have (12{,}837) less debt than renters.
For ownership, , so , with 495 degrees of freedom; . Reject . Residual plots were described as balanced around zero, with some high standardized residuals. Ownership has evidence of association but adds little explanatory improvement, illustrating that significance and predictive gain are distinct questions.
Adding four-level location
Location has Southeast, Southwest, Northeast, and Northwest levels. Use three indicators: for Northwest, for Northeast, for Southwest, and all three zero for Southeast, the baseline. In addition to the prior predictors, the location model had and standard error (18{,}589), only a slight improvement. Estimates included , , , , , , , and . The location coefficients are estimated debt differences from Southeast, controlling for the other variables.
The location tests gave:
| Comparison with Southeast | Coefficient | Standard error | (p) | Decision at | |
|---|---|---|---|---|---|
| Northwest | 4,229 | 3,471 | Do not reject | ||
| Northeast | 2,935 | 2,531 | Do not reject | ||
| Southwest | 5,211 | 2,881 | Do not reject |
None of the three location coefficients met the threshold. The model represents eight combinations of ownership and location, but that flexibility adds complexity. Since the three original quantitative predictors already explained substantial variation, the small gain from adding ownership and location may not justify all added parameters. Adding variables simply because they are available risks overfitting; use context, fit, individual evidence, and diagnostics together.
Key takeaways
- Ownership was statistically significant () but improved by only .
- Location added three comparisons to a baseline; none had , and rose only further.
- A strong starting model leaves less unexplained variation for new predictors to capture.
- Prefer a useful balance of explanatory performance and model simplicity over indiscriminate inclusion.
Worked Example: Flat Prices in Mysuru
For 30 flats in Jayalakshmi Puram, selling price is measured in lakhs, and area in square feet. The area-only model was , with . Bedroom count and bathroom count were not retained because they were highly correlated with area (multicollinearity).
Worked Example: Flat Prices in Mysuru
Premium location indicator
Let for a premium locality and zero otherwise. The two-predictor model improved to , adjusted , and standard error (versus ). , , and ; indeed . With 2 predictors and 27 error degrees of freedom, and .
with p-value close to zero for the overall test . Estimates were , , . A one-square-foot increase corresponds to an estimated lakh (₹8,700) higher price, holding location fixed. Premium location adds an estimated ₹86.2 lakh at the same area.
For area, , so , 27 degrees of freedom, . For premium location, , so , 27 degrees of freedom, . Both were significant. Standardized residuals were all below 3 in absolute value; plots against area, location, and fitted price were described as patternless and balanced.
Adding amenities and gated community
Now let for good amenities and for a gated community. With area and premium location retained, the model achieved , adjusted , and standard error . , , and . The transcript reports , , and with p-value close to zero. (The displayed is approximately ; using the rounded displayed values gives an near .)
The estimated equation is:
The spoken account gives the intercept once as and later writes ; the former is consistent with the reported coefficient. For a fixed set of other predictors, one more square foot is associated with lakh (₹7,800). Premium locality adds lakhs, amenities lakhs, and gated community lakhs in the equation (the spoken interpretation rounds the last coefficient to ).
Area and premium locality had p-values close to zero; amenities had ; gated community had . All met the criterion. Adding the categorical predictors changed the area coefficient from to , consistent with area alone previously absorbing some differences associated with omitted characteristics. Residual plots were described as random and standardized residuals below 3 in absolute value. This example shows that qualitative predictors can explain meaningful price variation, while bedroom/bathroom predictors need not be added when they overlap strongly with area.
Key takeaways
- Do not add highly correlated measures of the same underlying feature automatically.
- Premium location improved fit; location, amenities, and gated-community indicators were each significant in the expanded model.
- Interpret every slope holding the remaining predictors fixed and preserve the units of and .
- Validate the added fit with coefficient tests and residual diagnostics.
Worked Example: Stroke-Risk Score
A hospital data set has 25 observations. The response is a stroke-risk score from 1 to 100; predictors are patient age, blood pressure, and smoking history. Descriptive summaries were: risk mean , median , range –, standard deviation about ; age mean , median , range –, standard deviation about ; blood pressure mean , median , range –, standard deviation . The respective coefficients of variation for risk, age, and blood pressure were about , , and .
Worked Example: Stroke-Risk Score
Separate and joint quantitative models
Risk plotted separately against age and blood pressure showed trends but substantial scatter; each simple trend had . The age-only estimate was , with , , , and (p-value close to ). The blood-pressure-only model was , , , , and (p-value close to ). Standardized residuals in both were within , with no strong residual pattern reported.
Putting both predictors in one model yielded , adjusted , standard error , hence . , , and , so:
with p-value close to zero for . Both individual p-values were close to zero. The coefficients were for age and for pressure; their -statistics were (standard error ) and (standard error ), respectively. Both estimated effects were positive. Standardized residuals were between and , and residual plots against each predictor appeared random.
The joint can greatly exceed either separate : each one-predictor model leaves the other predictor out, while the joint model lets both explain response variation. Omitted predictors can distort the apparent relationship attributed to an included predictor; this is omitted-variable bias.
Adding smoking history
Let for a history of smoking and zero otherwise. The estimated model was:
where is age and is blood pressure. , adjusted , standard error , and . The reported sums of squares were , , . With three predictors, ; error degrees of freedom are , so .
with p-value close to zero for the joint null that all three slopes are zero. For age, and , 21 degrees of freedom, . The blood-pressure coefficient was also reported significant. For smoking, and , 21 degrees of freedom, with p-value reported close to zero; reject . At the same age and pressure, the fitted score is points higher for the smoking-history group. Standardized residuals remained within ; residual plots against age, pressure, and smoking indicator showed both signs without a notable pattern.
Key takeaways
- Weak one-at-a-time fit does not preclude a stronger joint model.
- The -test assesses whether all slopes are zero; -tests assess individual slopes conditional on the other predictors.
- Smoking adds an adjusted group difference of risk-score points in this fitted model.
- Residual checks accompany and significance tests; none alone establishes a complete model.
Worked Example: Monthly Online Spending
For 600 randomly selected customers, monthly website spend is recorded in thousands of rupees, annual salary in lakhs, married status is for married, and gender is for male. Mean and median spending were roughly ₹5,000, range ₹0–₹12,000, and standard deviation about ₹3,000. Salary mean and median were ₹18 lakh, range ₹12–₹25 lakh, and standard deviation about ₹4 lakh. Salary-spend correlation was .
Worked Example: Monthly Online Spending
Salary-only model
The fitted model was , with . Since the simple-regression is the squared correlation, . ANOVA gave , , and:
with p-value close to zero. Thus salary was statistically significant despite the low correlation and low explained proportion; the lecture retained it for subsequent models. Standardized residuals were between and , and the residual plot against salary had no strong pattern.
Add marital status
With X_2=1 for married, the reported fit was R^2=0.62, adjusted R^2 close to , standard error , MSE=3.36, and MSR=1{,}648:
with p-value close to zero. The fitted equation was:
Salary had t=5.16, standard error , and p-value close to zero. Marital status had t=30.6, standard error , and p-value close to zero. Since the response is in thousands of rupees, each additional lakh in salary corresponds to ₹0.10 thousand (₹100) higher predicted monthly spending, holding marital status fixed; a ₹10-lakh salary increase corresponds to ₹1,000. Married customers have ₹4.584 thousand (about ₹4,584) higher predicted spending than unmarried customers at the same salary. Residuals against salary and marital indicator were balanced; however, some standardized residuals were above or below , suggesting unexplained variation.
Gender comparison and combined model
Replacing marital status with X_2=1 for male gave R^2=0.17, adjusted R^2\approx0.164, standard error , MSE=7.4, MSR=442, and F=59.8 with p-value close to zero. The equation was:
Both salary and gender coefficients had p-values close to zero. At a fixed salary, the model estimated male customers’ monthly spend to be ₹2.2 thousand (₹2,200) higher. Despite significance, this model explained only about 16% of spending variation; the estimated group difference’s magnitude is less firmly informative when much variation remains unexplained. Residuals looked random, with most standardized residuals between and .
Including salary, married status X_2, and male indicator X_3 yielded R^2=0.76, standard error , MSE=2.16, MSR=1{,}339, and F=620, p-value close to zero. The transcript says adjusted R^2 was close to ; this conflicts with its R^2=0.76 and is retained as an apparent reporting inconsistency rather than silently corrected. The estimated equation was:
Salary t=6.8, married t=38.1 with standard error , and male t=18.2 with standard error ; all p-values were reported close to zero. At fixed values of the other predictors, each additional salary lakh predicts about ₹0.11 thousand (₹110) more monthly spend; married status predicts ₹4.58 thousand more; male status predicts ₹2.2 thousand more. Residuals against all three predictors were balanced, though some standardized residuals approached , indicating remaining variability and possible value in considering additional relevant predictors.
Key takeaways
- A small correlation or R^2 can coexist with statistically significant predictor evidence.
- Interpret monetary coefficients using the actual units: response in thousands, salary in lakhs.
- Marital status improved the model much more than gender alone; the combined model included both significant indicators.
- Significance does not imply that the model explains most individual variation or that every coefficient magnitude is estimated with high predictive accuracy.
Interaction Variables
A standard indicator model allows groups to have different intercepts but a common quantitative slope. An interaction allows the effect of one predictor to depend on another. It is often formed as the product of a continuous variable and a categorical variable.
Let X_1=1 for women and for men, X_2 be work experience in months, and Y salary. Include the product X_1X_2:
This remains a linear regression model because it is linear in the unknown coefficients , although the predictors are multiplied.
For men (X_1=0):
For women (X_1=1):
Thus is the women-men intercept difference at , is the men's experience slope, and is the difference in experience slopes. The gender indicator is a moderator because the salary-work-experience relationship depends on gender.
Interaction Variables
Worked salary example
For a 30-person sample, the fitted model was:
The model had R^2=0.92, adjusted R^2=0.91, overall F-test p-value small, and each predictor p-value below .
For women, substitute :
\widehat{Y}_{\text{women}}&=13{,}450.2-7{,}771.68+(293.8-241.6)X_2\\ &=5{,}678.52+52.2X_2\approx5{,}678.5+52.2X_2. \end{aligned}$$ For men, substitute $X_1=0$: $$\widehat{Y}_{\text{men}}=13{,}450.2+293.8X_2.$$ The fitted salary increase per extra month of experience is $52.2$ for women and $293.8$ for men. These are conditional slopes; the interaction coefficient $-241.6$ captures the difference between them. **Key takeaways** - Without interaction, groups share a slope; with interaction, the slope can vary by group. - The coefficient on the product term is the difference in slopes for a binary moderator. - A model can be nonlinear in predictors yet linear in its coefficients. - Substitute each indicator value to derive and interpret group-specific equations. #### Module Synthesis: Model Building and Inference Categorical variables extend multiple regression to mixed quantitative and qualitative data. Least squares estimation, multiple $R^2$, adjusted $R^2$, the overall $F$-test, individual $t$-tests, and residual analysis remain central. In a sample, these procedures support inferences about population relationships under the regression assumptions. Model building is iterative: propose predictors with substantive reason, estimate the model, interpret effects with units and comparisons explicit, assess tests and residuals, and decide whether improved explanation justifies added complexity. A large $R^2$ does not make every predictor useful; a statistically significant coefficient can coexist with low explanatory power. Correlated predictors can obscure individual contributions, while omitted relevant variables can distort coefficients. Qualitative data can add substantial explanatory value, as in the flat-price and spending examples, but other categories added little in the household-debt example. The broader purpose of business statistics is to summarize data, analyze it, make evidence-based inferences for decisions beyond the observed sample, and quantify uncertainty. This module’s regression tools build on descriptive statistics, probability, sampling distributions, confidence intervals, hypothesis tests, and simple and multiple regression. Statistical software reduces calculation effort; sound interpretation and decisions remain the analyst’s responsibility. **Key takeaways** - Use indicators for categories and interaction terms when a slope may depend on group membership. - Assess overall fit, individual predictor evidence, and residual behavior together. - Keep only predictors that contribute useful, defensible information; do not add variables merely because they were collected. - Regression supports population inference from sample data when its assumptions are appropriate.Test yourself on this module
Questions from these notes, one at a time, with feedback. Your weak topics are saved.