Multiple Linear Regression - II

Categorical Variables in Multiple Linear Regression

Many real-world datasets contain qualitative or categorical variables (e.g., brand, location, ownership). Including them in a regression model often improves fit and prediction. The core idea is to convert categories into numeric form using dummy (indicator) variables and then proceed with standard multiple linear regression.

The Dummy Variable Approach

A dummy variable takes only values 0 or 1 to represent membership in a category. For a categorical variable with two categories, a single dummy variable suffices:

$$x_2 = \begin{cases} 1 & \text{if Lenovo} \\ 0 & \text{if Dell} \end{cases}$$

The regression model becomes:

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \epsilon$$

where $x_1$ is a quantitative predictor (e.g., price) and $x_2$ is the dummy. The coefficient $\beta_2$ measures the average difference in the dependent variable between the two categories, holding $x_1$ constant.

Exam tip: For a categorical variable with $k$ categories, use exactly $k-1$ dummy variables to avoid perfect multicollinearity (the dummy variable trap). The omitted category becomes the reference group.

Worked Example: Basavaraja’s Customer Satisfaction Data

• Dependent variable $y$: Satisfaction score (0–100)
• Quantitative predictor $x_1$: Price paid (in ₹ lakhs)
• Categorical predictor: Brand (Lenovo vs. Dell)
• Dummy variable $x_2$: $1$ if Lenovo, $0$ if Dell
• Sample size: $n = 36$

Simple Regression (price only)

The estimated equation was:

$$\hat{y} = 48.43 + 1.32\,x_1$$

• $R^2 = 0.31$
• Standard error $= 5.19$
• $F = 18.9$, $p \approx 0$

Multiple Regression with Dummy Variable

Using the dummy variable, the estimated equation (from Excel Analysis ToolPak) is:

$$\hat{y} = 46.83 + 1.29\,x_1 + 3.93\,x_2$$

Model comparison

ANOVA

The overall $F$-test rejects $H_0: \beta_1 = \beta_2 = 0$, confirming a significant linear relationship.

Coefficient estimates and $t$-tests

Both predictors are individually significant. Interpretation:

• Price: For each additional ₹1 lakh, satisfaction score increases by 1.29 points, holding brand constant.
• Brand dummy: Lenovo customers score, on average, 3.93 points higher than Dell customers who pay the same price.

How Dummy Variables Change the Regression Lines

The model implies two parallel lines (same slope $\beta_1$) with different intercepts:

• Dell ($x_2=0$): $\hat{y} = 46.83 + 1.29 x_1$
• Lenovo ($x_2=1$): $\hat{y} = (46.83 + 3.93) + 1.29 x_1 = 50.76 + 1.29 x_1$

The vertical distance between the lines is exactly $3.93$ – the estimated brand effect.

Key Takeaways

• Dummy variables convert qualitative categories into 0/1 numeric values for regression.
• For two categories, one dummy is enough; the coefficient tells the average shift relative to the reference category.
• Adding a significant dummy improves $R^2$ and reduces standard error, as seen in the increase from 0.31 to 0.45.
• The $F$-test checks overall model significance; $t$-tests check each predictor (including the dummy) individually.
• Interpretation of a dummy coefficient: ceteris paribus (holding other predictors constant), the mean outcome differs by $\hat{\beta}_2$ between the two categories.

Interpretation of Coefficients (Categorical Variables)

When a categorical variable enters a multiple linear regression, each category shifts the intercept of the regression line (assuming no interaction with other predictors). The slope(s) for continuous predictors are shared across all categories, so the effect of a categorical predictor is a parallel shift of the regression plane.

Binary Dummy Variable (Two Categories)

For data with two brands (Dell and Lenovo), define one dummy variable:

[ x_2 = \begin{cases} 0 & \text{brand = Dell}\ 1 & \text{brand = Lenovo} \end{cases} ]

The model with price ($x_1$) and brand is

[ E(Y) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 ]

Substituting the two values of $x_2$ gives separate equations:

Interpretation:

• $\beta_1$ = effect of a one‑unit increase in price on mean satisfaction, holding brand constant.
• $\beta_2$ = mean difference in satisfaction between Lenovo and Dell, at any given price.
  • $\beta_2 > 0$: Lenovo has higher average satisfaction.
  • $\beta_2 < 0$: Dell has higher average satisfaction.
  • $\beta_2 = 0$: brand does not affect satisfaction.

Example (Chroma data – estimated model)

[ \hat{y} = 46.84 + 1.29, \text{Price} + 3.93, x_2 ]

Substituting the two brand cases:

• Dell ($x_2=0$): $\hat{y} = 46.84 + 1.29 \cdot \text{Price}$
• Lenovo ($x_2=1$): $\hat{y} = (46.84 + 3.93) + 1.29 \cdot \text{Price} = 50.77 + 1.29 \cdot \text{Price}$

Interpretation: On average, Lenovo satisfaction is 3.93 points higher than Dell, regardless of price.

---

Categorical Variable with $k > 2$ Categories

Rule: A categorical variable with $k$ levels requires $k-1$ dummy variables, each coded as 0 or 1. One category becomes the baseline (all dummies = 0); all other categories are compared to this baseline.

Example: Three Brands (Lenovo, Dell, Asus)

We need $k-1 = 2$ dummies:

[ x_1 = \begin{cases}1 & \text{Lenovo}\0 & \text{otherwise}\end{cases} \qquad x_2 = \begin{cases}1 & \text{Dell}\0 & \text{otherwise}\end{cases} ]

The baseline category (both dummies $=0$) is Asus.

Model: $E(Y) = \beta_0 + \beta_1 x_1 + \beta_2 x_2$

Interpretation (with Asus as baseline):

• $\beta_0$ = mean satisfaction for Asus.
• $\beta_1$ = mean difference between Lenovo and Asus.
• $\beta_2$ = mean difference between Dell and Asus.

Baseline choice is arbitrary. If we instead chose Lenovo as baseline, the dummy coding changes and so do coefficient interpretations – but the model’s fit and predictions remain identical.

Exam tip: Never create $k$ dummy variables for a $k$-level categorical variable – that introduces perfect multicollinearity (the dummy variable trap). Always use exactly $k-1$ dummy variables.

Key Takeaways

• Dummy variables allow categorical predictors to shift the intercept while keeping slopes constant.
• For a binary categorical, one dummy variable; its coefficient = difference in mean response between the two groups.
• For $k$ categories, use $k-1$ dummies (0/1 coding); the omitted category is the baseline.
• The estimated dummy coefficient = predicted difference in response between that category and the baseline, holding other predictors constant.
• Baseline choice is arbitrary; coefficient interpretations change relative to baseline, but predictions are unchanged.

Adding a Binary Dummy: Ownership

The starting model was a multiple regression with three continuous predictors:

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_3 + \varepsilon$

where $Y$ = debt, $X_1$ = first income, $X_2$ = monthly payment, $X_3$ = utilities.

Estimated equation from previous section:

$\hat{Y} = -140378 + 0.017 X_1 + 0.96 X_2 + 157.95 X_3$

$R^2 = 0.708$, standard error $= 18699$.

Intuition: Whether a household owns its home may affect debt – owners might have lower debt (e.g., paid-off property). To test this, add a dummy variable that flags ownership.

Model with ownership dummy:

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_3 + \beta_4 X_4 + \varepsilon$

where $X_4 = 1$ if the household owns the house, $0$ if it rents.

Results

Interpretation of $b_4$: Holding all other factors constant, a household that owns its home has $12837 less debt on average than a comparable renting household.

$p < 0.05$ → reject $H_0: \beta_4 = 0$. Ownership has a statistically significant linear association with debt. The overall F-test also remains significant.

Residual diagnostics: All residual plots (vs. each predictor and vs. fitted values) show random scatter; no pattern violations.

Exam tip: The dummy coefficient represents the shift in the intercept for the group coded 1. For two groups (own vs. rent), the model produces two parallel regression planes:

• Renters: $\hat{Y} = b_0 + b_1 X_1 + b_2 X_2 + b_3 X_3$
• Owners: $\hat{Y} = (b_0 + b_4) + b_1 X_1 + b_2 X_2 + b_3 X_3$ The slope coefficients of the continuous variables remain the same.

Adding a Multi‑Level Categorical Variable: Location

Intuition: Debt might vary by geographic region. The variable location has four categories: Southeast (SE), Northwest (NW), Northeast (NE), Southwest (SW). To include a $k$-level categorical variable, we need $k-1$ dummy variables. Choose SE as the baseline (reference) category.

Define three dummies

When all three are $0$, the property is in Southeast (baseline).

Extended model:

$Y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + \beta_3 X_3 + \beta_4 X_4 + \beta_5 X_5 + \beta_6 X_6 + \beta_7 X_7 + \varepsilon$

Results

Interpretation of location coefficients: Each dummy’s coefficient represents the average difference in debt compared to the baseline (SE), holding all else constant. For example, a household in the Northwest is predicted to have $4229 more debt than an otherwise identical household in the Southeast.

Significance tests: None of the three location dummy coefficients are statistically significant at $\alpha = 0.05$ (all $p > 0.05$). Therefore we cannot reject $H_0: \beta_5 = 0$, $H_0: \beta_6 = 0$, $H_0: \beta_7 = 0$. Adding location does not meaningfully improve the model.

Implicit regression equations: The model generates a separate equation for each combination of ownership and location – eight in total. For example, for a renting household in the Northwest:

$\hat{Y} = b_0 + b_1 X_1 + b_2 X_2 + b_3 X_3 + b_5$

For an owning household in the Southeast (baseline location):

$\hat{Y} = (b_0 + b_4) + b_1 X_1 + b_2 X_2 + b_3 X_3$

Lessons from the “Failures”

• Starting with an already strong model ($R^2 \approx 0.71$) makes it hard to improve with additional variables.
• Adding non‑significant categorical dummies increases complexity without real gain – risk of overfitting.
• Regression itself tells us which variables are relevant. The t‑tests and overall lack of improvement suggest that the three original continuous variables (first income, monthly payment, utilities) capture the dominant drivers of debt.
• Good model building requires balancing performance and complexity; not every collected variable deserves a place in the model.

Key takeaways

• Binary dummies (0/1) shift the intercept for one group; interpretation is the expected difference in $Y$ relative to the reference group.
• For a $k$-level categorical variable, create $k-1$ dummies; the omitted category becomes the baseline.
• Adding a categorical variable does not automatically improve a model – check individual t‑tests and $R^2$ / adjusted $R^2$.
• In the example, ownership was significant (small improvement), but location was not (all dummies $p>0.05$).
• A model with three continuous predictors already explained ~71% of debt variation; further gains are marginal.

Example: Categorical Variables in Flat Price Model

Adding categorical (qualitative) variables to a regression lets us capture non‑numerical factors like location, amenities, or neighbourhood type. In the flat‑price data from Mysuru (30 properties), the initial model used only area (square feet) and gave $R^2 = 0.44$ — nearly 56% of price variation was unexplained. The real‑estate analyst had three dummy variables: premium location, gated community, and good amenities, each coded as 1 = yes, 0 = no.

Motivation and Data

Quantitative variables (bedrooms, bathrooms) were dropped because of multicollinearity with area — a reminder that adding correlated predictors does more harm than good.

---

Model 1: Add Premium Location Only

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \varepsilon$$

Estimated equation: $\hat{y} = -3.92 + 0.087 x_1 + 86.2 x_2$

• $b_1$: Each additional sq.ft. → price higher by ₹8,700 (0.087 lakhs). $t = 5.31$, $p \approx 0$ → significant.
• $b_2$: Premium location adds ₹86.2 lakhs, holding area constant. $t = 4.11$, $p \approx 0$ → significant.
• Residual diagnostics: standardized residuals all $<3$ in absolute value, plots random.

Exam tip: When a dummy coefficient is large and significant, the qualitative factor has a major impact on the outcome — here location alone adds over ₹86 lakhs.

---

Model 2: Add All Three Dummy Variables

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_4 x_4 + \varepsilon$$

Estimated equation:

$$\hat{y} = -43.99 + 0.078 x_1 + 95.66 x_2 + 35.74 x_3 + 55.38 x_4$$

Coefficient significance (all $p$-values are low):

• All t‑statistics had $p < 0.05$ (amenities $p=0.04$, others near 0). Every variable should be retained.
• Omitted variable bias: The area coefficient dropped from 0.087 (Model 1) to 0.078 (Model 2). Earlier, area was partly picking up the effect of omitted categorical factors. Including them gives a cleaner estimate.

Residual diagnostics: Standardized residuals all within $\pm 3$, plots random across $x$'s and $\hat{y}$.

---

Interpreting Dummy Coefficients

With multiple dummies, the model implicitly contains $2^k$ different regression equations (one for each combination). For example, a property with all three features:

$$\hat{y} = -43.99 + 0.078 x_1 + 95.66 + 35.74 + 55.38 = 142.79 + 0.078 x_1$$

Each dummy adds a fixed shift to the intercept, while the slope for area stays common. This is the parallel‑lines interpretation.

---

Key Takeaways

• Categorical variables are essential in real‑estate models — they often explain more variance than quantitative predictors.
• Dummy coding (0/1) turns qualitative information into a numeric predictor; the coefficient is the expected change when the characteristic is present.
• Adding relevant dummies can dramatically improve $R^2$ (0.44 → 0.80) and reduce standard error.
• Always check for multicollinearity among predictors (e.g., bedrooms, bathrooms, and area) before including them.
• Omitted variable bias appears when excluded categorical factors are correlated with included variables — coefficient estimates for included variables may change when relevant dummies are added.
• Residual diagnostics remain essential: ensure plots are random and standardized residuals stay under 3 in absolute value.

Exam tip: When interpreting a dummy coefficient, always state “holding other variables constant.” The coefficient is the premium (or discount) associated with that category compared to the base case (all dummies = 0).

What are interaction variables?

An interaction variable is the product of two independent variables—often a continuous variable and a categorical variable. Including it in a regression model tests for conditional relationships: does the effect of one predictor depend on the value of another?

$\text{Interaction term} = x_1 \times x_2$

The model remains linear in the coefficients (e.g., $\beta_0, \beta_1, \beta_2, \beta_3$), even though it multiplies $x$’s. This is still a linear regression.

Why they matter: the moderation concept

A moderation variable changes the relationship between the dependent variable $y$ and another independent variable. The regression coefficient of the interacted variable ($x_2$) is contingent on the moderator ($x_1$). Two groups get different slopes.

Worked example: salary, gender, and work experience

From Jyoti Hegde’s survey: 30 respondents, variables:

• $y$ = salary (continuous)
• $x_1$ = gender (1 = women, 0 = men)
• $x_2$ = work experience in months (continuous)

Model:

$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 (x_1 \cdot x_2) + \varepsilon$

Separate equations (assuming statistical significance):

The marginal change in $y$ for a one‑unit increase in $x_2$ is:

• Women: $\beta_2 + \beta_3$
• Men: $\beta_2$

If $\beta_3 \neq 0$, the slope differs by gender.

Excel output & interpretation

Using Analysis ToolPack with inputs: salary (y), gender, work experience, and gender×experience (interaction).

Estimated regression equation:

$\hat{y} = 13450.2 - 7771.68\,x_1 + 293.8\,x_2 - 241.6\,(x_1\cdot x_2)$

Specific equations:

• Women ($x_1=1$): $\hat{y} = (13450.2 - 7771.68) + (293.8 - 241.6)\,x_2 = 5678.52 + 52.2\,x_2$

• Men ($x_1=0$): $\hat{y} = 13450.2 + 293.8\,x_2$

Interpretation: For each additional month of work experience, women’s salary increases by ₹52.2, whereas men’s salary increases by ₹293.8. The interaction term reveals a large gap in the return to experience by gender.

Exam tip: Always write out the separate group equations to interpret interaction coefficients clearly. The sign and magnitude of $\beta_3$ tell you how the slope changes for the category coded 1 relative to the baseline category (coded 0).

How interaction fits into the regression framework

The interaction term allows the relationship between $y$ and $x_2$ to depend on $x_1$ without splitting the data.

Key takeaways

• Interaction variable = product of two IVs; still linear in coefficients.
• Moderation variable alters the slope of another predictor.
• With a binary moderator, interaction produces two distinct regression lines.
• Interpret significance via $t$‑tests on the interaction coefficient.
• In the example, men’s salary grows 5.6× faster with experience than women’s (293.8 vs. 52.2).

Course Recap and Final Remarks

This section summarizes the entire Business Statistics for Entrepreneurs course — a comprehensive review of statistical methods for data-driven decision making. The central message: managers must back decisions with data analysis, not intuition alone, to quantify risk and gain competitive advantage.

Foundational Modules (1–3)

• Module 1: Data Types & Summaries
  • Qualitative vs. quantitative data; visualization; numerical summaries (mean, median, mode for central tendency; range, standard deviation, quartiles for dispersion).
• Module 2–3: Probability & Random Variables
  • Discrete random variables: Bernoulli, binomial, Poisson. Use probability mass function (PMF) to compute expectation, variance, cumulative/tail probabilities, and inverse calculations.
  • Continuous random variables: uniform, exponential, normal, standard normal. Probability density function (PDF) plays same role as PMF.
  • Linear combinations of random variables (especially normal). Relatives of normal: chi-square, t, F distributions — all share similar characteristics but serve specific inference purposes.

Statistical Inference

Module 4: From Sample to Population

• Sample statistics (sample mean $\bar{x}$, sample variance $s^2$, sample proportion $\hat{p}$) are random variables because samples are random.
• Their distributions are sampling distributions.
• Central limit theorem (CLT) is the bridge: even from a single random sample we can make probabilistic statements about population parameters ($\mu$, $p$, $\sigma^2$).

Module 5: Confidence Intervals & Hypothesis Tests

• Confidence intervals: point estimate $\pm$ margin of error for $\mu$ and $p$; for $\sigma^2$ a direct lower/upper bound.
• Hypothesis tests: structure null $H_0$ and alternative $H_a$ based on who makes the claim. Steps:
  1. Choose significance level $\alpha$.
  2. Assume $H_0$ true (at equality), compute test statistic (follows $t$, $z$, or $\chi^2$).
  3. Compute p-value (probability of Type I error).
  4. If p-value is small enough, reject $H_0$; otherwise, do not reject.
• Both methods allow inference about $\mu$, $p$, $\sigma^2$.

Regression Modeling

Module 6: Simple Linear Regression

• Model: $y = \beta_0 + \beta_1 x + \epsilon$
• Estimated equation: $\hat{y} = b_0 + b_1 x$ (least squares)
• Coefficient of determination $R^2$: proportion of variation in $y$ explained by the model.
• Validate assumptions about $\epsilon$ (homoscedasticity, normality, independence) via residual analysis; $t$-test and $F$-test check significance.
• Use Excel Analysis ToolPak.

Module 7: Multiple Linear Regression

• Model: $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p + \epsilon$
• $E(y) = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$
• Estimated: $\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + \dots + b_p x_p$
• Multiple coefficient of determination $R^2$; overall $F$-test; individual $t$-tests.
• Issues: multicollinearity, omitted variable bias; iterative model building; residual diagnostics for outliers and influential observations.

Module 8: Categorical Variables & Interaction Effects (Final Module)

• Extend multiple regression to include qualitative (categorical) independent variables alongside quantitative ones.
• Least squares estimation remains unchanged; evaluation uses same $R^2$, $F$-test, $t$-test, residual analysis.
• Adding categorical variables does not always improve the model — just because data exists doesn’t mean it should be included.
• Interaction variables (product of a categorical and a quantitative variable) help model interaction effects.

Final Advice

Exam tip: The course deliberately focuses on application and interpretation over theoretical derivation. For exams, prioritize converting business questions into probability/inference questions and correctly interpreting regression output (coefficients, $R^2$, p-values).

Key takeaways

• Decision-making without data is risky; statistics quantifies that risk.
• Sampling distributions and the central limit theorem are the foundation for inference.
• Confidence intervals and hypothesis tests are dual approaches for drawing conclusions about populations.
• Regression (simple → multiple → with categoricals) is the core tool for modeling relationships.
• Model building is iterative — add variables cautiously and validate assumptions via residual analysis.
• Categorical variables and interactions extend regression’s power without changing fundamental procedures.