Linear Regression

What Is Linear Regression?

Linear regression models the relationship between a continuous outcome variable (Y) and one or more predictor variables (X). It allows you to predict Y from X and quantify the relationship while controlling for other variables.

• ●β₀: Intercept — the predicted value of Y when all Xs are zero
• ●β₁, β₂, ...: Regression coefficients — change in Y for a one-unit increase in X, holding other variables constant
• ●ε: Residual error — the part of Y not explained by the model

Simple vs. Multiple Regression

• ●Simple linear regression: One predictor (X) predicts Y → Y = β₀ + β₁X
• ●Multiple regression: Two or more predictors → Y = β₀ + β₁X₁ + β₂X₂ + ...
• ●Multiple regression allows you to statistically control for confounders — this is its key advantage over simple correlation or t-tests.
• ●Example: Predicting HbA1c from age, BMI, and treatment group simultaneously

Interpreting Regression Coefficients

Each coefficient β represents the expected change in Y for a one-unit increase in that predictor, holding all other predictors constant.

• ●Continuous predictor: β = 0.8 means "each 1-unit increase in X is associated with a 0.8-unit increase in Y, controlling for other variables"
• ●Binary predictor (0/1): β = -1.2 means "category 1 has a predicted Y that is 1.2 units lower than category 0"
• ●Standardized β (beta): Coefficient expressed in standard deviations. Allows comparison of effect sizes across predictors with different units.
• ●95% CI for each β: If it excludes 0, the predictor is statistically significant at p < 0.05

R² and Model Fit

R² (R-squared, coefficient of determination) measures the proportion of variance in Y explained by the model. Adjusted R² penalizes for adding irrelevant predictors.

• ●R² = 0.60 means 60% of the variability in Y is explained by the model
• ●Adjusted R² is always ≤ R², and decreases when you add predictors that do not improve the model
• ●In behavioral/social sciences: R² = 0.1 (small), 0.3 (medium), 0.5 (large)
• ●In clinical prediction: aim for R² > 0.5–0.7 for a useful model

Assumptions of Linear Regression

1. 1.Linearity: The relationship between each predictor and Y must be linear. Check with partial regression plots.
2. 2.Normality of residuals: Residuals (Y - Ŷ) should be approximately normally distributed. Check Q-Q plot of residuals.
3. 3.Homoscedasticity: Variance of residuals should be constant across fitted values. Check residuals vs. fitted plot — look for a random cloud, not a funnel shape.
4. 4.Independence: Observations should be independent (no clustering, autocorrelation, or repeated measures without appropriate modeling).
5. 5.No multicollinearity: Predictors should not be highly correlated with each other. Check VIF (Variance Inflation Factor) — VIF > 10 is concerning.

Model Building Strategies

• ●Theory-driven (preferred): Select predictors based on prior knowledge, not data-driven selection
• ●Enter method: Put all predictors in simultaneously — most common in confirmatory research
• ●Stepwise/Backward: Let the algorithm select variables — prone to overfitting, not recommended for confirmatory studies
• ●LASSO/Ridge regression: Regularized regression for high-dimensional data (many predictors relative to sample size)
• ●Parsimony: Prefer simpler models that achieve similar R². Overfitting occurs when your model captures noise, not signal.