๐ Understanding Residual Normality in Regression
In regression analysis, we often make assumptions about the errors (residuals). One common assumption is that the residuals are normally distributed. But when does this assumption really matter, and what happens if it's violated?
๐ Background and Importance
The assumption of normality is tied to the Central Limit Theorem (CLT). The CLT states that the distribution of sample means approaches a normal distribution as the sample size increases, regardless of the shape of the population distribution. In regression, this helps justify the use of normal-based inference (hypothesis tests and confidence intervals) for the regression coefficients.
๐ Key Principles
- ๐งช Large Sample Sizes: If you have a large sample size (generally, $n > 30$ is considered sufficient), the CLT kicks in. Even if the residuals aren't perfectly normal, the sampling distribution of the regression coefficients will be approximately normal. This means your hypothesis tests and confidence intervals are still reasonably reliable.
- ๐ Small Sample Sizes: With small sample sizes, the normality assumption becomes more critical. If the residuals are severely non-normal, the p-values from your hypothesis tests and the coverage probabilities of your confidence intervals may be inaccurate.
- ๐ Focus on Inference: Normality of residuals is primarily important for statistical inference (hypothesis testing, confidence intervals). If your goal is only prediction, non-normal residuals are less of a concern, although they might suggest that your model isn't capturing all the information in the data.
- ๐ Outliers: Non-normality can often be caused by outliers. Identifying and addressing outliers can sometimes resolve the issue of non-normal residuals.
- ๐ก Transformations: If the residuals are non-normal, consider transforming your dependent variable (e.g., using a log transformation) to improve normality.
โ ๏ธ Consequences of Non-Normal Residuals
- ๐ Inaccurate P-values: The p-values associated with your regression coefficients may be incorrect, leading to wrong conclusions about statistical significance.
- ๐ Unreliable Confidence Intervals: Confidence intervals may not have the stated coverage probability (e.g., a 95% confidence interval might not contain the true parameter 95% of the time).
- โ๏ธ Less Efficient Estimates: The ordinary least squares (OLS) estimator, which is commonly used in regression, is the best linear unbiased estimator (BLUE) under the Gauss-Markov assumptions, which include normality. If normality is violated, OLS might not be the most efficient estimator.
๐ ๏ธ Checking for Normality
You can assess the normality of residuals using several methods:
- ๐ Histograms: Plot a histogram of the residuals and visually check if it resembles a normal distribution.
- ๐ Q-Q Plots: Create a Q-Q plot (quantile-quantile plot). If the residuals are normally distributed, the points will fall close to a straight line.
- ๐งช Statistical Tests: Perform statistical tests like the Shapiro-Wilk test or the Kolmogorov-Smirnov test to formally test the null hypothesis that the residuals are normally distributed.
๐ Real-world Examples
Example 1: Small Sample, Non-Normal Residuals
Suppose you're analyzing the relationship between advertising spending and sales for a small business with only 15 months of data. If the residuals from your regression are highly skewed, the p-values for the effect of advertising on sales may be unreliable.
Example 2: Large Sample, Non-Normal Residuals
Imagine you're studying the relationship between income and education level using a dataset of 1,000 individuals. Even if the residuals are slightly non-normal, the large sample size means that the p-values and confidence intervals for the effect of education on income are likely to be reasonably accurate.
๐ Conclusion
The importance of residual normality depends on your goals and the characteristics of your data. For statistical inference with small samples, normality is crucial. With large samples, the Central Limit Theorem provides some robustness against non-normality. Always assess the residuals and consider transformations or alternative modeling approaches if the normality assumption is severely violated.