๐ Definition of AIC and BIC
AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) are criteria used for model selection among a finite set of models. They estimate the quality of each model, relative to each of the other models, by balancing the goodness of fit with the complexity of the model. Lower values of AIC or BIC generally indicate a preferred model.
๐ History and Background
AIC was developed by Hirotugu Akaike in the 1970s, providing a method for model selection based on information theory. BIC, also known as the Schwarz criterion, was introduced by Gideon Schwarz in 1978 and is derived from Bayesian probability, aiming to estimate the Bayesian posterior probability of a model being true.
๐ Key Principles
- ๐งฎ Likelihood: Both AIC and BIC incorporate the maximum likelihood estimate of the model. This reflects how well the model fits the data.
- โ๏ธ Model Complexity (Number of Parameters): AIC and BIC penalize models with more parameters. This helps to prevent overfitting.
- โ๏ธ Trade-off: They balance the trade-off between goodness of fit and model complexity, aiming to select the model that generalizes best to unseen data.
๐ Formula for AIC
The formula for AIC is:
$AIC = 2k - 2ln(L)$
- ๐ k: The number of parameters in the model.
- ๐ L: The maximum value of the likelihood function for the model.
โ Formula for BIC
The formula for BIC is:
$BIC = ln(n)k - 2ln(L)$
- ๐ k: The number of parameters in the model.
- ๐ L: The maximum value of the likelihood function for the model.
- ๐ข n: The number of observations.
๐ AIC vs BIC: Key Differences
- ๐ง Sample Size Dependency: BIC has a stronger penalty for model complexity than AIC, especially with larger sample sizes. This means BIC tends to favor simpler models more often than AIC.
- ๐ฏ Consistency: BIC is consistent, meaning that as the sample size increases, it will select the true model (assuming the true model is in the candidate set). AIC is not consistent.
- ๐ก Use Cases: AIC is often preferred when the goal is prediction accuracy, while BIC is preferred when the goal is to identify the true model structure.
๐ Real-world Examples
Example 1: Stock Price Prediction
Suppose you are comparing two ARMA models for predicting stock prices. Model A has 3 parameters and a log-likelihood of -100. Model B has 5 parameters and a log-likelihood of -90. With 100 data points, we can calculate the AIC and BIC for each model:
For Model A:
- ๐งช $AIC = 2(3) - 2(-100) = 206$
- ๐ฌ $BIC = ln(100)(3) - 2(-100) = 3(4.605) + 200 = 213.815$
For Model B:
- ๐งช $AIC = 2(5) - 2(-90) = 190$
- ๐ฌ $BIC = ln(100)(5) - 2(-90) = 5(4.605) + 180 = 203.025$
Based on AIC, Model B is preferred as it has a lower AIC value. However, based on BIC, Model A is preferred because it has a lower BIC value.
Example 2: Sales Forecasting
A retail company wants to forecast monthly sales using time series models. They compare an ARIMA(1,1,1) model and an ARIMA(2,1,2) model. The ARIMA(1,1,1) model has a lower AIC, suggesting it provides a better balance between fit and complexity, and is chosen for forecasting.
๐ก Conclusion
AIC and BIC are valuable tools for model selection in time series analysis. They provide a quantitative way to balance model fit and complexity. Understanding their formulas, differences, and real-world applications is crucial for effective model building and forecasting. Choosing between AIC and BIC depends on the specific goals of the analysis and the characteristics of the data.