Difference between imputing with mean, median, and mode using fillna()

📚 Understanding Imputation with fillna()

When working with datasets, you'll often encounter missing values, represented as NaN (Not a Number). The fillna() method in pandas helps you replace these missing values with calculated values. Let's explore how to use it with mean, median, and mode.

📊 Imputing with Mean

The mean is the average of all values in a column. Using the mean to impute missing values involves replacing each NaN with the calculated average of the existing values in that column.

➕
• 🔢 Definition: The mean is calculated as the sum of all values divided by the number of values. Mathematically, it's represented as: $ \text{Mean} = \frac{\sum_{i=1}^{n} x_i}{n} $
➕
• 💻 Usage: df['column_name'].fillna(df['column_name'].mean(), inplace=True)
➕
• 💡 Pros: Simple and quick to calculate.
➕
• ⚠️ Cons: Sensitive to outliers, which can skew the mean and lead to biased imputation.

📈 Imputing with Median

The median is the middle value in a sorted list of numbers. If there are an even number of values, the median is the average of the two middle values. Using the median to impute missing values involves replacing each NaN with the calculated median of the existing values in that column.

➕
• 🔢 Definition: The median is the central value separating the higher half from the lower half of a data sample.
➕
• 💻 Usage: df['column_name'].fillna(df['column_name'].median(), inplace=True)
➕
• 🛡️ Pros: More robust to outliers than the mean.
➕
• ⚠️ Cons: May not accurately represent the central tendency if the data distribution is highly skewed or multimodal.

🧮 Imputing with Mode

The mode is the value that appears most frequently in a dataset. A dataset can have one mode (unimodal), multiple modes (multimodal), or no mode (if all values appear only once). Using the mode to impute missing values involves replacing each NaN with the most frequent value in that column.

➕
• 🔢 Definition: The mode is the value that appears most often in a dataset.
➕
• 💻 Usage: df['column_name'].fillna(df['column_name'].mode()[0], inplace=True) (Note: .mode() returns a Series, so we take the first element [0])
➕
• 💡 Pros: Useful for categorical data or discrete numerical data.
➕
• ⚠️ Cons: Not suitable for continuous numerical data with a relatively uniform distribution, as it might introduce bias.

Comparison Table of Mean, Median, and Mode Imputation

Feature	Mean	Median	Mode
Definition	Average of all values	Middle value in a sorted list	Most frequent value
Calculation	Sum of values / Number of values	Requires sorting; average of two middle values if even number of data points	Count occurrences of each value
Sensitivity to Outliers	Highly sensitive	Robust	Not applicable
Best Use Case	Normally distributed data with few outliers	Skewed data or data with outliers	Categorical or discrete data
Potential Bias	Can be biased by outliers	Less biased than mean, but can still introduce bias in skewed data	Can introduce significant bias if the mode is not representative

🔑 Key Takeaways

➕ 💡
• Mean: Use for normally distributed data without significant outliers.
➕ 🛡️
• Median: Preferred for data with outliers or skewed distributions.
➕ 🧮
• Mode: Best suited for categorical or discrete data.