Examples of Data Cleaning Techniques in Python

📖 Quick Study Guide: Python Data Cleaning Essentials

• 🔢 Missing Values: Common issues include NaN, None, or empty strings. Techniques involve imputation (filling with mean, median, mode, or a constant) or removal (dropping rows/columns with missing data). Pandas functions like .fillna() and .dropna() are key.
• 👯 Duplicate Data: Redundant entries can skew analyses. Identifying and removing duplicates ensures each observation is unique. Use Pandas' .duplicated() to find and .drop_duplicates() to remove.
• 📉 Outliers: Data points significantly different from others. They can distort statistical models. Detection methods include IQR (Interquartile Range) or Z-score. Handling involves removal, transformation, or capping/flooring.
• 📝 Inconsistent Data Formats: Variations in spelling, casing, or date formats (e.g., 'USA', 'U.S.A.', 'usa'; '2023-01-01', '01/01/2023'). Standardization is crucial. String methods (.lower(), .strip(), .replace()) and regex are useful.
• ↔️ Incorrect Data Types: Columns stored as the wrong type (e.g., numbers as strings, dates as objects). Can prevent mathematical operations or proper sorting. Use .astype() or pd.to_datetime()/pd.to_numeric().
• 📏 Data Validation: Ensuring data adheres to predefined rules or constraints (e.g., age cannot be negative, email must contain '@'). Custom functions or conditional filtering can identify invalid entries.
• 🛠️ Standardization/Normalization: Scaling numerical features to a standard range (e.g., 0-1 or mean=0, std=1) often for machine learning models, using techniques like Min-Max Scaling or Z-score normalization.

🧠 Practice Quiz

1. Which Pandas function is primarily used to fill missing (NaN) values in a DataFrame?

   1. .drop_duplicates()
   2. .replace()
   3. .fillna()
   4. .dropna()

2. What is a common technique for identifying outliers in a numerical dataset?

   1. Using .describe() to get mean and median.
   2. Calculating the Interquartile Range (IQR).
   3. Applying .value_counts() to the data.
   4. Converting data types with .astype().

3. You have a column 'Price' stored as strings (e.g., '$100.50'). Which method would you use to convert it to a numeric type in Python Pandas?

   1. df['Price'].str.replace('$', '').astype(float)
   2. df['Price'].fillna(0).to_numeric()
   3. df['Price'].drop_duplicates().astype(int)
   4. df['Price'].lower().astype(float)

4. Which of the following is NOT a direct method for handling duplicate rows in a Pandas DataFrame?

   1. df.drop_duplicates()
   2. df.duplicated()
   3. df.replace()
   4. df[~df.duplicated()]

5. If a column contains values like 'USA', 'U.S.A.', and 'united states', which data cleaning technique is most relevant to standardize these entries?

   1. Outlier detection
   2. Missing value imputation
   3. Inconsistent data formatting
   4. Data type conversion

6. What is the primary purpose of using pd.to_datetime() in data cleaning?

   1. To convert numerical data into string format.
   2. To convert object or string columns into datetime objects for proper time-series analysis.
   3. To fill missing date values with the current date.
   4. To remove duplicate date entries from a DataFrame.

7. When is Z-score normalization typically applied as a data cleaning or preprocessing step?

   1. To remove rows with missing values.
   2. To convert categorical features into numerical ones.
   3. To scale numerical features so they have a mean of 0 and a standard deviation of 1, often for machine learning algorithms.
   4. To identify and delete duplicate entries in a dataset.