Method 2: Imputing Missing Data

1. Loading the data

In [2]:
import pandas as pd
import numpy as np
In [3]:
df = pd.read_csv("Banking_Marketing.csv")
In [4]:
df.head()
Out[4]:
age job marital education default housing loan contact month day_of_week ... campaign pdays previous poutcome emp_var_rate cons_price_idx cons_conf_idx euribor3m nr_employed y
0 44.0 blue-collar married basic.4y unknown yes no cellular aug thu ... 1 999 0 nonexistent 1.4 93.444 -36.1 4.963 5228.1 0
1 53.0 technician married unknown no no no cellular nov fri ... 1 999 0 nonexistent -0.1 93.200 -42.0 4.021 5195.8 0
2 28.0 management single university.degree no yes no cellular jun thu ... 3 6 2 success -1.7 94.055 -39.8 0.729 4991.6 1
3 39.0 services married high.school no no no cellular apr fri ... 2 999 0 nonexistent -1.8 93.075 -47.1 1.405 5099.1 0
4 55.0 retired married basic.4y no yes no cellular aug fri ... 1 3 1 success -2.9 92.201 -31.4 0.869 5076.2 1

5 rows × 21 columns

In [5]:
df.isna().sum()
Out[5]:
age               2
job               0
marital           0
education         0
default           0
housing           0
loan              0
contact           6
month             0
day_of_week       0
duration          7
campaign          0
pdays             0
previous          0
poutcome          0
emp_var_rate      0
cons_price_idx    0
cons_conf_idx     0
euribor3m         0
nr_employed       0
y                 0
dtype: int64

2. Calculate the mean of the age column

In [6]:
age_mean = df.age.mean()
In [7]:
print("Mean of age column: ",age_mean)
Mean of age column:  40.023812413525256

3.Impute the missing data in the age column with the mean age value

In [8]:
df.age.fillna(age_mean,inplace=True)
In [9]:
df.isna().sum()
Out[9]:
age               0
job               0
marital           0
education         0
default           0
housing           0
loan              0
contact           6
month             0
day_of_week       0
duration          7
campaign          0
pdays             0
previous          0
poutcome          0
emp_var_rate      0
cons_price_idx    0
cons_conf_idx     0
euribor3m         0
nr_employed       0
y                 0
dtype: int64

4. Checking all the records in the dataset for which the 'duration' column is NA

In [10]:
df[df['duration'].isnull()]
Out[10]:
age job marital education default housing loan contact month day_of_week ... campaign pdays previous poutcome emp_var_rate cons_price_idx cons_conf_idx euribor3m nr_employed y
41185 42.0 admin. single university.degree unknown yes yes telephone may wed ... 3 999 0 nonexistent 1.1 93.994 -36.4 4.857 5191.0 0
41186 48.0 technician married professional.course no no yes telephone oct tue ... 2 999 0 nonexistent -3.4 92.431 -26.9 0.742 5017.5 0
41188 25.0 student single high.school no no no NaN may fri ... 4 999 0 nonexistent 1.1 93.994 -36.4 4.859 5191.0 0
41190 29.0 technician single Basic no yes n NaN may mon ... 1 999 0 nonexistent -1.8 92.893 -46.2 1.299 5099.1 0
41192 39.0 admin. married university.degree no no y cellular nov tue ... 2 999 0 nonexistent -0.1 93.200 -42.0 4.153 5195.8 0
41193 1.0 admin. married high.school no yes y cellular may thu ... 4 999 1 failure -1.8 92.893 -46.2 1.266 5099.1 0
41195 2.0 housemaid married Basic unknown no n NaN may thu ... 2 999 0 nonexistent 1.1 93.994 -36.4 4.860 5191.0 0

7 rows × 21 columns

5. Sort the values of duration in ascending order

In [11]:
df['duration'].sort_values(ascending=False).head()
Out[11]:
7802     4918.0
18610    4199.0
32880    3785.0
1974     3643.0
10633    3631.0
Name: duration, dtype: float64

6. Calculating the median value of the duration column

In [12]:
duration_med= df.duration.median()
In [13]:
print("The median of duration is: ",duration_med)
The median of duration is:  180.0

7. Imputing the median value of duration to all the NA fields in duration

In [14]:
df.duration.fillna(duration_med,inplace=True)
In [15]:
df.isna().sum()
Out[15]:
age               0
job               0
marital           0
education         0
default           0
housing           0
loan              0
contact           6
month             0
day_of_week       0
duration          0
campaign          0
pdays             0
previous          0
poutcome          0
emp_var_rate      0
cons_price_idx    0
cons_conf_idx     0
euribor3m         0
nr_employed       0
y                 0
dtype: int64

8. In the above steps both the 'duration' and 'age' column were numerical so we used mean and median to impute the missing values. However in this case 'contact' is a categorical value, so we will use Mode here to impute the missing values.

In [16]:
df['contact'].unique()
Out[16]:
array(['cellular', 'telephone', nan], dtype=object)
In [17]:
contact_mode = df.contact.mode()[0]
In [18]:
print("Mode for contact: ",contact_mode)
Mode for contact:  cellular

9.Imputing the Mode value of 'contact' to all the NA fields in 'contact'

In [19]:
df.contact.fillna(contact_mode,inplace=True)
In [20]:
df.isna().sum()
Out[20]:
age               0
job               0
marital           0
education         0
default           0
housing           0
loan              0
contact           0
month             0
day_of_week       0
duration          0
campaign          0
pdays             0
previous          0
poutcome          0
emp_var_rate      0
cons_price_idx    0
cons_conf_idx     0
euribor3m         0
nr_employed       0
y                 0
dtype: int64