Finding and removing Outliers in Data

In [1]:
import numpy as np
%matplotlib inline
import seaborn as sns

data = [12,32,45,51,54,59,65,74,77,87,121,139,321]

sns.boxplot(data)
Out[1]:
<matplotlib.axes._subplots.AxesSubplot at 0x1878843fd48>

1. Load the dataset

In [4]:
import numpy as np
import pandas as pd
%matplotlib inline
import seaborn as sns
In [5]:
data = pd.read_csv("german_credit_data.csv")
In [6]:
data.head()
Out[6]:
Unnamed: 0 Age Sex Job Housing Saving accounts Checking account Credit amount Duration Purpose
0 0 67 male 2 own NaN little 1169 6 radio/TV
1 1 22 female 2 own little moderate 5951 48 radio/TV
2 2 49 male 1 own little NaN 2096 12 education
3 3 45 male 2 free little little 7882 42 furniture/equipment
4 4 53 male 2 free little little 4870 24 car

2. Plotting a boxplot on the 'Age' column

In [7]:
sns.boxplot(data['Age'])
Out[7]:
<matplotlib.axes._subplots.AxesSubplot at 0x18789889d48>

As you could se from the above box plot , some data points are outliers for 'Age' column.

3.Calculating the 1st Quartile and the 3rd Quartile of the 'Age' column

In [8]:
Q1 = data['Age'].quantile(0.25)
Q3 = data['Age'].quantile(0.75)
In [9]:
print("The 1st and 2nd quartile value is {0:1f} and {1:1f} respectively".format(Q1,Q3))
The 1st and 2nd quartile value is 27.000000 and 42.000000 respectively

4.Calculating the Inter-quartile range

In [10]:
IQR = Q3 - Q1
In [11]:
print("The value of Inter Quartile Range is: ", IQR)
The value of Inter Quartile Range is:  15.0

5. Finding the Lower Fence and the Upper Fence

In [12]:
Lower_fence = Q1 - (1.5 * IQR)
Upper_fence = Q3 + (1.5 * IQR)
In [13]:
print("Lower Fence value: ",Lower_fence)
Lower Fence value:  4.5
In [14]:
print("The upper Fence value: ",Upper_fence)
The upper Fence value:  64.5

6. Checking the data which have the 'Age' data less than the Lower Fence or greater than the Upper Fence. Basically we are retrieving the outliers here

In [15]:
data[(data['Age']<Lower_fence)|(data['Age']>Upper_fence)]
Out[15]:
Unnamed: 0 Age Sex Job Housing Saving accounts Checking account Credit amount Duration Purpose
0 0 67 male 2 own NaN little 1169 6 radio/TV
75 75 66 male 3 free little little 1526 12 car
137 137 66 male 1 own quite rich moderate 766 12 radio/TV
163 163 70 male 3 free little moderate 7308 10 car
179 179 65 male 2 own little little 571 21 car
186 186 74 female 3 free little moderate 5129 9 car
187 187 68 male 0 free little moderate 1175 16 car
213 213 66 male 3 own little rich 1908 30 business
330 330 75 male 3 free little little 6615 24 car
430 430 74 male 1 own little NaN 3448 5 business
438 438 65 male 0 own little little 3394 42 repairs
536 536 75 female 3 own NaN little 1374 6 car
554 554 67 female 3 own little moderate 1199 9 education
606 606 74 male 3 own little NaN 4526 24 business
624 624 65 male 2 free little little 2600 18 radio/TV
723 723 66 female 1 own quite rich moderate 790 9 radio/TV
756 756 74 male 0 own little rich 1299 6 car
774 774 66 male 0 free quite rich rich 1480 12 car
779 779 67 female 2 own little moderate 3872 18 repairs
807 807 65 male 2 own NaN NaN 930 12 radio/TV
846 846 68 male 2 rent NaN NaN 6761 18 car
883 883 65 female 0 own little NaN 1098 18 radio/TV
917 917 68 male 3 own little little 14896 6 car

7. Checking the data which has 'Age' within the Lower fence and Upper Fence. Here were are negating the outlier data and printing only the potential data which is within the Lower and Upper Fence

In [16]:
data[~((data['Age']<Lower_fence)|(data['Age']>Upper_fence))]
Out[16]:
Unnamed: 0 Age Sex Job Housing Saving accounts Checking account Credit amount Duration Purpose
1 1 22 female 2 own little moderate 5951 48 radio/TV
2 2 49 male 1 own little NaN 2096 12 education
3 3 45 male 2 free little little 7882 42 furniture/equipment
4 4 53 male 2 free little little 4870 24 car
5 5 35 male 1 free NaN NaN 9055 36 education
... ... ... ... ... ... ... ... ... ... ...
995 995 31 female 1 own little NaN 1736 12 furniture/equipment
996 996 40 male 3 own little little 3857 30 car
997 997 38 male 2 own little NaN 804 12 radio/TV
998 998 23 male 2 free little little 1845 45 radio/TV
999 999 27 male 2 own moderate moderate 4576 45 car

977 rows × 10 columns