About the Dataset¶
The data was collected and made available by “National Institute of Diabetes and Digestive and Kidney Diseases” as part of the Pima Indians Diabetes Database. Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here belong to the Pima Indian heritage (subgroup of Native Americans), and are females of ages 21 and above.
The columns of the dataset are as follows:
- Pregnencies
- Glucose
- BloodPressure
- Skin Thickness
- Insulin
- BMI
- DiabetesPedigreeFunction
- Age
- Outcome
You can download the dataset from here: https://www.kaggle.com/kandij/diabetes-dataset
In this exercise, we will do hyper parameter tuning using Ridge and Lasso Regression. Before performing Ridge regression, we will perform Linear Regression to see the mean squared error which will give us an idea about the best fit line.
In this case the Outcome column will be out dependent variable.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
df = pd.read_csv("diabetes2.csv")
df.head()
df.shape
Shuffling the dataset to avoid any ordering issues¶
from sklearn.utils import shuffle
df_shuffle = shuffle(df)
df_shuffle.head()
Selecting the Dependent feature as Outcome and the rest as the independent feature¶
DV = 'Outcome'
y = df_shuffle[DV]
X = df_shuffle.drop(DV, axis = 1)
X.head()
y.head()
Linear Regression¶
Our goal is to calculate the difference between the actual dependent feature(y) and the predicted feature(ŷ). To calculate this we are using the cross_val_score and the parameter scoring='neg_mean_squared_error' will give us the difference for that.
We generally use a scorer object with the scoring parameter in the cross_val_score function. All scorer objects follow the convention that higher return values are better than lower return values.
- Thus metrics which measure the distance between the model and the data, like metrics.mean_squared_error, are available as neg_mean_squared_error which return the negated value of the metric.
In the below figure you could see the difference between the actual dependent feature(y) and the predicted feature(ŷ) is shown in the red arrow
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import GridSearchCV, cross_val_score
model = LinearRegression()
negMSE = cross_val_score(model,X,y, scoring='neg_mean_squared_error',cv=5)
mean_negMSE = np.mean(negMSE)
print("Negative Mean squared Error: ",mean_negMSE)
The above value for the Mean squared error looks pretty close to zero, which means our Linear Regression model has performed well. However let's if we could make our model better by hyper tuning it using Ridge Regression.
Ridge Regression¶
Here in Ridege Regression we have taken the alpha(λ-parameter) value as a bunch of possible random values. We are doing GridSearch to find the best parameters which will work for the Ridge Regressor.
Here also the value for the scoring parameter will be neg_mean_squared_error which measures the distance between the model and the data.
The formula for Ridge Regression is:
from sklearn.linear_model import Ridge
RidgeRegression = Ridge()
hyperParameters = {'alpha':[1e-15,1e-10,1e-8,1e-3,1e-2,1,5,10,20,30,35,40,45,50,55,100]}
ridgeRegressor = GridSearchCV(RidgeRegression, hyperParameters, scoring='neg_mean_squared_error', cv=5)
ridgeRegressor.fit(X,y)
print("Best value for lambda : ",ridgeRegressor.best_params_)
print("Best score for cost function: ", ridgeRegressor.best_score_)
Lasso Regression¶
For Lasso Regressor, we follow the same process as Ridge Regressor, the only difference is we are taking the magnitude of slope here instead of square of slope as in case of Ridge Regressor.
The formula for Lasso Regression is :
from sklearn.linear_model import Lasso
from sklearn.model_selection import GridSearchCV
LassoRegression = Lasso()
hyperParameters = {'alpha':[1e-15,1e-10,1e-8,1e-3,1e-2,1,5,10,20,30,35,40,45,50,55,100]}
LassoRegressor = GridSearchCV(LassoRegression, hyperParameters, scoring='neg_mean_squared_error', cv=5)
LassoRegressor.fit(X,y)
print("Best value for lambda : ",LassoRegressor.best_params_)
print("Best score for cost function: ", LassoRegressor.best_score_)
There is not a huge difference between the scores of Ridge and lasso in this case. But there are cases where there is significant difference between these 2.¶
We will split the dependent(y) features and the independent(X) features and then pass the test independent set to predict the dependent value¶
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.33, random_state = 10)
Predicting the y_test values by passing the test independent features to both ridge and lasso regressor to see which one performs better.
predict_ridge = ridgeRegressor.predict(X_test)
predict_lasso = LassoRegressor.predict(X_test)
We will plot a distance plot to show the probability distribution function for the difference between the actual(y) dependent feature and the predicted(ŷ) dependent feature through ridge regressor¶
import seaborn as sns
sns.distplot(y_test-predict_ridge)
We will plot a distance plot to show the probability distribution function for the difference between the actual(y) dependent feature and the predicted(ŷ) dependent feature through Lasso regressor¶
sns.distplot(y_test-predict_lasso)
