This post is part 1 of approaching a machine learning project. Given a problem statement, we will try to see how this can be solved using machine learning and what type of algorithm is best suited to solve it. Often, we tend to jump into solving the problem without understanding the underlying complexities. This increases the chances of inaccurate predictions. This post tries to change that mindset, and helps you to approach the problem from a investigative mindset. So, let’s dive into it right away!!
This post, including the code logic has been taken from the following book, as the objective is to teach and explain whatever I learn from this book :
To solve a problem, below are the important steps that one should keep in mind
- Understand the overall context.
- Gather relevant data.
- Analyze and visualize the data for insights.
- Process the data for machine learning applications.
- Choose a model and train it.
- Optimize your model for performance.
- Share your solution effectively.
- Implement, monitor, and sustain your system.
Understand the overall context
The problem statement here is to create a model that predicts housing prices in California using census data. This dataset includes various metrics for each block group in the state, such as population, median income, and median housing price. Block groups, which typically contain 600 to 3,000 residents, represent the smallest geographical unit for which the US Census Bureau releases sample data. For simplicity, we’ll refer to these as “districts.” The model should analyze this information and accurately predict the median housing price for any district based on the other available metrics.
Framing the problem right:
- The first step is to clarify the business objective; remember, building a model isn’t necessarily the end goal. How does the company plan to utilize and benefit from this model? Understanding this is crucial, as it will influence how you define the problem, select algorithms, determine performance metrics, and allocate time for fine-tuning. You find out, that the output of your model, which predicts the median housing price for a district, will be integrated into another Machine Learning system alongside various other data points. This subsequent system will assess the investment potential of specific areas. Getting this right is vital, as it has a direct impact on revenue.
- Next, inquire about the current solution, if one exists. This can provide a benchmark for performance and valuable insights into the problem at hand. Your boss shares that district housing prices are currently estimated manually by a team of experts. They compile up-to-date information for each district, and when precise median housing prices aren’t available, they resort to complex estimation rules. This process is costly and time-consuming, and the estimates often miss the mark, sometimes by over 20%. This inefficiency is why the company believes training a model to predict median housing prices based on available data could be beneficial. The census data appears to be a valuable resource, offering median housing prices and additional metrics for thousands of districts.
With this context, you can begin designing your system. First, you need to frame the problem:
- Is it supervised, unsupervised, or reinforcement learning?
- Is it a classification task, a regression task, or something else?
- Should you apply batch learning or online learning techniques?
In this case, it’s clearly a supervised learning problem since you have labeled training examples (each instance includes the expected output—the district’s median housing price). Furthermore, it qualifies as a regression task, as you are tasked with predicting a value. More specifically, this is a multiple regression problem because multiple features (such as population and median income) will inform the prediction. It is also a univariate regression problem, given that you are predicting a single value for each district; if multiple values were being predicted, it would be a multivariate regression problem. Lastly, since there isn’t a continuous stream of incoming data that requires rapid adjustments, and the dataset is manageable enough to fit in memory, a standard batch learning approach should suffice.
Selecting a Performance Measure
Root Mean Squared Error (RMSE) is a commonly used metric to evaluate the performance of regression models. It measures the average magnitude of the errors between predicted and actual values, providing insight into how well the model is performing.
The RMSE is calculated using the following steps:
- Calculate the Residuals: For each predicted value [math]\hat{Y}_i[/math] and actual value [math]Y_i[/math], compute the residual (error): [math]e_i = y_i – \hat{y}_i[/math]
- Square the Residuals: Square each of the residuals: [math]\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} e_i^2[/math], where n is the number of observation.
- Square Root of the Mean: Finally, take the square root of the mean squared error: [math]\text{RMSE} = \sqrt{\text{MSE}} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i – \hat{y}_i)^2}[/math]
Root Mean Squared Error (RMSE) gives greater weight to larger errors due to the squaring of the residuals in its calculation. Here’s a breakdown of how this works:
Squaring the Errors:
1. Residuals: For each prediction, the error is calculated as the difference between the actual value [math]Y_i[/math] and the predicted value [math]\hat{Y}_i[/math] (i.e., [math]e_i = y_i – \hat{y}_i[/math]).
2. Squaring: Each residual is squared, which means that larger errors become disproportionately larger:
– For example, if an error is 2, squaring it gives 4, while an error of 10 results in 100. This non-linear transformation emphasizes the impact of larger errors.
Mean Calculation:
3. Mean of Squared Errors: The Mean Squared Error (MSE) is calculated by averaging these squared errors:
[math]\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} e_i^2[/math]
Larger squared errors contribute significantly more to the mean compared to smaller ones.
Square Root:
4. Taking the Square Root: The final RMSE value is obtained by taking the square root of the MSE. While this step brings the RMSE back to the original units of the target variable, the influence of large errors remains due to their earlier squaring.
Summary:
Amplified Impact: Because RMSE squares the errors, larger discrepancies between predicted and actual values have a much larger impact on the final RMSE score. This means that a model with a few significant outliers can have a much higher RMSE than one with smaller, more uniformly distributed errors.
Sensitivity to Outliers: This characteristic makes RMSE particularly sensitive to outliers, making it an important consideration when evaluating model performance.
In practice, if you’re concerned about the influence of large errors or outliers, you might also consider alternative metrics like Mean Absolute Error (MAE), which treats all errors equally regardless of their size.
Mean Squared Error (MAE) :
Mean Absolute Error (MAE) is a commonly used metric for evaluating the accuracy of regression models. It measures the average magnitude of errors in a set of predictions, without considering their direction (i.e., whether they are overestimates or underestimates).
Formula:
The MAE is calculated using the following steps:
Calculate the Absolute Residuals: For each actual value [math]Y_i[/math] and the predicted value [math]\hat{Y}_i[/math], compute the absolute error:
[math]|e_i| = |y_i – \hat{y}_i|[/math]
Mean of the Absolute Errors: Take the average of these absolute errors:
[math]\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i – \hat{y}_i|[/math]
where n is the number of observations.
Key Characteristics:
- Interpretability: MAE is easy to interpret since it is in the same units as the target variable. For example, if predicting prices, an MAE of $10 means the predictions are, on average, $10 off.
- Sensitivity to Outliers: Unlike Root Mean Squared Error (RMSE), MAE treats all errors equally since it does not square the residuals. This makes MAE more robust to outliers, as large errors do not disproportionately influence the overall metric.
- Linear Measurement: Because MAE uses absolute values, it provides a linear score that reflects the average error magnitude without exaggerating the influence of larger errors.
Use Cases:
MAE is particularly useful in situations where:
- You want a straightforward measure of average error.
- You want to minimize the impact of outliers.
- Interpretability is a priority, as MAE provides a clear understanding of model accuracy.
Comparison with Other Metrics:
- MAE vs. RMSE: While RMSE gives more weight to larger errors (sensitive to outliers), MAE treats all errors uniformly, which can be beneficial in specific contexts.
- MAE vs. MSE: Like RMSE, Mean Squared Error (MSE) emphasizes larger errors due to squaring, whereas MAE does not.
Summary:
Mean Absolute Error (MAE) is a valuable metric for assessing the performance of regression models, providing a clear and interpretable measure of average prediction error while being less influenced by outliers compared to metrics like RMSE.
Review the assumptions : It’s essential to identify and verify the assumptions that have been made—either by you or others—throughout the process. This step can help catch significant issues early on. For instance, your model outputs district prices that will feed into a downstream Machine Learning system. We assume these prices will be used directly, but what if the downstream system actually categorizes them into groups like “cheap,” “medium,” or “expensive”? In that case, the precise price accuracy may be irrelevant; the model only needs to assign the correct category. If that’s the situation, the problem should have been framed as a classification task rather than a regression task. You certainly don’t want to discover this after investing months in developing a regression model.
