Big data science and machine learning organizations record many attributes/properties to avoid losing critical information. Each attribute has its own properties and valid ranges in which it can fall. For example, the speed of a motorcycle can be in the range of 0-200 KM/h, while the speed of a car can be in the range of 0-400 KM/h. Machine learning or deep learning models expect these ranges to be on the same scale in order to determine the importance of these properties without bias.
In this article, we will explore one of the important topics used in scaling different attributes for machine learning: normalization and standardization. There is often confusion among machine learning professionals about which one to use. We will attempt to clear up this confusion in this article.
Let's start with the fundamental question.
In machine learning, a feature is a measurable property or characteristic of an observed phenomenon that is used as input to train a model. For example, when training a model to predict the price of a flat, we might use the size of the flat as a feature. Adding the locality of the flat as an additional feature can improve the model's performance.
Normalization is the process of scaling all features to the same definite range. It is important to normalize the features because the numerical values of different features may vary greatly. If the ranges of the values of different features are significantly different, then some features may dominate the training process, leading to poor model performance. Normalization helps to ensure that all features are given equal weight and have an equal influence on the model.
Let’s go through one example to answer this question, which will open the mathematical angle supporting Normalization or Standardization.
Suppose we make a machine learning model to learn the function Y = θ1*X + θ0. We can use a supervised dataset, which includes input values (X) and corresponding output values (Y), to train a machine learning model. During the training process, the model will start with initial values for certain parameters (such as θ1 and θ0). These initial values may be randomly selected or manually set.
The model will then iteratively adjust these parameters in order to minimize the error between the predicted output values (Y^) and the actual output values (Y). The error is typically measured using a cost function, such as Mean Squared Error (MSE). Our goal is to find values for the parameters that minimize the error and improve the model's performance.
Let’s choose Mean Squared Error (MSE), our error function, also called a cost function. The formulae for MSE are given in the below equation, where n is the number of training samples.
1 MSE = --- * Σ (Y - Ŷ)^2, N (Y - Ŷ)^2 --> The Square of the difference between actual and predicted values
As Y is a function that depends upon two variables, θ1, and θ0, hence cost function will also depend on these two variables. In the GIF below, there is one dimension of the Cost function, and the rest two dimensions can be considered as θ1 and θ0.
At the start, suppose we are at position A (Shown in GIF above), and reaching position B is our ultimate goal as that is the minimum of the cost function. For that, the machine will tweak the values of θ1 and θ0.
But the machine can take infinite values for θ1 and θ0 if it randomly selects them at each step. We use optimizers to help the machine choose the following values of θ1 and θ0 to reach the minima quickly. Let’s choose gradient descent as our optimizer to learn the function Y = θ1*X + θ0. In gradient descent, we update any parameter value using the formulae below.
Let’s say we updated the value of θ1 and θ0 by using the above formulae. Then new values will be:
∂ J(θ0, θ1) θ0 := θ0 - ---------------, θ1 will be treated as constant ∂ θ0 ∂ J(θ0, θ1) θ1 := θ1 - ---------------, θ0 will be treated as constant ∂ θ1
ẟJ(θ0, θ1))/ẟθ1 and
ẟJ(θ0, θ1)/ẟθ0. Prediction error can be represented in the equation as error = (Ŷ — Y)
Cost function :
1 N J(θ0, θ1) = --- * Σ (Yi-Ŷi)^2 N i
Now let’s calculate the partial derivative of this cost function concerning two variables, θ1 and θ0.
After combining the equations and putting everything in the gradient descent formulae,
θ1 = θ1 - α * (2*error*X) θ0 = θ0 - α * (2*error)
The feature value X in the above update formula will affect the step size of the gradient descent. If the features are in different ranges, it will cause different step sizes for every feature. In the image below, X1 and X2 are two attributes that constitute the input variable X, i.e., X = [X1, X2]. Consider this X1 and X2 as two dimensions. To ensure the functionality of the gradient descent moves smoothly towards the minima and steps for gradient descent get updated at the same rate in every dimension, we scale the data before feeding it to the model.
Some machine learning algorithms are susceptible to Normalization or Standardization, and some are insensitive. Algorithms like SVM, K-NN, K-means, Neural Networks, or Deep-learning are susceptible to Normalization/Standardization. These algorithms use the spatial relationships (Space dependent relations) present among the data samples. Let’s take an example,
Students English marks out of 10 Maths marks out of 100 A 5 90 B 7 95 C 9 85 D 10 90 E 8 92 Distance (A , B) = √((5-7)^2 + (90-95)^2) = 5.385 Distance (B , C) = √((7-9)^2 + (95-85)^2) = 10.198
Let’s use the scaling technique and the percentage of marks instead of direct marks.
Students English marks in percent Maths marks in percent A 0.5 0.9 B 0.7 0.95 C 0.9 0.85 D 1.0 0.9 E 0.8 0.92 Distance (A , B) = √((0.5-0.7)^2 + (0.9-0.95)^2) = 0.2061 Distance (B , C) = √((0.7-0.9)^2 + (0.95-0.85)^2) = 0.2236
The scaled distances are closer and can be compared easily.
Algorithms like Decision trees, Random forests, or other tree-based algorithms are insensitive to Normalization or Standardization as they are applied to every feature individually and are not influenced by any other feature.
We know why scaling, so let’s see some popular techniques used to scale all the features in the same range.
This is the most used normalization technique in the machine learning industry. We bring every attribute in the defined range, starting from a and ending at b. We map every feature’s minimum and maximum value to the starting and ending range values. Range values [0, 1] and [-1, 1] are the most popular ones. Formula all three cases are given below:
Xi - mininum(X) Xi = ---------------------------, if the defined range is [0, 1] maximum(X) - mininum(X) Xi - mininum(X) Xi = 2* -------------------------- - 1, if the defined range is [-1, 1] maximum(X) - mininum(X) Xi - mininum(X) Xi = (b-a)* -------------------------- + a, if the defined range is [a, b] maximum(X) - mininum(X)
We generally do not implement these scaling techniques from scratch and use inbuilt functions from Scikit-learn:
from sklearn.preprocessing import MinMaxScaler data = [[-1, 2], [-0.5, 6], [0, 10], [1, 18]] scaler = MinMaxScaler() print(scaler.fit_transform(data)) ''' [[0. 0. ] [0.25 0.25] [0.5 0.5 ] [1. 1. ]] '''
For a custom range, we can use
scaler = MinMaxScaler(feature_range=(a, b)).
Here, we transform the features to start contributing proportionally in the updation step of gradient descent. If we see the formula below, we exponentially raise the value of X.
1 Xi = ----------------- 1 + exp(-Xi)
Standardization is another scaling technique in which we transform the feature such that the changed features will have mean (μ) = 0 and standard deviation (σ) = 1.
The formula to standardize the features in data samples is :
Xi - µ Xi = ----------- Xi
This scaling technique is also known as Z-Score normalization or Z-mean normalization. Unlike Normalization, Standardization techniques are not much affected by the presence of outliers (Think how!).
We generally use the inbuilt function
StandardScaler from scikit-learn to standardize the data like this:
from sklearn.preprocessing import StandardScaler scaler = StandardScaler() data = [[-1, 2], [-0.5, 6], [0, 10], [1, 18]] print(scaler.fit_transform(data)) ''' [[-1.18321596 -1.18321596] [-0.50709255 -0.50709255] [ 0.16903085 0.16903085] [ 1.52127766 1.52127766]] '''
In normalization, we have min and max operations in the formula. What if some outliers have significantly higher/lower magnitudes? They will affect the calculations, and hence normalization is much influenced by the presence of outliers. But if the data samples are large and there are some outliers, then the mean and standard deviation calculations will be affected by a smaller margin. Hence Standardization is less affected by outliers.
Now, we know two different scaling techniques. But sometimes, knowing more or having more options brings another challenge of choice. So we have a new question for us,
Let’s list the use cases where Normalization and Standardization would benefit.
In this article, we discussed the importance of scaling the features in a machine learning model. It is important to scale the features so that they are all on the same scale, as this helps the model to assign equal importance to all features and make predictions without bias. We provided two examples to demonstrate how scaling can improve the performance of a machine learning model. We also mentioned that scaling techniques can be challenging, even for experienced machine learning professionals. We hope you found this article helpful.
Enjoy Learning, Enjoy Algorithms!
Subscribe to get well designed content on data structure and algorithms, machine learning, system design, object orientd programming and math.
©2023 Code Algorithms Pvt. Ltd.
All rights reserved.