Different Types of Machine Learning Algorithms

Machine Learning involves finding connections or correlations between input and output data. If you were to ask about the various types of machine learning algorithms, you would receive a list of answers including Classification and Regression, Supervised and Unsupervised, Probabilistic and Non-probabilistic and many more. The critical question is: Why is machine learning classified into so many categories? To understand this, we need to understand the basic machine learning pipeline to solve a given problem.

The above image shows that five key components play a critical role in the pipeline, and the type of machine learning algorithm depends on the nature of these five components. So we can classify machine learning algorithms based on these five bases:

1. Based on the nature of input data
2. Based on the nature of problem statement
3. Based on the nature of algorithm
4. Based on the nature of solution
5. Based on the nature of output data

Let's understand each category in detail.

Types of machine learning models based on the nature of input data

Machine learning problems can be divided into four categories based on the type of input data used to train the algorithms.

Supervised Learning

Supervised Learning involves having input variables (X) and an output variable (Y). We use a machine learning algorithm to learn the relationship between the input and output. It is referred to as Supervised Learning because the training data acts like a teacher who guides the learning process. The figure below demonstrates this, where we provide explicit information in the form of labels for the input data.

Some examples of supervised learning algorithms are Linear Regression, Logistic Regression, Decision Trees, Random Forest, Support Vector Machines (SVM), Naive Bayes Classifier, k-Nearest Neighbors (k-NN), Neural Networks (MLP, CNN, RNN), etc.

Unsupervised Learning

In Unsupervised Learning, only input data (X) is available without corresponding output variables. No explicit output data is provided to guide the learning process, so the machine must find its own way to learn the mapping function. This is achieved by creating a pseudo output, such as similarities between data samples. Here machine tries to identify the underlying characteristics present in the input data samples as its output.

Unsupervised Learning approach is mainly used to dive deeper into data analysis. If we compare the above image with the image below, we don't have any information about the shape type of the input data. But our model can still segregate them based on their shape and size. This segregation, done by machines, can be termed Unsupervised Learning.

Some examples of unsupervised learning algorithms are K-Means Clustering, Hierarchical Clustering, Density-Based Clustering (DBScan), Apriori algorithm for association rule learning, Non-Negative Matrix Factorization (NMF), Principal Component Analysis (PCA).

Semi-Supervised Learning

Semi-Supervised Learning involves problems where a large amount of input data (X) is available, but only a portion of it is labelled (Y). For example, a dataset of images where some are labelled and some are not.

In the previous example, if the input data has explicit labels for the circle and triangle, but not for the rectangle and hexagon, this scenario falls under the category of Semi-Supervised Learning. Most of the real-world data lie in this category, as labelling data is time-consuming and requires expert human resources.

Examples of Semi-Supervised Learning algorithms are Self-training, Co-training, Multi-view learning, Transductive SVM, and Graph-based semi-supervised learning.

Reinforcement Learning (RL)

In Supervised Learning, we provide the learning algorithm with input-output pairs to train the model. In contrast, Reinforcement Learning involves training a machine agent to perform actions within a virtual environment to achieve a certain goal. For example, if we train a machine to play chess, the chessboard is the environment, and the machine is the agent.

The possible actions that the agent can take at any given stage are determined by the movements that the King can make. The agent selects the best action based on the environmental state and receives rewards or risks based on its choice. The goal of the algorithm is to maximize the rewards and minimize the risks by making the best possible moves. Through repeated interactions with the environment, the agent learns to improve its performance.

Some other examples of Reinforcement Learning are:

• Training autonomous robots to navigate and perform tasks
• Optimizing advertisement placement and recommendation systems
• Controlling autonomous vehicles, such as self-driving cars
• Optimizing energy consumption in smart homes.

Types of machine learning models based on the nature of problem

Machine learning problems can be classified into three different categories based on the type of problem being solved.

Classification Problem

Classification is a type of machine learning problem that involves assigning class labels to examples in a given problem domain. For example, classifying images into two categories, such as "Dog" and "Not a Dog", is a simple example of classification. The goal is to teach algorithms to accurately assign the appropriate class label based on the input data. Some other examples of classification problems are Spam email classification and Fraud detection in a financial transaction.

Regression Problem

Regression is a type of problem in machine learning that involves predicting continuous variables. A simple example of this would be predicting the temperature of a city, where the temperature can have any numerical value ranging from -50 to 50 degrees Celsius. Some other examples of Regression Problems are:

• Predicting housing prices based on location, square footage, and other factors.
• Forecasting stock prices based on past performance and economic indicators.
• Estimating the relationship between car performance and its engine specifications.
• Determining the relationship between the weather and crop yields.

Clustering Problem

Clustering is a problem that involves using machine learning algorithms to divide a set of data samples into a specific number of groups. A straightforward example of this would be grouping lemons based on their size. It's important to note that while clustering resembles classification, the key difference between these two algorithms is that classification is a form of supervised learning, while clustering is unsupervised learning.

Some examples of clustering problems are:

• Cluster customers into distinct groups based on the similarities of customer data.
• Divide the image into multiple segments, each corresponding to a different object or region in the image.
• Divide clusters of normal behaviour and data points that do not fit into any of the clusters (anomalies).
• Cluster stocks into groups based on their behaviour and performance.

Types of machine learning models based on the nature of algorithm

Machine learning can be categorized into three types based on the nature of the algorithm used in the machine learning process.

Classical Machine Learning

Algorithms that use statistical and mathematical equations to analyze the relationships between input and output data fall under the category of Statistical Machine Learning algorithms. These algorithms have the advantage of being explainable i.e. they can provide reasons for the predictions made for a given input.

Some examples of Statistical Machine Learning algorithms include K-means, Decision Trees, Random Forest, Support Vector Machine (SVM), and Linear Regression.

Neural Networks

Neural Network algorithms are inspired by the structure and function of the human brain. These algorithms use a complex mathematical model with many trainable parameters, such as weight and bias matrices, which are learned through the use of training data. Neural networks seem quite promising but they have limitations when the complexity of the model increases. This can make it difficult to capture complex dependencies like temporal or spatial dependencies.

Temporal dependencies refer to the dependence of the input sample at one-time step (t1) on the input sample at another time step (t2), while spatial dependencies refer to the dependence of input collected at one condition on input collected at different conditions.

Deep Learning

The basic principle of deep learning is similar to that of neural networks, but with improved architecture for the placement of hidden layers to overcome the limitations of neural networks. Deep learning algorithms have the ability to learn spatial and temporal relationships within training data. However, the main disadvantage of these algorithms is that they are not easily explainable.

Some examples of deep learning algorithms include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memories (LSTMs).

Types of machine learning models based on the nature of solution

We can classify machine learning algorithms into two categories based on the solution.

Naturally, ML algorithms are designed to learn the historical input data, make inferences from that historical data, and predict the output for future inputs. To predict the output, the model can take two approaches:

Parametric Models

Parametric models are models that consider only future inputs to make predictions about the output. These models rely on patterns observed in the training data and assume that the same patterns will apply to new, unseen data. Examples of parametric models include linear regression and neural networks.

In parametric models, there is a fixed number of parameters that do not change based on the number of input samples. For example, when learning a straight-line function, the parameters that need to be learned are the "slope" and "intercept," which are always two in number. The values of these parameters may change, but the number of parameters remains constant.

Non-Parametric Models

In non-parametric models, the prediction of output depends on both the input features and the previous outputs predicted by the model. These models derive the predicted output value from output values in similar scenarios found in the training data. Examples of non-parametric algorithms include K-Nearest Neighbor and Decision Trees.

It is a common misconception that non-parametric models have no parameters. However, this is not the case. Non-parametric models do have parameters, but the number of parameters depends on the number of input samples. If the number of samples increases or decreases, the number of parameters that need to be learned will also change.

Types of machine learning models based on the nature of output data

Based on the nature of the output, machine learning can be classified into two categories.

Probabilistic Models

Probabilistic models produce output in the form of probabilities, indicating confidence of the prediction. For example, in a classification problem, algorithms predict a label with a certain degree of confidence. For example, if a model looks at an image and reports that there is a 60% chance that a dog is present in the image. CNNs are one example of a probabilistic model.

In probabilistic models, we establish decision boundaries. If the model's confidence is greater than a certain threshold, such as 60%, then we consider that the model has predicted that class, otherwise we say that the model did not predict that class. The value of this threshold can be adjusted according to the user's needs.

Non-Probabilistic Models

This category of models predicts the output but does not provide a measure to know prediction quality. However, there are external methods to determine the error between the predicted and actual values. Decision Trees and Support Vector Machines (SVM) are examples of models that fall under this category.

In regression problems, the model directly provides the desired output in continuous form, but it does not indicate its level of confidence in the prediction. For example, if the model predicts that tomorrow's temperature will be 24.7 degrees Celsius, it does not specify how confident it is in that prediction.

Critical questions to explore

• How can the same model be classified as both a supervised model and a classification model?
• What distinguishes supervised learning from unsupervised learning?
• Can you explain the difference between classification and clustering?
• How does Reinforcement Learning differ from supervised, unsupervised, and semi-supervised learning?
• How are Deep Learning and Neural Networks different?

Conclusion

Having a clear understanding of the different types of machine learning algorithms is crucial for the success of a machine learning project. Each algorithm has its own strengths, weaknesses, and areas of applicability. Understanding these differences helps to select the most appropriate algorithm for a given problem and avoid common mistakes.

References: Machine Intelligence by Suresh Samudrala

Enjoy Machine Learning, Enjoy Algorithms!