In database replication, we create copies of the same database across multiple nodes (replicas). However, with multiple replicas, a question arises: How do we ensure that all the data ends up on all the replicas after every write? The most common solution is master-slave replication, also known as active/passive or single-leader replication.
There are two types of nodes in Master-Slave replication: Master and Slave. The single master (leader) node works as the primary database, while one or more slave (follower) nodes maintain copies of the Master’s data.
The Master-Slave replication is used in situations where the workload is read-heavy, and there is a need to distribute the read requests across multiple nodes to improve system performance. Since slave nodes can handle read requests, they can offload read traffic from the Master node and allow it to focus on processing write requests.
In master-slave replication, an essential aspect to consider is whether replication happens synchronously or asynchronously. To understand this, let’s consider what happens when a user updates their profile image on a social media website: the client sends an update request to the leader. As soon as the leader receives the request, it forwards the data change to the followers. Now, the critical question is: How do the followers update their data?
In synchronous replication, the client waits for confirmation from the leader that the update has been applied to all followers before receiving the response. In contrast, asynchronous replication allows the client to receive the response before all followers have been updated.
Disadvantage: If the follower doesn’t respond (due to a crash, network fault, or any other reason), the write cannot be processed.
Advantage: The follower is guaranteed to have an up-to-date copy of the data that is consistent with the leader. If the leader suddenly fails, we can be sure that the data is still available to the follower.
Disadvantage: If the leader fails and is not recoverable, any writes that have not yet been replicated to followers are lost. This means that a write is not guaranteed to be durable, even if it has been confirmed to the client.
Advantage: A leader can continue processing writes, even if all of their followers have fallen behind. Weakening durability may sound like a bad trade-off, but asynchronous replication is nevertheless widely used, especially if there are many followers or if they are geographically distributed.
Overall, the choice of synchronous or asynchronous replication depends on the trade-offs between consistency and performance. Synchronous replication provides stronger consistency guarantees, but it can also result in longer response times. On another side, asynchronous replication provides faster response times but may compromise consistency.
To handle the above situation, one solution is to read from the leader when reading something that the user may have modified. Otherwise, read from a follower. For example, in a social media platform, profile information is usually editable only by the user. So, it’s a good idea to always read a user’s own profile from the leader and any other user’s profile from a follower. Think!
In most cases, replication is very fast, and changes made to the leader database are quickly propagated to the followers. But there are certain situations where replication can become delayed. For example, if a follower is recovering from a failure, the system is operating at maximum capacity, or there are network problems between the nodes.
On the other hand, synchronous replication requires the leader database to block all write operations until the followers have acknowledged that the data has been received and stored. This can cause significant delays and performance issues if all followers are synchronous. If one of the replicas goes offline, the entire system would be affected.
A good solution would be to use semi-synchronous replication. In this setup, one follower is designated as synchronous, while the others are asynchronous. Here synchronous followers will update all data changes in real time and all asynchronous followers will update data eventually in the background. If the synchronous follower goes offline or slows down for any reason, one of the asynchronous followers can be promoted to take its place.
This configuration ensures that at least two nodes, the leader and one synchronous follower, have up-to-date copies of the data at all times. Let’s understand this via an example: Suppose a user updates their profile image on the website and there are one leader and two followers. Here replication to follower 1 is synchronous and replication to follower 2 is asynchronous.
Suppose we want to increase the number of followers or replace a failed follower node. How can we do this? How can we ensure that the new follower has an accurate copy of the leader’s data? Simply copying data from the leader node to the new follower node is not enough to guarantee data consistency because clients are continuously writing to the database. In other words, a standard data copy would see different parts of the database at different points in time.
One solution is to lock the leader or master database to ensure consistency during the copying process. However, this approach would not support the goal of high availability since it would make the leader database unavailable for writes during the copy process.
Fortunately, there is a way to set up a new follower without any downtime. Here are the steps:
The practical steps involved in setting up a follower can vary significantly depending on the DBMS system. In some systems, the process is fully automated, while in others, it may require a multi-step workflow that must be manually performed by an administrator.
Due to faults or errors, any node in a system can go down. So our goal is to keep the system running despite individual node failures and to minimize the impact of a node outage. The critical question is: How can we achieve high availability and reliability with leader-based replication? Let’s discuss this scenario separately in case of follower and leader failure.
Each follower maintains a log of the data changes it has received from the leader. This log helps the follower to identify the last transaction processed before the fault occurred.
So if a follower fails (event of a crash, restart, or temporary network interruption), it can connect to the leader and request all the data changes that occurred during the time when it was disconnected. Once it has applied these changes, it will have caught up to the leader and can resume receiving a stream of data changes as before.
This is a little trickier and requires three critical steps: 1) Detecting the failure of the leader node 2) Promoting one of the followers as a new leader 3) Configuring clients to send their writes to the new leader, and other followers to start consuming data changes from the new leader. This process is also called failover.
Step 1: Detecting leader failure: This can happen due to various reasons (crashes, power outages, network issues, etc). Since there is no foolproof way to detect the cause of failure, a timeout is used to assume that a leader node is dead if it doesn’t respond for a certain period of time, typically less than 30 sec or 1 minute.
Step 2: Choosing a new leader: This can be done through an election process where a new leader is chosen by a majority of the remaining replicas. To minimize data loss, the replica with the most up-to-date data changes from the old leader is usually chosen as the new leader. But getting all the nodes to agree on a new leader is a consensus problem. Note: We will discuss the idea of the consensus problem in a separate blog.
Step 3: Reconfiguring the system to use the new leader: Clients need to send their write requests to the new leader. If the old leader comes back online, it might still believe that it is the leader, unaware that it has been replaced by a new leader. The system needs to ensure that the old leader becomes a follower and recognizes the new leader.
Note: In case of semi-synchronous replication, we make the synchronous slave as a new master since we know that it is the most updated one and no data will be lost.
We will keep updating this blog with more insights on master-slave replication. If you have any queries/doubts/feedback, please write us at email@example.com. Enjoy learning, Enjoy system design, Enjoy algorithms!
Subscribe to get well designed content on data structure and algorithms, machine learning, system design, object orientd programming and math.
©2023 Code Algorithms Pvt. Ltd.
All rights reserved.