{"data":{"allMarkdownRemark":{"nodes":[{"id":"fed4d2ea-4911-500d-9ac2-120b6e48434d","frontmatter":{"title":"Introduction to Databases for Data Science","path":"/blog/introduction-to-database-for-data-science"},"html":"<p>With the increase in data generation, data has become an essential source of information for organizations. Companies are collecting as much data as possible in order to not miss any valuable information, which has pushed the limits of storage devices. This requires a more reliable and efficient method of storing the data.</p>\n<p>This is where databases come into play, providing a way to store, manage, and scale large amounts of data. This data is then used for analysis and to gain meaningful insights. In this blog, we will dive into the concept of databases, and explore how data science relies on them and their advantages.</p>\n<h3>Key takeaway from this blog</h3>\n<ul>\n<li>What is a database?</li>\n<li>Why do we need a database in data science?</li>\n<li>A case study on the evolution of Supermarkets.</li>\n<li>Benefits of using a database.</li>\n</ul>\n<h3>What is a Database?</h3>\n<p>A database is an organized way to store and manage data in various forms for easier access. You can think of a database as a large computer that stores data according to a pre-defined format or data model. The data model outlines the structure of the database and the rules for organizing and maintaining it.</p>\n<p>In order to perform actions on the data stored in a database, we need a Database Management System (DBMS). DBMS is the interface between the database and the end user, providing a platform to create, modify, and retrieve data. There are many different database management systems available, depending on the type of database being used.</p>\n<p>For example, relational databases, which store data in tables, can be managed by database management systems such as MySQL, Oracle, and IBM Db2. These are the most commonly used databases in Data Science, as much of the data we work with is presented in a tabular format. Non-relational databases, which store data in forms such as key-value pairs, column families, or graphs, can be managed by database management systems like MongoDB and Cassandra.</p>\n<p><img src=\"https://ucarecdn.com/a67bad42-82d2-46ad-b670-bd6ff753e27c/\" alt=\"What is difference between relational and non-relational databases?\" title=\"Databases for Data Science\"></p>\n<p>We will see the types of databases and how to select the best suited for our data science projects in the next blog. But first, let’s go through the benefits and needs of databases in data science.</p>\n<h3>Why do we need a database in data science?</h3>\n<p>A research conducted in 2020 found that an incredible three quintillion bytes of data are generated every day, leading to the creation of 65 zettabytes of data globally. This figure is expected to reach 181 zettabytes by 2025.</p>\n<p>Until the 1960s, we relied on spreadsheets such as Excel sheets to store and manage our data. However, without a structured method of storing data, it would be of little use. This is why databases were introduced to manage and store large amounts of data. </p>\n<p>There are two key reasons why databases have become so popular in recent years:</p>\n<ul>\n<li>The rapid increase in data generation</li>\n<li>The dependence of data science on data</li>\n</ul>\n<p>To better understand the importance of databases in our daily lives, let's take an example of supermarkets evolution.</p>\n<h4>Case study for the use case of Database</h4>\n<p>Earlier, shopping was not a very enjoyable experience as shops often lacked adequate space and customers had to wait or rely on the shopkeeper to find the items they were looking for. The introduction of supermarkets, however, changed that, as they made shopping much more pleasant by displaying all the products in a large space and making them easily accessible to customers. As the number of products and customers in supermarkets increased, the need for a database system to keep track of all the purchases became critical.</p>\n<p>After the rise of malls across the country, the 2008 recession hit, putting many of them on the brink of closing down. However, those that managed to survive realized the importance of incorporating new technologies like AI, Machine Learning, and Data Science in order to implement more structured and organized management strategies.</p>\n<p>Let's take some example:</p>\n<ul>\n<li>Data Science plays a crucial role in determining the products that should be placed in which section of the supermarket, the shelf life of each product, and the required quantity of products during different seasons. </li>\n<li>By analyzing sales data from different supermarket branches, supermarket owner can identify which products need to be stocked in larger quantities and during which months the sales are highest.</li>\n<li>An analysis of the sales data based on the types of products in the store revealed that fashion accessories, food and beverages, and home and lifestyle products were the most popular. Having a well-stocked inventory of these items is crucial for maintaining customer traffic.</li>\n<li>To determine the months with the heaviest customer traffic, a graph was plotted between the month and gross income. The analysis showed that the sales are highest in January and February, due to New Year offers, and follow a rising and falling trend every 2-3 months. This is because most customers tend to make bulk purchases every 3 months. Based on these insights, the supermarket owner could be advised to import goods during these high-traffic months.</li>\n</ul>\n<p><strong>Reference:</strong> <a href=\"https://www.kaggle.com/datasets/aungpyaeap/supermarket-sales\">https://www.kaggle.com/datasets/aungpyaeap/supermarket-sales</a></p>\n<p>Data science is extensively used to make these kinds of analyses and help businesses to grow. The success of these analyses depends on the data. And most importantly, this data should be easy to access, fast to retrieve, and structured. Database fulfils all these requirements.</p>\n<h3>Benefits of using a database in data science</h3>\n<p>Apart from providing storage space for data, databases have far more benefits.</p>\n<ul>\n<li>Reliability</li>\n<li>Multiple user access</li>\n<li>Remove data redundancy</li>\n<li>Data Scalability</li>\n</ul>\n<h4>Databases are Reliable</h4>\n<p>In today's world, where consumer policies dictate that financial fraud and identity theft can result from the misuse or inadequate protection of personal information, the use of databases is crucial for safeguarding user data. In other words, databases are a trustworthy source for storing data because they are designed to follow a predefined data model or schema that adheres to laws for collecting and protecting data. </p>\n<p>This ensures the safety of personal information and reduces the chances of collecting irrelevant or unnecessary data. Additionally, databases provide the ability to restore data if it is accidentally deleted or erased, ensuring the reliability of the data stored. ﻿</p>\n<p>Let’s take an example of an Employee-project database to get an idea of data schemas.</p>\n<p><img src=\"https://ucarecdn.com/8ccd2c02-4783-432f-b0d4-4841e9bc518d/\" alt=\"Employee project mapping example to show the use of database\" title=\"Example of Database in Data Science\"></p>\n<p>Schema is the logical representation of data storage in the database. It defines the structure of the database, including tables, relationships between tables, data type descriptions for columns, primary keys, and foreign keys. The data type defines what type of data can be stored in a particular column, such as an integer, string, or date/time. The primary key is a unique identifier for each record in a table, and the foreign key is a column that connects two tables by referencing the primary key in another table.</p>\n<p>For example, in the schema illustrated above, there are three tables: Employee, Department, and Project. Each table is identified by its own primary key and the data type of each column is specified.</p>\n<h4>Databases can handle multiple users</h4>\n<p>Now, after creating the schema for the data, the next challenge is to handle multiple access to the database. A stock exchange company is a good example. The company keeps track of the daily changes in the price of stocks. They decided to create a model to predict the next day's low and high stock prices. Different teams were assigned to create these models, and the one with the highest accuracy would be deployed. </p>\n<p>Now dynamic nature of stock prices, which change daily, presented a challenge. To make the models, each team needed access to the database to get the latest updates. Instead of assigning one employee to process these requests, each team was given access to the database using their unique access keys. </p>\n<p>This is an example of how databases can support multiple users.</p>\n<h4>Databases Reduce Data Redundancy</h4>\n<p>Data redundancy refers to the repetition of data, which is not efficient for businesses as it wastes storage space and weakens machine learning models. Companies often collect data through sensors or scraping tools, but these tools can sometimes cause data to be redundant or irrelevant. </p>\n<p>For example, consider a car equipped with a camera and sensors for the accelerator and brakes. If the accelerator sensor stops functioning and no longer records data when the accelerator is pressed, but the camera is still recording, the database can detect this issue and not permit the entry of a null value. It will also generate an alert.</p>\n<p><img src=\"https://ucarecdn.com/243397bf-ffca-419e-9db1-c01617d41fd6/\" alt=\"How do databases handle the data redundancy problem?\" title=\"Data redundancy in databases\"></p>\n<p>In some cases, companies introduce data redundancy intentionally, mostly when the shared data size is small. Hadoop is famously used to manage copies of data so that various teams can work on it separately. </p>\n<h4>Data Scalability</h4>\n<p>So far, databases have helped to solve all the major problems of data redundancy, multiple users, and data reliability. However, as the organization grows, data also grows, and soon we reach our storage limit. To overcome this challenge, databases have the property of scalability, which means they can increase or decrease their capacity to handle changes in user traffic.</p>\n<p>For example, popular social media platforms like Instagram, Facebook, and Twitter are experiencing an increasing number of users, resulting in the following issues:</p>\n<ol>\n<li>An increase in user requests to the CPU can lead to processing delays or even result in an inability to process them in a timely manner.</li>\n<li>The database server can become overloaded and run out of storage space</li>\n</ol>\n<p>The problem of increased requests can be solved through the use of caching and optimizing database queries. Caching involves storing frequently accessed data in lower-level storage called a cache. This allows for faster retrieval of data in the future, without having to access the central database.</p>\n<p>To solve the issue of database overload, two techniques can be used: vertical scaling and horizontal scaling. Vertical scaling involves increasing the computational power of the server by adding more storage space and CPUs while maintaining the same logical schema and database infrastructure. On the other hand, horizontal scaling expands storage capacity by adding parallel nodes to the central database server, thus reducing the load on the main server.</p>\n<h3>Conclusion</h3>\n<p>So far in our discussion, we have analyzed the importance of databases in data science by taking the example of supermarkets. We have highlighted the benefits of using a database system and the challenges that come along with it.</p>\n<p>Enjoy learning, Enjoy data science!</p>"},{"id":"d7893ddf-f537-540a-8028-1fa3af7edd5c","frontmatter":{"title":"Refresh Ahead Caching Strategy","path":"/blog/refresh-ahead-caching-pattern"},"html":"<p>In <a href=\"https://www.enjoyalgorithms.com/blog/cache-aside-caching-strategy/\">cache-aside</a> or <a href=\"https://www.enjoyalgorithms.com/blog/read-through-caching-strategy/\">read-through</a> patterns, if cached data get evicted due to expiration of TTL (time to live), cache will reload data from the database when there is a next read request for the same data. This is a scenario of a cache miss, which will increase read latency.</p>\n<p>Now the question is: Is there a way to prefetch data (from database to cache) before the expiration, so that request does not need to go to database? This is where Refresh-Ahead pattern comes into the picture.</p>\n<h3>How refresh-ahead caching works?</h3>\n<p>The goal of refresh-ahead pattern is to configure cache to asynchronously reload (refresh) the most recent version of data from database before the expiration (or close to its expiration). This refresh operation is done on a consistent time interval by some background job or a dedicated thread.</p>\n<p><img src=\"https://ucarecdn.com/10bbb7ce-8b2a-4a59-abc1-d00771a70d5c/\" alt=\"What is refresh-ahead caching pattern?\" title=\"Refresh-ahead caching strategy\"></p>\n<p>So refresh-ahead pattern ensures that cache entries that are likely to be accessed again soon are proactively available in the cache before they expire. Read requests for the same data will be served from cache with low latency. Now here is a dilemma: If there is a frequent refresh, we will get more up-to-date data but it will increase load on the database. On the other side, infrequent refresh can create a scenario of stale data in the cache. So we need to set appropriate refresh frequency to balance a tradeoff between data freshness and database load.</p>\n<p>One idea is to define refresh-ahead factor for the cache entries. This is a value between 0 and 1, representing a percentage of the TTL value. For example, suppose TTL for cached data is set to 120 seconds and refresh refresh-ahead factor is set to 0.5. </p>\n<ul>\n<li>If application read data before 60th seconds, nothing will happen and cache will return the data.</li>\n<li>If application read cached data after the expiration time (after 120 seconds), cache will perform a synchronous read from the database and return data to the application. Synchronous read means that the cache will wait for reload operation to complete before serving requested data to the application. So this read operation will be slow.</li>\n<li>If application read cached data after 60th seconds but before 120th seconds, cache will return the current value and asynchronously fetch the updated version from database. Here data is reloaded ahead of the TTL limit, so it will also reset the TTL (extending its stay in cache). Note: Here refresh process is asynchronous. So cache will not wait for data to be reloaded before serving it to the application.</li>\n</ul>\n<p>What will be the benefit of this approach? We mostly use cache to store frequently accessed data. If we totally rely on TTL value, then there can be a chance that we will be serving stale versions of frequently accessed data till the expiration of TTL. Even after the expiration, it will increase latency to fetch updated data from database. So using refresh-ahead pattern, frequently accessed cache entry will get reloaded asynchronously before its expiration. Due to this, subsequent reads of that entry will not face the latency of reloading from database. </p>\n<p>So refresh-ahead caching is useful when a large number of frequently accessed data needs to be updated regularly (or expected to change frequently) and keeping it up-to-date is crucial for the application’s performance and functionality. </p>\n<h3>Critical ideas to explore!</h3>\n<ul>\n<li>Is there some other way to implement this pattern?</li>\n<li>Rather than setting refresh factor for each cache entry, can we configure cache to asynchronously refresh a batch of frequently updated data on a scheduled interval using some background job?</li>\n<li>How can we monitor cache to improve the effectiveness of refresh ahead pattern? For example, suppose we observe the life time of cached data and based on some pattern or event, we configure cache to load the data even before the application requests it.</li>\n<li>How much complexity it will add to application code?</li>\n</ul>\n<h3>Advantages or benefits of refresh-ahead pattern</h3>\n<ol>\n<li>Ensure frequently read data is updated before expiration.</li>\n<li>Reduces waiting time for data retrieval from the database. This will lower the read latency for frequently accessed data compared to other techniques like read-through pattern.</li>\n<li>Reduces sudden latency spikes caused by TTL expiration. In other words, to some extent, it will solve the “thundering herd” problem associated with read-through caching!</li>\n<li>Asynchronous refresh is triggered only when a cache entry is accessed and close to its expiration time. This will optimize refresh efforts for actively used data and avoid wasteful reloading of less frequently accessed data.</li>\n<li>Refresh-ahead is especially useful if data is accessed by a large number of users. Values remain fresh in the cache and the latency that could result from excessive reloads from the database is avoided.</li>\n</ol>\n<h3>Limitations or drawbacks of refresh-ahead pattern</h3>\n<ul>\n<li>Implementing the refresh ahead pattern can add complexity to the application code, making it harder to maintain and debug. This is especially true if the caching logic becomes difficult to manage.</li>\n<li>It might add extra load on the cache and database if all keys refresh at the same time.</li>\n<li>The cache needs to accurately predict which cache items are likely to be needed in the future because inaccurate predictions can incur unnecessary database reads.</li>\n<li>Refresh-ahead might not be suitable for all access patterns. It may work well for certain read-heavy workloads, but the overhead of pre-fetching might not be justified for write-intensive access patterns.</li>\n</ul>\n<h3>Refresh-Ahead versus Read-Through Cache</h3>\n<ul>\n<li>Refresh-ahead cache proactively fetches data from storage before it is explicitly requested. On the other side, read-through cache fetches data from storage and populates the cache only when it is explicitly requested by the application.</li>\n<li>Refresh-ahead cache predicts and fetches data likely to be accessed in the future. On the other side, read-through cache fetches data from storage into the cache on-demand when a read request is made.</li>\n<li>Refresh-ahead cache reduces access latency since data is already available in the cache when needed. On the other side, read-through cache may initially experience higher latency.</li>\n<li>Refresh-ahead may introduce overhead if the system spends significant resources on prefetching data, competing with other tasks. But read-through cache has generally lower overhead since it fetches data only in response to actual read requests.</li>\n<li>Refresh-ahead is particularly useful for scenarios with predictable access patterns. On the other side, there are no such limitations with the read-through cache. It can work best for scenarios where future access patterns are less predictable.</li>\n</ul>\n<p>We hope you enjoyed the blog. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy algorithms, Enjoy system design!</p>"},{"id":"0ed68cf7-207d-58d6-be76-bf1d43a5f721","frontmatter":{"title":"Write-Behind Caching Strategy","path":"/blog/write-behind-caching-pattern"},"html":"<p>There is one big challenge with <a href=\"https://www.enjoyalgorithms.com/blog/write-through-caching-strategy/\">write-through caching</a>: the high latency of write operations because data needs to be updated in both places synchronously, i.e. first in the cache and then in the main database. One solution to improve write performance is to use write-behind (write-back) caching.</p>\n<p>The idea of write-behind caching is similar to the write-through cache, but there is one significant difference: the data written to the cache is asynchronously updated in the main database. In other words, the application will first write to the cache, and the cache will write to the database after some delay.</p>\n<p><img src=\"https://ucarecdn.com/5d8274a9-46ec-4bc2-a0b6-374c02c2cc04/\" alt=\"How write-behind caching strategy works?\" title=\"write-behind cache\"></p>\n<p>Let's understand this from another perspective! In a write-through cache, when many write requests come in in a short period, the database can easily become a bottleneck. So, write-behind caching allows faster cache writes with eventual data consistency between cache and database. In the meantime, any read from the cache will still get the latest data.</p>\n<p>Therefore, application does not need to wait for the database update, i.e. application only writes data to the cache, and cache acknowledges the write immediately. Cache keeps track of the changes made and writes them back to the main database at a later time (maybe during less busy periods). This asynchronous update will reduce the latency of the write operation.</p>\n<p>Now the question is: How can we implement asynchronous update from cache to database? One idea is to use a time-based delay, where cache waits for a predefined period before performing the update to database. Another idea is to use an entry-based delay, where cache waits until a certain number of new data entries are accumulated before updating the database.</p>\n<h3>Advantages of write-behind pattern</h3>\n<ul>\n<li>Write-behind cache improves the performance of write operations, so it is a good choice when dealing with write-heavy workloads. Additionally, in combination with <a href=\"https://www.enjoyalgorithms.com/blog/read-through-caching-strategy/\">read-through pattern</a>, write-behind is also suitable for mixed workloads involving both read and write operations. Explore and think!</li>\n<li>Cache acknowledges writes immediately and postpones database updates. So this will reduce strain on the cache. Even if database experiences downtime or failures, application can still function and serve read and write requests from the cache. The updates that have not yet been written to the database are queued in cache and can be synchronized with the database once it recovers. This enhances the overall availability of the system and mitigates the impact of database failures.</li>\n<li>Write-Back caching also reduces the workload on the database, as the database is updated less frequently. Now the question is: Can we further reduce the load on database?</li>\n</ul>\n<h3>Approaches to offload database</h3>\n<p>The idea is to use these strategies in combination with the write-behind approach to optimize the writing process and reduce workload on the database by handling write spikes more gracefully.</p>\n<h4>Using rate limiting</h4>\n<p>When a lot of write requests come in all at once, it can overwhelm the database. To avoid this, we can use rate limiting in a write-behind cache to put a cap on the number of write requests the database can handle per second or per minute.</p>\n<p>This will spread out the write workload over a more extended period, ensuring a steady flow of writes to the database rather than a sudden surge during peak periods. It will give the database enough time to catch up and process the write requests without getting overloaded.</p>\n<h4>Using batching and coalescing technique</h4>\n<p>We can use batching and coalescing techniques in the write-behind cache to reduce the number of write requests. Batching combines multiple write operations into a single write while coalescing consolidates multiple updates on the same data into a single update.</p>\n<p>This means that instead of immediately writing each change to the database, the cache can group them and write them as a single operation. Consequently, this will decrease the overall number of database writes, effectively reducing the load on the database. The good thing is: It will also save costs when the database provider charges based on the number of requests made.</p>\n<h4>Using time shifting</h4>\n<p>Databases can experience \"rush hours\" when lots of data are being written or modified simultaneously. So time shifting is an idea to strategically move the process of writing data to less busy times or time intervals other than peak usage times. This will allow the system to avoid becoming overwhelmed during high contention periods.</p>\n<h3>Disadvantages and limitations of write-behind pattern</h3>\n<ul>\n<li>Delay between cache write and database write introduces a period during which data in the cache is not yet reflected in the database. During this period, cache store the most recent data and the database may store old data i.e. cache may be ahead of the database in terms of data consistency. So we should use write-behind cache when application is designed to tolerate inconsistent data during delays.</li>\n<li>If there is a cache failure or system crash, recently written data in the cache may be permanently lost. This can affect overall functioning of the application.</li>\n<li>In write-behind cache, if a failure occurs during delayed write to the database after the cache write has been acknowledged to the application, there will be a chance of data loss or data inconsistency. How can handle this situation? One idea is to implement retry mechanisms or use delayed writes with an appropriate timeout.</li>\n</ul>\n<h3>Write-behind vs Write-through cache</h3>\n<p>The decision between write-through and write-behind caching strategies is related to one of the key trade-offs: data consistency vs write performance. Here is a critical comparison between both approaches:</p>\n<ol>\n<li>Write-through maintains consistency with cache and database. On the other side, write-behind can lead to temporary inconsistency with database and cache.</li>\n<li>Since data is written immediately to both cache and database, write-through caching provides strong data durability. In the event of a system failure, the updated data will be available in the database. On the other side, there is a potential risk of data loss in write-behind caching, especially in the event of a sudden power failure or cache failure before the data is updated in the database.</li>\n<li>Write-through introduces high latency for write operations because data needs to be written to both the cache and the database. On the other side, write-behind caching usually provides low latency for write operations compared to write-through because data is first written to the cache and application can proceed without immediate write delays.</li>\n<li>Write-through is well-suited for highly sensitive data or scenarios where data consistency and integrity are top priorities. On the other side, write-behind caching is suitable for non-critical data or scenarios where write performance for write-heavy workloads is crucial.</li>\n</ol>\n<h3>Critical ideas to think and explore!</h3>\n<ul>\n<li>How does the write-behind caching approach handle conflicts when multiple write requests are received for the same data within the delay period?</li>\n<li>Is there a need to implement TTL or other cache eviction policies in the write-behind caching?</li>\n<li>What are some specific use cases where write-behind caching is not recommended?</li>\n<li>Examples of caching technologies that are well-suited for implementing write-behind pattern.</li>\n<li>Are there any best practices or guidelines for determining the appropriate delay time for the asynchronous updates from the cache to the database?</li>\n<li>How does the write-behind caching approach impact data integrity and consistency in scenarios where multiple caches are distributed across different nodes or regions?</li>\n<li>Can we combine write-behind caching with other caching strategies like <a href=\"https://www.enjoyalgorithms.com/blog/cache-aside-caching-strategy/\">cache-aside</a> or read-through caching, to achieve even better performance optimizations?</li>\n</ul>\n<p>We hope you enjoyed the blog. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy algorithms, Enjoy system design!</p>"},{"id":"f89dee12-583d-58ef-8d0a-227260b649e4","frontmatter":{"title":"Write-Around Caching Strategy","path":"/blog/write-around-caching-pattern"},"html":"<p>In <a href=\"https://www.enjoyalgorithms.com/blog/write-through-caching-strategy/\">write-through</a> or <a href=\"https://www.enjoyalgorithms.com/blog/write-behind-caching-pattern/\">write-behind</a> cache, if data is unlikely to be read again soon after the write operation, cache may become filled with rarely used data (cache pollution). So, in the case of such data access pattern, we can use write-around strategy for efficient cache usage.</p>\n<p>In write-around caching, when application needs to write new data, it bypasses the cache and directly writes it to the underlying data source. Consequently, data is not stored in the cache during the write operation. When a read request is made to the cache, and if there is a cache miss, cache will fetch data from the database, update its corresponding entry, and then return data to the application.</p>\n<p><img src=\"https://ucarecdn.com/52e92dc2-d666-4d33-8f87-cdbeecde8842/\" alt=\"How write-around caching strategy works?\" title=\"Write-around caching pattern\"></p>\n<p>Now let's consider a critical scenario: Suppose we want to update data that is already present in the cache. Following write-around idea, we would update the old data in the database. However, a key issue arises when a user attempts to read the same data again. The application would return the older data from cache rather than the newly stored data from the database. How can we handle this scenario? Let's think!</p>\n<p>We can think about three possible options:</p>\n<ol>\n<li>Tolerate arbitrarily stale data in cache (usually not practical).</li>\n<li>One brute force idea is: For every read, cache will check and confirm with the database whether data is stale or not. However, this approach will increase read latency by a huge margin, which is practically not good for a read-heavy workload! So in this scenario, it is better to use write-through caching to avoid another query to the database.</li>\n<li>A more viable solution is to update the database and invalidate corresponding cache entry. By doing so, we force a cache miss on the next read request. This is one of the variation of write-through cache, which is called write-invalidate policy. </li>\n</ol>\n<p>Normally, it doesn't make much sense to implement a pure write-around cache because it is time-consuming and slow. But if we use the write-around pattern with <a href=\"https://www.enjoyalgorithms.com/blog/cache-aside-caching-strategy/\">cache-aside</a> or <a href=\"https://www.enjoyalgorithms.com/blog/read-through-caching-strategy/\">read-through</a> patterns, it will make some sense. Here is the reason: Suppose we have some data that will not be read anytime soon. In this situation, we never want write of such data to go through cache. </p>\n<p>Since cache generally has a limited size, by not caching unimportant data, we leave more room for the data that actually matters for performance i.e. frequently accessed data or read heavy data. In other words, we never want rarely read data to fill the cache because it can evict parts of the active data that you actually want to be cached. Is there some other way to handle the above scenario of stale data in the write-around cache? Explore and think!</p>\n<h3>Advantages or benefits of write-around caching</h3>\n<ul>\n<li>As we have seen above, write-around caching helps to improve cache efficiency by preventing the cache from occupying infrequently accessed data, write-only data that may never be reused, or data that is unlikely to be read again soon after the write operation.</li>\n<li>With cache-aside or read-through patterns, write-behind caching works best for read-heavy workloads. This approach helps us implement lazy loading (storing data in the cache based on demand).</li>\n<li>By bypassing the cache and writing data directly to the database, write-around pattern will keep the database consistently up to date. Even in the case of cache failure or power failure, it reduces the risk of data loss since the data is already safely stored in the database.</li>\n<li>By bypassing the cache, write-around caching helps prevent unnecessary cache evictions and reduces cache thrashing. Note: Cache thrashing occurs when the cache is constantly evicting and loading data due to its limited size.</li>\n</ul>\n<h3>Limitations or drawbacks of write-around caching</h3>\n<ul>\n<li>Read operations for recently written data will experience a cache miss (higher latency) since data must be fetched from the database.</li>\n<li>If data is constantly updated in the database, and corresponding cache entry is not invalidated, then subsequent read operations will access the old or stale version of the data. This will create cache coherence issues and lead to inconsistent results. So, we need to implement a proper cache invalidation scheme to ensure consistency between cache and database.</li>\n<li>Even if we invalidate cache after the database write, it will increase the chances of a cache miss if data is updated constantly in the database. So, this is not a good choice for a write-heavy workload.</li>\n<li>It does not benefit from the cache coalescing or buffering effects, which are techniques of combining or delaying multiple write requests to reduce the load on the database.</li>\n<li>Determining which data to cache and which to bypass requires an understanding of the data access patterns. Depending on the accuracy of these predictions, write-around caching may or may not be an optimal choice.</li>\n</ul>\n<h3>Critical ideas to think and explore further!</h3>\n<ul>\n<li>How does write-around caching ensure consistency between the cache and the underlying data source?</li>\n<li>In what scenarios would hybrid caching, combining write-around caching with other caching techniques, be a more effective solution?</li>\n<li>How do data size and access patterns affect the performance of the write-around cache?</li>\n<li>What are the best practices for monitoring performance and optimizing write-around cache usage?</li>\n<li>What are the other limitations of write-around caching?</li>\n<li>How can cache invalidation schemes be implemented effectively to maintain consistency between the write-around cache and the database?</li>\n<li>What are the trade-offs between write-around caching and other caching strategies in terms of data durability and performance under different workloads?</li>\n</ul>\n<p>We hope you have enjoyed the blog.  If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, enjoy algorithms, enjoy system design!</p>"},{"id":"dadcca8c-32a0-5e44-a823-b436f16669cd","frontmatter":{"title":"Availability: System Design Concept","path":"/blog/availability-system-design-concept"},"html":"<p>Many of us may have experienced moments where we could not access certain applications due to an outage or unavailability. Recently, YouTube faced a global outage that stopped users from streaming videos for about an hour. You may wonder about the reason and how one can prevent it from happening. To understand it better, let's explore the idea of availability.</p>\n<h3>What is Availability?</h3>\n<p>The availability of a distributed system is the percentage of time in a given period that a system is available to perform its task and function under normal conditions. Let's understand it from another perspective. Various components in a distributed system are spread across multiple nodes or locations. So, availability is the ability of a system to remain operational despite failures within its components.</p>\n<p>One way to look at it is how resistant a system is to failures. The percentage of availability that a system requires depends on the usage of the system. For example, Air Traffic Control are among the systems that require high availability because a single error in directing aeroplanes can lead to catastrophic results. On the other side, systems which are not prone to failures can work well with fewer availability requirements. The idea is simple: High availability comes with a cost, so we have to optimize according to our needs.</p>\n<h3>How is Availability Measured?</h3>\n<p>We can measure the availability of a distributed as the percentage of a system’s uptime in a given time period i.e. dividing the total uptime by the total uptime and downtime in a given period.</p>\n<p>Availability = Uptime / (Uptime + Downtime).</p>\n<h4><strong>The Nine’s of Availability</strong></h4>\n<p>We usually measure availability in terms of Nines rather than percentages. If availability is 99.00 percent, it is said to have “2 nines” of availability, and if it is 99.9 percent, it is called “3 nines,” and so on. A system with 5 nines (i.e., 99.999%) is a Gold Standard of Availability. Let's take a look at different Nines of Availability.</p>\n<p><img src=\"https://ucarecdn.com/cf5e3146-79a9-411b-be72-97b4cca49de0/\" alt=\"The nine’s of availability in system design\" title=\"Availability of a system\"></p>\n<h3>How do we achieve High Availability?</h3>\n<p>To increase availability, we can use <strong>redundancy</strong> by duplicating or adding additional components (servers or storage). For example, a system with two identical web servers behind a load balancer can continue operating even if one of the servers goes down because the load balancer can redirect traffic to the remaining server. So by adding redundancy, we can make the system more resilient to failure.</p>\n<ul>\n<li><strong>Passive Redundancy:</strong> Only some of the components are active at any given time and backup components are available in case of a failure. If some component fails, the backup component will take over and become active.</li>\n<li><strong>Active Redundancy:</strong> Multiple active components work simultaneously to perform the task. In the event of a failure of one of the active components, the other active components can take over.</li>\n</ul>\n<p>Redundancy alone is not enough to guarantee high availability. Failure detection and alerting mechanisms must also be in place to identify failures. For this, we should continuously monitor system health and regularly perform high-availability testing, so that we can take corrective action whenever one of the components in the system becomes unavailable.</p>\n<p>Here are some other strategies to ensure high availability:</p>\n<ul>\n<li>Use load balancing to prevent server overloading. It can also help us to monitor server health.</li>\n<li>If possible, implement mechanisms for automatic failover. If one component fails, another takes over its function automatically without manual intervention.</li>\n<li>Replicate data across multiple locations to avoid outages and make the system resilient against disasters. Replication can be synchronous or asynchronous, depending on the requirements.</li>\n</ul>\n<p>There is a trade-off between the availability of a system and its performance. To achieve high availability, we often implement redundancy or disaster recovery strategies, which can degrade system performance (higher latency or lower throughput). For example, implementing redundancy requires replicating data or tasks across multiple resources, which can increase latency.</p>\n<h3>Difference between high availability and fault tolerance</h3>\n<p>Both high availability and fault tolerance are strategies used to achieve high uptime, but they approach the problem differently. High availability is about the system's ability to remain operational with minimal downtime. On the other side, Fault tolerance is about the system's ability to continue functioning normally even in the event of a failure.</p>\n<ul>\n<li>Fault tolerance requires multiple systems that run in parallel. In the event of a failure, another system can take over without any loss of uptime. This requires advanced hardware that can detect component faults and enable the systems to operate in coordination. However, it may take longer for complex networks and devices to respond to malfunctions, and technical issues that result in a system crash may also cause the failure of redundant systems running in parallel.</li>\n<li>High availability, on the other hand, can also use the software-based approach to minimize server downtime rather than relying on hardware redundancy. This can be more flexible and easier to implement but it may not provide the same level of protection against system failures.</li>\n</ul>\n<p>Thanks to Chiranjeev and Navtosh for their contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"7bfa04eb-9a9c-51ba-ba19-daef3d603024","frontmatter":{"title":"Caching Strategy: Cache-Aside Pattern","path":"/blog/cache-aside-caching-strategy"},"html":"<p>Cache-aside is one of the commonly used caching strategies, where cache and database are independent, and it is the responsibility of the application code to manage cache and database to maintain data consistency. Let’s understand this from another perspective!</p>\n<p>Some applications use <a href=\"https://www.enjoyalgorithms.com/blog/read-through-caching-strategy/\">read-through</a> or <a href=\"https://www.enjoyalgorithms.com/blog/write-through-caching-strategy/\">write-through</a> patterns, where cache system defines the logic for updating or invalidating the cache and works as a transparent interface to the application. If the cache system does not provide these features, it is the responsibility of the application code to manage cache lookups, cache updates on writes, and database queries on cache misses. </p>\n<p>This is where the cache-aside pattern comes into the picture: the application interacts with the cache and database, and the cache doesn’t interact with the database at all. So cache is “kept aside” as a scalable in-memory data store.</p>\n<h3>Read operation in cache-aside pattern</h3>\n<p>For reading the data, application first checks the cache. If data is available (cache hit), the cached data will be returned directly to the application. If data is not available (cache miss), application code will retrieve data from the database, update the cache, and return data to the user.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/800/1*3DDetzMbVd7HfLU_xrCCbA.png\" alt=\"How cache-aside pattern works?\" title=\"Cache-aside Pattern\"></p>\n<h3>Write operation in cache-aside pattern</h3>\n<p>Now, the critical question is: How do we perform write operations? For writing data (add, modify, or delete), we can think about these two approaches.</p>\n<h4>Writing to the database and Invalidating the cache (Similar to write-around)</h4>\n<p>One approach is to update the database and invalidate or evict corresponding data in the cache (if it exists in the cache). This will ensure that the next read operation will retrieve updated data from the database. So, when there is a need to read the same data again, there will be a cache miss, and application will retrieve data from the database and add it to the cache.</p>\n<p>Here is one observation: After updating the database, data will not be stored in the cache until there is a next read request for the same data. So this strategy will load data on-demand into the cache (lazy-loading). It makes no assumptions about which data an application will require in advance.</p>\n<ul>\n<li>As highlighted above, data is not immediately updated in the cache, which can result in a cache miss for the next read. This will introduce additional latency. But subsequent read requests for the same data can be served efficiently from the cache. In simple words, this strategy can be beneficial if read-to-write ratio is very high or application re-reads the same data frequently after database write.</li>\n<li>There can be a chance of increased load on the database. For example, if multiple requests trigger cache misses simultaneously, it could lead to a spike in database queries.</li>\n</ul>\n<p>There is a critical question: Can we first invalidate the cache, and then write to the database? If we invalidate the cache first, there is a small window of time when a user might fetch the data before the database is updated. So this will result in a cache miss (because data was removed from the cache), and earlier version of data will be fetched from the database and added to the cache. The can result in stale cache data.</p>\n<h4>Writing to the database and updating the cache immediately (Similar to write-through)</h4>\n<p>This idea is similar to the write-through strategy: After updating the database, application also update cache with the latest modified data. This will improve subsequent read performance, reduce the likelihood of stale data, and ensure data consistency between cache and database.</p>\n<ul>\n<li>However, this will increase write latency because data needs to be written to both cache and database. Nonetheless, this overhead can be acceptable in systems with read-heavy workloads.</li>\n<li>Since cache is updated with the latest data after database update, subsequent read requests are more likely to hit the cache, thereby reducing the number of read requests to the database.</li>\n<li>This approach is commonly used in systems that prioritize data integrity.</li>\n</ul>\n<h3>Advantages of cache-aside pattern</h3>\n<ul>\n<li>Provides flexibility and control to the application code. Due to this, application developers can optimize caching based on the specific needs of the application and decide what data is cached and for how long.</li>\n<li>Cache-aside pattern is resilient to the failure of the cache layer. If the cache layer fails, application can still access data from the database.</li>\n<li>As the application grows and more resources are needed to handle increased traffic, we can add more caching servers to distribute the load. In other words, in the cache-aside pattern, we can easily scale the cache layer independently of the database.</li>\n<li>Due to separation of cache and database, we can use different data models in cache and database.</li>\n<li>With write-around pattern, cache-aside pattern can load data on demand into the cache from database. Due to this, cache can only contain data that application actually requests, which helps in the optimal use of cache capacity. </li>\n<li>With write-through pattern, cache-aside pattern can ensure that the data in the cache is up to date as far as possible. In other words, the risk of serving outdated or inconsistent data from the cache is reduced by this hybrid strategy.</li>\n<li>Anybody can understand the execution flow of the code. On the other side, cache provider requirements are as minimum as possible. Due to this, one can easily migrate from one to another cache provider.</li>\n<li>This can be a good option for scenarios where data access patterns are not easily predictable.</li>\n</ul>\n<h3>Limitations of cache-aside pattern</h3>\n<ul>\n<li>The responsibility of managing the cache in application code can bring some challenges. Developers need to ensure proper cache handling, including cache invalidation, eviction policies, and handling concurrent requests. This can add complexity and increase the risk of bugs or errors.</li>\n<li>When using lazy loading, data is loaded into the cache only after a cache miss. So if there are frequent updates in the database, there will be frequent cache misses.</li>\n<li>When using cache-aside with write-through, there will be high latency for the write because data will be updated in both cache and the database. So this strategy will be inefficient for write-heavy workloads. On the other side, we are writing at both places, this can fill the cache with undesired data. This may lead to high cache miss, inefficient use of cache or evictions of some frequently used data. How to handle this scenario? Explore and think!</li>\n</ul>\n<p>In the cache-aside pattern, applications can ensure that the cache data is up to date as far as possible. But in a practical scenario, cached data can be inconsistent with the data in database. For example, an item in the database can be changed at any time by some external process, and this change might not be reflected in the cache until the next read of the same item. This can be a major issue in the system that replicates data across data stores and synchronization occurs frequently. </p>\n<p>So application developers should use proper cache invalidation schemes or expiration policies to handle the data inconsistency. Here is another example: Suppose you have successfully updated the cache but database fails to update. So the code needs to implement retries. In the worst case, during unsuccessful retries, cache contains a value that database doesn’t.</p>\n<h3>Cache-aside as a Local (in-memory) Cache</h3>\n<p>If an application repeatedly accesses the same data, cache-aside can be used in the local (in-memory) caching environment. Here local cache is private to each application instance. Here is a challenge: If multiple instances are dealing with the same data, each instance might store its copy of data in its local cache. Due to this, cached data across instances could become inconsistent. For example, if one instance updates the shared database, the other instance's caches won’t be automatically updated, resulting in stale or outdated data being used.</p>\n<p>To address this limitation of local caching, we recommend exploring the use of distributed caching. In this caching, system maintains a centralized cache accessible to all instances of the application. Due to this, all instances can read and update the same cached data (data consistency). If required, we can also spread cache across multiple nodes (improves scalability and fault tolerance).</p>\n<h3>Lifetime of cached data and cache expiration policy</h3>\n<p>What will happen when cache data is not accessed for a specified period? As discussed above, there can be a chance that data become stale. To solve this inconsistency problem, one solution is to implement expiration policy in the application code.</p>\n<p>In cache expiration policy, each item in the cache is associated with a time-to-live (TTL) value, which specifies how long the item can remain in the cache before it is considered expired. When the data expires, the application can be forced to fetch the latest data from the main data source and update the cache with the fresh data. By doing this on consistent time intervals, application can ensure that data is up-to-date.</p>\n<p>But it may result in unnecessary evictions if data remains valid for a longer period than TTL. So the question is: What should be the best value of the TTL? For this, we need to ensure that the expiration policy should match the access pattern of data in the application.</p>\n<ul>\n<li>If TTL is too short, there will be a lot of cache misses and applications will frequently need to retrieve data from the database. In other words, low TTL can reduce chances of inconsistencies but can result in high latencies and reduced performance.</li>\n<li>If TTL is too long, then cached data is likely to become stale. In other words, large TTL will ensure system performance but increases the chances of inconsistencies.</li>\n</ul>\n<p>Two ideas are important to determine appropriate TTL value: 1) Understanding the rate of change of the underlying data. 2) Evaluating the risk of outdated data being returned to your application. For example, we can cache static data (rarely updated data) for longer TTL and dynamic data (that changes often) for smaller TTL. This lowers the risk of returning outdated data while still providing a buffer to offload database requests.</p>\n<h4><strong>TTL with Jitter</strong></h4>\n<p>When caching multiple items, if all items have the same fixed TTL, they will expire simultaneously after that specific duration. This can lead to a “cache stampede” or “thundering herd,” where all the expired items are requested to be refreshed from database at the same time. Such a sudden surge in requests can cause high load and strain on the database.</p>\n<p>To mitigate this problem, TTL with Jitter technique (TTL = Initial TTL value + jitter) introduces randomness in TTL values of individual cached items. Instead of having the same fixed TTL for all items, a small random delta or offset is added to each item’s TTL. This effectively spreads out the expiration times of the cached items, so they will not expire simultaneously.</p>\n<h4>Critical ideas to think!</h4>\n<ul>\n<li>What are various use cases of cache control headers?</li>\n<li>How can we determine the optimal TTL value for different types of data in cache?</li>\n<li>How can we evaluate the effectiveness of expiration policy by monitoring cache hit and miss rates?</li>\n<li>How do you determine the appropriate range for TTL jitter?</li>\n<li>How do you handle the scenario where an item’s TTL expires, but application doesn’t immediately refresh the cache with fresh data from main data source?</li>\n<li>How do you address scenarios where cached items have different rates of change in database?</li>\n<li>How do you handle cases where main data source experiences significant delays or downtime, and application is unable to fetch fresh data to update the cache?</li>\n</ul>\n<h3>Cache Replacement Policy</h3>\n<p>What will happen when cache is full? We should implement cache replacement policies in the application code to handle this situation. The goal is to evict some existing cache entries to make space for new ones. There are two common cache replacement policies:</p>\n<ol>\n<li>LRU (Least Recently Used) Eviction: Least recently accessed items are removed from the cache when it reaches its capacity. If an item hasn’t been accessed for a specified period, it will likely be among the least recently used items (a candidate for eviction).</li>\n<li>LFU (Least Frequently Used) Eviction: Least frequently accessed items are removed from the cache. If an item has not been accessed for a specified period, its frequency of access will be low (a candidate for eviction).</li>\n</ol>\n<h3>Priming the cache</h3>\n<p>Instead of waiting for actual data requests from application to trigger caching, some applications proactively load the cache with relevant data beforehand. This is an idea of cache priming: Populating the cache with data that an application is expected to require during its startup or initialization process.</p>\n<p>One of the primary reasons for priming the cache is to quickly speed up the system. The reason is simple: When actual requests arrive, there is a higher chance that the requested data is already in the cache. So in scenarios with heavy read traffic, this will reduce initial read latency and initial load on the database.</p>\n<p>The effectiveness of cache priming depends on understanding the data access patterns and choosing the right data to preload into the cache. Over-priming or preloading unnecessary data can lead to wasted cache space and reduced overall cache performance.</p>\n<p>Please write in the message below if you want to share some feedback or if you want to share more insight. Enjoy learning, Enjoy system design!</p>"},{"id":"e8327632-e1b3-5326-8543-1acb19d581f1","frontmatter":{"title":"Introduction to Graph Database","path":"/blog/graph-database-system-design"},"html":"<p>In various real-life applications, we need to understand the relationships between data elements i.e. a graph-like arrangement of various entities. One of the good examples is social media applications. The critical question is: How to store such data and efficiently get answers to various queries? One of the best ways is to store data in graph databases which provide an efficient way to organize and analyze such data.</p>\n<ul>\n<li>Graph databases organize information into nodes (data entities) and edges (relationships). Edges can represent various relationships like parent-child, friend-friend, user-follower, ownership, actions, dependencies, etc. The good thing is: There are no restrictions on the number and types of relationships that can be associated with a node.</li>\n<li>Graph databases help us to efficiently traverse hierarchies, identify connections, and uncover relationships. That’s why they are used in applications where things are interconnected.</li>\n<li>In distributed systems, a graph database is a NoSQL database because it does not use the tabular structure of a traditional SQL database.</li>\n</ul>\n<h3>Use cases in real-life applications</h3>\n<ul>\n<li>Graph databases are used in recommendation engines to store relationships between different categories of information, such as customer interests, purchase history, and preferences. Based on this, these systems analyze user behaviour to provide personalized recommendations.</li>\n<li>Fraud detection systems use graph databases to analyze connections between individuals and their purchase activities to enhance transaction security. For example, they can detect patterns like multiple individuals linked to a single email address or various people sharing an IP address despite living in different locations.</li>\n<li>Social media platforms use graph databases to easily find people (nodes) and their relationships (edges). For instance, they can determine the 'friends of friends' of a particular person.</li>\n<li>In life sciences, graph databases can be used to understand complex relationships between genes, proteins, and diseases, leading to advancements in drug discovery and personalized medicine.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/421dc1bb-08f5-443b-82a3-347cbaaac3ea/\" alt=\"Graph database real life examples\" title=\"Real life example of graph database\"></p>\n<h3>When do we prefer graph databases over relational databases?</h3>\n<p>Let’s take an example of social media. If we use relational databases, we need to create a “users” table to store data about each user (name, email, password, etc.) and a “connections” table to store data about relationships between users (date they connected and type of relationship like friend, family, etc.).</p>\n<p>In such a scenario, we might need to perform a JOIN operation between the “users” and “connections” tables to analyze relationships. This can be time-consuming and resource-intensive if there are large amounts of data and relationships. For example, if we want to find all friends of a user, we need to perform a JOIN operation that matches the user’s ID with the IDs of all their friends in the “connections” table.</p>\n<p>On the other hand, the rigid schema of relational databases can limit the flexibility and scalability of the database. For example, if we want to add a new type of relationship, we need to modify the schema of the “connections” table. This can be a complex and time-consuming process. In other words, this can make it difficult to adapt to changing requirements and increase latency.</p>\n<p>To solve the above issues, we can use graph databases as a better solution. </p>\n<ul>\n<li>Graph databases eliminate the need for time-consuming JOIN operations. It can explore connections between nodes quickly because relationships are stored in the database, instead of being calculated each time a query is made. For example, if we want to find all the friends of a user, we can simply start from the node and follow edges (friend relationship) to reach all the friends. This will be an efficient process than performing a JOIN operation in a relational database.</li>\n<li>Graph databases have a flexible schema, which helps us to easily add new types of relationships without modifying the schema. This makes it easier to adapt to changing requirements.</li>\n</ul>\n<h3>Some popular graph databases</h3>\n<ul>\n<li><strong>Neo4j</strong> is a highly efficient and scalable graph database with powerful performance.</li>\n<li><strong>Amazon Neptune</strong> is a fully managed graph database service offered by Amazon Web Services.</li>\n<li><strong>ArangoDB</strong> is an open-source multi-model database.</li>\n<li><strong>TigerGraph</strong> is a fast and scalable graph database that is specifically designed for use cases such as real-time fraud detection, recommendation systems, and network analysis.</li>\n<li><strong>RedisGraph</strong> is an open-source graph database built on top of popular in-memory data store, Redis.</li>\n<li><strong>GraphQL</strong> is a query language for APIs, which provides a complete and understandable description of the data and gives clients the power to ask for exactly what they need.</li>\n</ul>\n<h3>Advantages of graph database</h3>\n<ul>\n<li>Provide a flexible data model that simplifies the representation of complex relationships.</li>\n<li>Process and analyze large amounts of graph data in real time (useful for real-time applications).</li>\n<li>Offer fast performance for complex graph queries (useful for fast and efficient data analysis).</li>\n<li>Easily integrate with other data sources (useful for analyzing data from multiple sources).</li>\n<li>Handle dynamic and changing data (useful for applications where data is constantly changing).</li>\n</ul>\n<h3>How graph database is implemented under the hood?</h3>\n<p>At a high level, a graph database is a collection of nodes and edges, which are stored in a data structure like an adjacency list or adjacency matrix. However in a typical implementation at a low level, a graph database will use an index to efficiently store and retrieve the nodes and edges. For example, it can use a hash table or B-tree to index nodes based on some unique ID. When performing a query, the graph database will use indexes to quickly locate relevant nodes, and then traverse the edges to find related nodes.</p>\n<p>In addition to indexing, graph databases can also use optimization techniques to improve performance like, caching frequently-used data, using algorithms like breadth-first or depth-first search to traverse the graph, or partitioning the graph into smaller sub-graphs to reduce the size of the data that needs to be processed.</p>\n<h3>Some common queries on graph database</h3>\n<ul>\n<li><strong>Neighbourhood queries:</strong> Find all nodes and edges that are directly connected to a specific node. For example, finding all friends of a user on a social network.</li>\n<li><strong>Pathfinding queries:</strong> Find the shortest path between two nodes in the graph. For example, finding the shortest path between two cities in a transportation network.</li>\n<li><strong>Pattern matching queries:</strong> Find all instances of a specific pattern in the graph. For example, finding all triangles in a social network (three users who are friends with each other).</li>\n<li><strong>Centrality queries:</strong> Find important nodes in the graph based on some property. For example, find the most influential users in a social network based on the number of connections they have.</li>\n<li><strong>Clustering queries:</strong> Find group of nodes that are densely connected to each other, but less connected to other nodes in the graph. For example, finding clusters of users who are friends with each other but not friends with users outside the cluster.</li>\n</ul>\n<p>These are just a few examples of the types of queries that can be performed on a graph database. The specific queries will depend on the nature of the data and the requirements of the application.</p>\n<p>If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"4a535739-0699-56e8-a4b6-ca010a9b8431","frontmatter":{"title":"Bloom Filter Data Structure: Implementation and Application","path":"/blog/bloom-filter"},"html":"<h3>What is Bloom Filter?</h3>\n<p>Bloom filter is a space-efficient probabilistic data structure that tells whether an element may be in a set or definitely is not. If we look up an item in the Bloom filter, we can get two possible results.</p>\n<ul>\n<li>The item is not present in the set: True negative.</li>\n<li>The item might be present in the set: Can be either a False positive or True positive.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/6b480bbc-b167-4623-8b88-028ce54ab209/\" alt=\"How bloom filter works?\" title=\"What is bloom filter?\"></p>\n<h3>How does Bloom Filter work?</h3>\n<p>Suppose we want to compare two strings. Instead of storing the entire string, we compute the hash of the string and then compare the hashes. Computing hash and then comparing them takes O(1) time instead of the O(n) time to compare the strings directly. If both hashes are unequal, we can definitely say that strings are unequal. Otherwise, both strings may be equal!</p>\n<p>One drawback of this method is the limited range of the hash function. The hash function h1 ranges from 0 to r1–1. We cannot uniquely identify more than r1 strings with this method, as at least two strings will have the same hash value. To compensate for this, we can use multiple hash functions. </p>\n<p>Suppose we use k hash functions, h1, h2, …, hk, pass two strings through these hash functions, and compute their hashes. If all the hashes are identical for both strings, there is a very high probability that both strings are the same. On the other hand, even if one hash does not match, we can say that the strings are different.</p>\n<p>Bloom filter is an array that stores 0s and 1s. In the image below, each cell represents a bit. The number below the bit is its index for a bloom filter of size 10.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*KoUC9Iz9PCWgXSQHB8GDJg.png\" alt=\"Empty bloom filter\"></p>\n<h3>Inserting an item to the bloom filter</h3>\n<p>In order to add an element, we need to pass it through k hash functions. Bits are set at the index of the hashes in array. For example, we want to add the word “coding”. After passing it through three hash functions, we get the following results.</p>\n<ul>\n<li>h1(“coding”) = 125</li>\n<li>h2(“coding”) = 67</li>\n<li>h3(“coding”) = 19</li>\n</ul>\n<p>Suppose that the size of our bloom filter is m = 10. We need to take mod of 10 for each of these values so that the index is within the bounds of the bloom filter. Therefore, indexes at 125%10 = 5, 67%10 = 7 and 19%10 = 9 have to be set to 1.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*RKFr0rPXuHMMvKQRoceMcg.png\" alt=\"Inserting an item to the bloom filter example\" title=\"Insertion in bloom filter example\"></p>\n<h3>Testing membership of an item in Bloom filter</h3>\n<p>If we want to test the membership of an element, we need to pass it through same hash functions. If bits are already set for all these indexes, then this element might exist in the set. However, even if one index is not set, we are sure that this element is not present in the set.</p>\n<p><img src=\"https://ucarecdn.com/085159cf-35cb-40bb-8237-293efa8ec932/\" alt=\"Testing membership of an item in bloom filter\" title=\"Bloom filter membership testing \"></p>\n<p>Let’s say we want to check the membership of “cat” in our set. Furthermore, we have already added two elements, “coding” and “music”, to our set.</p>\n<ul>\n<li>Coding has the hash output {125, 67, 19} from the three hash functions, and as discussed above, the indexes {5, 7, 9} are set to 1.</li>\n<li>Music has the hash output {290, 145, 2} and the indexes {0, 2, 5} are set to 1.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*IW83w-lMrCgAQP4HMrgtCA.png\"></p>\n<p>We pass “cat” through the same hash functions and get the following results.</p>\n<ul>\n<li>h1(“cat”) = 233</li>\n<li>h2(“cat”) = 155</li>\n<li>h3(“cat”) = 9</li>\n</ul>\n<p>So we check if the indexes {3, 5, 9} are all set to 1. As we can see, even though indexes 5 and 9 are set to 1, 3 is not. Thus we can conclude with 100% certainty that “cat” is not present in the set.</p>\n<p>Now let’s say we want to check existence of “gaming” in our set. We pass it through same hash functions and get the following results.</p>\n<ul>\n<li>h1(“gaming”) = 235</li>\n<li>h2(“gaming”) = 60</li>\n<li>h3(“gaming”) = 22</li>\n</ul>\n<p>We check if the indexes {0, 2, 5} are all set to 1. We can see that all of these indexes are set to 1. However, we know that “gaming” is not present in the set. So this is a false positive. Can we predict the probability of these false positives? Yes, and it is explained below.</p>\n<p>Let n = number of elements, m = length of the bloom filter, k = number of hash functions.</p>\n<p>Assume that the hash function selects each index with equal probability. The probability that a particular bit is not set to 1 by a specific hash function during the insertion of an element is 1-(1/m). If we pass this element through k hash functions, the probability that the bit is not set to 1 by any of the hash functions is (1 -(1/m))^k. After inserting inserted n elements, the probability that a particular bit is still 0 is (1 -(1/m))^(kn). Therefore the probability that it is one is 1 - (1 -(1/m))^(kn).</p>\n<p>Here we want to test the membership of an element in the set. So each of the k array positions computed by the hash functions is 1 with a probability 1 -(1- (1/m))^(kn). The probability of all of them being 1, which would result in a false positive, is (1 -(1 -(1/m))^(kn))^k. This is also known as the error rate.</p>\n<p>As we can see, if m tends to infinity, the error rate tends to zero. Another way to avoid a high false-positive rate is to use multiple hash functions.</p>\n<h3>Bloom Filter Python Implementation</h3>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\"><span class=\"token keyword\">import</span> <span class=\"token module\">mmh3</span> # murmurhash<span class=\"token operator\">:</span> is faster <span class=\"token keyword\">for</span> blooms\n<span class=\"token keyword\">import</span> <span class=\"token module\">math</span>\n\n<span class=\"token keyword\">class</span> <span class=\"token class-name\">BloomFilter</span><span class=\"token punctuation\">(</span>object<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n\n    def <span class=\"token function\">__init__</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> m<span class=\"token punctuation\">,</span> k<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        self<span class=\"token punctuation\">.</span>m <span class=\"token operator\">=</span> m # size of bloom filter\n        self<span class=\"token punctuation\">.</span>k <span class=\"token operator\">=</span> k # number of hash functions\n        self<span class=\"token punctuation\">.</span>n <span class=\"token operator\">=</span> <span class=\"token number\">0</span> # total count of the elemnts inserted in the set\n        self<span class=\"token punctuation\">.</span>bloomFilter <span class=\"token operator\">=</span> <span class=\"token punctuation\">[</span><span class=\"token number\">0</span> <span class=\"token keyword\">for</span> i in <span class=\"token function\">range</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">.</span>m<span class=\"token punctuation\">)</span><span class=\"token punctuation\">]</span>\n        \n    def <span class=\"token function\">_setAllBitsToZero</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        self<span class=\"token punctuation\">.</span>n <span class=\"token operator\">=</span> <span class=\"token number\">0</span>\n        <span class=\"token keyword\">for</span> i in self<span class=\"token punctuation\">.</span>bloomFilter<span class=\"token operator\">:</span>\n            self<span class=\"token punctuation\">.</span>bloomFilter<span class=\"token punctuation\">[</span>i<span class=\"token punctuation\">]</span> <span class=\"token operator\">=</span> <span class=\"token number\">0</span>\n            \n    def <span class=\"token function\">getBitArrayIndices</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> item<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        <span class=\"token string\">\"\"</span>\"\n\t\thashes the key <span class=\"token keyword\">for</span> k defined<span class=\"token punctuation\">,</span>\n\t\treturns a list of the indexes which have to be set\n        <span class=\"token string\">\"\"</span>\"\n\n        indexList <span class=\"token operator\">=</span> <span class=\"token punctuation\">[</span><span class=\"token punctuation\">]</span>\n        <span class=\"token keyword\">for</span> i in <span class=\"token function\">range</span><span class=\"token punctuation\">(</span><span class=\"token number\">1</span><span class=\"token punctuation\">,</span> self<span class=\"token punctuation\">.</span>k <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n            indexList<span class=\"token punctuation\">.</span><span class=\"token function\">append</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">(</span><span class=\"token function\">hash</span><span class=\"token punctuation\">(</span>item<span class=\"token punctuation\">)</span> <span class=\"token operator\">+</span> i <span class=\"token operator\">*</span> mmh3<span class=\"token punctuation\">.</span><span class=\"token function\">hash</span><span class=\"token punctuation\">(</span>item<span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">%</span> self<span class=\"token punctuation\">.</span>m<span class=\"token punctuation\">)</span>\n        <span class=\"token keyword\">return</span> indexList\n        \n    def <span class=\"token function\">add</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> item<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        <span class=\"token string\">\"\"</span>\"\n\t\tInsert an item in the filter\n        <span class=\"token string\">\"\"</span>\"\n        \n        <span class=\"token keyword\">for</span> i in self<span class=\"token punctuation\">.</span><span class=\"token function\">getBitArrayIndices</span><span class=\"token punctuation\">(</span>item<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n            self<span class=\"token punctuation\">.</span>bloomFilter<span class=\"token punctuation\">[</span>i<span class=\"token punctuation\">]</span> <span class=\"token operator\">=</span> <span class=\"token number\">1</span>\n        \n        self<span class=\"token punctuation\">.</span>n <span class=\"token operator\">+=</span> <span class=\"token number\">1</span>\n    \n    def <span class=\"token function\">contains</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> key<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        <span class=\"token string\">\"\"</span>\"\n\t\treturns whether item exists in the set <span class=\"token operator\">or</span> <span class=\"token operator\">not</span>\n        <span class=\"token string\">\"\"</span>\"\n\n        <span class=\"token keyword\">for</span> i in self<span class=\"token punctuation\">.</span><span class=\"token function\">getBitArrayIndices</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n            <span class=\"token keyword\">if</span> self<span class=\"token punctuation\">.</span>bloomFilter<span class=\"token punctuation\">[</span>i<span class=\"token punctuation\">]</span> <span class=\"token operator\">!=</span> <span class=\"token number\">1</span><span class=\"token operator\">:</span>\n                <span class=\"token keyword\">return</span> False\n        <span class=\"token keyword\">return</span> True\n        \n    def <span class=\"token function\">length</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        <span class=\"token string\">\"\"</span>\"\n\t\treturns the current size of the filter\n        <span class=\"token string\">\"\"</span>\"\n\n        <span class=\"token keyword\">return</span> self<span class=\"token punctuation\">.</span>n\n        \n    def <span class=\"token function\">generateStats</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        <span class=\"token string\">\"\"</span>\"\n\t\tCalculates the statistics of a filter\n\t\tProbability of False Positives<span class=\"token punctuation\">,</span> predicted <span class=\"token boolean\">false</span> positive rate<span class=\"token punctuation\">,</span> n<span class=\"token punctuation\">,</span> m<span class=\"token punctuation\">,</span> k<span class=\"token punctuation\">.</span>\n        <span class=\"token string\">\"\"</span>\"\n\n        n <span class=\"token operator\">=</span> <span class=\"token keyword\">float</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">.</span>n<span class=\"token punctuation\">)</span>\n        m <span class=\"token operator\">=</span> <span class=\"token keyword\">float</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">.</span>m<span class=\"token punctuation\">)</span>\n        k <span class=\"token operator\">=</span> <span class=\"token keyword\">float</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">.</span>k<span class=\"token punctuation\">)</span>\n        probability_fp <span class=\"token operator\">=</span> math<span class=\"token punctuation\">.</span><span class=\"token function\">pow</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">(</span><span class=\"token number\">1.0</span> <span class=\"token operator\">-</span> math<span class=\"token punctuation\">.</span><span class=\"token function\">exp</span><span class=\"token punctuation\">(</span><span class=\"token operator\">-</span><span class=\"token punctuation\">(</span>k<span class=\"token operator\">*</span>n<span class=\"token punctuation\">)</span><span class=\"token operator\">/</span>m<span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">,</span> k<span class=\"token punctuation\">)</span>\n\n        <span class=\"token function\">print</span><span class=\"token punctuation\">(</span><span class=\"token string\">\"Number of elements entered in filter: \"</span><span class=\"token punctuation\">,</span> n<span class=\"token punctuation\">)</span>\n        <span class=\"token function\">print</span><span class=\"token punctuation\">(</span><span class=\"token string\">\"Number of bits in filter: \"</span><span class=\"token punctuation\">,</span> m<span class=\"token punctuation\">)</span>\n        <span class=\"token function\">print</span><span class=\"token punctuation\">(</span><span class=\"token string\">\"Number of hash functions used: \"</span><span class=\"token punctuation\">,</span> k<span class=\"token punctuation\">)</span>\n\n        <span class=\"token function\">print</span><span class=\"token punctuation\">(</span><span class=\"token string\">\"Predicted Probability of false positives: \"</span><span class=\"token punctuation\">,</span> probability_fp<span class=\"token punctuation\">)</span>\n        <span class=\"token function\">print</span><span class=\"token punctuation\">(</span><span class=\"token string\">\"Predicted false positive rate: \"</span><span class=\"token punctuation\">,</span> probability_fp <span class=\"token operator\">*</span> <span class=\"token number\">100.0</span><span class=\"token punctuation\">,</span> <span class=\"token string\">\"%\"</span><span class=\"token punctuation\">)</span>\n        \n    def <span class=\"token function\">reset</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">)</span><span class=\"token operator\">:</span>\n        <span class=\"token string\">\"\"</span>\"\n\t\tResets the filter <span class=\"token operator\">and</span> clears old values <span class=\"token operator\">and</span> statistics\n        <span class=\"token string\">\"\"</span>\"\n\n        self<span class=\"token punctuation\">.</span><span class=\"token function\">_setAllBitsToZero</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span></code></pre></div>\n<h3>Bloom Filter Visualization With Example</h3>\n<p><img src=\"https://ucarecdn.com/6ff1c5ef-b146-4650-a7a8-8497d94932e1/\" alt=\"Bloom filter explained with example\" title=\"Bloom filter visualization\"></p>\n<h3>Bloom Filter Applications</h3>\n<p>Bloom filters can be used as the first layer of filtering. The advantage is that they require constant space. They can be helpful where we want to know if something is definitely not present or possibly present somewhere. For example, we want to query a large database from a rotating hard drive that is slow to read. Suppose there is a high probability that the thing we want to query is not present in the database, then before querying, we can check for this item in the bloom filter. We can avoid the expensive query if the bloom filter says that this element is definitely not present in the database.</p>\n<ul>\n<li>Facebook uses bloom filters to avoid one-hit wonders. One-hit wonders are web objects requested by users just once. Using a Bloom filter to detect the second request for a web object and caching it only on its second request prevents one-hit wonders from entering the local storage. For example, searching for “coding” often can be stored in the local storage. However, if you search for something only once, like “giraffe”, it should not be stored in the local storage as it will be a one-hit-wonder.</li>\n</ul>\n<p>Using an <a href=\"https://www.enjoyalgorithms.com/blog/implement-least-recently-used-cache/\">LRU Cache</a> here will remove some older but frequent results from the local storage for one-hit wonders, negatively affecting the user experience. </p>\n<ul>\n<li>Medium uses Bloom filters in its Recommendation module to avoid showing those posts that the user has already seen.</li>\n</ul>\n<h3>Summary</h3>\n<p>A Bloom filter is a data structure designed to tell rapidly and memory-efficiently whether an element is present in a set. The tradeoff is that it is probabilistic; it can result in False positives. Nevertheless, it can definitely tell if an element is not present. Bloom filters are space-efficient; they take up O(1)space, regardless of the number of items inserted as the bloom filter size remains the same. However, their accuracy decreases as more elements are added. Insert, and lookup operations are O(1) time.</p>\n<p><strong>Enjoy learning, Enjoy algorithms!</strong></p>"},{"id":"44aa72de-85dd-53ce-9b66-7c14ff45a619","frontmatter":{"title":"Introduction to Database Replication (Types and Advantages)","path":"/blog/introduction-to-database-replication-system-design"},"html":"<p>The concept of database replication has evolved in the last two decades due to the increasing use of distributed databases. But its core principles are more and less the same over time. One reason is: The fundamental constraints of distributed systems have not changed significantly. In this blog, we will discuss the definition, types, advantages and disadvantages of database replication.</p>\n<h3>What is Database Replication?</h3>\n<p>Database replication is the process of keeping multiple copies of the same data in different servers (replicas) so that if one server goes down, other servers can continue to serve data without any interruption or downtime.</p>\n<ul>\n<li>Database replication provides several advantages like high performance, availability and reliability.</li>\n<li>If the data that we are replicating does not change over time, then replication is easy: we just need to copy the data to every node once and we are done. All the difficulty in replication lies in handling changes to replicated data.</li>\n<li>We can create a replica of an entire database or a subset of the database.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/41b13bc0-8674-44c4-a8a2-2abb2a10dea3/\" alt=\"Database replication in distributed systems\" title=\"database replication\"></p>\n<h4>Understanding database replication with an example</h4>\n<p>Suppose there is an online store that has a single database server that stores all the data related to the products, orders and customers. If this server fails due to some reason (hardware issues, software issues, etc.) the whole website will be unavailable.</p>\n<p>To solve this problem, we can use database replication. The website owner can set up multiple database servers that replicate data from the primary database. If the primary server goes down, other servers can take over and continue to serve requests.</p>\n<h3>Database replication techniques based on architecture</h3>\n<p>It's important to ensure that any updates made to the data on one node are reflected on all other nodes. So, we need efficient techniques that can ensure that all nodes have the most up-to-date version of the data.</p>\n<p>There are three types of database replication techniques based on architecture: Single-leader replication, multi-leader replication and no-leader replication. Each strategy has its advantages and disadvantages.</p>\n<h4><strong>Single-leader Architecture</strong></h4>\n<p>This is also known as <strong>Master-Slave</strong> or <strong>Active-Passive replication</strong>. </p>\n<p>In single leader replication, there is a single leader (master) and several follower replicas (slaves). All write requests are served by the leader node and all read requests are served by the leader or any of the follower nodes. The best idea would be to use the leader only for write requests and distribute read requests across followers. This will remove some load from the leader.</p>\n<ul>\n<li>After serving the write request, the leader node sends a stream of data changes to the follower nodes to update their current state of data.</li>\n<li>As write operations are concentrated on a single leader node. If the leader node fails, the system may become unavailable until a new leader is elected.</li>\n<li>During the replication process, we need to handle one of the critical problems: reads from follower nodes may not reflect the latest changes as there can be some delay in the replication process (replication lag).</li>\n<li>This architecture is a good option when read-write ratio is very high.</li>\n</ul>\n<p>We have covered this idea of replication in a separate blog: <a href=\"https://www.enjoyalgorithms.com/blog/master-slave-replication-databases/\">Complete overview of Master-Slave Replication</a>.</p>\n<p><img src=\"https://ucarecdn.com/6f775641-707c-49dd-bcd1-75b2fb2aa419/\" alt=\"How single-leader replication technique works?\" title=\"single-leader replication or master-slave replication\"></p>\n<h4><strong>Multi-leader Architecture</strong></h4>\n<p>There is a one problem with leader-based replication: If the leader is unavailable due to some reason, we cannot perform write operations to the leader until we upgrade another follower replica as a leader. So a common solution to such a problem is to use multi-leader replication. This is also called <strong>master-master or active-active replication.</strong></p>\n<p>Here clients can send write requests to one of the leader nodes and each leader works as a follower for the other leader. So whenever a leader node performs the write operation, it will forward streams of data change to all other leader nodes.</p>\n<p>There are some complexities in multi-leader architecture. For example, conflicts can arise when two or more leader nodes receive conflicting write requests simultaneously. So it’s important to have conflict resolution mechanisms in place to ensure data consistency. <strong>Note:</strong> In the near future, we will discuss multi-leader in detail in a separate blog.</p>\n<p><img src=\"https://ucarecdn.com/de618c9d-103a-4131-a2e4-9af2244adb47/\" alt=\"How multi-leader replication technique works?\" title=\"multi-leader replication or master-master replication\"></p>\n<ul>\n<li>When data needs to be available in multiple regions, a multi-leader architecture can help reduce latency and improve performance. The idea is simple: Each region can designate its own leader node.</li>\n<li>By having multiple leader nodes, a multi-leader architecture can provide redundancy and ensure that the system remains available even if one or more nodes fail. It can also enable write scaling because write requests can be distributed across multiple nodes.</li>\n</ul>\n<h4><strong>No-leader Architecture</strong></h4>\n<p>This is also known as <strong>leader-less replication</strong>. In this architecture, clients send write requests to several nodes and read from several nodes in parallel. There is no concept of a leader in this approach, which allows any replica to directly accept writes from clients. </p>\n<p>Leaderless replication can provide high availability and fault tolerance. Because there is no single point of failure, the system can continue to function even if some nodes fail. It can also provide high read and write throughput because requests can be distributed across multiple nodes. </p>\n<p>This method also poses challenges for synchronisation, as it can be difficult to ensure that all nodes have the same view of the data at all times. In addition, handling conflicts that may arise from concurrent writes can be complex, and careful design is necessary to ensure data consistency.</p>\n<p><strong>Note:</strong> In the near future, we will discuss multi-leader in detail in a separate blog.</p>\n<h3><strong>Asynchronous</strong> replication</h3>\n<p>In this strategy, the leader node responds immediately to the client after updating its own copy of the data, without waiting for the changes to be propagated to the followers. But there is a risk of data loss without the client’s knowledge because the confirmation comes before the main replication process. The idea is: replication happens in the background and the leader asynchronously propagates the changes to the followers. If the leader node crashes, changes in data that are not propagated are lost permanently. </p>\n<p>Despite this disadvantage, asynchronous replication is the default strategy for most data stores because it offers flexibility. The idea is: Client is blocked only for the duration that the write happens on the leader!</p>\n<h3><strong>Synchronous</strong> replication </h3>\n<p>In synchronous replication, once the leader node updates its own copy of the data, it initiates the write operation on its followers: Followers receive the update, apply changes to their copy of data, and then send confirmation to the leader. Once the leader receives confirmation from all followers, it responds to the client and completes the operation.</p>\n<p>Synchronous replication ensures that followers are always in sync and consistent with the leader, making this setup fault-tolerant by default. Even if the leader crashes, the entire data is still available to the followers. So the system can easily promote any one of the followers as the new leader and continue to function as usual.</p>\n<p>One major disadvantage of synchronous replication is that the client and the leader can remain blocked if a follower becomes non-responsive due to a crash or network partition. In other words, the leader will continue to block all writes until the affected followers become available again. So having a large number of followers in this setup can result in longer block times for the client.</p>\n<p>There are many trade-offs to consider with replication. For example, when to use synchronous or asynchronous replication and how to handle failed followers. Explore the master-slave replication blog.</p>\n<h3>Full replication</h3>\n<p>In this method, we copy the entire original database at every replica. This makes data highly available and decreases query execution time (data can be fetched from any closest replica). But it can be slow to update the replicas because the entire database needs to be copied at every replica’s location.</p>\n<p>Full replication is useful when users at different locations need to see the same view of the data. For example, users looking for match scores need to see the same details about the match, regardless of their location.</p>\n<h3>Partial replication</h3>\n<p>In partial replication, we store a copy of only a selected part of data from the original database at each replica. So the type and importance of the data determine the number of replicas required. Here update process of each replica is fast because each replica only receives a portion of the entire database.</p>\n<p>But the problem is: If the local replica does not contain some required data, it needs to be fetched from the original database. This can increase the query execution time. So, partial replication is useful when one wants to provide an isolated view of data based on their location.</p>\n<p>For example, suppose an online fashion retailer sells clothing items in different regions of the world. The retailer may have different inventories in different regions depending on local demand. By replicating only the relevant data to each location, the retailer can ensure that customers in each region see only the relevant products that are available.</p>\n<h3>Advantages of database replication</h3>\n<ul>\n<li>Replication helps us scale out the number of machines that can serve read and write queries, which increases throughput and allows more queries to be processed in parallel.</li>\n<li>It helps us keep data geographically close to users, which reduces latency. The best example is CDN.</li>\n<li>If one of the database servers is destroyed by a natural disaster, data is still preserved. We do not need to worry about data loss because data is replicated across multiple locations.</li>\n<li>By replicating data across different database servers, website remains in operation even if a database server is down due to maintenance or some other reasons.</li>\n</ul>\n<h3>Disadvantages of database replication</h3>\n<ul>\n<li>Replicating data across multiple servers can increase the complexity of the database system.</li>\n<li>Implementing and maintaining a replication system can be expensive, as it requires additional hardware, software, and IT resources.</li>\n<li>Replication lag can occur when changes made to the master database are not immediately propagated to the replicated databases. This can result in inconsistent data across different instances of the database.</li>\n<li>Data conflicts can occur when changes are made to the same data in different instances of the database. This can result in data inconsistencies or even data loss if conflicts are not properly resolved.</li>\n<li>Replication systems require maintenance, monitoring, troubleshooting, and periodic updates. This can add to the overall maintenance overhead of the database system.</li>\n</ul>\n<p><strong>References</strong></p>\n<ul>\n<li>Designing data-intensive applications by Martin Kleppmann </li>\n<li>Web Scalability for Startup Engineers by McGraw-Hill </li>\n<li>Understanding distributed systems by Roberto Vitillo</li>\n</ul>\n<p>We will keep updating this blog with more insights on database replication. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"c5998bb7-5bee-5c5f-b82a-e7260c16ae8c","frontmatter":{"title":"Introduction to Hadoop Framework for Big Data","path":"/blog/introduction-to-hadoop-framework"},"html":"<h3>Introduction</h3>\n<p>Big Data is a stage of data where traditional techniques fail to analyze and perform analysis. So we need a new tech stack that can handle the analytical and operational needs of Big Data. Hadoop is one such technique specially designed to serve the purpose of storing and processing massive databases efficiently. It is also frequently mentioned technology in job descriptions for Data Scientist positions.</p>\n<p><strong>Key takeaways from this blog</strong></p>\n<p>After going through this blog, you will learn following things:</p>\n<ul>\n<li>What is Hadoop, and what makes it so important? </li>\n<li>What challenges do engineers working on BIG Data face, and how does Hadoop provide solutions? </li>\n<li>What are HDFS, YARN, and MapReduce in the Hadoop Framework? </li>\n<li>What are the challenges of using Hadoop? </li>\n<li>What are the various applications built on top of the Hadoop Framework?</li>\n</ul>\n<p>Let’s start by knowing more about Hadoop.</p>\n<h3>What is Hadoop?</h3>\n<p>Hadoop is an open-source framework that enables large-scale data processing for BIG Data. But what does open-source software mean? Open-source software is a type of software whose source code is available to the public, allowing anyone to inspect, modify, and enhance the base code. This type of software is continuously improved through community-driven development, providing many benefits, including cost savings, flexibility, and a larger pool of developers working to improve the software.</p>\n<p>Hadoop has evolved into a powerful technology with the contributions and support of many contributors and users. The literal meaning of Hadoop is a yellow elephant, given by its inventor after his son’s toy. It is also known as Apache Hadoop, as it is a top-level project of the Apache Software Foundation and is licensed under Apache License 2.0.</p>\n<p>But why? Why is Hadoop so important in the context of Big Data? Let’s understand.</p>\n<h3>Why is Hadoop so important?</h3>\n<ul>\n<li>Hadoop supports all three formats of datasets: Structured, Unstructured, or Semi-Structured.</li>\n<li>Hadoop consists of a cluster of machines performing tasks. Because of that, it provides a scalable solution for handling massive datasets. For example, if our website sees a sudden surge of incoming users, it can add more machines (as nodes) in the same Hadoop cluster to manage the load.</li>\n<li>With the help of Hadoop, many organizations can use the stored data to perform various analyses simultaneously.</li>\n<li>It is protected from hardware failures. If a node from the cluster goes down, the processes automatically get redirected to other free nodes.</li>\n</ul>\n<p>Why do traditional systems fail to process Big Data, or what are the challenges of Big Data handled by Hadoop? Let’s see.</p>\n<h3>How Hadoop handles the challenges of Big Data?</h3>\n<p>The three main challenges faced by engineers working on BIG Data are:</p>\n<h4>Challenge 1: Huge Database with heterogeneous nature</h4>\n<p>Big Data comprises various datasets classified into structured, unstructured, and semi-structured. Previously, datasets were primarily present in a structured format and were very easily manageable. Storing structured format datasets on a single system is easy. However, modern databases are more complex. They are comprised of video, audio, textual and in a semi-structured format, and they need non-conventional approaches.</p>\n<p><strong>Solution:</strong> Hadoop efficiently manages the diverse nature of modern datasets by separating them according to their type and storing them on multiple low-end hardware. The default storage capacity for one hardware is 128 MBs. For example, if we have 1000 MBs of data to store, it would require eight low-end hardware (128 * 7 + 104 * 1). Additionally, Hadoop creates multiple copies of these datasets to ensure that the data is automatically shifted to other available hardware if there is a hardware failure. This decentralized approach makes it more reliable.</p>\n<h4>Challenge 2: Traditional ways of processing Data is Very Slow</h4>\n<p>Big Data is a large and varied collection of data, which makes it difficult to process it simultaneously. So this would take a significant amount of time to execute.</p>\n<p><strong>Solution:</strong> To handle this efficiently, Hadoop enables parallel data processing on multiple low-end hardware. For this, Hadoop divides the analysis tasks among multiple hardware and then combines the results from all the hardware.</p>\n<h4>Challenge 3: Process Overload and mismanagement of Jobs performed on hardware</h4>\n<p>In traditional data processing methods, resources such as CPU, GPU, RAM, and ROM are not taken into account, making it challenging to manage job assignments and schedule them efficiently.</p>\n<p><strong>Solution:</strong> Hadoop addresses this issue using YARN’s resource management framework (Yet Another Resource Negotiator). YARN allocates the necessary resources to the process based on their availability. We will learn about this in the later part of this blog.</p>\n<p><img src=\"https://ucarecdn.com/b18b016d-5da0-4567-8479-b3c6d58f1990/\" alt=\"How hadoop framework works?\" title=\"What is hadoop?\"></p>\n<p>Out of these challenges in BIG Data, various modules are designed in Hadoop to address them.</p>\n<h3>Hadoop Modules</h3>\n<p>There are mainly four modules present in Hadoop:</p>\n<h4>1. Hadoop Distributed File Systems (HDFS)</h4>\n<p>Hadoop Distributed File System (HDFS) is a distributed storage system used in Hadoop. For example, if any human resource is overloaded from work, we can hire additional resources to manage the load and increase productivity. Similarly, in BIG Data, when the storage of one machine gets overloaded, Hadoop distributes the data samples across multiple machines to manage it effectively. All these systems form a cluster in Hadoop, which is done with the help of HDFS. </p>\n<p>HDFS has two main components: the <strong>Name Node</strong> and the <strong>Data Node</strong>. The Name Node and Data Nodes work together using a Master-slave mechanism, where the Name Node is the Master, and the Data Node is the Slave. The Name Node is responsible for managing the smooth functioning of the Data Nodes.</p>\n<p>The Data Nodes read, write, process, and replicate the Big Data stored in them according to their capacity. They also send heartbeat signals to the Name Node to communicate their status, whether they are functioning or not. This provides high security and tolerance against faults. If one hardware goes down, the data is automatically transferred to other available hardware.</p>\n<p><img src=\"https://ucarecdn.com/b1cf4653-d016-47bf-b32b-ba5369eaa6f0/\" alt=\"What is Hadoop Distributed File Systems or HDFS in Hadoop?\" title=\"HDFS working representation\"></p>\n<p>The Name Node is connected to multiple Data Nodes, which use standard or low-end hardware to store the Big Data. But why low-end hardware is used instead of standard cloud servers?</p>\n<p>Using low-end hardware for Data Nodes is a cost-effective solution compared to standard cloud servers. An enterprise server version costs around $10k per terabyte with full processing power. In the case of BIG Data, where 100 TBs of data is not uncommon, standard cloud servers would require hundreds of servers and cost millions of dollars.</p>\n<h4>2. MapReduce</h4>\n<p>MapReduce is a framework that provides support for parallel processing in Hadoop. While discussing the challenges with BIG Data processing, we mentioned the processing speed would be very slow as the data is enormous. But what if we distribute the processing across multiple systems and later combine the results? This would be highly efficient as compared to traditional ways. That’s what Hadoop does with the help of the MapReduce technique.</p>\n<p>There are two components in MapReduce: the <strong>Map</strong> task and the <strong>Reduce</strong> task. The map task takes the input data and transforms it for key-value operations. For example, in the dataset below, we need to count the number of dogs, cats, and mice entries in the list.</p>\n<p><img src=\"https://ucarecdn.com/150cc3e6-49b7-46b8-acb2-39700ed31bfe/\" alt=\"How MapReduce works in Hadoop and what are the map and reduce tasks?\" title=\"Map and Reduce task example in MapReduce Hadoop\"></p>\n<p>Hadoop first splits the entire list into smaller sets and stores them in different low-end hardware. Then at the Map stage, each data sample is represented as a ‘key’, and the value of ‘1’ is given to it. This is the same as the dictionary data type in Python {‘key’: ‘1’}. Then it operates on these key-value pairs; for example, sorting is one kind of operation on this key-value pair. After performing operations, it brings similar keys together. Finally, aggregation of results from parallel processing happens at the reduce phase, and the final output is prepared. That’s MapReduce for us.</p>\n<h4>3. Yet Another Resource Negotiator (YARN)</h4>\n<p>In traditional data analysis and pre-processing methods, we were not thinking about the available resources. For example, while filtering the outliers, we were not concerned about the CPU, GPU or RAM usage. But in BIG Data, we need some mechanism to optimize these resources smartly, and Hadoop has a solution.</p>\n<p>YARN (Yet Another Resource Negotiator) is the management module for Hadoop. It is designed to coordinate among large numbers of machines working together in a Hadoop cluster. YARN continuously monitors and manages the cluster nodes and systematically allocates resources to prevent any system from becoming overloaded. It also schedules jobs and tasks that need to be performed on Big Data, ensuring efficient use of resources and smooth operation of the system.</p>\n<p><img src=\"https://ucarecdn.com/86e073aa-6950-46b8-83e4-df78f189774b/\" alt=\"What are the components of YARN in Hadoop?\" title=\"YARMN components\"></p>\n<p>It comprises four main components: <strong>1. the resource manager</strong> <strong>2. the node manager</strong> <strong>3. the application master</strong>, and <strong>4. the containers</strong>.</p>\n<ul>\n<li><strong>Containers:</strong> It holds physical resources such as CPU, RAM, and ROM.</li>\n<li><strong>Resource Managers:</strong> We, as clients, request the resource manager to perform a specific operation or analysis on Big Data.</li>\n<li><strong>Application master:</strong> When the resource manager receives the job, the application master requests the necessary resources from the node manager.</li>\n<li><strong>Node Manager:</strong> The node manager manages resources within the containers and allocates them to the resource manager when they become available.</li>\n</ul>\n<p>The smooth functioning of all these components in YARN helps Hadoop perform operations efficiently and effectively.</p>\n<h4>4. Hadoop Common</h4>\n<p>Hadoop Common is the central component of the Hadoop framework. It includes all the standard utilities, libraries and APIs that support YARN, HDFS, and MapReduce. It also contains the necessary Java libraries to start the Hadoop system. </p>\n<p>Hadoop Common also includes the documentation and source code of the framework and a section for contributions showcasing software built on top of Hadoop. It is freely available to the public, and one can visit the <a href=\"https://github.com/apache/hadoop\">GitHub repository for Hadoop</a> for more detailed information.</p>\n<p>Now let’s combine the working of all these modules and see how it works.</p>\n<h3>How Hadoop works?</h3>\n<p>Let’s understand the complete picture of the working of Hadoop step-wise. Please remember that there are three main challenges of Big Data, and Hadoop is trying to solve them: <strong>Bigger size</strong>, <strong>single heavy process</strong>, and <strong>resource wastage</strong>. Let’s see how Hadoop will help:</p>\n<ul>\n<li>Hadoop will split the entire data into smaller datasets and store them in distributed file systems over multiple low-end hardware (HDFS). By default, the data size that can be stored in one hardware is 128 MBs. There will be various HDFS clusters where it will replicate each dataset 3 times (by default) to avoid any cases of data loss because of hardware failure.</li>\n<li>As the dataset is enormous, we can not run the entire operation at once. For example, suppose a process on one data sample takes 1 milli-second, and in Big Data, we have 100 Million data samples. Then if we run our code on the entire data, it will take <strong>1/1⁰³ * 100 Million seconds,</strong> equivalent to <strong>28 hours of a continuous process.</strong> If any data sample from 100 Million samples is corrupt, we will have to perform the entire process again from the start. With MapReduce’s help, Hadoop solves this issue by performing parallel processing on each hardware and finally aggregates the results provided by each HDFS and stores the result in different hardware. This is the job of the MapReduce module.</li>\n<li>Whenever a client requests to operate on data, resource managers efficiently allocate sufficient resources from the clusters and avoid any case of overloading on any hardware.</li>\n<li>After completing the reduce task for all the data nodes, the output is written back to HDFS.</li>\n</ul>\n<h3>Which companies use Hadoop Systems?</h3>\n<p>Hadoop is widely used by tech companies that collect and manage large amounts of data. Some well-known companies that use Hadoop systems include Google, Meta, Netflix, Uber, Twitter, British Airways, and the US National Security Agency. As a result, Hadoop has become an essential tool for data scientists working at these companies.</p>\n<p>If Hadoop is so efficient and charming, why don’t we use Hadoop everywhere? Let’s understand the reason for that.</p>\n<h3>What are the various applications built over the Hadoop framework?</h3>\n<p>Some of the most popular components present in the Hadoop ecosystem are:</p>\n<h4>HBase</h4>\n<ul>\n<li>Apache HBase was designed with two primary goals: 1. Hosting large tables (Billions of rows X Millions of columns) on low-end hardware, 2. Real-time read or write access for the vast dataset.</li>\n<li>It is a non-relational (NoSQL) distributed database built on top of the Hadoop Distributed File System (HDFS).</li>\n<li>It provides fast, horizontally scalable, fault-tolerant BIG Data access. Horizontal scaling means adding additional machines into the cluster, and vertical scaling means adding extra power (CPU/GPU memory, RAM, etc.) to the existing system.</li>\n<li>Some famous use cases of HBase are online analytics, time-series data, and real-time data processing.</li>\n</ul>\n<p>HBase provides a high-performance, low-latency, and flexible data model, making it an attractive option for Big Data storage and management.</p>\n<h4>Hive</h4>\n<ul>\n<li>Apache Hive is a Data warehouse built on top of the Hadoop framework to store and process massive datasets efficiently. A warehouse can be correlated with a central information store, using which we can perform analysis efficiently.</li>\n<li>Hive supports querying and analyzing large datasets stored in HDFS using HiveQL, a SQL-like query language. This SQL-like interface makes it preferable for data scientists and analysts as they only need to learn some of the new frameworks to perform queries on databases.</li>\n<li>It is designed to handle enormous datasets (Petabytes) and support distributed computing using MapReduce.</li>\n<li>Hive is also extensible, allowing developers to add custom functions to support their specific data processing needs.</li>\n</ul>\n<h4>Cassandra</h4>\n<ul>\n<li>Apache Cassandra is the only distributed NoSQL database management system that provides an always-on availability. Availability in databases refers to the condition when an organization’s data is available for the end-users.</li>\n<li>It allows organizations to store large amounts of data across multiple servers with no single point of failure.</li>\n<li>\n<p>It has a proven linear scalable nature, which means we can anytime scale horizontally or vertically as per the user’s traffic in a linear manner. One example of a linearly scalable system is shown in the image below, where Netflix uses Cassandra to scale the system as the write-traffic increases linearly (horizontally).</p>\n<p><img src=\"https://ucarecdn.com/34f33685-b6bd-4d72-9cf1-a8c5fabaceed/\" alt=\"Netflix uses Cassandra&#x27;s linear scalable property to manage the surge in traffic. \" title=\"Cassandra linear scalability\"></p>\n</li>\n</ul>\n<p>Source: <a href=\"https://netflixtechblog.com/benchmarking-cassandra-scalability-on-aws-over-a-million-writes-per-second-39f45f066c9e\">Netflix Blog</a><img src=\"\"></p>\n<ul>\n<li>It can be easily integrated with other Big Data technologies like Apache Hadoop and Apache Spark.</li>\n</ul>\n<h4>Pig</h4>\n<p>While discussing MapReduce in Hadoop, we saw how Map and Reduce tasks are performed on BIG Data to do the parallel processing. We discussed a simple use-case of sorting there, but in actual, these tasks can be complex to write.</p>\n<ul>\n<li>Apache Pig is a high-level platform for creating MapReduce programs that process and analyze large data sets. It provides an easier-to-use interface for writing complex data transformations, such as aggregation, summarization, and sorting, which can be difficult to express using MapReduce alone.</li>\n<li>Pig is often used for big data processing, mining, and machine learning tasks. It allows developers to express complex data flows and logic in a simple script.</li>\n<li>Pig also supports parallel processing on large datasets, making it efficient and effective for big data analytics. As a result, Pig is a popular tool among data engineers and data scientists working with big data.</li>\n</ul>\n<h4>Spark</h4>\n<p>One challenge with Hadoop was the need for real-time analytics support. Hadoop only supports batch processing and the need for a technique better suited for real-time data processing, leading to the development of Apache Spark.</p>\n<ul>\n<li>Apache Spark has an advanced analytical engine, making it 100 times faster than native Hadoop.</li>\n<li>It is widely used for fast data analysis and applying machine learning algorithms on large datasets, making it a popular choice for Big Data processing.</li>\n</ul>\n<p>We will learn about Apache Spark and Pyspark in our next blog.</p>\n<h3>Conclusion</h3>\n<p>Hadoop has become a valuable tool for companies that handle large amounts of data. It offers benefits such as scalability, storage, speed, and analytics. In this article, we provided an overview of the basics of Hadoop and how it works. We also highlighted some popular applications built on top of the Hadoop framework that supports various use cases of Big Data. We hope you found the information helpful and enjoyed reading the article.</p>\n<h4>References</h4>\n<ol>\n<li><a href=\"https://aws.amazon.com/emr/details/hadoop/what-is-hadoop/\">https://aws.amazon.com/emr/details/hadoop/what-is-hadoop/</a></li>\n<li><a href=\"https://hadoop.apache.org/docs/stable/index.html\">https://hadoop.apache.org/docs/stable/index.html</a></li>\n</ol>\n<h4>Enjoy Learning!</h4>"},{"id":"9d78cf9f-86e0-5964-b272-00b5405d6ab5","frontmatter":{"title":"Master-Slave Replication (Single Leader Replication)","path":"/blog/master-slave-replication-databases"},"html":"<p>In <a href=\"https://www.enjoyalgorithms.com/blog/introduction-to-database-replication-system-design/\">database replication</a>, we create copies of the database across multiple nodes. But with multiple replicas, a question arises: How do we ensure that all the data ends up on all the replicas after every write? The most common solution is master-slave replication, also known as <strong>active/passive</strong> or <strong>single-leader replication.</strong></p>\n<h3>What is Master-Slave Replication?</h3>\n<p>There are two types of nodes in Master-Slave replication: Master and Slave. The single <strong>master (leader)</strong> node works as the primary database, while one or more <strong>slave (follower</strong>) nodes maintain copies of the Master’s data.</p>\n<ul>\n<li>The master node is responsible for handling write queries while slave nodes are responsible for handling read queries. Note: If required, the master node can also perform read queries.</li>\n<li>Whenever a write operation is performed on the master node, it is replicated to all the slave nodes to keep the data consistent across the entire system.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/1abbe9bf-f823-468b-b230-408f74aa0aa1/\" alt=\"How master-slave replication works?\" title=\"Master Slave Replication in DBMS\"></p>\n<h4>The idea of master-slave replication works as follows:</h4>\n<ul>\n<li>When clients want to perform a write query, they send their requests to the leader, which first writes the new data to its local storage.</li>\n<li>Whenever the leader writes new data to its local storage, it also sends the data change to all of its followers as part of a replication log. Each follower takes the log from the leader and updates its local copy by applying all write in the same order as they were processed on the leader.</li>\n<li>If only one slave database is available and it goes offline, read operations will be temporarily directed to the master database, and a new slave database will replace the old one. If multiple slave databases are available, read operations are redirected to other healthy slave databases and a new database server will replace the old one.</li>\n<li>If the master database goes offline, a slave database will be promoted to be the new master. Now, all write operations will be executed on the new master database, and a new slave database will replace the old one for data replication immediately. <strong>Note:</strong> Promoting a new master is more complicated as the data in a slave database might not be up to date. How to update the missing data? We will discuss this idea in the later section of this blog.</li>\n</ul>\n<h3>Use cases of master slave replication</h3>\n<p>Master-Slave replication is used in situations where the workload is read-heavy and there is a need to distribute read requests across multiple nodes to improve performance. Since slave nodes can handle read requests, they can offload read traffic from the master and allow it to focus on processing write requests.</p>\n<ul>\n<li>This architecture provides fault tolerance i.e. the system can continue functioning even if the master or any slave node fails.</li>\n<li>Master-Slave replication is used in relational databases like MySQL, PostgreSQL, and Oracle, as well as in NoSQL databases like MongoDB, Cassandra, RethinkDB, Espresso, etc.</li>\n<li>Leader-based replication is not restricted to only databases. Distributed message brokers like Kafka and RabbitMQ also use it to provide highly available queues.</li>\n</ul>\n<h3>Synchronous Vs Asynchronous Master-Slave Configuration</h3>\n<p>A critical aspect in master-slave replication is to consider whether replication happens synchronously or asynchronously. To understand this, let’s consider what happens when a user updates their profile image on a social media website: the client sends an update request to the leader. As soon as the leader receives the request, it forwards the data change to the followers. Now, How do the followers update their data?</p>\n<p>In synchronous replication, the client waits for confirmation from the leader that the update has been applied to all followers before receiving the response. In asynchronous replication, the client receives the response before all followers have been updated.</p>\n<h4>Advantages and disadvantages of synchronous replication</h4>\n<p><strong>Disadvantage:</strong> If the follower doesn’t respond (due to a crash, network fault, or any other reason), the write cannot be processed.</p>\n<p><strong>Advantage:</strong> The follower is guaranteed to have an up-to-date copy of the data that is consistent with the leader. If the leader suddenly fails, we can be sure that the data is still available to the follower.</p>\n<p><img src=\"https://ucarecdn.com/7703fb87-6e75-4c6e-9739-d63e4a2da8e4/\" alt=\"How synchronous replication works in master-slave configuration?\" title=\"synchronous replication in leader-based configuration\"></p>\n<h4>Advantages and disadvantages of asynchronous replication</h4>\n<p><strong>Disadvantage:</strong> If the leader fails and is not recoverable, any writes that have not yet been replicated to followers are lost i.e. a write is not guaranteed to be durable, even if it has been confirmed to the client.</p>\n<p><strong>Advantage:</strong> A leader can continue processing writes, even if all of their followers have fallen behind. Weakening durability may sound like a bad trade-off, but asynchronous replication is nevertheless widely used, especially if many followers are geographically distributed.</p>\n<p><img src=\"https://ucarecdn.com/ed0f25e6-bddb-463d-9bf7-6b928b1335bc/\" alt=\"How asynchronous replication works in master-slave configuration?\" title=\"asynchronous replication in leader-based configuration\"></p>\n<p>Overall, the choice of synchronous or asynchronous replication depends on the <strong>trade-offs between consistency and performance</strong>. Synchronous replication provides stronger consistency guarantees, but it can also result in longer response times. On another side, asynchronous replication provides faster response times but may compromise consistency.</p>\n<h4><strong>Critical ideas to think in asynchronous replication</strong></h4>\n<ul>\n<li>It can be a serious problem in asynchronous replication to loose data if the leader fails. For this reason, researchers have continued to investigate replication methods that provide good performance and availability without sacrificing data integrity. One example of such a method is <strong>chain replication</strong>, which is used in Microsoft Azure Storage.</li>\n<li>When a user makes a write, the new data may not immediately reach all followers. If the user tries to read the same data immediately after making the write, it may not be available to the follower, leading the user to think that their data was lost. To handle this situation, one solution is to read from the leader when reading something that the user may have modified. Otherwise, read from a follower. For example, in a social media, profile info is usually editable only by the user. So, it’s a good idea to always read a user’s own profile from the leader and any other user’s profile from a follower.</li>\n</ul>\n<h3>A common ground: Semi-synchronous replication</h3>\n<p>In most cases, replication is fast and changes made to the leader are quickly propagated to the followers. But there are certain situations where replication can become delayed. For example, if a follower is recovering from a failure, the system is operating at maximum capacity, or there are network problems between the nodes.</p>\n<p>On the other hand, synchronous replication requires the leader to block all write operations until the followers have acknowledged that the data has been received. This can cause delay and performance issues if all followers are synchronous. If one of the replicas goes offline, the entire system would be affected.</p>\n<p>A good solution would be <strong>semi-synchronous replication</strong>. In this setup, one follower is designated as synchronous, while the others are asynchronous. Here synchronous followers will update all data changes in real time and all asynchronous followers will update data eventually in the background. If the synchronous follower goes offline or slows down, one of the asynchronous followers can be promoted to take its place.</p>\n<p>This configuration ensures that at least two nodes, the leader and one synchronous follower, have up-to-date copies of the data. Let’s understand this via an example: Suppose a user updates their profile image on the website and there are one leader and two followers. Here replication to follower 1 is synchronous and replication to follower 2 is asynchronous.</p>\n<p><img src=\"https://ucarecdn.com/9aca87e9-3c24-4b0a-9e42-497c4e03ca34/\" alt=\"How semi-synchronous replication works in master-slave configuration?\" title=\"semi-synchronous replication in leader-based configuration\"></p>\n<ul>\n<li><strong>In the case of follower 1:</strong> The leader waits until follower 1 has confirmed that it received the write before reporting success to the user, and before making the write visible to other clients. </li>\n<li><strong>In the case of follower 2:</strong> The leader sends the message but doesn’t wait for a response from follower 2. The diagram shows that there is a substantial delay before follower 2 processes the message.</li>\n</ul>\n<h3>Setting up new followers in master slave replication</h3>\n<p>Suppose we want to increase the number of followers or replace a failed follower node. How can we do this? How can we ensure that the new follower has an accurate copy of the leader’s data? Simply copying data from the leader node to the new follower node is not enough to guarantee data consistency because clients are continuously writing to the database. In other words, a standard data copy would see different parts of the database at different points in time.</p>\n<p>One solution is to lock the leader or master database to ensure consistency during the copying process. But this approach would not support the goal of high availability since it would make the leader database unavailable for writes during the copy process.</p>\n<p>Fortunately, there is a way to set up a new follower without any downtime. Here are the steps:</p>\n<ol>\n<li>Take a <strong>consistent snapshot</strong> of the leader database at a specific point in time. Most databases have this feature to take consistent snapshots without locking the entire database.</li>\n<li>Copy the snapshot to the new follower node. This ensures that the new follower has a complete and accurate copy of the leader data at the time the snapshot was taken.</li>\n<li>Connect the follower to the leader and request all the data changes that have occurred since the snapshot was taken. The new follower associate the snapshot with a specific position in the leader <strong>replication log</strong> to request all the data changes since that point. This position is known by different names in different databases like the log sequence number in PostgreSQL and the binlog coordinates in MySQL.</li>\n<li>Finally, the new follower will process the backlog of changes since the snapshot. This process is complete once the new follower has caught up with the leader and can continue processing data changes.</li>\n</ol>\n<p>The practical steps involved in setting up a follower can vary significantly depending on the DBMS system. In some systems, the process is fully automated, while in others, it may require a multi-step workflow that must be manually performed by an administrator.</p>\n<h3>Handling node failure and outage in master slave replication</h3>\n<p>Due to faults or errors, any node in a system can go down. So our goal is to keep the system running despite individual node failures and to minimize the impact of a node outage. The critical question is: How can we achieve high availability and reliability with leader-based replication? Let’s discuss this scenario separately in case of follower and leader failure.</p>\n<h4>Handling follower failure </h4>\n<p>Each follower maintains a log of the data changes it has received from the leader. This log helps the follower to identify the last transaction processed before the fault occurred. So if a follower fails, it can connect to the leader and request all the data changes that occurred during the time when it was disconnected. Once it has applied these changes, it will have caught up to the leader and can resume receiving a stream of data changes as before.</p>\n<h4>Handling leader failure</h4>\n<p>This is a little trickier and requires three critical steps: 1) Detecting the failure of the leader node 2) Promoting one of the followers as a new leader 3) Configuring clients to send their writes to the new leader, and other followers to start consuming data changes from the new leader. This process is also called <strong>failover.</strong></p>\n<p><strong>Step 1: Detecting leader failure:</strong> This can happen due to various reasons (crashes, power outages, network issues, etc). Since there is no foolproof way to detect the cause of failure, a timeout is used to assume that a leader node is dead if it doesn’t respond for a certain period of time, typically less than 30 sec or 1 minute.</p>\n<p><strong>Step 2: Choosing a new leader:</strong> This can be done through an election process where a new leader is chosen by a majority of the remaining replicas. To minimize data loss, the replica with the most up-to-date data changes from the old leader is usually chosen as the new leader. But getting all the nodes to agree on a new leader is a consensus problem. <strong>Note:</strong> We will discuss the idea of <strong>the consensus problem</strong> in a separate blog.</p>\n<p><strong>Step 3: Reconfiguring the system to use the new leader:</strong> Clients need to send their write requests to the new leader. If the old leader comes back online, it might still believe that it is the leader, unaware that it has been replaced by a new leader. The system needs to ensure that the old leader becomes a follower and recognises the new leader.</p>\n<p><strong>Note:</strong> In case of semi-synchronous replication, we make the synchronous slave as a new master since we know that it is the most updated one and no data will be lost.</p>\n<h4>Some critical challenges in handling leader failure!</h4>\n<ul>\n<li>There can be a situation in asynchronous replication, where the new leader has not received all the writes from the old leader before it fails. When the former leader rejoins the cluster after a new leader has been chosen, there may be a possibility of conflicting writes. The critical question is: What should happen to those writes? Think and explore!</li>\n<li>In some cases, it is possible for two nodes in a distributed system to believe that they are the leader, which is known as a <strong>split-brain scenario</strong>. This situation can be quite tricky because both leaders may accept writes without any process for resolving conflicts, which can lead to data loss or corruption. How to prevent or handle this split-brain scenario?</li>\n<li>What is the right timeout before the leader is declared failed? A longer timeout means a longer time to recover in the case where the leader fails. However, if the timeout is too short, there could be unnecessary failovers. For example, a temporary increase in load could increase the response time of the node to more than timeout, or a network glitch could cause delayed packets. If the system is already struggling with high load or network problems, an unnecessary failover is likely to make the situation worse, not better.</li>\n<li>There are no easy solutions to these problems. For this reason, some operations teams prefer to perform failovers manually, even if the software supports automatic failover. So, is it better to use Multi-leader replication instead? Problems like node failures, unreliable networks, and trade-offs around replica consistency, durability, availability, and latency are fundamental problems of study in distributed systems.</li>\n</ul>\n<p><strong>References</strong></p>\n<ul>\n<li>Designing data-intensive applications by Martin Kleppmann </li>\n<li>Web Scalability for Startup Engineers by McGraw-Hill </li>\n<li>Understanding distributed systems by Roberto Vitillo</li>\n</ul>\n<p>We will keep updating this blog with more insights on master-slave replication. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"1a5b753b-4e05-515a-87ee-c3e109b5bcc3","frontmatter":{"title":"CAP Theorem in Distributed Database Systems","path":"/blog/cap-theorem-in-system-design"},"html":"<p>Almost a couple of decades ago, Eric Brewer introduced the idea that there exists a trade-off among consistency, availability and partition tolerance. This trade-off is known as the CAP Theorem! CAP theorem is an an important concept that helps architects in designing distributed database systems.</p>\n<h3>What is CAP Theorem?</h3>\n<p>The CAP theorem states that distributed databases can have at most two of the three properties in a given time: Consistency, Availability and Partition Tolerance. In other words, it is impossible for a distributed computer system to simultaneously guarantee all three properties.</p>\n<ol>\n<li><strong>Consistency</strong> means that all nodes should have the same data at any given time. When a client performs a read operation, they should receive the most recent write value. So to maintain consistency, the node where the write operation is performed has the responsibility to replicate the data to all other nodes. If there is a failure to maintain consistency, there will be some outdated data in the application.</li>\n<li><strong>Availability</strong> means that a distributed database system should always be able to respond to client requests i.e. every request should receive a response, without the guarantee that it contains the most recent write. So the system remains operational even in the event of failures. Some techniques to achieve high availability are redundancy, replication, load balancing, etc.</li>\n<li><strong>Partition tolerance</strong> means that a distributed database system should continue operating even if there is a network partition. So what is network partition? Sometimes due to disruption in the network (network failures or other factors), nodes get separated into groups so that they can not communicate with each other. This is called network partition.</li>\n</ol>\n<p>Network failures can disrupt the system's ability to access and update the data. So network partitions are unavoidable and partition tolerance is a must-have requirements in distributed DBMS. That's why most of the time, we need to choose between consistency (all nodes see the same data) or availability (the system remains operational and responsive).</p>\n<p>Maintaining consistency can come at the cost of increased latency and reduced availability, because system may need to wait for all nodes to agree on the current state before responding to client requests. On the other side, maintaining availability can lead to data inconsistencies because nodes may return stale data.</p>\n<h3>CAP Theorem Database Architecture</h3>\n<p>NoSQL databases are highly distributed and provide horizontal scalability. They can rapidly scale across a growing network of multiple interconnected nodes. Due to this, the idea of the CAP theorem is highly relevant in the case of NoSQL databases! </p>\n<p>But as discussed above, one can only have two of the three available functionalities. So based on various choices, here are three different systems based on the CAP theorem.</p>\n<ul>\n<li><strong>CP System:</strong> This focuses more on consistency and partition tolerance. So these systems are not available most of the time. When any issue occurs in the system, it has to shut down the non-consistent node until the consistency problem is resolved, and during that time, it is not available.</li>\n<li><strong>AP System:</strong> This focuses more on availability and partition tolerance rather than consistency. When any issue occurs in the system, then it will no longer remain in a consistent state. However, all the nodes remain available, and affected nodes might return a stale data, and the system will take some time to become eventually consistent.</li>\n<li><strong>CA System:</strong> This focuses more on consistency and availability across all nodes rather than partition tolerance. Fault-Tolerance is the basic necessity of any distributed system. So it is almost rare to use a CA type of architecture for any practical purpose.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/55d3d7cb-3f95-48ba-88d6-b9b367d104ae/\" alt=\"What is CAP Theorem?\" title=\"CAP Theorem\"></p>\n<h3>Use Cases of CAP Theorem</h3>\n<ul>\n<li>MongoDB focuses on a CP database style. By default, it maintains consistency while compromising on availability: If you read, then you will always get the most recent value of write. There is a reason: By default, MongoDB is a single-leader system, where all reads will go to the leader. If required, MongoDB provides the option to enable read operations from the follower nodes. In that case, MongoDB becomes eventually consistent and it will work as an AP system!</li>\n<li>Cassandra focuses on an \"AP\" database style, which concentrates entirely on availability and partition tolerance rather than consistency. In other words, Cassandra will prioritise the ability to always respond to client requests and provide eventual consistency.</li>\n<li>Microservices-based applications often rely on the CAP theorem to design the most efficient databases. For example, if horizontal scalability is essential with eventual consistency, an AP database like Cassandra can help meet deployment requirements. On the other hand, if the application depends heavily on data consistency like a payment service, it may be better to opt for a relational database like PostgreSQL, which focuses on a CP database style.</li>\n</ul>\n<p><strong>Note:</strong> Some modern database systems rarely fall completely into any one of these categories because they provide configurations to select replication methods, consistency level, etc. This can have an effect on all three parameters. The truth is: Most of the systems have the goal to achieve all combinations of these three states. So rather than classify a database system as CP or AP, we need to think in terms of the options they provide for tuning these properties, based on our use case.</p>\n<h3>Conclusion</h3>\n<p>The CAP theorem provides an understanding of the challenges and trade-offs involved in designing distributed database systems. It guides decision-making, database selection, and system design to achieve the desired balance between consistency, availability, and partition tolerance.</p>\n<p>Additional reading: <a href=\"https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/brewer-cap.pdf\">CAP Twelve Year Later: How rules have changed?</a></p>\n<p>Thanks to Suyash for his contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"32b97ae2-71e5-5d37-86db-24acccb25816","frontmatter":{"title":"Introduction to Caching and Caching Strategies","path":"/blog/caching-system-design-concept"},"html":"<p>Have you ever experienced a website taking longer to load on the first visit but significantly faster on subsequent visits? This phenomenon is related to caching: an idea to improve the performance of distributed systems. In this blog, we will discuss the caching concept, how it works and its types.</p>\n<h3>What is caching?</h3>\n<p>Caching is the process of storing frequently accessed data in a temporary storage called cache to improve the speed of data access. Now, the question is: What is a cache? A cache is a high-speed storage that stores a small proportion of critical data so that requests for that data can be served faster.</p>\n<ul>\n<li>Accessing data from the main database will take a considerable time (due to disk I/O or network latency). So, when data is accessed for the first time, system will store that data in a faster memory, i.e., the cache. If there is a further request for the same data, system will serve it directly from the cache.</li>\n<li>So one of the critical goals of caching is to make data retrieval fast by accessing data from the cache instead of retrieving it again from the main database. This will significantly improve performance.</li>\n<li>Cache memory is costly, so its size is much smaller compared to the size of the main database. So, what type of data should we store in the cache? One basic idea is to store frequently accessed data.</li>\n</ul>\n<h3>How caching works?</h3>\n<p>Now, let's understand how caching works in practice when a user requests some data.</p>\n<ul>\n<li>The system will first check the cache to see if the requested data is already there. If the data is present (<strong>cache hit</strong>), the system will return the data from the cache.</li>\n<li>If the requested data is not present in the cache (<strong>cache miss</strong>), the system will retrieve the data from the original database. As discussed earlier, this process is slower compared to cache access.</li>\n<li>After fetching data from the database, the system will store the fetched data in the cache for future use so that subsequent requests for the same data can be served more quickly.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/1080/1*KNlUFMg72Ziy4QbbPpOwcw.png\" alt=\"What is caching in system design?\" title=\"How caching works in system design?\"></p>\n<p>The above idea is one of the commonly used caching strategies: <strong><a href=\"https://www.enjoyalgorithms.com/blog/cache-aside-caching-strategy/\">cache-aside strategy</a></strong>. Here cache and database are independent, and it is the responsibility of the application code to manage operations on the cache and database. There can be other types of caching strategies like read-through strategy, write-around strategy, write-back strategy, and write-through strategy. We will discuss these concepts later in this blog.</p>\n<p><strong>Note:</strong> Cache hits and misses are critical metrics for measuring cache performance.</p>\n<ul>\n<li>A high cache hit rate means the cache is effectively storing and retrieving the data.</li>\n<li>A high cache miss rate implies that we are not using the cache effectively. In this situation, we can consider adjusting the cache size or using some other techniques to minimize cache misses.</li>\n</ul>\n<h3>Real-life examples of caching</h3>\n<ul>\n<li><strong>Web browsers</strong> use caching to store frequently accessed HTML, CSS, JavaScript and images.</li>\n<li><strong>Content Delivery Networks (CDNs)</strong> store static files like images and videos and serve them from locations closer to the user. This reduces the time it takes to retrieve data and reduces latency.</li>\n<li><strong>DNS caching</strong> helps to improve the speed of accessing web pages by storing the IP address of a domain name in a cache. In other words, instead of performing a DNS query every time, an IP address can be quickly retrieved from the cache.</li>\n</ul>\n<h3>Cache Eviction Policy</h3>\n<p>Cache eviction policies are algorithms that manage data stored in a cache. When the cache is full, some data needs to be removed in order to make room for new data. So cache eviction policy determines which data to remove based on certain criteria. There are several cache eviction policies:</p>\n<ul>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/implement-least-recently-used-cache/\">Least Recently Used (LRU)</a>: Removes least recently used data.</li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/least-frequently-used-cache/\">Least Frequently Used (LFU)</a>: Removes least frequently used data.</li>\n<li>Most Recently Used (MRU): Removes most recently used data.</li>\n<li>Random Replacement (RR): Randomly selects a data item and removes it to make space.</li>\n<li>First In First Out (FIFO): This algorithm maintains a queue of objects in the order that they were added into the cache. When a cache miss occurs, one or more objects are removed from the head of the queue, and a new object is inserted at the tail.</li>\n</ul>\n<h3>Solution of Cache Invalidation using various Caching Strategies</h3>\n<p>When data in the database is constantly being updated, it is important to ensure that the cache is also updated to reflect these changes. Otherwise, the application will serve outdated or stale data. So, we use cache invalidation techniques to maintain the cache consistency.</p>\n<p>We mostly use five types of caching strategy to solve the problem of cache invalidation: Cache-aside strategy, Read-through strategy, Write-through strategy, Write-around strategy and Write-back strategy. We have already discussed the cache-aside strategy in the above section. So let's discuss the rest of them.</p>\n<h4><a href=\"https://www.enjoyalgorithms.com/blog/read-through-caching-strategy/\">Read-through strategy</a></h4>\n<p>In this strategy, cache is present between the application code and the database. In case of a cache miss, it retrieves data from the database, stores that data in the cache and returns it to the application. So read-through caches are also one of the good choices for read-heavy systems.</p>\n<p>Both cache-aside and read-through look similar, but there are differences. In cache-aside, application code is responsible for retrieving data from the database and storing it in the cache. In read-through, this is supported by the cache provider. So read-through strategy simplifies the application code by abstracting away the complexity of cache management.</p>\n<p>In both strategies, the most basic approach is to write directly to the database. Due to this, cache will be inconsistent with the database. So what is the solution? </p>\n<ul>\n<li>One idea is to use the time to live (TTL), so cache will continue serving stale data until expiry of TTL.</li>\n<li>If we need to ensure data consistency, we can follow the write strategy discussed below.</li>\n</ul>\n<h4><a href=\"https://www.enjoyalgorithms.com/blog/write-through-caching-strategy/\"><strong>Write through</strong> strategy</a></h4>\n<p>In write-through cache, writes are first made to the cache and then to the database. If both writes are successful, write operation is considered successful. This will ensure that cache and database remain consistent and reduces the risk of data loss in case of database crash.</p>\n<ul>\n<li>Write-through cache comes with the trade-off of increased write latency. This is because data must be written to two separate places! So this strategy is not a good choice for write-heavy applications.</li>\n<li>This is best for applications that frequently re-read data. Despite the increased write latency, this approach provides consistent data and lower read latency.</li>\n</ul>\n<h4><strong>Write around</strong> strategy</h4>\n<p>In write-around cache, write operations bypass the cache and go directly to the database rather than being written to the cache first. This technique does not ensure that data in the cache is always up-to-date because updated data may not be available in the cache.</p>\n<ul>\n<li>This is used where write performance is more critical than read performance, or where write activity is heavy and data is not frequently accessed. In such cases, cache is only updated when the data is read. This will reduce the number of write operations on the cache and the risk of cache overloading.</li>\n<li>This is not suitable for applications that frequently write and re-read the most recent data. In this situation, cache misses will occur frequently, which can cause slower read times or higher read latency.</li>\n</ul>\n<h4><strong>Write back</strong> strategy</h4>\n<p>The write-back cache is used in systems with high write activity to improve write performance. Writes are temporarily stored in a cache layer, where they are quickly verified and then asynchronously written to the database. This results in lower write latency and higher write throughput.</p>\n<p>However, this technique carries the risk of data loss if cache layer fails, because cache is the only copy of the written data. To minimize this risk, it is recommended to have multiple cache replicas that acknowledge the write. This way, if one cache fails, data can still be recovered from another cache.</p>\n<h3>Different types of caching used in system design</h3>\n<h4>Browser caching</h4>\n<p>If you're wondering how websites load quickly on subsequent visits, one of the reasons is browser caching. This idea temporarily stores resources like images, HTML and JavaScript files within a cache in a web browser. When you revisit the same website, the browser will retrieve these resources from the cache. This is also known as client-side caching.</p>\n<p>Browser cache has limited capacity and is set to store resources for a specific time duration. When the cache reaches its capacity or resources reach their expiration date, the browser will automatically clear the cache and retrieve updated resources from the network during the next visit. Users can also manually clear their browser cache if it becomes full.</p>\n<h4>Web server caching</h4>\n<p>Web server caching is used to improve performance by storing resources on the server side. This reduces the load on the server. There are several ways to implement web server caching i.e. reverse proxy cache and key-value store such as Memcached or Redis.</p>\n<p>The reverse proxy cache acts as an intermediary between the browser and the web server. When a user makes a request, the reverse proxy cache checks if it has a copy of the requested data. If yes, it will serve a cached version to the user rather than forwarding the request to the web server.</p>\n<p>We can also use key-value database to cache application data. These databases are typically accessed by the application code. Unlike reverse proxies, which cache HTTP responses for specific requests, key-value databases can cache any user-specific data or frequently accessed data based on need.</p>\n<h4>Content Delivery Network (CDN)</h4>\n<p>Content Delivery Network is designed to improve the delivery speed of static content like web pages, images, videos, and other media files. These proxy servers are located in strategic locations around the world to reduce the distance between the end user and the origin server (reducing latency).</p>\n<p>When a user requests content from a website that uses a CDN, CDN fetches the content from the origin server and stores a copy of it. If the user requests the same content again, CDN serves content directly rather than fetching it again from the origin server.</p>\n<p>Think CDN like a chain of grocery stores: Instead of going all the way to farms where food is grown, which could be hundreds of miles away, customers can go to their local grocery store. Grocery store stocks food from faraway farms, allowing customers to get the food they need in a matter of minutes, rather than days.</p>\n<h4>Distributed caching</h4>\n<p>Distributed caching is the practice of using multiple caching servers that are spread across a network. Unlike traditional caches, which are usually limited to the memory of a single machine, distributed cache can scale beyond the memory limits of a single machine by linking together multiple machines (distributed clusters).</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/1080/1*KOu-FfSUpVHufIrvJfmPzQ.png\" alt=\"What is distributed caching in system design?\" title=\"How distributed caching works in system design?\"></p>\n<p>In distributed caching, each caching server maintains a portion of the cached data, and requests for data are directed to the appropriate server based on a hashing algorithm or some distribution strategy.</p>\n<ul>\n<li>Distributed caching is useful in an environment of high data access because distributed architecture can incrementally add more machines to the cluster. This will increase the cache size with the growth in data.</li>\n<li>Distributed caching provides several benefits: faster response times, scalability, increased availability, and better fault tolerance. It is used in large-scale applications and cloud-based architectures.</li>\n</ul>\n<h3>Advantages of caching</h3>\n<ul>\n<li>By storing frequently accessed data, caching reduces the need to retrieve data from slower data sources. This will result in faster access and low latency.</li>\n<li>Caching increases throughput because it provides much higher request rates compared to the main database. For example, a cache instance can serve hundreds of thousands of requests per second.</li>\n<li>Caching offloads the backend databases by serving frequently requested data from the cache. Due to this, database servers can handle more requests simultaneously. If required, cache layer can scale horizontally by distributing the workload across multiple cache servers. So caching improves scalability.</li>\n<li>By reducing the load on backend databases, caching reduces the overall cost of the system, especially if the backend database charges per throughput.</li>\n<li>By storing relevant data in the cache, users can continue to access data or content even when the network connection is unreliable or completely offline. This is particularly useful for mobile applications. For example, if you install the EnjoyAlgorithms website as a progressive web app on mobile, you can access the content even if you are offline.</li>\n</ul>\n<h3>Disadvantages and limitations of caching</h3>\n<ul>\n<li>When there is a data update in the main database, the cached copy will be outdated. So there is a risk of data inconsistency. For this, we need to carefully choose perfect cache invalidation or cache expiration strategies to ensure data consistency.</li>\n<li>Cache misses introduce latency! So it's important to keep them low compared to cache hits to maximize the benefits of cache. If cache misses are not well managed, the caching can become nothing more than overhead!</li>\n<li>When a cache is initially empty, it needs to \"warm up\" by populating it with frequently accessed data. This can cause temporary performance degradation because the cache needs time to build up its contents. During this time, users may experience slower response times until the cache becomes populated with the most accessed data.</li>\n<li>The caching only improves the performance of read operations. When it comes to the write operation, most of the time system needs to perform write on both cache and the main database. This process will not improve write performance.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>In summary, caching is a useful technique for improving performance, reducing cost, and increasing the scalability of a system by storing frequently accessed data in fast storage called cache!</p>\n<p>Thanks to Chiranjeev and Navtosh for their contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy system design!</p>"},{"id":"88857f4d-5eab-5507-9d5e-966d499ca1f9","frontmatter":{"title":"Caching Strategy: Read-Through Pattern","path":"/blog/read-through-caching-strategy"},"html":"<p>Unlike <a href=\"https://www.enjoyalgorithms.com/blog/cache-aside-caching-strategy/\">cache-aside pattern</a>, which requires application logic to fetch and populate the cache, Read-Through cache manages data retrieval from database on behalf of the application. Here application will interact with the cache system that acts as an intermediary between the application and database.</p>\n<p>One of the critical benefits of read-through pattern is the simplification of application code. It offloads the responsibility of handling cache misses and data fetching to the cache layer. Due to this, application will only interact with the cache as if it were the primary data source.</p>\n<h3>How read-through pattern works?</h3>\n<p>When an application requests data from the cache, cache will first check if data is present (cache hit). If yes, then cached data will be returned directly to the application. If data is not present (cache miss), cache will act as a mediator and retrieve data from the database. Now, cache will store the retrieved data and return it to the application as the result of the original read request. This will allow subsequent read requests for the same data to be served efficiently from the cache with low latency.</p>\n<p><img src=\"https://ucarecdn.com/4a02dc04-3572-413b-8d60-49e49b5d4683/\" alt=\"How read through caching pattern works?\" title=\"What is read through caching strategy?\"></p>\n<p>For writing data in read-through cache, we can follow these strategies based on the requirements: <a href=\"https://www.enjoyalgorithms.com/blog/write-around-caching-pattern/\">Write-around</a>, <a href=\"https://www.enjoyalgorithms.com/blog/write-through-caching-strategy/\">Write-through</a>, and <a href=\"https://www.enjoyalgorithms.com/blog/write-behind-caching-pattern/\">Write-back (or write-behind)</a> strategy. The choice of the best write approach depends on the tradeoff in terms of data consistency and write performance. Explore and think!</p>\n<h3>Advantages or benefits of Read-through cache</h3>\n<ul>\n<li>Read-through cache significantly reduces the read latency for frequently accessed data, especially when original data source is located remotely (geographically distant data centres). That’s why it is used in a system where data access pattern is read-heavy.</li>\n<li>With write-through pattern, read-through cache provides an automatic way to ensure data consistency. On the other side, read-through with write-around pattern does not need complete knowledge of all possible data that might be requested in the future. Instead, it can cache data as read requests come in (lazy loading). This will allow system to load data into the cache on demand and avoids filling up cache with data that isn’t requested.</li>\n<li>Since read-through caching serves data from the cache whenever possible, it reduces the number of direct requests to the underlying data source.</li>\n<li>We can configure the read-through cache to automatically reload data from the database when it expires or when corresponding data changes in the database. So in the situation of high traffic, there can be low chances of cache misses for these cache items.</li>\n<li>It simplifies application code and abstracts away the complexity of cache management and data retrieval. So, developers don’t need to implement complex logic to handle caching manually.</li>\n<li>In case of a node failure, we can easily replace the node with a new empty node, so application can continue to function. But latency will increase at the start because there will be a high chance of a cache miss for some initial read queries. One solution for this delay problem is to warm up the cache by populating data that is expected to be frequently requested in the near future.</li>\n</ul>\n<h3>Limitations or drawbacks of Read-through cache</h3>\n<ul>\n<li>Whenever a new read request comes or data is requested for the first time or after TTL has ended, it will always result in a cache miss. The idea is simple: In all such situations, read request will go to the database. This will add extra latency: Three network round trips 1) Check the cache 2) Retrieve from the database 3) Add data into the cache.</li>\n<li>With write-through caching, read-through caching can lead to the caching of data that is infrequently or never accessed again. This situation can consume valuable cache space that could have been used for more relevant and frequently accessed data.</li>\n<li>Just like cache-aside, it is also possible for data to become inconsistent between the cache and the database. For example, if there is a change in the database and cache key is not expired yet, it can throw stale data to the application. One solution is to use proper write strategies like write-through caching and configuring proper eviction strategy to ensure data consistency.</li>\n<li>As discussed above, we can automatically reload data in the cache from the database when it expires. The problem is: Simultaneous expiration of a lot of cache data can lead to a lot of database requests (cache stampede). These simultaneous requests cause a spike in load on the database. What is the solution? Think and explore!</li>\n</ul>\n<h3>Read-Through versus Cache-Aside Pattern</h3>\n<p>While read-through and cache-aside cache look very similar, there are some major differences:</p>\n<ul>\n<li>Read-through caching places the caching logic within the cache system itself and reduces the burden on application to manage caching logic. Cache-aside places caching logic within the application. The application decides when to cache data when to evict data, and how to handle cache misses. This will provide more control to the application.</li>\n<li>In read-through cache, application interacts with the cache as if it were the primary data source. Application is unaware of whether data is already cached or not, as cache manages this internally. In cache-aside, application must explicitly check the cache before accessing the data source.</li>\n<li>In cache aside pattern, if application does not handle cache updates correctly, there could be possibility of data inconsistencies. So it is the responsibility of application developer to write logic for cache invalidation or updating cached data correctly. On the other side, When data is updated in the data source, read-through cache can be easily configured to automatically update or invalidate corresponding cached entry.</li>\n</ul>\n<h3>Some critical ideas to explore!</h3>\n<ul>\n<li>Which write strategy will work best with the read-through caching: Write-around, Write-through, and Write-back? What are the key tradeoffs, pros and cons of each combination?</li>\n<li>How can we use the read-through pattern in the distributed caching environment? What are the key challenges and advantages?</li>\n<li>What are the techniques to handle the issue of cache stampede in read-through caching?</li>\n<li>How to determine the appropriate TTL to balance performance and data consistency?</li>\n<li>Best practices for configuring cache size and eviction policies in read-through caching.</li>\n<li>How read-through caching handle situations when data source experiences performance issues?</li>\n</ul>\n<p>Please write in the message below if you want to share some feedback or if you want to share more insight. Enjoy learning, Enjoy system design!</p>"},{"id":"5e6715dd-2660-5395-b1e5-3d1d6e40daf8","frontmatter":{"title":"SQL vs NoSQL Databases: What is the Difference? ","path":"/blog/sql-vs-nosql"},"html":"<p>There are two types of databases: SQL and NoSQL. Each has its own strengths and weaknesses. SQL databases are the best choice for consistency, complex query and ability to handle structured data, while NoSQL databases are the best for scalability, flexibility and ability to handle both structured and unstructured data. So the decision between SQL and NoSQL will depend on requirements: data size, data types, relationships between data, query complexity, data consistency, availability, scalability, etc.</p>\n<p>Let's move forward to learn more about properties, advantages, disadvantages and use cases of both databases. We will first start with the SQL databases and then we will discuss NoSQL databases.</p>\n<p><img src=\"https://ucarecdn.com/b0aa13df-e279-459b-80bb-3c27742382d5/\" alt=\"SQL vs NoSQL Database\" title=\"SQL vs NoSQL Comparison\"></p>\n<h3>SQL Databases</h3>\n<p>In a relational database, data is organised into tables. Each table has columns that define the kinds of information it stores, like customer name or ID. Each row is one record, i.e. a set of values for those columns.</p>\n<p>Tables can be connected through keys. For example, an orders table can include a customer ID that refers to a record in the customers table. This will help to link one customer to many orders. Relationships can also connect records in other ways, like many-to-many relationships.</p>\n<ul>\n<li>SQL (Structured Query Language) is the language used to work with relational databases. It lets you retrieve, insert, update, and delete data. SQL can also define and change the database structure. For example, creating tables, specifying relationships and adding indexes to help queries run faster.</li>\n<li>Relational databases are well suited to queries that combine data from multiple tables, filter and sort results, or calculate summaries like totals and averages. Their query engines use details about the tables and their relationships to plan how to retrieve the requested data.</li>\n<li>Before storing data, you define a schema: the database structural blueprint. It specifies the tables, their columns and data types, how tables relate and any rules the data must follow.</li>\n</ul>\n<p>Suppose a database has two tables:</p>\n<div class=\"gatsby-highlight\" data-language=\"sql\"><pre class=\"language-sql\"><code class=\"language-sql\"><span class=\"token keyword\">CREATE</span> <span class=\"token keyword\">TABLE</span> Customers <span class=\"token punctuation\">(</span>\n    id   <span class=\"token keyword\">INTEGER</span> <span class=\"token keyword\">PRIMARY</span> <span class=\"token keyword\">KEY</span><span class=\"token punctuation\">,</span>\n    name <span class=\"token keyword\">VARCHAR</span><span class=\"token punctuation\">(</span><span class=\"token number\">100</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">NOT</span> <span class=\"token boolean\">NULL</span>\n<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\n<span class=\"token keyword\">CREATE</span> <span class=\"token keyword\">TABLE</span> Orders <span class=\"token punctuation\">(</span>\n    id          <span class=\"token keyword\">INTEGER</span> <span class=\"token keyword\">PRIMARY</span> <span class=\"token keyword\">KEY</span><span class=\"token punctuation\">,</span>\n    customer_id <span class=\"token keyword\">INTEGER</span> <span class=\"token operator\">NOT</span> <span class=\"token boolean\">NULL</span><span class=\"token punctuation\">,</span>\n    total       <span class=\"token keyword\">DECIMAL</span><span class=\"token punctuation\">(</span><span class=\"token number\">10</span><span class=\"token punctuation\">,</span> <span class=\"token number\">2</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">NOT</span> <span class=\"token boolean\">NULL</span><span class=\"token punctuation\">,</span>\n\n    <span class=\"token keyword\">FOREIGN</span> <span class=\"token keyword\">KEY</span> <span class=\"token punctuation\">(</span>customer_id<span class=\"token punctuation\">)</span> <span class=\"token keyword\">REFERENCES</span> Customers<span class=\"token punctuation\">(</span>id<span class=\"token punctuation\">)</span>\n<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span></code></pre></div>\n<p>Customers.id uniquely identifies each customer. Orders.customer<em>id stores the ID of the customer who placed each order. The schema defines customer</em>id as a foreign key that references Customers.id.<br>\nThat rule prevents the database from storing an order for a customer who does not exist.</p>\n<p>For example, if there is no customer with ID 42, the database rejects an order whose customer_id is 42. The schema can also define what happens if someone tries to delete a customer who has orders—for example, reject the deletion or delete the related orders.</p>\n<p>Schemas sometimes need to change with the evolution of the application. Suppose you want to add a required status column to Orders. Existing orders do not yet have a status, so adding the column as mandatory immediately could fail or leave invalid data. A managed change might:</p>\n<ol>\n<li>Add the column in a way that allows existing rows to remain valid.</li>\n<li>Fill in a status for existing orders.</li>\n<li>Make the column required once every order has a valid value.</li>\n</ol>\n<p>These planned database changes are called schema migrations. Managing them carefully helps keep old records consistent with the new rules and lets the application and database change safely together.</p>\n<p>SQL databases can <strong>scale vertically</strong> i.e. we can increase the capacity of the same machine by adding more CPU, RAM, or other hardware. Horizontal scaling is not well-supported in traditional SQL databases because they only work with one main database server and replicas. So how can we do it? There are two options:</p>\n<ul>\n<li>One idea is to write partition logic in the application code. This will add complexity because we need to manage multiple connections and query the appropriate partition. </li>\n<li>Another idea is to use <a href=\"https://www.mysql.com/products/cluster/\">MySQL Cluster</a> or <a href=\"https://vitess.io/\">Vitess</a> for MySQL or <a href=\"https://www.citusdata.com/\">Citus</a> for PostgreSQL. Their built-in partitioning helps us grow the database without adding partitioning logic to the application. For example, Vitess enables live re-partitioning with minimal read-only downtime. But they are not without limitations!</li>\n</ul>\n<h4>ACID Compliance</h4>\n<p>SQL databases exhibit ACID compliance (Atomicity, Consistency, Isolation, Durability). ACID compliance guarantees the integrity of data transactions and ensures high reliability. So this makes them suitable for financial systems and applications that require reliable transaction processing.</p>\n<ul>\n<li><strong>Atomicity:</strong> Each transaction is treated as a single unit of work and all changes to the database made by a transaction must either succeed or fail as a whole. If any part of the transaction fails, all changes made by the transaction must be rolled back.</li>\n<li><strong>Consistency:</strong> Transaction must preserve the consistency of the database and any changes made by a transaction must not violate any integrity constraints!</li>\n<li><strong>Isolation:</strong> Concurrent transactions do not interfere with each other. In other words, each transaction must be executed in isolation, without any interference from other transactions.</li>\n<li><strong>Durability:</strong> Once a transaction is committed, its changes are permanent and cannot be undone, even in the case of system failures.</li>\n</ul>\n<h3>NoSQL Databases</h3>\n<p>NoSQL (not only SQL) is a term for databases that use data models other than the relational model. Depending on the system, they can store data like documents, key-value pairs, graphs, or wide-column records. In other words, NoSQL databases are designed to handle large volumes of structured, unstructured and semi-structured data.</p>\n<ul>\n<li>NoSQL databases have flexible schemas: records do not always need to follow one fixed set of fields. This can make it easier to represent varied data or adapt as application requirements change. </li>\n<li>Schema changes may still require application updates, data migrations, or planned downtime.</li>\n<li>Some NoSQL databases provide fast reads and writes for workloads that fit their data model, which makes them useful for applications that need fast data access. Performance depends on the database, query patterns, data size and system design. </li>\n<li>For example, a relational database may normalise data across several tables to reduce duplication. A query that needs detail from those tables may use joins, which can be costly for complex workloads. A NoSQL design might store related information together, which can help it to retrieve in one operation. This can speed up the access pattern, though it may duplicate data and make updates complicated.</li>\n<li>NoSQL databases are designed to scale horizontally: data and requests can be distributed across multiple servers and capacity can be increased by adding more machines. This can increase storage and help to handle more read and write traffic. Such distribution also improves availability. Note: NoSQL databases vary in how they distribute data and what guarantees they provide.</li>\n</ul>\n<p>There are several <strong>types</strong> <strong>of NoSQL databases</strong>: key-value stores, document-oriented, column-oriented, and graph databases. Each one is optimised for different use cases. Let's understand some of them.</p>\n<ul>\n<li>Due to simplicity and high performance, <strong>key-value stores</strong> are best for fast data retrieval. For example, e-commerce can use <strong>Redis</strong> to cache frequently accessed data: product catalogues, user sessions, etc.</li>\n<li>Due to flexible schema and the ability to handle variety of data types, <strong>document-oriented databases</strong> are best for storing unstructured data in JSON format. For example, a blogging platform can use <strong>MongoDB</strong> to store data related to blogs: title, content, author, date, tags, and other metadata.</li>\n<li>Due to efficient aggregation and ability to query specific columns, <strong>column-oriented databases</strong> are best to store structured data in a columnar format. For example, data warehousing applications can use <strong>Apache Cassandra</strong> to store sales data, financial data, customer data or any other structured data that requires complex aggregations for generating reports.</li>\n<li>Due to the efficient modelling of relationships and ability to handle complex graph queries, <strong>graph databases</strong> are the best for storing interconnected data. For example, social media can use <strong>Neo4j</strong> to store data related to users and followers: User profiles, posts, comments, likes, and other interactions.</li>\n</ul>\n<p>Some other popular NoSQL databases are HBase, Couchbase, DynamoDB, Apache CouchDB, and Riak. These days, community support for these databases is growing at a very fast rate.</p>\n<h4>BASE Properties</h4>\n<p>NoSQL databases follow various trade-offs based on the <a href=\"https://www.enjoyalgorithms.com/blog/cap-theorem-in-system-design/\">CAP theorem</a>. Most of the time, they follow BASE compliance (Basically Available, Soft state, Eventually consistent), which prioritises availability and partition tolerance over immediate consistency.</p>\n<ul>\n<li><strong>Basically Available:</strong> The system guarantees availability even if it returns outdated data. In other words, the system always provides a response and remains available even in the event of a failure.</li>\n<li><strong>Soft State:</strong> The state of the data can change over time due to the lack of immediate consistency. So the system allow temporary inconsistencies across nodes during updates or failures.</li>\n<li><strong>Eventually Consistent:</strong> The data is replicated to different nodes and eventually reaches a consistent state, but consistency is not guaranteed at a transaction level. Until the system eventually converges to a consistent state, data reads are still possible, even though they may not reflect updated data. </li>\n</ul>\n<p>So in other words: Strict consistency may not always be desirable. So BASE compliance provides a more relaxed idea to maintain data consistency (eventual consistency) in NoSQL databases. </p>\n<p>But the above scenario is not always the case in modern NoSQL databases. Sometimes they also prefer strong consistency over availability. This is another side of a practical story based on the CAP theorem! These days, NoSQL databases provide configurations to change degrees of consistency levels to handle various requirements. We highly recommend to explore tradeoffs related to C, A and P  in NoSQL databases like Redis, MongoDB, Cassandra, HBase and Neo4j.</p>\n<h3>When to use SQL Databases?</h3>\n<ul>\n<li>When data is structured and fits naturally into tables with defined relationships.</li>\n<li>When you need complex queries (joins, filtering, grouping, or sorting).</li>\n<li>When consistency and data integrity are important.</li>\n<li>When the data relationships and requirements are well understood and relatively stable.</li>\n</ul>\n<h3>When to use NoSQL Databases?</h3>\n<ul>\n<li>When we need to scale up and scale down because of changing requirements.</li>\n<li>When we require complex data models that cannot be accommodated by a relational model. </li>\n<li>When high availability is priority and data consistency is not a major concern.</li>\n<li>When we need to constantly add new features and data types.</li>\n<li>When records have varying fields or the data model is expected to evolve frequently.</li>\n<li>When the application needs to handle large volumes of distributed data.</li>\n<li>When low-latency reads or writes are important for specific, well-understood access patterns.</li>\n</ul>\n<h3>Drawbacks of SQL databases</h3>\n<ul>\n<li>Scaling vertically means upgrading the server CPU, memory, or storage, which can become costly. Horizontal scaling is possible too, but may require extra architecture or database tooling.</li>\n<li>Schema changes need to be managed. A defined schema can keep data consistent, but changing tables or constraints may require migrations and careful coordination with the application.</li>\n<li>Growing data volume can require tuning. Performance can decline without suitable indexes, query optimization, partitioning, or additional resources.</li>\n<li>Not suitable for storing and querying unstructured data.</li>\n<li>Rely heavily on data consistency and integrity constraints.</li>\n</ul>\n<h3>Drawbacks of NoSQL databases</h3>\n<ul>\n<li>NoSQL databases do not support ACID properties. This can be a constraint when strong data consistency and integrity are required. Note: Some NoSQL databases like MongoDB can integrate ACID rules.</li>\n<li>Not optimised for reducing data duplication. This can result in larger database sizes if the same data is stored redundantly across multiple nodes. But this trade-off can be acceptable when performance and scalability are more important. Note: Many NoSQL databases do support compression and other optimisation techniques to reduce storage requirements.</li>\n<li>A single type of NoSQL database may not be able to cover all use cases. So one has to operate with multiple databases and data models. For example, graph databases are excellent for analysing relationships in data but may not provide retrieval of data like range queries.</li>\n<li>NoSQL databases do not have any standard query language like SQL. In other words, they lack the standard interface like SQL. For example, many NoSQL databases have unique data manipulation languages constrained by specific structures. So there is poor consistency between NoSQL languages.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/443b5f08-bc29-4900-ba4c-5a8d680e817a/\" alt=\"Difference Between SQL and NoSQL\" title=\"SQL Vs NoSQL\"></p>\n<h3>Combining the best of SQL and NoSQL</h3>\n<p>In some cases, it may be useful to use both SQL and NoSQL databases in a single application to take advantage of each. For example, YouTube stores video content in a NoSQL database and user metadata and other information in a SQL database. This allows them to leverage the flexibility and scalability of NoSQL databases for storing large amounts of unstructured data, while also taking advantage of the structured data and complex query capabilities of SQL databases.</p>\n<p>There are also databases that provide features of both SQL and NoSQL databases. For example, MySQL Document Store combines the structure of an SQL database with the features and flexibility of a NoSQL database. Similarly, MongoDB also offers ACID transactions. This can be useful for applications that need to handle both structured and unstructured data.</p>\n<p>If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"ff88822c-59d6-5a8b-9a7d-ccac54769f52","frontmatter":{"title":"Client Server Architecture","path":"/blog/client-server-architecture"},"html":"<h3>What is client-server architecture?</h3>\n<p>Client-server is a type of distributed system that consists of clients and servers. The server is responsible for hosting, managing and delivering services to clients. Here clients are connected to the server and communicate with it over the internet using a computer network. So when a client needs a service, it sends a request to the server, which processes the request and sends a response back to the client.</p>\n<h3>Client-server example</h3>\n<p>Let's take an example of a service like Gmail. Here client can be the web browser or email interface which interacts with the mail servers. The client can send HTTP requests to the mail server for fetching emails, composing new emails, displaying the received data, etc. The mail server handles the storage, retrieval, and manipulation of the data.</p>\n<p>So when you send an email, the client sends a request to the mail server, which processes the request and stores the email in your account. Similarly, when you request to view your inbox, the client sends a request to the server, which retrieves emails and sends them back to the client for display. Overall, client-server architecture is a good example of the separation of concerns: the server handles the heavy data processing, while the client focuses on providing a user-friendly interface.</p>\n<h3>Components of client-server architecture</h3>\n<p>A complex client-server architecture can have several components. At the basic level, there are four important components: client, load balancer, servers and network protocols.</p>\n<ul>\n<li><strong>Client:</strong> A software application that requests resources and services made available by servers. It runs on a user's local computer or a remote machine and connects to a server.</li>\n<li><strong>Server:</strong> A software program that receives and processes requests from clients. It operates on a remote machine and can be accessed by a user's local computer or workstation. For example, a web server is a type of server that receives and processes requests from web clients (browsers).</li>\n<li><strong><a href=\"https://www.enjoyalgorithms.com/blog/load-balancers-in-system-design/\">Load balancer</a></strong> is responsible for distributing incoming requests across a group of servers to manage traffic and optimize resource usage.</li>\n<li><strong><a href=\"https://www.enjoyalgorithms.com/blog/network-protocols-concept/\">Network protocols</a></strong> are used to facilitate communication between clients and servers.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/1080/1*SBH_Y5t32ixv8C_F1MVYzA.png\" alt=\"example of client server architecture\" title=\"Client server architecture visualization\"></p>\n<h3>How does client-server model work?</h3>\n<p>The data flow in a client-server architecture is unidirectional and forms a cycle. It begins when a client requests data from the server and the server processes the request by querying a database for relevant data. The database then returns the data to the server, which processes it and sends it back to the client. Clients cannot communicate directly with each other.</p>\n<p>Here is a simple example of how a client (browser) interacts with servers in a client-server architecture:</p>\n<ol>\n<li>User enters a website URL into the browser.</li>\n<li>Browser sends a request to the DNS server to look up the IP address of the web server.</li>\n<li>DNS server sends the IP address of the web server to the browser.</li>\n<li>Browser sends an HTTP/HTTPS request to the IP address of the web server.</li>\n<li>Web server sends the necessary files for the website back to the browser.</li>\n<li>Browser renders the files and displays the website.</li>\n</ol>\n<h3>Client-server architecture types</h3>\n<h4>Two-Tier Architecture</h4>\n<p>There are only two components in a two-tier architecture: client and server. Here client directly communicates with the server. So this architecture is simple and easy to set up, which makes it suitable for small applications. As the application grows in size, it will be less efficient and take longer to process requests.</p>\n<p>To understand this better, you can think client as a customer who goes to a fast-food restaurant and directly places an order with the cook (server). The cook then prepares and serves the food to the customer. So this process can be less efficient if there are many customers or if the customer wants a more complex meal.</p>\n<h4>Three-Tier Architecture</h4>\n<p>There are three components in a three-tier architecture: client, application server and database server. Here client sends a request to the application server, application server processes the request and communicates with the database server to retrieve or update the data. So this architecture can work well for medium to large-scale applications and provides better scalability and fault tolerance.</p>\n<p>To understand better, you can think of the client as a customer who goes to a restaurant and places an order with a waiter (application server). The waiter then communicates with the kitchen staff (database server) to prepare the food. This will help the restaurant system to serve efficiently because multiple waiters and kitchen staff can be added to serve more customers.</p>\n<h4>N-Tier Architecture</h4>\n<p>There can be multiple tiers in N-tier architecture: presentation, application, data tiers, etc. Each tier is responsible for a specific function and they communicate with each other to perform the required tasks. So this architecture is suitable for large and complex applications because it can provide better scalability and fault tolerance than the 2-tier and 3-tier architectures.</p>\n<p>To understand better, you can think client as a customer who visits a restaurant with a well-organised hierarchy of staff. Each staff has a specific role, such as host (presentation), waiter (application), and kitchen staff (data tier). The host greets the customer and shows them to their table, the waiter takes their order and communicates with the kitchen staff to prepare the food and the kitchen staff delivers the food to the table.</p>\n<h3>Client-Server vs Peer-to-Peer Architecture</h3>\n<p>Here are some major differences between peer-to-peer and client-server architecture:</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/1080/1*a4GW0oovqhdNweLYyImcdg.png\" alt=\"Client server vs peer to peer architecture\" title=\"Peer to peer vs client server architecture\"></p>\n<h3>Advantages of client-server architecture</h3>\n<ul>\n<li>We can store and manage data in one central location. So this will help us to provide complete control over processes and make it easy to share data across different platforms. Users can access any file stored in the central storage at any time.</li>\n<li>Data passed between the client and services provided by the server are entirely at the discretion of the programmer. Due to this, there are many ways to use this architecture to solve future problems. On the other side, this can be easily combined with other types of architectures on the client or server side.</li>\n<li>The system can be updated based on changes in functional and non-functional requirements without altering the architecture or disrupting service.</li>\n<li>Clients only make requests to the server with their input data, so they don't see how the server will handle the request. It may seem like a single, central server to the user.</li>\n<li>Servers do not usually need to shut down for long periods because server uptime is possible during maintenance via server duplication. Additionally, if multiple servers provide the same services, the system can still function even if one or more servers fail.</li>\n<li>Client-server architecture is capable of adding or removing servers in the network (horizontal scaling) or migrating to larger and faster server machines (vertical scaling).</li>\n</ul>\n<h3>Disadvantages of client-server architecture</h3>\n<ul>\n<li>Centralised control can lead to increased chances of failure. When many clients send simultaneous requests to the server, it can overload the server and slow down performance. This can also lead to server failure, causing the entire system to go down.</li>\n<li>Servers are more powerful than client computers, which means they are more expensive. They also require person with networking and infrastructure knowledge to manage the system.</li>\n<li>The client-server architecture is vulnerable to Denial of Service (DoS) attacks because the number of servers is typically smaller than the number of clients.</li>\n<li>Data packets may be modified during transmission, leading to the potential loss of useful information.</li>\n</ul>\n<h3>Use cases of client server architecture</h3>\n<ul>\n<li>Client-server architecture is suitable for applications that require separation of concerns between the client and server and the ability of systems to exchange information. </li>\n<li>It is often used in systems that require functional separation, where clients and servers have their own responsibilities. For example: Validation may be handled and managed on the client side, while the server is responsible for executing the client's request and returning the result. As a result, both the client and server can assist in implementing abstract functions without interfering with each other's capabilities. </li>\n<li>Separation of functionality allows each layer to work more efficiently at a large scale. For this, modern client-server architecture uses ideas like load balancing, sharding, partitioning, caching, etc.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>We hope you enjoyed this blog. We covered the fundamental concept behind client-server architecture, how it works, and how it can be an efficient way of interaction. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, enjoy system design, enjoy algorithms!</p>"},{"id":"53665752-c7fa-5572-85ff-6620cfb5aa05","frontmatter":{"title":"Consistent Hashing (An Idea to Scale Distributed Systems)","path":"/blog/consistent-hashing-in-system-design"},"html":"<p>This blog discusses consistent hashing, which come in handy while scaling out the distributed systems. This idea is frequently applied in designing real-time applications and solving system design questions.</p>\n<h3>What is Consistent Hashing?</h3>\n<p>Before diving deep into Consistent Hashing, let’s understand what hashing is? Hashing is used to efficiently retrieve information and improve the performance of insert, delete and search operations. It uses hash function to map data to a storage pool based on key values. For example, we can generate a random number and use it as a key to map data to a server by taking the modulo of the total number of servers. But this approach is effective when the number of servers are fixed.</p>\n<p>In distributed systems, the number and location of servers may change, which makes it necessary to have a more efficient way of handling requests for a scalable application. Consistent hashing is an improvement over normal hashing that addresses these issues.</p>\n<p>In consistent hashing, user and server locations are virtually represented in a circular structure called a hash ring. The hash ring is considered infinite and can accommodate any number of servers, which are assigned to random locations based on a hash function. This allows for flexible allocation of servers and efficient handling of requests.</p>\n<h3>Why do we need Consistent Hashing?</h3>\n<p>The traditional hashing method is not well-suited for use in distributed systems where multiple users are requesting services from multiple servers. This is because it assumes that the number of servers and their mapping locations are fixed, which is not the case in a distributed system. If a server fails or goes offline, the traditional method requires significant computation to remap requests to different servers, which can affect the throughput and latency of the service.</p>\n<p>In distributed systems, multiple nodes interact with each other and the number of nodes can change due to changes in traffic or maintenance. For example, if we have five nodes and there is an increase in traffic, we may need to add two more nodes to the system, bringing the total to seven. Using traditional hashing, we would need to recompute the mapping of requests, as the hash was previously based on five nodes but now we have seven. Similarly, if the number of nodes decreases, we would need to recompute the mapping again. This is inefficient and requires a lot of computation and data reshuffling. </p>\n<p>So in situations where the number of operational servers is not fixed, the traditional hashing method becomes increasingly inefficient, since it requires more and more computation and reassignment of requests. To address these shortcomings, the concept of consistent hashing was developed as a more dynamic solution.</p>\n<h3>How Consistent Hashing Works?</h3>\n<p>Consistent Hashing helps us in effective distribution of resources by ensuring minimum reorganisation of requests in any failure. Here we use a hash function to map servers to locations in a virtual ring. The position of the server is just a random position obtained using the hash function. </p>\n<p>Consistent Hashing is organised in the following manner:</p>\n<ul>\n<li>It maps both servers and keys to positions on the same circular hash space called a ring.</li>\n<li>Servers are hashed using their IP addresses and assigned positions on the ring based on the hash function's output.</li>\n<li>Hash each key using the same hash function to determine its position on the ring.</li>\n<li>Starting at the key’s position, move clockwise until you reach the first server. Assign the key to that server. If you reach the end of the ring, wrap around to the beginning. Note: The key doesn't need to land exactly on a server, it is assigned to the next server clockwise.</li>\n</ul>\n<p>So in this manner, the keys are assigned to the server in Consistent Hashing. The beauty of Consistent Hashing comes when we add or remove servers.</p>\n<h4>Addition of new server</h4>\n<p>Imagine the hash space as a circle. Both servers and keys are hashed to positions on this circle. To find a key’s server, start at the key’s position and move clockwise to the first server. Suppose the clockwise order is: Server 4 → Server 1 → … → Server 4. So keys between Server 4 and Server 1 are assigned to Server 1.</p>\n<p>Now add Server 5 between Server 4 and Server 1: Server 4 → Server 5 → Server 1 → … → Server 4. Only keys in the section from Server 4 up to Server 5 now map to Server 5. Keys between Server 5 and Server 1 still map to Server 1, and the mapping of keys in other sections of the ring remains unchanged. In other words, some keys previously assigned to Server 1 move to Server 5; not every key after Server 4 does.</p>\n<p>This limited reassignment helps us scale: the system can add a server without redistributing every key. The new server can take on some of the workload, which can increase throughput and reduce latency if the existing servers were overloaded.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/800/1*TCyBjcM0inmdwUyBKNeW5Q.png\" alt=\"addition of new server in consistent hashing\" title=\"consistent hashing visualisation 1\"></p>\n<h4>Removal of server</h4>\n<p>On a consistent hash ring, each key is assigned to the first server encountered when moving clockwise from the key position. So if a server fails, the system treats it as unavailable and continues the clockwise search to the next active server. Suppose Server 1 is the next active server clockwise after Server 4.</p>\n<ul>\n<li>Before failure: → Server 4 → Server 1 →...</li>\n<li>After failure: →Server 4 is unavailable → Server 1 →...</li>\n</ul>\n<p>Now keys that were assigned to Server 4 are routed to Server 1, and keys assigned to other servers are unchanged. This can help keep the service available, but the newly asigned server need to handle the extra traffic. One more thing: Consistent hashing only redirects requests and does not recover data stored on the failed server. To preserve access to that data, systems need to replicate it to other servers. So replication, health checks and capacity planning work together with consistent hashing to ensure fault tolerance.</p>\n<h3>Non-Uniform Distribution</h3>\n<p>In basic consistent hashing, each server is mapped to one position on the ring. The keys in the clockwise interval after one server and up to the next are assigned to that next server. These intervals can vary greatly in size. If one server interval is much larger than the others, it will have more keys and handle more workload. In this scenario, an overloaded server can become a bottleneck and may be more likely to fail.</p>\n<p>So, what is the solution? Here is the idea: use virtual nodes to improve distribution. Instead of placing each server at just one position, place it at several positions on the ring. These positions are called virtual nodes. For example, Server A might be represented by A-1, A-2, and A-3, each hashed to a different location.</p>\n<p>A key is still assigned to the first server position encountered when moving clockwise. But each server owns several smaller ring intervals scattered around the circle, so its keys come from multiple parts of the ring.</p>\n<ul>\n<li>Physical servers: A, B, C</li>\n<li>Virtual nodes on ring: A-1 → C-1 → B-1 → A-2 → B-2 → C-2 → ...</li>\n</ul>\n<p>When a physical server fails, all of its virtual nodes are removed from the ring. The keys assigned to those positions move to the next active virtual nodes clockwise. The other key assignments generally remain unchanged. Adding more virtual nodes usually improves balance, though it does not guarantee a perfectly uniform distribution. Also, virtual nodes are not the same as data replicas: they distribute ownership across the ring, while data replication stores copies of data on multiple servers for availability and recovery.</p>\n<h3>Conclusion</h3>\n<p>Consistent hashing is an important technique for distributing data or requests across servers. It makes scaling easier by limiting how many keys need to be reassigned when servers are added or removed. Combined with replication and health checks, it can also help a system remain available when a server fails.</p>\n<p>Thanks to Suyash for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"f261c131-82b4-5475-9caa-6bb243ca6a71","frontmatter":{"title":"Web Crawler System Design","path":"/blog/web-crawler"},"html":"<h3>What is Web Crawler?</h3>\n<p>A web crawler (also known as a spider) is a system for downloading, storing, and analyzing web pages. It performs the task of organizing web pages so that users can easily find the information. For this, web crawlers collect a few web pages and follow links associated with those pages to gather new content. That's why web crawlers are one of the critical components of search engines. These search engines compile a collection of web pages, index them, and help users search the index for pages that match their queries.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*1P6RNLLO517jAVIfDkf82w.png\"></p>\n<h3>Use cases of web crawler</h3>\n<p><strong>Search engine indexing:</strong> Search engines uses web crawlers to collect web pages and generate a local index. For example, Google uses Googlebot web crawler.</p>\n<p><strong>Web archiving</strong>: We can use web crawlers for collecting web-based information and storing it for future use. The US Library of Congress and the EU web archive often use crawlers for this purpose.</p>\n<p><strong>Web monitoring</strong>: Web crawlers can monitor the internet for copyright and trademark violations. For example, Digimarc uses crawlers to identify and report pirated activities.</p>\n<h3>Requirements of the System</h3>\n<p>A simple design of a web crawler should have following functionalities:</p>\n<ul>\n<li>Given a set of URLs, visit the URL and store the web page.</li>\n<li>Now, extract URLs in these web pages.</li>\n<li>Append new URLs extracted to the list of URLs to be visited.</li>\n<li>Repeat the process.</li>\n</ul>\n<h3>System analysis and characteristics</h3>\n<p>When designing a web crawler system, we should consider some important characteristics:</p>\n<ul>\n<li>System should be scalable because there are billions of web pages that need to be crawled and analyzed. In other words, a web crawler should efficiently handle this task using parallelization.</li>\n<li>Web crawler should be robust to handle challenges like poorly formatted HTML, unresponsive servers, crashes, and malicious links.</li>\n<li>Web crawler should avoid making too many requests to a website within a short time interval.</li>\n<li>System needs to be extensible so that it can be flexible to any future changes. For example, the ability to crawl images or music files may be required in the future.</li>\n<li>Web crawler should be manageable and reconfigurable i.e. it should have a good interface for monitoring crawl like statistics about hosts and pages, crawler speed, and sizes of the main data sets. This is important for monitoring the performance of the web crawler.</li>\n</ul>\n<h3>Estimation of scale and constraints</h3>\n<p>To estimate the scale of a web crawling system, we can make assumptions based on a discussion with the interviewer. For example:</p>\n<ul>\n<li>Assuming that 1 billion web pages need to be crawled every month, we can estimate the average number of queries per second (QPS) to be approximately 400 pages per second: QPS = 1,000,000,000 / 30 days / 24 hours / 3600 seconds = 400.</li>\n<li>We can assume that the peak value of queries per second is twice the average i.e. 800 pages/second.</li>\n<li>Assuming that average web page is 500 Kb in size, we can estimate the storage required per month to be 500 TB: 1-billion-page * 500k = 500 TB storage per month.</li>\n<li>If we assume that the data needs to be stored for five years, then total storage needed would be 30 PB: 500 TB * 12 months * 5 years = 30 PB.</li>\n</ul>\n<h3>High Level Design</h3>\n<p>Once we have a clear understanding of requirements and have made estimations on the scale, we can design a high-level design of the system. This will outline the structure and components of the web crawler.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*F9QnGqmAMcXEr1lQeSiqVA.png\" alt=\"high-level architecture of a distributed web crawler\" title=\"Web crawler high level design\"></p>\n<p>Let us explore the individual component of the system:</p>\n<p><strong>Seed URLs:</strong> To begin the crawl process, we need to provide a set of seed URLs to the web crawler. One way to do this is to use a website's domain name to crawl all of its web pages. To make system more efficient, we should be strategic in choosing the seed URL because it can impact the number of web pages that are crawled. The selection of the seed URL can depend on factors like geographical location, categories (entertainment, education, sports, food), content type, etc.</p>\n<p><strong>URL Frontier:</strong> Component that explores URLs to be downloaded is called the URL Frontier. One way to crawl the web is to use a breadth-first traversal, starting from the seed URLs. We can implement this by using the URL Frontier as a first-in first-out (FIFO) queue, where URLs will be processed in the order that they were added to the queue (starting with the seed URLs).</p>\n<p><strong>HTML Fetcher:</strong> HTML fetcher component is responsible for downloading web pages corresponding to a given URL provided by the URL Frontier. It does this by using a network protocol like HTTP or HTTPS. In simple words, HTML fetcher retrieves the actual web page content that needs to be analyzed and stored.</p>\n<p><strong>DNS Resolver:</strong> Before a web page can be downloaded, URL must be translated into an IP address. For this, HTML fetcher component initiates download process by calling the DNS Resolver. After this, DNS Resolver converts URL into the corresponding IP address, which is then used to access the web page.</p>\n<p><strong>HTML Parser:</strong> After HTML fetcher has downloaded a web page, we need to parse, analyze, and validate the content to ensure the integrity and security of the data. For this, HTML Parser will check issues like poorly formatted HTML or malware that could cause problems with the storage system. Through this process, we ensure that the data being stored is of high quality.</p>\n<p><strong>Duplicate Detection:</strong> Studies have shown that around 30% of web pages contain duplicate content, which can lead to inefficiencies in the storage system. To avoid this problem, we can use a data structure to check for redundancy in the downloaded content. For example, we can use MD5 hashing to compare the content of pages that have been previously seen, and check if the same hash has occurred before. This can help to identify and prevent the storage of duplicate content.</p>\n<p><strong>Data Storage:</strong> After web pages have been downloaded and parsed, they need to be stored in a storage. The choice of a specific storage will depend on the use cases. For example, if we want to use the content for offline archiving or analysis, we can store data with a low-cost cloud storage. On the other side, if we want to use content for real-time search, we can store the data in a large distributed database like HDFS, Google's BigTable, or Apache Cassandra. The key thing is: Regardless of the storage system we choose, it is important to ensure that we have enough space to store large amounts of data that is likely to be collected.</p>\n<p><strong>Caching:</strong> To improve the efficiency of web crawler, we can use a cache to store recently processed URLs. This allows us to quickly look for a URL in the cache rather than crawling the web to find it again. The type of cache will depend on the specific use case for the web crawler.</p>\n<p><strong>URL Extractor:</strong> URL Extractor parse and extract links from HTML pages. Once the links have been extracted, they are filtered and added to the URL Frontier. This process allows web crawler to expand the scope of its crawl by following the links on each web page and collecting new content. In other words, URL Extractor enables system to discover new content and continue expanding the collection of web pages.</p>\n<p><strong>URL Filter:</strong> URL filter is used to filter out unwanted content types, faulty links, and URLs from unsafe sites. This helps to ensure that the system only collects high-quality, relevant content. The URL filter can be customized to meet the specific needs of the web crawler like excluding certain content types or blocking access to unsafe sites. In simple words: By using a URL filter, we can improve the efficiency of the web crawler by limiting the amount of unnecessary or irrelevant content that is collected.</p>\n<p><strong>URL Detector:</strong> URL Detector is used to filter out URLs that have already been visited. This helps to prevent the system from processing the same URL repeatedly. We can use various techniques to implement the URL Detector like Bloom Filters and Hash Tables. These techniques efficiently identify already visited URLs, so that they can be skipped in the crawling process.</p>\n<p><strong>URL Storage:</strong> URL storage is used to store the URLs of web pages that have already been visited. This will help system to keep track of the already processed URLs, so that they can be skipped in the future.</p>\n<h3>Workflow of Web Crawler System</h3>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*T5EJQVrn8GVuPmA0RaM_uQ.png\" alt=\"Web crawler system workflow\" title=\"Workflow of a web crawler system\"></p>\n<p>The crawling process of a web crawler consists of several worker threads that perform repeated cycles of work. The following is a summary of the steps involved in a work cycle:</p>\n<ol>\n<li>At the beginning, worker thread sends seed URLs to the URL Frontier. URL Frontier then retrieves URLs according to its priorities and politeness policies.</li>\n<li>HTML Fetcher module is called to fetch URLs from the URL Frontier.</li>\n<li>HTML Fetcher calls DNS resolver to resolve the host address of the web server associated with the URL.</li>\n<li>HTML Parser parses the HTML page and analyzes content of the page.</li>\n<li>The content is validated and then passed to the duplicate detection component to check for duplicity.</li>\n<li>The duplicate detection component checks the cache, and if the content is not found there, it checks the data storage to see if the content is already stored there.</li>\n<li>If content is not in the storage, web page is sent to the Link Extractor, which extracts any outgoing links from the page.</li>\n<li>The extracted URLs are passed to the URL filter, which filters out unwanted URLs such as file extensions of no interest or blacklisted sites.</li>\n<li>After links are filtered, they are passed to the URL Detector component.</li>\n<li>URL Detector checks if a URL has already been processed. If it has, no further action is needed. If URL has not been processed, it is added to the URL Frontier to be crawled in a future work cycle.</li>\n</ol>\n<p>URL Frontier is responsible for managing the queue of URLs that are waiting to be crawled. It assigns specific positions to the URLs in its data structure based on certain priority rules. These rules can be customized to meet the specific needs of the web crawler. For example, URL Frontier might prioritize URLs from certain domains or categories, or it might prioritize URLs that have been waiting in the queue for a longer time.</p>\n<p>Thanks to Chiranjeev and Navtosh for their contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"1b15075e-e0a6-54a3-be5b-f28d538562ba","frontmatter":{"title":"Database Partitioning in System Design","path":"/blog/data-partitioning-system-design-concept"},"html":"<p>Database partitioning is the backbone of distributed database management systems. It is a process of dividing a large dataset into several small partitions placed on different machines. In other words, It is a way of partitioning data like index-organised tables into smaller pieces so that data can be easily accessed.</p>\n<ul>\n<li>It distributes data across several partitions to improve availability, scalability and query processing performance. The combined data from all partitions is the same as the data from the original database.</li>\n<li>The partition architecture is transparent to the client application, where the client application keeps talking to the database partitions as if it was talking to a single database.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*zDfepdN0iQeqwhBTYeKS2w.png\" alt=\"Data partitioning in system design\" title=\"Data partitioning\"></p>\n<h2><strong>What are the problems solved by database partitioning?</strong></h2>\n<p>With the growth in services and user base, it becomes tricky for a single database server to function efficiently. We may experience lower performance with the architecture of a single database server. Here is some situation that could arise:</p>\n<ul>\n<li>Database operations become slower.</li>\n<li>Network bandwidth starts reaching the saturation level. </li>\n<li>The database server starts running out of disk space at some point.</li>\n</ul>\n<p>Database partition helps us fix all the above challenges by distributing data across several partitions. Each partition may reside on the same machine (coresident) or different machines (remote). The idea of co-resident partitioning is to reduce individual indexes size, and the amount of I/O needed to update records. Similarly, the concept of remote partitioning is to increase the bandwidth access to data by having more RAM, avoiding disk access, or having more network interfaces and disk I/O channels available.</p>\n<p>A high level view of partitioned tables</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*bNuOCjSXMyP0Updlk9ZsjA.gif\"></p>\n<h2><strong>When to partition a table?</strong></h2>\n<p>There are several scenarios when partitioning data can be beneficial:</p>\n<ul>\n<li>When data is too large to fit in the existing database.</li>\n<li>When database indexes grow continuously, it will impact the query performance. So one can partition the database and distribute the data across multiple partitions. This will improve query performance by reducing the amount of data that needs to be accessed for each query.</li>\n<li>When new data is added on a daily basis like a table containing historical data where only the current month's data is updated and the other 11 months data are read-only. In other words, when historical data is queried less frequently than recent data, we can use partitioning to separate older data into different partitions. This will improve query performance because now queries can be targeted to specific partitions based on the time range.</li>\n<li>When there is a need to distribute data across different storage devices to improve scalability and parallelism (executing queries in parallel across multiple partitions).</li>\n<li>When there is a need to handle specific data access patterns. For example, one can partition data based on customer segments, geographical regions, or product categories to optimize data retrieval.</li>\n</ul>\n<p>However, not all cases require data partitioning. So it is important to carefully analyse the specific needs of the system before deciding whether or not to use partitioning.</p>\n<h2>Why do we need data partitioning?</h2>\n<ul>\n<li>By partitioning the database, we can ensure high availability. Here individual partitions can be managed independently, so if one partition becomes unavailable, the other partitions can still execute database queries. This will avoid a single point of failure for the entire dataset.</li>\n<li>Keeping data in different partitions helps the database administrator do backup and recovery operations on each partition, independent of the other partitions. This could allow the active partition to be made available sooner so access to the system can continue while the inactive data is still being restored.</li>\n<li>As traffic increases, the performance of the service can decrease. So data partitioning helps us to scale out the service and remove any limitations on scalability.</li>\n<li>We can improve security by storing sensitive and non-sensitive data in different partitions. </li>\n<li>As discussed above, It will improve performance because instead of querying the entire database, now system can query smaller partitions.</li>\n<li>Data partitioning divides tables and indexes into smaller, more manageable units. So this \"divide and conquer\" approach of data management helps us simplify the maintenance of particular table partitions.</li>\n</ul>\n<h2>Data Partitioning Methods</h2>\n<p>There are various data partitioning strategies. Let’s have a look at each one of them.</p>\n<h3><strong>Horizontal Partitioning or Database Sharding</strong></h3>\n<p>Horizontal partitioning (also known as database sharding) is a strategy for splitting table data horizontally based on the range of values defined by a partition key. Here we divide the table into smaller and more manageable tables, with each row of the table being assigned to one of the partitions. </p>\n<ul>\n<li>The partition key is responsible for distributing the data among all the partitions. When a query is made using the partition key, the database will determine which partition it needs to query.</li>\n<li>We need to balance the number of requests between partitions to ensure that none become overloaded.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/a221285e-863e-4d00-8622-8741b7d809e7/\" alt=\"Horizontally partitioning or sharding of a database based on partition key\" title=\"Horizontally partitioning\"></p>\n<p>Suppose there is a large database containing multiple rows of customer data that has a slow query performance. So we can think of partitioning the table into two separate tables horizontally. The first table would contain the first half of the customer data, and the second table would contain the second half. Now query will go to either partition 1 or partition 2, depending on the partition key. For example, suppose we store the contact details for customers. In that case, we can keep the contact info starting with the name A-H on one partition and contact info starting with the name I-Z on another partition.</p>\n<p><strong>The benefit of the horizontal partition:</strong> The horizontal partitioning is the most fundamental partitioning method. It divides the database into separate partitions that have the same schema as the original database. So this makes it easy to answer queries without having to combine data from multiple partitions.</p>\n<p><strong>The disadvantage of the horizontal partition:</strong> Data may not be evenly distributed across the partitions. For example, if there are many more customers with names that fall in the range of A-H than in the range I-Z, the first partition may experience a much heavier load than the second partition.</p>\n<h3><strong>Vertical Partitioning</strong></h3>\n<p>Vertical partitioning (also known as normalization) divides a table into smaller tables based on columns. For example, in a social media application like LinkedIn, a user's profile data, list of connections, and articles they have written can be placed on separate partitions using vertical partitioning i.e. first partition for user profile data, second partition for the list of connections and third partition for the articles.</p>\n<ul>\n<li>Vertical partitioning can store different types of data in separate partitions. This can be useful when some data is more critical or sensitive than other data. For example, we could store passwords, salary information, and other sensitive data in a separate partition to provide additional security controls.</li>\n<li>Vertical partitioning can be a great help when our data is stored on a solid-state drive (SSD). The idea is: If certain columns are not frequently queried, we can partition the table vertically and move those less frequently used columns to a different location. This can help to reduce the I/O and performance costs associated with fetching frequently accessed items.</li>\n<li>Overall, vertical partitioning helps us separate slow-moving data from more dynamic data. Slow-moving data is a good candidate for caching in memory.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/f5b0da7a-e5ef-4278-9034-0d5db1c9fbfb/\" alt=\"Vertical database partitioning\" title=\"Vertical partitioning of a database\"></p>\n<p>There are a few disadvantages to using vertical partitioning:</p>\n<ul>\n<li>It may be necessary to combine data from multiple partitions to answer a query, which can increase the operational complexity. For example, if a profile view request requires data from a user's profile, connections, and articles, this data will need to be retrieved from separate partitions and combined.</li>\n<li>If the website experiences additional growth, it may be necessary to further partition a database across multiple servers. This can be time-consuming and may require additional resources.</li>\n</ul>\n<p>The choice of which type of partitioning to use depends on the structure of the data. In some cases, it may be useful to combine both horizontal and vertical partitioning. For example, if we have a large dataset of customer information with different data types, we could use vertical partitioning to divide the database and horizontal partitioning to divide the customer information.</p>\n<h2>Data Partitioning Criteria</h2>\n<p>There are a large number of criteria available for data partitioning. Most of them use partition keys to identify partitions. Some of the popular criteria are range-partitioning, list-partitioning, hash partitioning, etc.</p>\n<h3><strong>Range Based Partitioning</strong></h3>\n<p>In range partitioning, data is organized into partitions based on ranges of values of the partition key i.e. each partition contains rows with values for the partition key within a specific range. The ranges are typically contiguous and do not overlap, where each range specify a lower and upper bound for the partition. Any partition key equal to or higher than the upper bound of the range are added to the next higher partition.</p>\n<p>Range partitioning is used in a few specific cases:</p>\n<ul>\n<li>When there is a need to organize data based on date and time. For example, a table with a date column as the partition key might have a January-2022 partition that contains rows with partition key values from 01-Jan-2022 to 31-Jan-2022.</li>\n<li>When data is regularly added to the database and there is a need to remove old data. By dividing the partitions into date ranges, it becomes easier to remove old data. For example, we could delete all rows relating to employees who stopped working for the company before 1991. This can be more efficient for a table with many rows than running a delete query to remove employee data &#x3C;= 1990.</li>\n</ul>\n<h3><strong>Hash-Based Partitioning</strong></h3>\n<p>In hash partitioning, rows are divided into different partitions based on a hashing algorithm. This is different from range partitioning, which groups database rows based on continuous indexes.</p>\n<p>Hash partitioning can be used in a few different ways:</p>\n<ul>\n<li>We can use a client request's IP address or application ID as an input to a hash function to generate a hash value. This value determines which database partition to use for the request. For example, if we have 4 database partitions and each request contains an application ID, we could perform a modulo operation on the application ID with 4 and take the remainder to determine the partition to use.</li>\n<li>Hash partitioning is a good method for distributing data evenly across partitions.</li>\n<li>It is an easy-to-use alternative to range partitioning, especially when the partitioning data is not historical or does not have an obvious partition key.</li>\n</ul>\n<p>One disadvantage of hash partitioning is that it can be expensive to dynamically add or remove database servers. For example, if we want to add more partitions, we may need to remap some of the keys and migrate them to a new partition, which requires changing the hash function. During this process, a large number of requests may not be served, resulting in downtime until the migration is complete. We can solve this problem using consistent hashing!</p>\n<h3><strong>List Based Partitioning</strong></h3>\n<p>In list partitioning, each partition is defined and selected based on a list of values for a particular column, rather than a set of contiguous ranges. Some key points:</p>\n<ul>\n<li>The partition key can only consist of a single table column.</li>\n<li>It is easy to group and organize unrelated data sets in different partitions.</li>\n<li>We can partition data based on a specific column like region column, to ensure that data for each region is stored in a single partition. For example, we could store all customers from India in one partition and customers from other countries in different partitions.</li>\n</ul>\n<p>As an example, consider a table with data for 20 video stores distributed among 4 regions:</p>\n<ul>\n<li>Region: India, ID Numbers: 3, 5, 6, 9, 17</li>\n<li>Region: USA, ID Numbers: 1, 2, 10, 11, 19, 20</li>\n<li>Region: Japan, ID Numbers: 4, 12, 13, 14, 18</li>\n<li>Region: UK, ID Numbers: 7, 8, 15, 16</li>\n</ul>\n<p>Using list partitioning, we could partition the table so that rows belonging to the same region are stored in the same partition. This will help us to easily add or drop records relating to specific regions from the table.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*B4cq6ccYzDeoFMZXdx4v1A.gif\" alt=\"Data partitioning criteria\" title=\"Criteria for data partitioning\"></p>\n<h3><strong>Composite Partitioning</strong></h3>\n<p>Composite partitioning is a method of partitioning data based on two or more partitioning techniques. In this method, data is first partitioned using one technique, and then each partition is further divided into sub-partitions using the same or a different method. Here all sub-partitions of a given partition together represent a logical subset of the data.</p>\n<p>Composite partitioning can be a useful for organizing and managing large datasets. It can help to improve the performance and scalability of the database by providing more precise control over data placement. </p>\n<p>There are several types of composite partitioning:</p>\n<p><strong>Composite Range-Range Partitioning:</strong> This method performs range partitioning based on two table entries. For example, we could first partition the data by date and then sub-partition the range by price.</p>\n<p><strong>Composite Range-Hash Partitioning:</strong> This method first partitions the data using range partitioning and then sub-partitions it using hash partitioning within each range partition.</p>\n<p><strong>Composite Range-List Partitioning:</strong> This method first partitions the data using range partitioning and then sub-partitions it using list partitioning within each range partition.</p>\n<p><strong>Composite List-Range Partitioning:</strong> This method performs range sub-partitioning within a list partition. For example, first perform list partitioning by country name and then perform range sub-partitioning by date.</p>\n<p><strong>Composite List-Hash Partitioning:</strong> This method sub-partitions list-partitioned data using the hash partitioning technique.</p>\n<p><strong>Composite List-List Partitioning:</strong> This method performs list partitioning based on two table dimensions. For example, perform list partitioning by country name and list sub-partitioning by customer account status.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*1aYojTvdxVJejBqmpBiuqg.gif\" alt=\"Composite partitioning of a database\" title=\"Composite partitioning\"></p>\n<h3>Effective Data Partitioning Design</h3>\n<ul>\n<li><strong>Query Processing</strong>: Effective data partitioning strategies can enhance query performance by using smaller datasets and leveraging parallelism. In addition to that, partitioning simplifies backup and recovery processes by focusing on smaller components rather than the entire database.</li>\n<li><strong>Application Considerations</strong>: Data partitioning's effectiveness heavily relies on specific requirements. While it can enhance availability, scalability, and performance, it introduces complexity to service design. So it's crucial to analyse data access, querying, and modification patterns to select the optimal partitioning approach.</li>\n<li><strong>Rebalancing Partitions</strong>: With increasing system traffic, services may experience disproportionate loads. Consequently, there might be a need to redefine partitioning strategies and migrate data from old partitions to new ones to maintain balance.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>Data Partitioning is the backbone of modern distributed data management systems. Data Partitioning proves very effective in improving the availability, scalability, and performance of the system. In this blog, we tried to present a full conceptual understanding of Data Partitioning. Hope you liked it. Please share your views in the comments below.</p>\n<p><strong>References</strong></p>\n<ul>\n<li><a href=\"https://docs.microsoft.com/en-us/azure/architecture/best-practices/data-partitioning\">https://docs.microsoft.com/en-us/azure/architecture/best-practices/data-partitioning</a></li>\n<li><a href=\"https://docs.oracle.com/cd/B28359_01/server.111/b32024/partition.htm\">https://docs.oracle.com/cd/B28359_01/server.111/b32024/partition.htm</a></li>\n<li><a href=\"https://dev.mysql.com/doc/mysql-partitioning-excerpt/5.7/en/partitioning.html\">https://dev.mysql.com/doc/mysql-partitioning-excerpt/5.7/en/partitioning.html</a></li>\n<li>Book: Designing Data-Intensive Applications</li>\n</ul>\n<p>Enjoy learning, Enjoy system design!</p>"},{"id":"f28d21df-5cf5-5d7d-82c4-891731ee39f5","frontmatter":{"title":"API Rate Limiter System Design","path":"/blog/design-api-rate-limiter"},"html":"<h3>What is a rate limiter?</h3>\n<p>A rate limiter restricts the number of events that a certain user or device can perform within a given time range. In other words, a rate limiter limits the number of requests a sender can send in a particular period. So it is implemented as a protective mechanism to prevent excessive use of services. Once the upper limit is reached, the rate limiter blocks further requests. </p>\n<p>In this blog, we will discuss components and algorithms related to the design of a rate limiter.</p>\n<h3>Why rate limiter?</h3>\n<ul>\n<li>Ensure that servers are not overburdened. So using a rate restriction per user ensures fair and reasonable use of resources like CPU, memory, network bandwidth, database connections, etc.</li>\n<li>Helps a system to maintain a consistent performance and remain responsive even during peak loads.</li>\n<li>Avoid resource starvation due to a Denial of Service (DoS) attack.</li>\n<li>In cloud-based systems or services where usage is billed based on resource consumption, rate limiting can help us control costs by preventing excessive usage.</li>\n</ul>\n<h3>Requirements</h3>\n<p>It’s critical to ask questions and clarify needs and restrictions with the interviewer during a system-design interview. Here are a few examples:</p>\n<ul>\n<li>Is the rate limiter configured to throttle API requests based on IP, user ID, or other parameters?</li>\n<li>What is the overall size of the system?</li>\n<li>Is the rate limiter on the client or the server-side API?</li>\n</ul>\n<h3>High-level design</h3>\n<p>Let’s keep things simple and communicate using a basic client-server approach. Here is a critical question: where should the rate limiter be placed? One can implement a rate limiter on either the client or server side.</p>\n<ul>\n<li>Client-side implementation: Hostile actors can quickly falsify client requests. So the client is an unstable venue to impose rate restrictions. Furthermore, we may not have complete control over client implementation.</li>\n<li>Server-side implementation: On the server side, a rate restriction is shown in the diagram below.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*USFG2voaDy9RmuZotqzBGw.png\" alt=\"Server-side rate limiting\" title=\"Rate limiting Server-side implementation\"></p>\n<p>There is an alternative to the client and server-side implementations. We construct a rate limiter middleware, which throttles requests to your APIs, rather than establishing a rate limiter on the API servers.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*iXLmGIrDhu8UA5jaFIe7FA.png\" alt=\"Rate limiter middleware\" title=\"Middleware of Rate limiter\"></p>\n<p>The underlying concept of rate-limiting algorithms is simple. We need a counter at the highest level to track how many requests are submitted from the same user, IP address, etc. The request is rejected if the counter exceeds the limit.</p>\n<p>Where should we keep the counters? Due to the slowness of disc access, using the database is not a smart option. So we can use an in-memory cache to make it quick and support a time-based expiration technique. For example, Redis is a popular choice for rate-limiting, where “INCR” and “EXPIRE” are two commands that can be used to access in-memory storage.</p>\n<ul>\n<li>INCR: Used to increase the stored counter by 1.</li>\n<li>EXPIRE: Used to set a timeout for the counter. The counter is automatically deleted when the timeout expires.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*ShYMdK2BVNoN44YLJsuq-A.png\" alt=\"How rate limiting works?\" title=\"Rate limiter high level design\"></p>\n<ul>\n<li>A request is sent to rate limiting middleware by the client.</li>\n<li>The rate-limiting middleware retrieves the counter from the associated Redis bucket and determines whether or not the limit has been reached.</li>\n<li>The request is refused if the limit is reached.</li>\n<li>The request is forwarded to API servers if the limit is not reached. </li>\n<li>In the meantime, the system adds to the counter and saves it to Redis.</li>\n</ul>\n<h3>Rate limiting algorithms</h3>\n<p>Now, let’s explore several rate-limiting algorithms available for implementing API rate limiting. Suppose we need to create an API Rate Limiter that limits API calls based on the user, where one user can send a fixed number of requests in a fixed time range. Based on the use case, we can change the implementation according to our criteria.</p>\n<h4>Token Bucket Algorithm</h4>\n<p>The idea of the token bucket algorithm works by maintaining a bucket with a fixed capacity of tokens. Here, tokens are added to the bucket at a constant rate, and each token represents permission to perform a certain action (such as making an API request).</p>\n<p>When an API request comes in, the system checks if there are tokens available in the bucket. If there is at least one token available, the system removes one token from the bucket and processes the request. If there are no tokens available, the request is either rejected or delayed until tokens are available.</p>\n<p>The token bucket gets refreshed or refilled after a specified time interval, which determines how quickly the bucket refills with tokens. For example, if a user is allocated 10 tokens per minute, they can make up to 10 requests per minute. Once the user uses up all their tokens within the time interval, they will be denied further requests until the bucket is refilled at the next interval.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*CvkTqyxVqWLw1W1DxRnW-Q.png\" alt=\"Token bucket algorithm for rate limiting\" title=\"Token bucket rate limiter\"></p>\n<ul>\n<li>Suppose there is a limit of 3 requests per minute per user.</li>\n<li>The user makes the first request at 10:00:00, and the available token is 3. So the request is approved, and the available token count is reduced to 2.</li>\n<li>Now, the user’s 2nd request arrives at 10:00:10, and the available token is 2. So the request is permitted, and the token count is decremented to 1.</li>\n<li>The 3rd request arrives at 10:00:35, with a token count of 1 available, so the request is granted, and the token count is decremented to 0.</li>\n<li>The 4th request arrives at 10:00:45, and the available count is already 0. So the API access is denied.</li>\n<li>Now, at 10:01:00, the count will be refreshed with the available token count of 3.</li>\n</ul>\n<p><strong>Pros:</strong> Memory friendly and simple to use. <strong>Cons:</strong> A race condition counter may cause an issue in a distributed system due to concurrent requests from the same user.</p>\n<p>You can explore the sample implementation for the token bucket using <a href=\"https://android.googlesource.com/platform/frameworks/base.git/+/master/core/java/com/android/internal/util/TokenBucket.java\">this link</a>.</p>\n<h4>Leaky Bucket Algorithm</h4>\n<p>We can understand the leaky bucket algorithm idea using this analogy: Suppose there is a bucket with a hole in the bottom. Water (API requests) flows into the bucket (queue), but it can only drain out through the hole at a fixed rate. If water flows into the bucket faster than it drains out, eventually, it will overflow.</p>\n<p>So, this algorithm uses a queue, which can be thought as a bucket that holds the requests. This queue is used to control that flow: requests are entered into the queue as they arrive (analogous to water being added to the bucket). These requests are then removed from the queue (first come, first served) at a fixed rate (comparable to water leaking from the bucket). If the queue is full (or leaked), any incoming request is rejected until there is space in the queue. Overall, this algorithm ensures that output is processed at a fixed rate.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*6hLdrngMTxAPTfc0NNODYw.png\" alt=\"Leaky bucket algorithm for rate limiting\" title=\"Leaky bucket rate limiter\"></p>\n<p>If 3 requests/user is allowed and the queue size is 3, the API Rate Limiter will block the 4th Request.</p>\n<h4>Fixed Window Counter</h4>\n<p>In this algorithm, we will keep a bucket per unit time window and a counter to track the number of requests made in that time window. What is the meaning of the time window here? Suppose you have a fixed time window of 1 minute, it means that the rate limit applies within each minute interval.</p>\n<p>Here we also set a threshold for the number of requests allowed within each time window. When a request is made, the algorithm increments the counter. Once the value of the counter reaches the value of the threshold, further requests are either delayed or rejected, until the start of the next time window when the counter resets.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*RD19FFIgP8pVhLe18NrwOw.png\" alt=\"Fixed window counter algorithm for rate limiting\" title=\"Fixed window counter rate limiter\"></p>\n<p>In the above diagram, a new request is seen at 00:00:10. So we get the count from the current bucket 00:00-00:01 (which is 0) and check if processing one more request exceeds the permissible limit of 3. If not (which is the case here), the bucket's count is incremented by 1 and the request is allowed to be processed. When a request arrives at 00:00:40 and the window count is 3, no request is authorised and an exception is triggered.</p>\n<p>Space complexity: O(1)  for storing the count for the current window.\nTime complexity: O(1)  for get and simple atomic increment operation.</p>\n<p><strong>Pros:</strong></p>\n<ul>\n<li>Easy to put into action.</li>\n<li>Smaller memory footprint, because all that’s done is store the counts.</li>\n<li>Can leverage Redis-like technology with built-in concurrency.</li>\n</ul>\n<h4>Sliding Window Logs</h4>\n<p>The algorithm maintains a log of request timestamps. This log can be implemented using a data structure like a queue or a circular buffer. One of the best ideas is to store these timestamps in a cache for faster retrieval and manipulation of the data.</p>\n<ul>\n<li>When a new request arrives, algorithm checks the timestamps in the log. Any timestamps older than the start of the current time window (i.e., timestamps that fall outside the defined time interval) are outdated and are removed from the log. This ensures that the log only contains timestamps relevant to the current time window.</li>\n<li>Once outdated timestamps have been cleared, the timestamp of the new request is added to the log. This effectively updates the log with the latest event.</li>\n<li>After updating the log with the new timestamp, the algorithm checks the size of the log against a predefined threshold (number of allowed requests within the defined time window). If the log size is equal to or less than this allowed count, the incoming request is accepted. Otherwise, if the log size exceeds the allowed count, the new request is dropped or rejected.</li>\n</ul>\n<p>For example, suppose the rate limiter allows 2 requests per minute, and the first request arrives at 1:00:00. Now, the second request arriving at 1:00:20 is allowed because the log has a size of 2, and it falls within the time range of 1:00:00 - 1:01:00. Log = [1:00:00, 1:00:20].</p>\n<ol>\n<li>The third request arriving at 1:00:45 is rejected because the log size is 3, exceeding the allowed size of 2. The timestamp of this rejected request is still maintained in the log. Log = [1:00:00, 1:00:20, 1:00:45].</li>\n<li>Subsequently, the 4th request arrives at 1:01:25. This request is allowed because outdated requests (1:00:00 and 1:00:20) are removed from the log, reducing its size to 2. Log = [1:00:45, 1:01:25].</li>\n<li>However, the 5th and 6th requests, arriving at 1:01:35 and 1:01:40 respectively, are rejected because the log size exceeds 2. Nevertheless, they are maintained in the log since they are within the 1-minute window. Log = [1:00:45, 1:01:25, 1:01:35, 1:01:40].</li>\n<li>The 7th request arrives at 1:02:30. Outdated requests (1:00:45 and 1:01:25) are removed, yet this request is rejected due to the log size being 3, still exceeding the limit of 2. However, like the previously rejected requests, it is maintained in the log as it falls within the 1-minute window. Log = [1:01:35, 1:01:40, 1:02:30].</li>\n</ol>\n<p><strong>Space complexity: O(Max requests seen in a window time)</strong>. In a time window, all of the requests’ timestamps are saved.</p>\n<p><strong>Time complexity: O(Max requests seen in a window time)</strong> — A portion of timestamps is deleted.</p>\n<p><strong>Pros:</strong> It works flawlessly</p>\n<p><strong>Cons:</strong></p>\n<ul>\n<li>Memory usage is high. To manage numerous users or huge window times, all of the request timestamps must be kept for a window time, which necessitates a lot of memory.</li>\n<li>Removing earlier timestamps has high time complexity.</li>\n</ul>\n<h4>Sliding Window Counters</h4>\n<p>The low processing cost of the fixed window algorithm is combined with the increased boundary constraints of the sliding log in this hybrid approach. To begin, we track a counter for each fixed window, similar to the fixed window technique. To reduce surges of traffic, we account for a weighted value of the previous window’s request rate based on the current date.</p>\n<p>We keep track of each user’s request counts by using various fixed time windows, such as 1/60th the size of our rate limit’s time frame. If we have a minute rate limit, for example, we can record a count for each second and calculate the sum of all counters in the previous minute when we get a new request to determine the throttling limit.</p>\n<ol>\n<li>Remove any counters that are more than one minute old.</li>\n<li>If a request is received in the current bucket, the counter will be increased.</li>\n<li>If the current bucket has reached its throat limit, the request will be blocked.</li>\n</ol>\n<p>Space Complexity: O(number of buckets).</p>\n<p>Time Complexity: O(1).</p>\n<p><strong>Pros:</strong></p>\n<ul>\n<li>Because only the counts are saved, there is no big memory footprint.</li>\n</ul>\n<p><strong>Cons:</strong></p>\n<ul>\n<li>Only works for non-strict look-back window times, such as smaller unit times.</li>\n</ul>\n<h3>References</h3>\n<ol>\n<li><a href=\"https://cloud.google.com/solutions/rate-limiting-strategies-techniques\">Rate-limiting strategies and techniques — Google Cloud</a></li>\n<li><a href=\"https://blog.cloudflare.com/counting-things-a-lot-of-different-things/\">How we built rate-limiting capable of scaling to millions of domains</a></li>\n</ol>\n<p>Thanks to <strong>Navtosh Kumar</strong> for his contribution to creating the first draft of content. Please write in the message below if you find anything incorrect, or if you want to share more insight. Enjoy learning, Enjoy algorithms!</p>"},{"id":"14e5ed2c-82a7-5599-9b31-943367d6b8fc","frontmatter":{"title":"Dropbox System Design","path":"/blog/design-dropbox"},"html":"<h3>What is Dropbox?</h3>\n<p>Dropbox is a cloud storage system that allows users to store their data on remote servers. The remote servers store files durably and securely, and these files are accessible anywhere with an internet connection.</p>\n<h3>Requirements of the System</h3>\n<h4><strong>Functional Requirements</strong></h4>\n<ol>\n<li>Users should be able to sign up using their email addresses and subscribe to a premium plan. If they don’t subscribe, they will get 1 GB of free storage.</li>\n<li>Users should be able to upload and download files from any device.</li>\n<li>Users should be able to share files and folders with other users.</li>\n<li>Users should be able to upload files up to 1 GB.</li>\n<li>The system should support automatic synchronization across devices.</li>\n<li>The system should support offline editing. Users should be able to add/delete/modify files offline, and once they come online, changes should be synchronized to remote servers and other online devices.</li>\n<li>ACID-ity is required: Atomicity, Consistency, Isolation, and Durability of all file operations should be guaranteed.</li>\n</ol>\n<h4><strong>Non-functional requirements</strong></h4>\n<ul>\n<li>The system should be highly reliable. Any file uploaded should not be lost.</li>\n<li>The system should be highly available.</li>\n</ul>\n<h3>Capacity Estimation</h3>\n<p>Let’s do some back-of-the-envelope calculations.</p>\n<p><strong>Assumptions</strong></p>\n<ul>\n<li>Total number of users = 500 million.</li>\n<li>Total number of daily active users = 100 million.</li>\n<li>Average number of files stored by each user = 200.</li>\n<li>The average size of each file = 100 KB.</li>\n<li>Total number of active connections per minute = 1 million.</li>\n</ul>\n<p><strong>Storage Estimations</strong></p>\n<ul>\n<li>Total number of files = 500 million * 200 = 100 billion</li>\n<li>Total storage required = 100 billion  * 100 KB = 10 PB</li>\n</ul>\n<h3>High-Level Design</h3>\n<p>The Dropbox system needs to deal with a huge volume of read and write data, and their ratio will remain almost the same. So while designing the system, we should focus on optimizing the data exchange between client and server.</p>\n<p>At a high level, we need to store files and their metadata information like file name, file size, directory, etc., and who this file is shared with. So, we need servers to help the client upload/download files to cloud storage and some servers to facilitate updating metadata about files and users. We also need some mechanism to notify all clients whenever an update happens to synchronize their files.</p>\n<p>As shown in the diagram below, block servers will work with the clients to upload/download files from cloud storage, and metadata servers will keep metadata of files updated in an SQL or NoSQL database. Synchronization servers will handle the workflow of notifying all clients about different changes for synchronization.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*unKlNEfSwqZb_SqDGT-bAg.png\" alt=\"High level design of dropbox system\" title=\"Dropbox high level design\"></p>\n<h3>Detailed Design</h3>\n<h4>1. Client Application</h4>\n<p>The client application is responsible for monitoring the changes in the workspace. It interacts with the synchronization service to process metadata updates like changes in the file name or contents. It is also responsible for indexing the file, sending the updated chunks to the cloud storage, and retrieving the same if other clients have updated the file.</p>\n<p><strong>Major components of the client application:</strong></p>\n<ul>\n<li><strong>Watcher</strong>: Responsible for monitoring and synchronizing the files and folders for any create, update or delete operation and then informing the indexer component to handle the changes.</li>\n<li><strong>Chunker</strong>: A chunker's job is to split the files and the incremental changes into chunks of a suitable size of ~4MB. These chunks can be later joined in the same order to reconstruct the original file. This component can intelligently sense the changes that are done on the file and transmit only those parts to the storage server.</li>\n<li><strong>Indexer:</strong> Responsible for listening to the Watcher and maintaining the information about the chunks of files. The indexer also syncs this data with the Metadata storage server using a synchronization service to successfully store chunks in cloud storage.</li>\n<li><strong>Internal Database</strong>: Keeps a record of the chunks and associated metadata. It allows for offline operations when the client is not connected to the Dropbox server.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*7Y0smzuX1ewPpzArjxfbLg.png\" alt=\"Client application components in dropbox system design\" title=\"Components of the client application in dropbox design\"></p>\n<h4>2. Metadata Service</h4>\n<p>The metadata database is responsible for maintaining the version and metadata information about files/chunks, users, and workspaces. It can be a relational database such as MySQL or a NoSQL database service such as DynamoDB. </p>\n<p>Regardless of the type of database, the synchronization service should provide a consistent view of the files using a database, especially if more than one user is working with the same file simultaneously. </p>\n<p>Since NoSQL data stores do not support ACID properties in favour of scalability and performance, we need to incorporate the support for ACID properties in our synchronization service's logic if we opt for this kind of database. However, using a relational database can simplify the synchronization service implementation as they natively support ACID properties.</p>\n<h4>3. Synchronization Service</h4>\n<p>It is one of the critical components of cloud storage design. It processes all the client's updates on the file and synchronizes those updates across all the devices. It updates the client's local database to be in sync with the metadata stored on the server. All Dropbox clients, including desktop, mobile, and web clients, talk to the synchronization service to get updates from the server or push updates to the server. </p>\n<p>This way, all clients are in sync with the master copy stored in the Dropbox cloud. When the client is offline, all updates are stored locally, and when the client becomes online, the synchronization service syncs the data to metadata storage. The same is subsequently pushed to other clients or shared workspace users. </p>\n<p>It is also possible that two clients have made changes to the same file offline. Dropbox handles such scenarios by creating a conflicted copy and saving it with the editor’s username and the save date. Users will be required to resolve that conflict manually.</p>\n<h4>4. Message Queuing Service</h4>\n<p>An important part of our architecture is a messaging middleware that will handle substantial reads and writes. So a scalable message queuing service that supports asynchronous message-based communication between clients and the synchronization service instances best fits our application's requirements. </p>\n<ul>\n<li>The message queuing service supports asynchronous and loosely coupled communication between distributed components.</li>\n<li>This service will provide high performance, high scalability, and persistently store any number of messages in a highly available and reliable queue.</li>\n<li>It also provides load balancing and elasticity for multiple instances of the synchronization service. </li>\n</ul>\n<p>The figure below illustrates two types of queues that are used in our message queuing service. The <strong>Request Queue</strong> is a global queue that is shared among all clients. Clients’ requests to update the metadata database through the synchronization service will be sent to the request queue. </p>\n<p>The response queues that correspond to individual subscribed clients are responsible for delivering the client's update messages. Since a message will be deleted from the queue once received by a client, we need to create separate <strong>Response Queues</strong> for each client to share an update message that should be sent to multiple subscribed clients.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*b7OGbjsy_qwf85xAUc8wSA.png\" alt=\"Dropbox system design message queuing service\" title=\"Message queuing service in dropbox system design\"></p>\n<h4>5. Cloud Storage</h4>\n<p>Cloud storage stores the chunks of the files uploaded by the users. Clients directly interact with cloud storage to send and receive objects using the cloud provider's API. The separation of the metadata from the object storage enables our reference architecture to use any cloud storage as the back-end data store.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*4WVp8WD9AaoyhZzLRuk2_A.png\" alt=\"Dropbox system design cloud storage architecture\" title=\"Cloud storage architecture dropbox system design\"></p>\n<h3><strong>References for further reading</strong></h3>\n<ul>\n<li><a href=\"https://www.dropbox.com/business/trust/security/architecture\">Dropbox Architectur</a><a href=\"https://dropbox.tech/infrastructure/streaming-file-synchronization\">e</a></li>\n<li><a href=\"https://dropbox.tech/infrastructure/streaming-file-synchronization\">Streaming File Synchronization</a></li>\n<li><a href=\"https://www.youtube.com/watch?v=PE4gwstWhmc\">How We've Scaled Dropbox</a></li>\n</ul>\n<p>Thanks to Navtosh for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"9d46eaf7-d62f-5ecb-bc1c-53d95594470a","frontmatter":{"title":"Designing Instagram: System Design Interview Question","path":"/blog/design-instagram"},"html":"<p>Instagram is a social media platform that enables users to share photos and videos with others. On the platform, content creators can choose to set the visibility of their posts (photos/videos) to either private or public, and other users can interact with the posts by liking or commenting on them.</p>\n<p>Instagram also offers a range of other functionality, including the ability for users to follow each other, see each other's news feeds, and search for content across the entire platform. Other features available on Instagram include image editing, location tagging, private messaging, push alerts, group messaging, hashtags, filters, and more.</p>\n<p>In this blog, we will look at how to design a simplified version of the Instagram system with features such as photo sharing, following, and news feeds.</p>\n<h3>Requirements of the system</h3>\n<h4><strong>Functional requirements</strong></h4>\n<ul>\n<li>Users should be able to upload and view photos.</li>\n<li>The system should allow users to search for photos based on their titles.</li>\n<li>Users should be able to follow each other.</li>\n<li>Each user should have a custom news feed that displays the top photos from the users they follow.</li>\n</ul>\n<h4><strong>Non functional requirements</strong></h4>\n<ul>\n<li>The system should prioritize high availability and low latency while viewing photos. To improve availability, we can trade off consistency, meaning it's acceptable if a user doesn't see an image immediately.</li>\n<li>The system should be highly scalable and optimized for read-heavy workloads, with a high read-to-write ratio.</li>\n<li>The system should be reliable, ensuring that no uploaded photos or videos are lost.</li>\n<li>The system should be optimized for accessing popular posts.</li>\n<li>The system should be compatible with a wide range of devices, support multiple languages, and work well with varying internet bandwidths.</li>\n</ul>\n<h3>Capacity estimation</h3>\n<p>It's important to consider that read requests will be significantly more frequent than write requests, with a ratio of approximately 100 to 1.</p>\n<ul>\n<li>Let's assume there are 500 million users registered on the platform, with 1 million active users per day.</li>\n<li>If 5 million images are posted daily, this translates to an average of 57 photos being uploaded per second (5M / (246060)).</li>\n<li>If the average photo size is 150 KB, then the daily storage usage is 716 GB (5M * 150KB).</li>\n<li>If we assume the service will be active for ten years, the total space required will be approximately 2.6 PB (716GB * 365 * 10).</li>\n</ul>\n<h3>High Level Design</h3>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*tHnk4INWGdTcKJL8gYYfyg.png\" alt=\"Designing Instagram high level design\" title=\"High level design of Instagram system design\"></p>\n<p>The user service is responsible for managing user onboarding, login, and profile-related actions. It runs on a MySQL database, which was selected because the data is structured in a relatively relational way and the system is optimized for read-heavy workloads, which MySQL is well-suited to handle.</p>\n<p>The user service is connected to a Redis database, which stores all user data. When the user service receives a request, it first checks Redis for the requested information and returns it to the user if it is found.\nIf the data is not present in Redis, the user service will check the MySQL database, retrieve the data, and insert it into Redis before returning it to the user. Additionally, a similar process can be followed whenever new users or information are added to the database. </p>\n<p>This approach allows the user service to quickly access and returns the most up-to-date data to the user while maintaining a cache of frequently accessed information in Redis to improve performance.</p>\n<h4>System Components</h4>\n<p>Our system will be composed of multiple microservices, each responsible for a specific task. To store the data, we will use a graph database like Neo4j. We've chosen this data model because our data contains complex relationships between elements like users, posts, and comments, which can be represented as nodes in the graph. The edges of the graph can be used to record relationships like follows, likes, and comments. We may also use columnar databases like Cassandra to store information such as user feeds, activities, etc.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*18X120QrSQj29kAoiNogtw.png\" alt=\"Components of Instagram system design\" title=\"Designing Instagram system components\"></p>\n<h3>Data flow and API design</h3>\n<h4>Data flow</h4>\n<ol>\n<li>The user sends a request to the API.</li>\n<li>The load balancer receives the request and forwards it to an app server.</li>\n<li>The app server receives the request and performs input validation.</li>\n<li>If the input is valid, the app server attempts to fulfill the request.</li>\n<li>If successful, the app server returns an OK response with or without the requested data. If there is an issue, it returns a specified error response.</li>\n</ol>\n<h3>API design</h3>\n<ul>\n<li><strong>signup</strong> (username, first<em>name, last</em>name, salted<em>password</em>hash, phone_number, email, bio, photo): adds the user to the user table</li>\n<li><strong>login</strong> (username, salted<em>password</em>hash): logs in the user and updates the last login time</li>\n<li><strong>search_user</strong> (search<em>string, auth</em>token): returns public user data for the given search string (can be searched in user first name, last name, and username)</li>\n<li><strong>get<em>user</em>by_id</strong>(user<em>id, auth</em>token): returns public user data for the given user ID</li>\n<li><strong>follow_user</strong>(user<em>id, target</em>user<em>id, auth</em>token): adds follow data to the database</li>\n<li><strong>add_post</strong>(file, caption, user<em>id, auth</em>token): uploads the file to the file storage server</li>\n<li><strong>delete_post</strong>(user<em>id, post</em>id, auth_token): deletes the given user's given post along with its metadata (using soft delete)</li>\n<li><strong>get_feed</strong>(user<em>id, count, offset, timestamp, auth</em>token): returns the top posts after the given timestamp of users followed by the given user according to count and offset</li>\n<li><strong>get<em>user</em>posts</strong>(user<em>id, count, offset, auth</em>token): returns the posts of the given user according to count and offset</li>\n<li><strong>post_like</strong>(user<em>id, post</em>id, auth_token): adds the given post ID to the given user's likes</li>\n<li><strong>post_unlike</strong>(user<em>id, post</em>id, auth_token): removes the given post ID from the given user's likes</li>\n<li><strong>add_comment</strong>(user<em>id, post</em>id, comment): adds a comment to the given user's comment on the given post</li>\n<li><strong>delete_comment</strong>(user<em>id, comment</em>id): deletes the given user's comment with the given comment ID</li>\n</ul>\n<h3>Database Design</h3>\n<p>It's important to clearly define the database structure early in the interview process to help understand the flow of data between different components and determine how to segment the data.</p>\n<p>We need to store data about users, their posted images, and the people they follow. To efficiently retrieve recent photos from the photo table, which stores all data related to a photo, we will create an index on (PhotoID, CreationDate).</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*MSRy_GMUFaYK7gwaKUSl7g.png\" alt=\"Database design of Instagram system\" title=\"Instagram system database design\"></p>\n<p>One option for storing the data described above, which requires joins, is to use a relational database management system (RDBMS) like MySQL. However, RDBMSs can have challenges with scaling. Instead, we could store photos in a distributed file system like HDFS or S3.</p>\n<p>To take advantage of NoSQL's features, we can store the schema described above in a distributed key-value store. We can create a table with a \"key\" of \"PhotoID\" and a \"value\" of an object containing all the metadata for the photo, such as PhotoLocation, UserLocation, CreationTimestamp, etc. This allows us to store and retrieve data using a simple key-value interface.</p>\n<p>To know who owns which photo, we need to store relationships between users and pictures. We also need to keep track of who a user follows. We can use a wide-column datastore like Cassandra 28 for both of these tables. The ‘key’ for the ‘UserPhoto’ table would be ‘UserID,’ and the ‘value’ would be the user’s list of ‘PhotoIDs,’ kept in distinct columns. The ‘UserFollow’ table will follow a similar pattern.</p>\n<p>Like other key-value stores, Cassandra maintains a set number of replicas to ensure reliability. Deletes are also not implemented immediately in key-value stores, as data is typically retained for a certain number of days to allow for undeletion before being permanently erased from the system. This helps to ensure data consistency and recoverability.</p>\n<h3>News Feed Generation</h3>\n<h4>Generating news feed</h4>\n<p>Designing a customized newsfeed for each user that showcases the most recent post from each user they are following is a critical aspect of an Instagram-like service. For the sake of simplicity, let's assume that each user and their followers upload 200 unique photos per day. This means that a user's newsfeed will consist of a combination of these 200 unique photographs, followed by the reputation of previous submissions. This allows the user to see the most recent and relevant content from the users they follow.</p>\n<p>To generate a news feed for a user, we will first retrieve the metadata (such as likes, comments, time, location, etc.) of the most recent 200 photographs and pass it to a ranking algorithm. This algorithm will use the metadata to determine the order in which the photos should be displayed in the news feed. This allows the user to see the most relevant and engaging content at the top of their feed.</p>\n<p>One disadvantage of the news feed generation approach described above is that it requires simultaneously querying a large number of tables and ranking them based on predefined criteria. This can result in higher latency, meaning it takes a longer time to generate a news feed. To improve performance, we may need to optimize the queries and ranking algorithms, or consider alternative approaches such as pre-computing and caching the results.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*joJoJmbF8PpNcNlwGWu15Q.png\" alt=\"News feed generation of Instagram system\" title=\"Instagram system news feed generation \"></p>\n<p>To address the latency issues with the news feed generation algorithm described above, we can set up a server that pre-generates a unique news feed for each user and stores it in a separate news feed table. When a user wants to access their news feed, we can simply query this table to retrieve the most recent content. This approach reduces the need to query and rank a large number of tables in real-time, improving the performance and responsiveness of the system.</p>\n<h4>Serving the news feed</h4>\n<p>We have now discussed how to create a news feed. The next challenge in designing the architecture of an Instagram-like service is determining how to deliver the generated news feed to users.</p>\n<p>One approach is to use a <strong>push</strong> mechanism, where the server alerts all of a user's followers whenever they upload a new photo. This can be done using a technique called long-polling. However, this approach may be inefficient if a user follows a large number of people, as the server would need to push updates and deliver notifications frequently.</p>\n<p>An alternative approach is to use a <strong>pull</strong> mechanism, where users refresh their news feeds (send a request to the server) to see new content. However, this can be problematic because new posts may not be visible until the user refreshes, and many refreshes may return empty results.</p>\n<p>A <strong>hybrid</strong> approach combines the benefits of both push and pull mechanisms. For users with a large number of followers (such as celebrities), the server can use a pull-based approach. For all other users, the server can use a push-based approach. This allows for efficient delivery of updates while minimizing the burden on the server.</p>\n<h3>Load balancing</h3>\n<p>To handle user requests, we need to use a load balancer to distribute requests among app servers. One way to do this is to use a round-robin technique, where requests are distributed in a rotating fashion. However, this approach can be problematic if a server is unavailable, as requests may still be sent to it. To prevent this, we can implement a \"heartbeat\" system in which each server pings the load balancer at a set interval to indicate that it is not down.</p>\n<p>Load balancers are also necessary for database and cache servers, which may also be distributed. To route requests to the appropriate server, we can use a technique called consistent hashing. This involves mapping each request to a specific server based on user-specific information. This helps to ensure that requests are routed to the correct server and that the load is evenly distributed.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*nMJiMDgpqkSNwjmgAsXCQQ.png\" alt=\"Load balancing in Instagram system design\" title=\"Instagram system design load balancing\"></p>\n<p>To evenly distribute the load among servers, we can use a load balancing algorithm called the \"Least Bandwidth Method.\" This algorithm selects the server with the least amount of traffic (measured in megabits per second) to handle a request.</p>\n<p>Load balancers can be placed at two points in the system: between the client and the server, and between the database and the server. This allows for efficient routing of requests and helps to ensure that the system can handle a large volume of traffic.</p>\n<p>Thanks to Navtosh Kumar for his contribution in creating the first version of this content.  In case of any queries and feedback, feel free to write us at contact@enjoyalgorithms.com. Enjoy learning, enjoy system design!</p>"},{"id":"d1321856-79da-52e1-a6f2-225ae60ec0d9","frontmatter":{"title":"Design Pastebin: System Design Interview Question","path":"/blog/design-pastebin-system-design-interview-question"},"html":"<p><strong>Asked In:</strong> Amazon, Ola Cabs</p>\n<h3>What is Pastebin?</h3>\n<p>Have you ever wanted to store data online and make it accessible to others? If so, then you may be interested in the Pastebin system design. Pastebin is a service that lets users post various types of content, such as text and images, on the internet and share them with others using a unique URL. The creator of the content can also update it if they are logged in.</p>\n<h3>Key Requirements</h3>\n<h4><strong>Functional Requirements</strong></h4>\n<ul>\n<li>Users should be able to generate a unique URL by pasting content into the system.</li>\n<li>The content and URLs should expire after a specified amount of time</li>\n</ul>\n<h4><strong>Non-Functional Requirements</strong></h4>\n<ul>\n<li>System should be reliable and have high availability.</li>\n<li>System should be available in real-time with minimal latency.</li>\n<li>Paste URL should not be predictable.</li>\n</ul>\n<p>As we know what our essential requirements are, it’s time to estimate capacity of the system.</p>\n<h3>Capacity Estimation</h3>\n<p>Since this system is expected to receive more reads than writes, let's consider a read-to-write ratio of 10:1. Suppose we have 10 million users, and each user makes one write request per month on average. This means we can expect the following traffic:</p>\n<ul>\n<li>10 million write requests per month.</li>\n<li>100 million read requests per month (10 times the number of write requests).</li>\n<li>1.2 billion write requests over 10 years (10 million per month * 12 months/year * 10 years).</li>\n<li>Suppose the maximum size per paste is 10 MB, resulting in 100 TB of paste per month (10 million pastes * 10 MB/paste).</li>\n<li>1.2 PB of paste content over a year (100 TB/month * 12 months/year).</li>\n<li>Total 12PB of data will be accumulated over 10 years (1.2 PB/year * 10 years).</li>\n</ul>\n<h3><strong>High Level Design</strong></h3>\n<p><img src=\"https://ucarecdn.com/b2380e83-af8d-4ded-a25d-2788c83c5575/\" alt=\"Pastebin high level design\" title=\"Design pastebin system\"></p>\n<p>In the Pastebin system, we will have two main APIs: One for writing content and another for handling reading requests. These APIs can be implemented using either the SOAP or REST architecture. </p>\n<h4><strong>Write API:</strong> write(apiKey, content, expiryDate)</h4>\n<ul>\n<li><strong>apiKey:</strong> A unique API key associated with each user.</li>\n<li><strong>content:</strong> The user provides the content that they want to paste.</li>\n<li><strong>expiryDate:</strong> The expiry date indicates the time after which the content should expire.</li>\n<li>The API returns a short URL that can be used to access the content. If the input is invalid, an error code will be returned.</li>\n</ul>\n<p>To create a paste, the client sends a request to the web server, which forwards it to the write API server. The write API server then performs the following actions:</p>\n<ul>\n<li>Generates a unique URL.</li>\n<li>Stores the URL in the pastes table of the SQL database.</li>\n<li>Stores the content in an object store and returns the URL.</li>\n</ul>\n<h4>Read API: read (shortUrl)</h4>\n<ul>\n<li><strong>shortUrl:</strong> The URL is provided by the write API to access the content.</li>\n<li>The API returns the original paste content or an error code if the URL is incorrect.</li>\n</ul>\n<p>To view a paste, the client sends a request to the web server, which forwards it to the read API server. The read API server then performs the following actions:</p>\n<ul>\n<li>Checks the SQL database for the generated URL.</li>\n<li>Retrieves the paste content from the object store if the URL is present in the database.</li>\n<li>Returns an error message to the user if the URL is not found.</li>\n</ul>\n<h4>Delete API: delete (apiKey, URL)</h4>\n<ul>\n<li><strong>apiKey:</strong> A unique API key provided to each user.</li>\n<li><strong>URL:</strong> The URL of the content that the user wants to delete.</li>\n<li>Delete API returns a boolean value indicating the success or failure of the operation.</li>\n</ul>\n<p>In addition to these essential services, we also have two additional APIs that handle real-time analytics and the deletion of expired content.</p>\n<h3>Database Design</h3>\n<p>Both the content and short URL need to be stored. The average size of each content is 10 MB. As previously calculated, our service will need to store a total of 12 PB of data over the course of 10 years, which is too much to store in a traditional database. Therefore, we will use an object storage service, such as Amazon S3, to store the content and a relational database to store the URLs.</p>\n<h4>Database Structure</h4>\n<p>The database has the following structure:</p>\n<p><strong>User Data</strong></p>\n<ol>\n<li>User ID: A unique identifier for each user.</li>\n<li>Name: The name of the user.</li>\n<li>Email: The email address of the user.</li>\n<li>Password: The password of the user.</li>\n<li>Creation Date: The date on which the content was shared.</li>\n</ol>\n<p><strong>Content Data</strong></p>\n<ol>\n<li>User ID: A unique identifier for each user.</li>\n<li>Short URL: A unique, 7-character short URL.</li>\n<li>Paste path: The URL path of the S3 bucket.</li>\n<li>Expiration date: The time after which the content expires.</li>\n<li>Last accessed: The last time the content was accessed.</li>\n</ol>\n<p>The User ID serves as the primary key in this structure.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/1080/1*3Swh5Tgmbwnjch0UIYIu6Q.jpeg\" alt=\"database design of pastebin system\" title=\"pastebin system database design \"></p>\n<p>For storing paste URLs, we have two options: a relational database like MySQL or a NoSQL database. Since we need fast read and write speeds but do not have many dependencies or relationships among our data, a NoSQL database (like Key-Value Store) is the optimal choice for our system.</p>\n<p>Relational databases are very efficient when there are many dependencies and complex queries, but they can be slow for our purposes. On the other hand, NoSQL databases may not be as good at handling relationship queries, but they are faster overall.</p>\n<h3><strong>Low Level Design</strong></h3>\n<p>The key components of the pastebin system are generating short URLs and storing content. Let's dive into these topics in more detail.</p>\n<h4><strong>Generating Short URL</strong></h4>\n<p>Every time a user creates a paste, the system needs to generate a unique, short URL. For this, we can use BASE64 encoding because all BASE64 characters are URL-safe (i.e., they are within the range of [A-Z], [a-z], [0-9], '+', and '/').</p>\n<ul>\n<li>To generate a unique URL, we can use the MD5 hash algorithm. MD5 is a cryptographic hash function algorithm that takes input data of any length and produces a fixed-size output of 128 bits or 16 bytes.</li>\n<li>We take the MD5 hash of the user's IP address and timestamp (or randomly generated data) to create a unique URL for the paste. Now we encode the MD5 result using BASE64, which will be 22-character string. Note: Base64 encoding will use 6 bits to represent each character. So Base64 encoding of MD5 output will give 128/6 i.e. ~ 22 character output.</li>\n<li>N﻿ow for choosing the short URL, we can choose the first 6 characters of the resulting Base64 encoding.</li>\n</ul>\n<p>The critical idea to explore: What is the MD5 hash algorithm? How does it work? What are its use cases in system design?</p>\n<h4><strong>The drawback of the above approach</strong></h4>\n<p>One issue with this algorithm is that the generated URL may be repeated. In such a case, we need to regenerate a new key and keep trying until we don't encounter any failure due to the duplicate key. To address this, we can use a Key Generating Service (KGS) to ensure that all keys inserted into the key database are unique and prevent collisions or duplications.</p>\n<p>Key Generation Service (KGS) generates random six-letter strings beforehand and stores them in some database. Whenever we want to store a new paste, we can select an already generated key from the DB and use it. KGS will also ensure that all keys inserted into DB are unique. </p>\n<ul>\n<li>KGS can use two tables to store keys: One for unused keys and one for all used keys. As soon as KGS provides keys to an application server, it can move them to the used keys table, ensuring that each server gets unique keys.</li>\n<li>We can have a standby replica of KGS to prevent a single point of failure. This replica can take over and generate and provide keys whenever the primary server fails.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/f682b64d-e2b1-4be9-a9c7-2a37e9b79ce9/\" alt=\"high level architecture of pastebin system design\" title=\"high level design of pastebin system\"></p>\n<h3>Scaling the System</h3>\n<p>Scalability, reliability, availability, and performance are critical business requirements for this service. It should be able to handle a large number of requests and provide a fast, seamless experience for users. To achieve this, we will need to scale our database and have additional servers to handle the increased traffic.</p>\n<h4>Database Sharding</h4>\n<p>If we use a single database to store all of the data, the service will be more prone to failure. To prevent this, we can partition the data across multiple machines or nodes to store the billions of URLs.</p>\n<p>To partition the database, we can use a technique called hash-based partitioning. This involves using a hash function to distribute the URLs into different partitions. We need to decide how many shards to create and choose an appropriate hash function that maps each URL to a specific partition or shard number.</p>\n<h4>Caching</h4>\n<p>Since we have ten times more read operations than write operations, we can use caching to speed up the service. Caching involves storing frequently accessed URLs in memory for faster access. For example, if a URL appears on a trending page on a social networking website, it is likely to be visited by many people. In this case, we can store the content of the URL in our cache to avoid delays. Services like Memcached or Redis can be used for this purpose.</p>\n<p>There are several considerations to keep in mind when designing the cache:</p>\n<ul>\n<li>When the cache is full, we need to replace less frequently used URLs with more popular ones. We can use the Least Recently Used (LRU) method to implement this.</li>\n<li>The cache should be kept in sync with the original content. If any changes are made to the original content, they should be reflected in the cache.</li>\n</ul>\n<p>We can use a distributed cache to address these issues.</p>\n<h4>Load Balancing</h4>\n<p>There may be instances where a single server receives a large number of requests, causing the service to fail. To prevent this single point of failure, we can use load balancing.</p>\n<p>There are many algorithms available for distributing the load across servers, but in this system, we can use the least bandwidth method. This algorithm directs incoming requests to the server with the least amount of traffic.</p>\n<p>In addition to load balancing, we also need to replicate all databases and servers to ensure that the service remains highly consistent and does not go down in case of any discrepancies.</p>\n<h3>Conclusion</h3>\n<p>Pastebin system is a complex and highly scalable service to design. In this blog, we have covered the fundamental concepts necessary for building such a service. We hope you like the content. Please do share your views.</p>\n<p>Thanks to Chiranjeev and Suyash for their contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"417d7332-fce9-566e-b1fc-8bd291969c1f","frontmatter":{"title":"Design Key-Value Store","path":"/blog/design-key-value-store"},"html":"<h3>What is a Key-Value Store?</h3>\n<p>A key-value database is a non-relational database that stores data using a simple key-value system. It stores data as a collection of key-value pairs, with the key serving as a unique identifier for the value. Both key and value can be any type of object, from simple to complex compound objects. </p>\n<p>Key-value databases are highly partitionable and can scale horizontally to a larger extent than other databases. For example, Amazon DynamoDB will allocate additional partitions to a table if an existing partition becomes full and more storage space is needed.</p>\n<h3><strong>Requirements of the System</strong></h3>\n<p>Coming up with features that the system should offer is the initial stage of any system design interview. As an interviewee, we should try to think of as many features as possible that our system should support.</p>\n<ul>\n<li>Data size: Size of values is too large to be held in memory, and we should leverage the external storage for them. However, we can still keep the data keys in memory.</li>\n<li>Let’s assume a few 100 TB.</li>\n<li>Should support updates.</li>\n<li>Let’s assume that there is an upper cap of 1GB to the size of the value.</li>\n</ul>\n<h4><strong>Functional Requirements</strong></h4>\n<ul>\n<li>Set a value for a key (or update the value if it already exists).</li>\n<li>Retrieve the value that the key specifies.</li>\n<li>Remove the value from the key (unset the key).</li>\n</ul>\n<h4><strong>Non Functional Requirements</strong></h4>\n<ul>\n<li><strong>High scalability:</strong> System should be able to handle a sudden increase in load by adding more instances.</li>\n<li><strong>Consistency:</strong> System should provide a consistent and correct response to every request.</li>\n<li><strong>Durability:</strong> System should not lose data in the event of network partition failures.</li>\n<li><strong>Availability:</strong> System should not have a single point of failure. According to the CAP theorem, it should be a CP system, which means consistency takes precedence over availability.</li>\n</ul>\n<h4><strong>Capacity Estimation</strong></h4>\n<p>This is usually the second part of a design interview: Coming up with the estimated numbers of how scalable our system should be. Important parameters to remember for this section are the number of queries per second and the data which the system will be required to handle. Let’s assume:</p>\n<ul>\n<li>Total Storage: 100 TB.</li>\n<li>Total queries per second (QPS): 100K.</li>\n</ul>\n<h3>How would we design a simple key-value storage system in a single machine?</h3>\n<p>A hash table is a type of data structure that allows for fast read and write access to key-value pairs. It is simple to use and is supported by most programming languages. However, the disadvantage of using a hash table is that the entire data set must be stored in memory, which may not be practical for large data sets.</p>\n<p>There are two common solutions for dealing with large data sets in a hash table:</p>\n<ul>\n<li>Compressing the data: This involves reducing the amount of space needed to store the data by using techniques such as saving references rather than actual data, using smaller data types (e.g. float32 instead of float64), and using specialized data representations like bit arrays or vectors.</li>\n<li>Storing data on a disk: When it is not possible to fit the entire data set in memory, it is possible to store a portion of the data on a disk and keep the frequently accessed data in memory. This can be thought of as a caching system to optimize performance.</li>\n</ul>\n<h3>Distributed key-value storage</h3>\n<p>Designing a distributed key-value storage system is a complex task, especially when it comes to scaling over multiple machines. This is an important issue to consider when dealing with large amounts of data, as it will likely require a distributed system to support it. Let’s look at how a distributed key-value storage system could be designed.</p>\n<p>To store large amounts of data, we need to divide it among multiple machines. This is known as data partitioning. One way to do this is to use a coordinator machine to direct clients to the machine that has necessary resources. However, the challenge is to effectively divide the data among the machines and come up with a good data partitioning approach.</p>\n<h4>Database Sharding</h4>\n<p>To store 100TB of data, a single system may not be able to handle such a large amount. If we want to use a single machine, it would have to handle all requests, which would cause performance issues. So what is the solution? To avoid this, we can divide data into smaller pieces and distribute them across multiple machines (shards). This way, we can better manage the load and improve the performance.</p>\n<p>Let's take an example of messaging application where each user has their own inbox. In this situation, we can store data in a denormalized manner by sharding based on each user as a row. Here all data for a particular user will be stored in the same place, which can reduce latency when retrieving information. However, it is important to consider the sharding criteria carefully to avoid potential consistency issues, as the same data will be stored in multiple places.</p>\n<p>On the other side, if we choose to store the data in a normalized form, we will need to perform join across tables and rows to retrieve information. This can be undesirable because it may result in high latency and limited indexing support. So storing data in a denormalized manner will be more efficient due to the reduced latency, but it is important to take care of consistency issues.</p>\n<p>Ensuring consistent data access is critical, and having only one copy of the data may not be enough. To achieve acceptable latency, it's important to distribute the data across multiple shards so that each shard has a reasonable load. Both reads and writes should function similarly to a single database, with an additional layer that resolves the relationship between rows, shards, and machine IPs. This layer helps identify which shard a particular row belongs to and which machine to query or write to.</p>\n<p>The main concern with this approach is what happens if the machine hosting a particular shard fails? While maintenance downtime is acceptable, if a machine fails and the hard disk becomes corrupted, we risk losing data. This scenario could be disastrous, resulting in the loss of all communications if the shard is an important one. To mitigate this risk, it's necessary to keep multiple copies of the data we're writing. This ensures data redundancy and high availability, reducing the risk of data loss in the event of a machine failure.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*6Z30bFUPh-0YeBbjviFQuA.png\"></p>\n<h4>System availability</h4>\n<p>One important measure to consider when evaluating a distributed system is system availability. What happens if one of our workstations breaks for some reason (hardware or software issues, for example), and how does this affect our key-value storage system?</p>\n<p>We won’t be able to return the correct response if someone requests resources from this system, it appears. When developing a side project, we may overlook this difficulty. However, if we’re serving millions of people from a large number of servers, this happens frequently, and we can’t afford to restart the server manually every time. This is why, in today’s distributed systems, availability is critical. So, how would we go about dealing with this problem?</p>\n<p>Of course, we can write more robust code with test cases. However, our program will always have bugs. In addition, hardware issues are even harder to protect. The most common solution is a replica. By setting machines with duplicate resources, we can significantly reduce system downtime. If a single machine has a 10% of chance to crash every month, then with a single backup machine, we reduce the probability to 1% when both are down.</p>\n<h4>Replica VS Sharding</h4>\n<p>The replica appears to be very similar to sharding at first glance. So, what’s the connection between these two? When developing a distributed key-value store, how would we decide between replica and sharding?</p>\n<p>First and foremost, we must understand the aim of these two strategies. Because a single machine can only store so much data, sharding is used to spread data over numerous machines. The replica serves as a backup for the system in the event of a failure. With that in mind, a replica won’t assist if a single machine can’t store all of the data.</p>\n<h4>Consistency</h4>\n<p>We can strengthen the system by introducing replicas. Consistency, on the other hand, is a problem. Let’s imagine we have replica A2 for machine A1. How can we be certain that A1 and A2 have the same information? When adding a new entry, for example, we must update both machines. However, one of them may fail the write operation. As a result, A1 and A2 may accumulate a significant amount of conflicting data over time, which is a significant issue.</p>\n<p>There are a few options available here. The first option is for the coordinator to keep a local copy. When a resource is updated, the coordinator keeps a copy of the new version. If the update fails, the coordinator can retry the procedure.</p>\n<p>The commit log is another option. If we’ve been using Git, we’re probably already familiar with the concept of a commit log. Essentially, each node machine will record a commit log for each transaction, which acts as a track of all changes. So, if we wish to update an entry in machine A, we’ll have to go through the commit log first. Then, a separate software will process all of the commit logs (in a queue). We can recover if an operation fails since we can look at the commit log.</p>\n<h3><strong>Redis Architecture</strong></h3>\n<p>Redis is a key-value data store that is widely used by major IT companies around the world. It runs in memory and is supported by Amazon Elastic Cache, making it a powerful and useful tool for key-value data storage.</p>\n<h4>What is In-Memory Key-Value Store?</h4>\n<p>In the case of an in-memory key-value store like Redis, the key-value pairs are saved in primary memory (RAM). This means that Redis stores data in RAM as key-value pairs, where the value can be a string, list, set, sorted set, or hash and the key must be a string. Here are a few examples of key-value pairs in Redis:</p>\n<ul>\n<li>Key: \"name\", Value: \"John\"</li>\n<li>Key: \"age\", Value: \"30\"</li>\n<li>Key: \"location\", Value: \"New York\"</li>\n</ul>\n<p>Redis allows for a variety of value types and stores data in primary memory, making it a fast and flexible tool for storing and retrieving data.</p>\n<h4>Advantages and Disadvantages of Redis over DBMS</h4>\n<p>In database management systems, all data is typically stored in secondary storage, which can make read and write operations slower. Redis, on the other hand, saves everything in primary memory (RAM), which is much faster for data read and write operations. However, it is important to note that Redis has limitations due to the limited size and cost of primary memory compared to secondary storage. </p>\n<p>Redis cannot store large files or binary data and is best suited for small amounts of textual data that need to be accessed, changed, and inserted quickly. If we try to write more data than the available memory allows, we may encounter errors. Overall, Redis is a useful tool for fast read and write operations with small amounts of textual data, but it is important to consider its limitations when using it to store and access data.</p>\n<p>Thanks to Navtosh Kumar for his contribution in creating the first version of this content. Enjoy learning, Enjoy algorithms, Enjoy system design!</p>"},{"id":"fead2126-c846-5999-8d23-62412b1da6b3","frontmatter":{"title":"Design QR Code Generator for a Grocery Store","path":"/blog/design-qr-code-system"},"html":"<h3>What’s a QR Code System?</h3>\n<p>QR (quick response code) is a machine-readable 2-dimensional barcode consisting of an array of black and white squares, typically used for storing URLs or other information for reading by the camera on a reader e.g smartphone devices. </p>\n<ul>\n<li>QR code payment is one of the contactless payment methods to transfer funds from the buyer’s wallet to the seller’s wallet or account. here Payment is performed by QR code at POS from the mobile app.</li>\n<li>Shoppers only need a mobile phone with a camera and a mobile payment app to scan a QR code and pay. It is quick and easy to use and has increased convenience and security benefits.</li>\n</ul>\n<h3>System Requirements</h3>\n<p>Let’s start with building a simpler version of the QR code payment transaction system. There are two types of users in our system: 1) Sellers 2) Buyers (Customers). </p>\n<ul>\n<li>Buyer and seller registers in the system. </li>\n<li>The seller downloads the QR code and displays it in front of his store. </li>\n<li>The buyer goes to the store and purchases items. (Stored in Buyer transaction table) </li>\n<li>Buyers scan the QR code displayed in the store through the phone app. </li>\n<li>The buyer enters the amount and clicks OK to proceed. </li>\n<li>Fund transferred from Buyer to Seller </li>\n<li>SMS notification sent to Seller and Buyer for confirmation </li>\n<li>Transaction completes.</li>\n<li>Extended: What changes do we need in our system to extend the system for dynamic QR code (Not in scope).</li>\n</ul>\n<h3>Use cases</h3>\n<p>The system should allow these basic use cases:</p>\n<ol>\n<li><strong>Create QR code</strong>: We need to create a unique QR code for each seller and provide it to the seller which can be displayed at the store. A customer would use this unique QR code to scan using his/her mobile camera through an app and make the payment to the seller account/wallet.</li>\n<li><strong>Read QR code</strong>: A customer at the grocery store would use his/her mobile phone camera to scan the QR code displayed at the grocery store. This scanned image needs to be decoded to get the seller information to which the customer’s payment would go to.</li>\n<li><strong>Register buyers and sellers</strong>: Both buyer and seller would have to register themselves first in the system.</li>\n<li><strong>Transfer fund from buyer to seller</strong>: The system needs to transfer funds from the buyer’s wallet to the seller’s wallet.</li>\n</ol>\n<h3>Database schema</h3>\n<p>We need to store data about sellers of a grocery store, their wallets, and transactions, buyers, wallet, and buyers' transactions. </p>\n<p>Ref: <a href=\"https://dbdiagram.io/d/5f8cc7e53a78976d7b7820fd\">https://dbdiagram.io/d/5f8cc7e53a78976d7b7820fd</a></p>\n<p><strong>Seller Schema</strong></p>\n<ul>\n<li>Seller Table: SellerID, SellerName, TaxId, Phone, Email, AC_Number, BankName, BankDetails</li>\n<li>Seller_wallet Table: SellerWalletID, SellerId, Balance</li>\n<li>Seller_transaction Table: ID, BuyerId, PurchasedItemID, Amount</li>\n</ul>\n<p><strong>Buyer Schema</strong></p>\n<ul>\n<li>Buyer Table: BuyerId, BuyerName, Phone, Email, BankName, bank details</li>\n<li>Buyer_Wallet Table: BuyerWalletId, BuyerId, Balance</li>\n<li>Buyer_Transaction Table: ID, SellerId, BuyerId, PurchasedItem, Amount, Status</li>\n</ul>\n<p>A straightforward approach for storing the above schema would be to use an RDBMS like MySQL since we require joins. But relational databases come with their challenges, especially when we need to scale them. For details, please take a look at SQL vs. NoSQL.</p>\n<h3>Capacity Estimation and Constraints</h3>\n<p>The number of sellers is significantly less than that of buyers. Let’s assume:</p>\n<ul>\n<li>There is a 1:10 ratio for sellers to buyers.</li>\n<li>We have 500M total users(buyers), and 100M daily active users(buyers).</li>\n<li>On average each user makes 2 transactions/day using QR payment methods at different grocery stores. So a total of 200M transactions/day.</li>\n<li>The size of each entry in the transaction table is 1KB. This would require us 200M*1KB = 200GB of total storage per day. </li>\n</ul>\n<p>Size of transaction table to store 10 year data = 200GB*365*10 = 730TB</p>\n<p>Size of buyers table is 500M (total users)* 1KB (data per user)= 500GB total.</p>\n<h3>Key Service APIs</h3>\n<p><strong>APIs for Seller Application</strong></p>\n<ul>\n<li>RegisterSeller: Each seller needs to register itself before it’s able to use the system. This API would create a database entry for sellers in the seller schema or return an error.</li>\n<li>CreateAndGetQRCode: The API will create and provide a unique QR code to the seller which contains the seller data and can be read by a QR code scanner. </li>\n<li>CreateSellerWallet: This API is responsible to create a wallet for the seller where money from the buyer would go.</li>\n<li>TransferFundBuyerToBank: This API will connect PaymentGateway to transfer money from buyer wallet to bank account.</li>\n</ul>\n<p><strong>APIs for the Buyer Application side</strong></p>\n<ul>\n<li>GetDataFromQRCode: This will read and decrypt the QR code and get the data into the buyer’s app.</li>\n<li>RegisterBuyer: This will be used to register a customer and update the buyer’s schema table with the information provided in the API.</li>\n<li>CreateBuyer Wallet: This will create the buyer’s wallet schema and store phone number, email, address, bank details.</li>\n<li>TransferFundToBuyerWallet: This API will connect PaymentGateway to transfer funds from the buyer’s bank account to its wallet.</li>\n<li>TransferFundFromBuyerToSeller: This API will createSaga. It will also update the seller’s and buyer’s schema and complete or abort the transaction sending success or failure code to the buyer’s application. </li>\n</ul>\n<h3>High-Level Design</h3>\n<p>Buyers and Sellers would use a relational database schema. We would use a payment gateway to transfer funds to or from buyers' bank accounts. We would also use a saga pattern for the distributed payment transaction.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/1200/1*RuuVxqtoy_nY_lum4enRnA.png\" alt=\"qr code system high level design\" title=\"High level design of qr code system for a grocery shop\"></p>\n<h3>Transfer Fund From Customer to Seller</h3>\n<p>We would use the <a href=\"https://microservices.io/patterns/data/saga.html\">saga</a> pattern here. A saga is a sequence of local transactions where each transaction updates data within a single service.</p>\n<ul>\n<li>We would use an API TransferFundFromBuyerToSeller that first create a Pending status in the buyer_transaction table in buyerSchema.</li>\n<li>It then creates createSaga in parallel. This tries to go into the seller schema and adds money to the seller’s wallet.</li>\n<li>CreateSaga sends the command to the seller_wallet table to update the wallet Amount +sales amount</li>\n<li>createSaga gets messages as updates<em>pass or update</em>fail.</li>\n<li>If update<em>pass then a) buyer</em>transaction table in buyer schema is updated b) wallet table is decremented and c) finally pending status is removed.</li>\n<li>If updated<em>fail then the Buyer</em>transaction table in buyer schema is updated with “canceled” status.</li>\n</ul>\n<h3>Scaling the System</h3>\n<p><strong>CDN:</strong> system servers need to deliver many javascript assets, images, etc. Decoupling these assets delivery to CDN can offload work of servers improving server capacity.</p>\n<p><strong>Service on Docker and Docker on PODS:</strong> The number of requests to the system gradually increases over time. As the traffic increases even during peak hours like evening/morning, we need to handle additional traffic using auto-scaling by adding docker automatically based on CPU usage threshold or just traffic at each server’s load balancer level.</p>\n<p><strong>Load balancers:</strong> we would use load balancers at each key service to distribute traffic using the least conns strategy.</p>\n<p><strong>Sharding the database:</strong> if we shard the data based on the buyer’s id, then we can keep all transactions of a user in the same shard. But as the size of the transaction table is much larger than that of the user’s data table, therefore, we can choose to shard the data based on transactionId which can be integer incremented value based on timestamp. This is more useful as we can retrieve transaction data from any user from a specific shard just by looking at the timestamp range of the request.</p>\n<p>Thanks to Chiranjeev for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"4cec4b04-4064-5c4a-ad04-6ca1d8796326","frontmatter":{"title":"Design Typeahead (Autocomplete) System","path":"/blog/design-typeahead-system"},"html":"<h3><strong>What is Typeahead?</strong></h3>\n<p>Typeahead, or autocomplete, is a feature that allows an application to predict the rest of a word as a user is typing it. In many graphical user interfaces, the user can accept a suggestion by pressing the tab key or scroll through multiple suggestions by using the down arrow key. </p>\n<p>Autocomplete is commonly used in search functions across a variety of platforms, including social media and search engines such as Facebook, Instagram, and Google. It is a convenient tool that can save time and improve the user experience by providing relevant suggestions as the user types.</p>\n<h3>Requirements</h3>\n<h4><strong>Functional Requirements</strong></h4>\n<p>Our service should provide the top 10 terms that begin with the user's current query as they type. The suggestions should be sorted based on the frequency and recentness of the phrases or queries. This means that the most commonly used and recently used terms that match the user's input will be prioritized in the list of suggestions.</p>\n<h4><strong>Non-Functional Requirements</strong></h4>\n<ul>\n<li>High availability: The system should be able to handle disruptions and continue to function without significant downtime.</li>\n<li>Real-time suggestions: The user should be able to see suggestions in near-instantaneous time, with a target of within 200 milliseconds.</li>\n<li>Scalability: The system should be able to handle a large volume of traffic without experiencing performance issues.</li>\n</ul>\n<h3>Estimation of Scale and Constraints</h3>\n<h4><strong>Traffic estimations</strong></h4>\n<p>If we build a service on the same scale as Google, we can expect around 5 billion searches per day, or about 60,000 queries per second. However, we can estimate that only 20% of these searches will be unique, with many duplicates. If we only want to index the top 50% of search phrases, we can exclude less frequently searched queries. </p>\n<p>For example, if we want to develop an index for 100 million different terms, we can estimate the storage requirements as follows:</p>\n<ul>\n<li>Average query size: 15 characters (assuming 3 words per query with an average length of 5 characters per word)</li>\n<li>Storage space per query: 30 bytes (assuming 2 bytes per character)</li>\n<li>Total storage space: 100 million * 30 bytes = 3 GB</li>\n</ul>\n<p>We can expect some growth in the data over time, but we can also exclude phrases that are no longer searched. If we assume a 2% increase in new requests every day and retain our index for the previous year, the total storage would be approximately 25 GB: 3 GB + (0.02 * 3 GB * 365 days). This calculation takes into account both the initial storage requirements and the growth in data over time.</p>\n<h3><strong>System APIs</strong></h3>\n<ol>\n<li><strong>get_suggestions(prefix)</strong>: This function takes in a prefix (i.e., the initial characters that have been typed by the user in a search query) and returns a list of suggestions based on that prefix.</li>\n<li><strong>add<em>to</em>db(query)</strong>: This function updates the database with a new, unique, and trending search query that has been searched above a certain threshold. The input for this function is the query that is being added to the database.</li>\n</ol>\n<h3>High Level Design</h3>\n<p>To meet the real-time requirement for our autocomplete system, we need to be able to quickly add new search queries to our database. This means that in addition to developing a system for providing suggestions to the user, we also need to incorporate popular search queries into our database. This will allow users to receive suggestions based on both current and popular searches, ensuring that the autocomplete feature is up-to-date and relevant.</p>\n<p>As a result, we’ve divided our autocomplete into two sections:</p>\n<ol>\n<li><strong>Making recommendations</strong>: This component is responsible for taking in a search query or prefix and returning the seven most frequently searched keywords that match the input.</li>\n<li><strong>Collecting information</strong>: This component is responsible for collecting and organizing user input requests in real-time. While it may not be feasible to process large datasets in real-time, this approach can serve as a starting point. We will later explore a more realistic approach for handling larger data sets.</li>\n</ol>\n<p>The autocomplete feature works by sending a query to the application servers every time the user types a character into the search field. The query get_suggestions() retrieves the user's top seven options from the database and displays them as suggestions.</p>\n<h3>Data Structure to Store Search Queries</h3>\n<p>To efficiently handle a large number of queries with minimal latency, we need a system that can store our data in a way that allows for fast querying. We cannot use a traditional database for this purpose, so we need to use an in-memory data structure that is optimized for fast access.</p>\n<p>One effective data structure for this purpose is a trie, also known as a prefix tree. A trie is a tree-like structure that stores phrases by representing each character in the phrase as a node in the tree. The root node represents an empty string, and each child node stores a character and has 26 children, one for each possible character. This allows the trie to efficiently store and retrieve strings based on their prefixes.</p>\n<p>In our case, the trie data structure is well-suited for storing and querying search phrases because it allows us to efficiently find and retrieve the most frequently searched terms that match a given prefix.</p>\n<p>If we need to store ‘camping, campus, cart’ in the trie, for example, it will look like this:</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*3U-xXC93Oer4PUlSyPnC2A.png\" alt=\"Trie data structure for storing search queries in Autocomplete System \" title=\"How Trie storing search queries in a Typeahead System \"></p>\n<p>If the user has typed ‘cam,’ our service can traverse the trie to the node ‘m’ to find all the terms that start with this prefix (e.g., cam-ping, cam-pus, etc.).</p>\n<p>We need to design an efficient solution for indexing a large amount of data, as the tree we expect to create will be massive and traversing even a sub-tree may take a long time. For example, the search term \"system design interview questions\" has 30 levels. Our solution needs to have a high level of efficiency because we have strict latency constraints.</p>\n<p>To improve search speed, we can save the top ten suggestions for each node and return them to the user. However, this will require a significant increase in storage capacity. Instead of saving the full phrases, we can save space by storing references to the terminal nodes and keeping track of the frequency of each reference. To find the suggested terms, we can use the parent reference from the terminal node.</p>\n<p>To construct the trie efficiently, we can build it from the bottom up. Each parent node will first call all of its child nodes recursively to compute the top suggestions and counts. Then, the parent node will combine the top suggestions from all of its children to decide on its own top choices.</p>\n<p>If we have a daily search volume of five billion, this translates to around 60,000 requests per second. Updating the trie on every query will use a lot of resources and slow down our read requests. One solution to this problem is to update the trie offline after a set period of time.</p>\n<p>We can also track the frequency of new queries as they come in. We can either log every query or sample and log every 1,000th query. It is acceptable to only log every 1,000th searched term if we don't want to show terms that have been searched fewer than 1,000 times.</p>\n<p>To regularly process all of the logging data, we can set up a Map-Reduce (MR) job. For example, we can run this MR job once an hour to calculate the frequency of all terms searched in the previous hour. Then, we can use this information to update our trie with the new data. To do this, we can update the trie's current snapshot with all of the new terms and their frequencies. Note: It is important to perform this update offline, as we do not want the process of updating the trie to obstruct our read queries.</p>\n<p>There are two possibilities available to us:</p>\n<ol>\n<li>We can make a copy of the trie on each server and update it. Once the update is complete, we can start using the new copy and discard the old one.</li>\n<li>We can set up each trie server as a master-slave arrangement. While the master is serving traffic, we can update the slave. Once the update is complete, we can make the slave the new master and update the old master. This way, both the new master and the updated old master will be able to serve traffic.</li>\n</ol>\n<h4>How can we keep the frequency of typeahead suggestions up to date?</h4>\n<p>We need to update the frequencies of our typeahead suggestions, which are stored with each node in the trie. Instead of recreating all search phrases from scratch, we can simply update the changes in frequency. For example, if we want to track all phrases searched in the last 10 days, we can adjust the counts by subtracting the counts from the time period that is no longer included and adding the counts for the new period. We can do this using the Exponential Moving Average (EMA) of each term, which gives more weight to the most recent data.</p>\n<p>When we add a new term to the trie, we will go to the terminal node for that phrase and increase its frequency. If the search term has now entered the top 10 queries for additional nodes, we will need to update the top 10 queries for those nodes as well. To do this, we will need to travel all the way up to the root from the node, checking if the current query is in the top 10 for each parent node. If it is, we will adjust the frequency accordingly. If the current query is not in the top 10, we will check if it has a high enough frequency to be included. If it does, we will add the new phrase and remove the term with the lowest frequency.</p>\n<h4>What is the best way to delete a term from the trie?</h4>\n<p>Suppose we need to delete a term from the trie due to legal concerns, hatred, or piracy, we can handle this in two ways. First, during the normal update process, we can completely delete the offending phrases from the trie. In the meantime, we can also add a filtering layer to each server that will remove any such terms before sending them to users. This will ensure that users do not have access to the deleted terms until they are fully removed from the trie.</p>\n<h4>Different ranking criteria for suggestions</h4>\n<p>When ranking phrases, we must consider more than just a simple count. Other criteria such as freshness, user location, language, demographics, and personal history must also be taken into account. These factors can help us provide more relevant and accurate search results to users.</p>\n<h3>Caching</h3>\n<p>Caching is a performance optimization technique that involves storing frequently accessed data in a temporary storage area, called a cache, in order to speed up future access to that data. In the context of searching, before making a query to a database, the server will first check a cache to see if the search terms and their corresponding results are already stored there. If they are, the server can retrieve the results from the cache. This can help reduce the overall latency of the search process.</p>\n<p>One way to implement caching for frequently searched terms is to place a cache in front of the database servers that contain the most commonly searched terms and their suggestions. To ensure that the cache stays up to date, it is important to use a caching strategy like Least Recently Used (LRU), which removes the least recently used items from the cache in favor of more recent items. This is based on the idea that recent search results are more likely to be searched again in the future.</p>\n<h3>Database Partitioning</h3>\n<p>There are a few different approaches we can take to partition our data in order to distribute the load across multiple servers. One approach is to <strong>partition the data based on the first letter</strong> of the search term. For example, we could store all words beginning with the letter 'A' on one server, and all words beginning with the letter 'B' on another server. This approach has the advantage of being easy to implement, but it can result in uneven partitioning, with some servers handling significantly more traffic than others.</p>\n<p>Another approach is to <strong>use a hash function</strong> to determine which server a particular search term should be stored on. The hash function generates a server number based on the search term, which allows us to store the term on the appropriate server. This approach can result in a more evenly distributed load, but it requires us to send queries to all of the servers in order to generate a complete list of suggestions, which can increase latency.</p>\n<p>As a result, we can take the first strategy. We can combine those characters with fewer terms onto one server.</p>\n<h3>Replication and Load Balancer</h3>\n<p>To ensure that our autocomplete service can handle a large volume of traffic and remain available even if one of our servers goes down, we need to implement replicas and a load balancer. The replicas are copies of our trie servers, which allows us to distribute the load across multiple servers and improve fault tolerance. If one server goes down, the others can continue to handle traffic.</p>\n<p>The load balancer is responsible for managing the distribution of traffic across the servers. It tracks our data partitioning method and redirects incoming traffic to the appropriate server based on the prefix of the search term. This helps to ensure that the load is evenly distributed across the servers and that the service remains available even if one server becomes overwhelmed.</p>\n<h3>Fault Tolerance</h3>\n<p>If one of the trie servers goes down, we can use a master-slave arrangement to maintain availability. In this arrangement, we have a primary server (the master) that handles traffic and a secondary server (the slave) that serves as a backup. If the master server goes down, the slave can take over after failover to ensure that the service remains available.</p>\n<p>Additionally, if any server needs to be restarted, it can use the last snapshot of the trie to recreate it. This ensures that we don't lose any data and that the service can continue to function without interruption. By implementing these measures, we can ensure that our autocomplete service remains available and can handle a large volume of traffic.</p>\n<p>Thanks to Navtosh Kumar for his contribution in creating the first version of this content. Please write in the message below if you want to share more insight. Enjoy learning, Enjoy system design :)</p>"},{"id":"8b77137a-ef7c-5799-a579-c999bd6437fc","frontmatter":{"title":"Twitter System Design","path":"/blog/design-twitter"},"html":"<p>Twitter is a social media platform where users can post short messages called \"tweets\" and interact with each other. These tweets can be up to 280 characters long and can include text, photos, or links. Users can also follow other users and receive notifications when they post new tweets. In this article, we will discuss how to design a system similar to Twitter. First, we will outline the key requirements for our system.</p>\n<h3>Key Requirements</h3>\n<p>Before designing a system like Twitter, it's important to identify the essential features and the requirements we want to meet with our design. In this article, we will cover the following features:</p>\n<ul>\n<li>Posting tweets: Users should be able to create and post tweets, which may include text, photos, or links.</li>\n<li>Tweet delivery: The service should be able to push new tweets to the followers of the user who posted them and send push notifications to alert followers of the new tweets.</li>\n<li>Timeline view: Users should be able to view their own timeline (a feed of their own tweets) as well as the home timeline (a feed of tweets from the accounts they follow).</li>\n<li>High availability: The service should be highly available, meaning it should be able to handle a large number of users and requests without experiencing downtime.</li>\n</ul>\n<h3>Capacity Estimation</h3>\n<p>Assuming that our Twitter system has:</p>\n<ul>\n<li>0.5 billion total users.</li>\n<li>100 million daily active users (DAU).</li>\n<li>50 million new tweets are created every day.</li>\n</ul>\n<p>The system would generate a total of 4 billion tweet views per day, assuming that, on average, a user visits their timeline 1 time a day, visits 3 other people's pages, and each page has 10 tweets. This is calculated using the formula: The total number of tweet views/day = Number of DAU * (Number of timeline visits per day + Number of other people's pages visited per day) * Number of tweets per page = 100 million * (1 + 3) * 10 = 4 billion.</p>\n<p>So our twitter system is more read-heavy than write-heavy, meaning it receives more requests to read tweets than to create them. Therefore, it is critical to optimize for fast reads of tweets.</p>\n<p>Assuming that each tweet is 10 KB, the total daily storage needed would be approximately 5 TB. This calculation is done using the formula: Total storage needed per day = Tweet size * Number of tweets per day = 10KB * 50 Million = 5 TB. For 10 years of storage, we need 10 years * 365 days/year * 5 TB/day = 18.5 PB.</p>\n<h3>High Level Design</h3>\n<p>At a high level, a Twitter-like service requires a system with multiple application servers to handle client requests. These servers are placed behind load balancers, which distribute incoming traffic evenly among the servers. On the backend, a powerful database is needed to store tweets and support a high number of reads. In addition, the system needs file storage to store photos and videos</p>\n<p>Our platform’s general high-level architecture is as follows:</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*UQDE2Z-wd_kr84RJSFu31g.png\" alt=\"Designing Twitter high-level design\" title=\"High-level design of Twitter system design\"></p>\n<p>The platform offers a range of services that are designed to handle specific functions. To support all the intended use cases, each service provides a set of APIs that can be used by client apps or other services. These services also have their own data stores for storing relevant data. </p>\n<p>When a service receives a request, such as a new tweet or a request to follow a user, it validates the request and stores the data in its data store. In addition, the service sends a message in the form of an event to the messaging service, which can be used by other services to update their state. In this way, the platform is able to handle a wide range of use cases and ensure that data is stored and processed efficiently.</p>\n<h4><strong>Data model and storage</strong></h4>\n<p>When deciding on a data storage solution for a large platform with many users, it is important to consider both SQL and NoSQL options. However, due to scalability issues with SQL solutions, a NoSQL solution may be more suitable. In this case, we can use a wide column data store like Cassandra to store data such as tweets, comments, and likes. </p>\n<p>For entities like users and their followers, we may consider using a graph-based data store solution like Neo4j or Cassandra. By using the appropriate data store for each type of data, we can ensure that the system is able to handle a large volume of data and scale effectively.</p>\n<h3>Detailed Components Design</h3>\n<p>Our platform’s major system components are as follows:</p>\n<p><strong>User service:</strong> This provides APIs for managing users and their interactions, such as following or unfollowing other users. In some cases, it may be beneficial to create a separate User Follower Service to handle these interactions. This separation of responsibilities can help to ensure that the system is scalable and easy to maintain.</p>\n<p><strong>Tweet service:</strong> This provides APIs for creating and storing tweets in the data store and managing comments and likes on tweets. Depending on the needs of the application, this service may be further divided into Tweet Comments and Tweet Like Services to improve organization and scalability. </p>\n<p><strong>Feeds service:</strong> This is responsible for calculating the tweets that will appear on a user's timeline. It has an API that returns these tweets so that the timeline can be displayed, and it also supports pagination to allow users to browse their timeline in chunks. This makes it easier for users to navigate through their timeline and view the tweets that are most relevant to them. The service also has the ability to calculate the timeline quickly and efficiently, ensuring that the system can scale to support a large number of users.</p>\n<p><strong>Timeline generation:</strong> To display a user's timeline, the system must show the most recent posts from the user's followers. Generating the timeline on demand can be slow for users with many followers, as the algorithm must query, merge, and rank a large number of tweets. To improve performance, the system should pre-generate the timeline and store it in memory. This allows the timeline to be retrieved quickly when the user loads the page, rather than having to generate it on demand. </p>\n<p>To ensure that the pre-generated timelines are regularly updated, the system can use dedicated servers to constantly create and store them in memory. By using this approach, users' timelines are not compiled on demand, but rather regularly updated and stored in memory for quick retrieval.</p>\n<p><strong>Timeline updates:</strong> If the system generates timelines for all users in the same way, users with many followers may experience delays in receiving new postings. To address this issue, the system can prioritize users with new updates when generating timelines. When a new tweet is added to the message queue, it is picked up by timeline generator services, which re-generate the timeline for all followers. This approach allows users to receive new postings quickly, even if they have a large number of followers.</p>\n<p>There are several approaches that can be used to publish new posts to users:</p>\n<ul>\n<li><strong>Pull model:</strong> In this approach, clients can request data regularly or whenever they need it manually. The disadvantage of this method is that new information is not displayed until the client submits a pull request. Additionally, most pull requests will return an empty response if there is no new data, which can waste resources.</li>\n<li><strong>Push model:</strong> With this approach, the system sends a notification to all of a user's followers as soon as they post a tweet. However, this technique can be challenging when a user has millions of followers, as the server must simultaneously transmit updates to a large number of individuals.</li>\n<li><strong>Hybrid model:</strong> A combination of the pull and push models can be used to balance the benefits and drawbacks of each approach. For example, the system could provide updates directly to users with only a few hundred or thousands of followers, while celebrities with a larger number of followers could rely on their fans to disseminate updates. This hybrid approach can help to ensure that the system is able to scale and support a large number of users efficiently.</li>\n</ul>\n<h3>System Optimization</h3>\n<p>In this section, we will explore various optimization strategies to improve the performance, scalability, redundancy, and other non-functional aspects of our architecture. These approaches will help us ensure that the system is able to meet the needs of a large number of users and provide a reliable, high-quality experience.</p>\n<h4><strong>Database sharding</strong></h4>\n<p>Given the large number of users and tweets, it is not feasible to store all of the data for a service on a single machine. To address this challenge, it is important to define a partitioning strategy to divide the data into smaller sections, or partitions, which can be replicated and stored on different cluster hosts. This process, known as data sharding, helps to improve the speed of data nodes by reducing the number of nodes that must be searched in order to find a particular document. </p>\n<p>There are several different approaches that can be used to split the data, including partitioning by user, by tweet, or by some other criterion. By carefully considering the needs of the application and the characteristics of the data, it is possible to design an effective partitioning strategy that helps to ensure that the system is able to scale and support a large number of users efficiently.</p>\n<p><strong>Sharding based on UserID</strong></p>\n<p>One possible approach to data partitioning is to link each user to a specific server based on their UserID. This strategy keeps all of a user's tweets, likes, and followers, among other things, on the same server. However, this approach can be problematic when some users (such as celebrities) are particularly popular, as it can result in a disproportionate amount of data and access being concentrated on a subset of servers. To address this issue, it may be necessary to adopt a different partitioning strategy that is better suited to the characteristics of the data and the needs of the application.</p>\n<p><strong>Sharding based on TweetID</strong></p>\n<p>Another approach to data partitioning is to map each tweet to a specific server based on its TweetID. This technique stores the tweet information on the server that is responsible for that particular tweet. To find tweets, the system must query all servers and receive a collection of tweets from each one. While this approach helps to avoid the problem of hot users by distributing the data evenly across servers, it can increase latency because all servers must be queried in order to find a particular tweet. As a result, this technique may not be the best option for applications that require fast search performance.</p>\n<p><strong>Sharding based on tweet creation time</strong></p>\n<p>Storing tweets based on the time they were created allows us to quickly retrieve the most recent tweets by querying only a small number of servers. However, this approach can result in uneven traffic load distribution. For example, when writing new tweets, all incoming traffic will be directed to a single server, while the other servers will be idle. On the other hand, when reading tweets, the server holding the most recent data may experience higher load compared to servers holding older data, leading to performance issues and inefficiency in handling large amounts of traffic.</p>\n<h4>Database caching</h4>\n<p>Caching is a useful technique for improving the performance and scalability of a system. It works by storing precomputed data in a cache, which can be accessed quickly in response to requests for that data.</p>\n<p>The Feed Service is a good candidate for introducing caching in order to improve the speed of the system. One option for storing cached data is to use a solution such as Redis or Memcache. These solutions can be used to cache the chronology for each user, potentially storing the entire user timeline or a significant portion of it. </p>\n<p>By preloading the cache with this data, it is possible to reduce the latency when a user accesses their feed, improving their experience. To further improve performance, the system can preemptively request and cache the next batch of feed data for a user as they are browsing their feed. This can help to reduce the time it takes to load new tweets and enhance the overall user experience.</p>\n<h4>Load balancing</h4>\n<p>To ensure that incoming requests are evenly distributed across multiple data centers, we can use DNS load balancing. This is particularly useful for applications that receive requests from a variety of geographic regions. Each service should have a load balancer in front of it to distribute requests to various service nodes based on capacity estimates. </p>\n<p>There are several load balancing methods that can be used, such as round-robin and least connection, to determine how to distribute the load across the available nodes. Additionally, because our services are stateless, we can add or remove nodes from the cluster without losing any data, making it easier to scale the system as needed.</p>\n<h4>Replication and Fault Tolerance</h4>\n<p>To optimize for read-heavy workloads, we can use multiple slave database servers for each partition of the database. These slave servers will only be used for handling read traffic, while all write operations will be directed to the master database and then replicated to the slave database. This approach not only helps to distribute read traffic, but also provides fault tolerance.</p>\n<h3>Conclusion</h3>\n<p>In this blog post, we discussed how to design a system similar to Twitter. We covered topics such as how to handle client requests, how to store and manage data, and how to optimize the system for performance and scalability. We hope you enjoyed reading this blog post and learned something new.</p>\n<p>Thanks to Suyash Namdeo for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"579ee644-4c0d-56e1-b43d-343335414c1a","frontmatter":{"title":"Design WhatsApp Messenger: System Design Interview Question","path":"/blog/design-whatsapp"},"html":"<h3>What is Whatsapp?</h3>\n<p>WhatsApp is a social messaging platform that enables users to send messages to one another. It has become an integral part of many people's daily lives, but how does it work? What are the principles behind its design and functioning?</p>\n<p>In this blog, we will answer these fundamental questions and explore the basics of how WhatsApp works. We will also discuss the generic architecture of WhatsApp, which can serve as a foundation for designing other chat applications.</p>\n<h3>Key Requirements</h3>\n<p>WhatsApp is a highly scalable system that is accessed frequently by a large number of users around the world, so it is essential that it is designed efficiently in order to remain reliable and operational. Therefore, it is important to identify the key requirements of the system.</p>\n<p>Some of the basic requirements for WhatsApp include:</p>\n<ul>\n<li>The ability to support one-on-one conversations.</li>\n<li>The ability to show last seen and message acknowledgement (sent, delivered, and read).</li>\n<li>Media support (images/videos) and end-to-end encryption.</li>\n</ul>\n<p>Let’s figure out the capacity estimation of our required service.</p>\n<h3>Capacity Estimation</h3>\n<p>Our goal is to build a highly scalable platform that can support a large volume of traffic. To do this, we need to consider various factors, such as the number of messages and users, peak traffic levels, and data storage requirements.</p>\n<p>Let's assume the following:</p>\n<ul>\n<li>10 billion messages are sent per day by 1 billion users.</li>\n<li>At peak traffic, there are 700,000 active users per second.</li>\n<li>At peak traffic, there are 40 million messages per second.</li>\n<li>On average, each message has 160 characters, resulting in 1.6 TB (10B * 160) of data per day.</li>\n<li>The system is expected to be in service for 10 years, requiring approximately 6 PB (10 * 1.6B * 365) of storage.</li>\n<li>The application will consist of multiple microservices, each performing a specific task. Let's assume that the latency for sending a message is 20 milliseconds, and that each server can handle 100 concurrent connections. Based on these assumptions, we would need a fleet of 8000 (40M * 20ms / 100) servers to support the chat service.</li>\n</ul>\n<h3>High Level Design</h3>\n<p>In the WhatsApp system, there are two primary services that form the core of the system: the chat service and the transient service. The <strong>chat service</strong> manages all traffic related to online messages sent by users. This includes handling incoming messages, sending messages to recipients, and maintaining the status of messages (such as whether they have been sent, delivered, or read).</p>\n<p>The <strong>transient service</strong>, on the other hand, deals with traffic when the user is offline. This includes storing messages that are sent to a user while they are offline and delivering them once the user comes back online. It also handles other tasks related to offline users, such as managing the \"last seen\" status and message acknowledgement.</p>\n<p>Here's a summary of how these two services work together to ensure that messages are delivered efficiently and reliably:</p>\n<ul>\n<li>When a user sends a message, the chat service is responsible for checking whether the recipient is online. If the recipient is online, the chat service delivers the message immediately.</li>\n<li>If the recipient is offline, the chat service passes the message to the transient service. The transient service stores the message in a separate storage until the recipient comes online.</li>\n<li>When the recipient comes online, the transient service delivers the stored message to the chat service, which then sends it to the recipient.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/800/1*edjThKWIvg8nX44oeI8ARQ.png\" alt=\"WhatsApp High-Level Design\" title=\"How WhatsApp works?\"></p>\n<h3><strong>High Level API Design</strong></h3>\n<p>The messaging system described will have two main APIs for sending and viewing messages. These APIs will use the REST (representational state transfer) architecture. REST is a popular choice for designing APIs because it is lightweight, easy to understand, and can be implemented using the standard HTTP protocol.</p>\n<h4><strong>Sendmessage (fromUser, toUser, clientMetaData, message)</strong></h4>\n<p>This API is used for sending messages from one user to another.</p>\n<ol>\n<li>fromUser: The user who is sending the message.</li>\n<li>toUser: The user who is receiving the message.</li>\n<li>clientMetaData: Metadata about the client, such as information about the device or platform being used.</li>\n<li>message: The actual message being sent.</li>\n</ol>\n<h4><strong>Conversation(userId, offset, messageCount, TimeStamp)</strong></h4>\n<p>This API displays conversations in a thread, similar to the view you see when you open WhatsApp. To avoid fetching too many messages at once, the API retrieves a limited number of messages for a single user in each call. The offset and message count parameters help control how many messages are retrieved.</p>\n<p>Parameters:</p>\n<ul>\n<li>userId: a unique identifier for the user</li>\n<li>offset: used to retrieve previous messages</li>\n<li>messageCount: the number of messages to be displayed</li>\n<li>TimeStamp: the last time the messages were updated</li>\n</ul>\n<h3>How features like last seen, single tick, and double tick work?</h3>\n<p>The acknowledgment service is responsible for implementing these features by continuously generating and checking acknowledgement responses. Based on the responses received, the service determines how to proceed.</p>\n<p><strong>Single Tick:</strong> When a message from User A reaches User B, the server sends an acknowledgement signal to confirm that the message has been sent.</p>\n<p><strong>Double Tick:</strong> After the server sends a message to User B through the appropriate connection, User B will send an acknowledgement back to the server to confirm receipt of the message. The server will then send another acknowledgement to User A, which will be displayed as a double tick.</p>\n<p><strong>Blue Tick:</strong> When User B reads the message, they will send another acknowledgement to the server to confirm that they have read it. The server will then send another acknowledgement message to User A, which will be displayed as a blue tick.</p>\n<p><strong>Last Seen Feature:</strong> This feature relies on the <strong>heartbeat mechanism</strong>, which sends continuous messages to the server every 5 seconds. The server uses these messages to maintain a table of the last-seen status of various users, which any other user can easily retrieve to check their last-seen status.</p>\n<h3>Design of Key Features</h3>\n<h4><strong>One-to-One Communication</strong></h4>\n<p>This is an essential component of the chat service that allows one user to send messages to another user. Here's how it works:</p>\n<p>Suppose Alice wants to send a message to Bob. The message is sent to the chat server that Alice is connected to. Alice receives an acknowledgement from the chat server that the message has been sent. The chat server then asks the data storage to fetch information about the chat server that Bob is connected to. The chat server of Alice then forwards the message to the chat server of Bob, and the message is delivered to Bob using a push mechanism. Bob then sends an acknowledgement back to the chat server of Alice, which informs Alice that the message has been delivered. If Bob reads the message again, a new acknowledgement is sent to Alice to confirm that the message has been read.</p>\n<h4><strong>User Activity Status</strong> </h4>\n<p>This figure shows a mechanism for maintaining a connection between the client and the server. A connection is established between the server and the client using web sockets, which create a bidirectional connection. Heartbeats are sent through these connections to monitor the user's activity status.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/800/1*YbIkFYY2_lNxUzPDARhDRg.png\"></p>\n<h4>End-to-End Encryption</h4>\n<p>End-to-End encryption is a feature that ensures that only the communicating users can read the messages. This is achieved through the use of public keys shared among all the users participating in the communication.</p>\n<p>For example, suppose Alice and Bob are communicating with each other in a channel. Both Alice and Bob have each other's public keys, and they each have their own private key that is not shared. When Alice wants to send a message to Bob, she encrypts the message using Bob's public key. This message can only be decrypted using Bob's private key, which he keeps to himself. Similarly, Alice can only decrypt messages sent by Bob. In this way, only Alice and Bob can read each other's messages, and the server acts only as a mediator in the process.</p>\n<h3>Bottlenecks</h3>\n<p>All systems are vulnerable to failures, and it is important to have measures in place to handle such situations. In order to handle a large volume of traffic, the service must remain active and fault-tolerant to handle any bottlenecks that may arise. In our service, the chat and transient servers are essential components, and it is necessary to address any challenges that may arise in their operation.</p>\n<ul>\n<li><strong>Chat Server Failure:</strong> The chat server is the core component of our system and is responsible for delivering messages to users who are online. It maintains connections with users, so if it fails, it will affect the entire architecture. There are two ways to handle the chat server's failure: transferring the TCP connections to another server, or allowing users to automatically initiate a new connection in case of a connection loss.</li>\n<li><strong>Transient Storage Failure:</strong> The transient storage component is also prone to failures, which can affect the entire service. If this component fails, it can result in the loss of messages that were in transit to offline users. To prevent this, we can replicate each user's temporary storage to ensure that no messages are lost. When the user comes back online, the replica can be used to process their messages. If the original server becomes available again, the original and replica instances of the user's transient storage are merged to create a unique store for storing messages.</li>\n</ul>\n<h3>Optimizations</h3>\n<p><strong>Latency:</strong> To provide a smooth and better customer experience, the messenger service must be real-time. This means that latency needs to be minimized, which can be achieved by using caching to store frequently queried data. A distributed cache like Redis can be used to cache user activity status and recent chats in memory. This helps to improve the performance of the service.</p>\n<p><strong>Availability:</strong> It is important for our service to remain available as much as possible. To ensure that our system is fault-tolerant, we can store multiple copies of transient messages. If any message is lost, it can be easily retrieved from its replicas. This helps to maintain the availability of the system and prevent any disruption in service.</p>\n<h3>Further Requirements</h3>\n<p>The current version of our system only supports a few basic features, but it can easily be extended to support group chats, video and phone calls, and the ability to view each other's status or stories. Additionally, we can also extend the system to allow for payments or transactions. All of these additional features require advanced concepts that are beyond the scope of this blog. We will cover these functionalities in the second part of this blog.</p>\n<p>Thanks to Suyash Namdeo for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"5bb544d7-2ec2-5981-879b-c424eeb2d81d","frontmatter":{"title":"Design URL Shortening Service like TinyURL","path":"/blog/design-a-url-shortening-service-like-tiny-url"},"html":"<p><strong>Asked in:</strong> Google, Facebook, Amazon, Adobe</p>\n<h3>Let’s understand the problem</h3>\n<p>Design a URL Shortening service like Tiny URL. The goal is to design a highly scalable service that could allow users to create shorter URLs, given long URLs, and have read and write functionality.</p>\n<h4><strong>What is Tiny-URL?</strong></h4>\n<p>Tiny-URL is a URL-shortening web service that creates shorter aliases for long URLs. Whenever the user visits the short URL, the user will be redirected to the original URL.</p>\n<p>Before jumping to the solution, as an interviewee, we should always discuss with the interviewer what he/she wants. This brings us to a discussion about the requirements and features of the system.</p>\n<h3>Requirements of the System</h3>\n<p>The URL shortening service like Tiny-URL should have the following features:</p>\n<h4><strong>Functional Requirements</strong></h4>\n<ul>\n<li>Users should be able to generate a shortened URL from the original URL.</li>\n<li>The short link should redirect users to the original link.</li>\n<li>Users should have the option to give a custom short link for their URL.</li>\n</ul>\n<h4><strong>Design Goals</strong></h4>\n<ul>\n<li>If the system fails, it will imply that all the short links will not function. So our system should be highly available.</li>\n<li>URL redirection should happen in real-time with minimal latency.</li>\n<li>Shortened links should not be predictable in any manner.</li>\n</ul>\n<h4><strong>Extended/Optional Goals</strong></h4>\n<ul>\n<li>The service should be <strong>REST API</strong> accessible.</li>\n<li><strong>Analytics:</strong> How many times the URL is visited?</li>\n<li>Users should be able to specify the <strong>expiration</strong> <strong>time</strong> of the URL.</li>\n</ul>\n<h3>System Analysis</h3>\n<p>This section will examine the estimate of the number of monthly requests, storage, and so on. The most important thing to keep in mind here is that our system will be read-heavy, meaning the number of read requests per second can be up to 1000 times greater than the number of writing requests. We will assume that our read/write ratio is 100:1.</p>\n<h4>key components</h4>\n<p>The following are the key components of the URL-shortening application:</p>\n<ol>\n<li><strong>Clients:</strong> Web Browsers/Mobile apps. They will communicate with the backend servers via the HTTP protocol.</li>\n<li><strong>Load Balancer:</strong> To distribute the load evenly among the backend servers.</li>\n<li><strong>Web Servers:</strong> Multiple instances of web servers will be deployed for horizontal scaling.</li>\n<li><strong>Database:</strong> It will be used to store the mapping of long URLs to short URLs.</li>\n</ol>\n<h4>Estimation of Scale and Constraints</h4>\n<p><strong>Traffic Estimation:</strong> We can assume that we will have 500 million new URL shortening requests per month. With a 100:1 read/write ratio, we can expect 50 billion redirections during the same period (100 * 500 million = 50 billion).</p>\n<ul>\n<li>If we calculate this value for each second, i.e., queries per second (QPS) = 500 million / (30 days * 24 hours * 3600 seconds) = ~200 URLs/s.</li>\n<li>Considering the 100:1 read/write ratio, URL redirections per second will be = 100 * 200 URLs/s = 20K/s.</li>\n</ul>\n<p><strong>Storage Estimation:</strong> Let's assume that we are willing to store our data (short URL + long URL) for ten years. Then, the number of URLs we will be storing would be = 500 million * 10 * 12 = 60 billion URLs.</p>\n<p>Now, let's assume the average URL length to be 120 characters (120 bytes), and we add 80 more bytes to store the information about the URL. The total storage requirement would be = 60 billion * 200 bytes = 12TB.</p>\n<h3>Encoding URL</h3>\n<p>What is the minimum length of the shortened URL to represent 60 billion URLs? We will use (a-z, A-Z, 0-9) to encode our URLs, which leads us to use base62 encoding. Now, the question arises: what should be the length of the short URL generated to cater to 60 billion requests? Let us find out.</p>\n<p>The longer our key, the more URLs we have, and the less we have to worry about our system ever running out. However, the point of this system is to keep our URL as short as possible. Let's stick with <strong>7</strong> because even if we consumed 1000 keys per second, it would take 111 years to run out!</p>\n<h4><strong>Encoding</strong></h4>\n<p>To generate a unique short URL, we can compute it using the unique <strong>hash (MD5, SHA256, etc.</strong>) of the original URL and then encode it using <strong>base62</strong>. If we use the MD5 algorithm as our hash function, it will produce a <strong>128-bit hash value</strong>. After base62 encoding, we will get a string with more than seven characters. We can take the first 7 characters for the key.</p>\n<p>MD5(Long URL) -> base62encode(128-bit hash value) -> <strong>FE66AB7</strong>1IT...</p>\n<p>There is a problem with this approach, as it could (however unlikely) result in duplications or collisions in the database. One solution to this problem is to use a counter, which keeps track of a count (0-3.5 trillion) so that in each case if we encode the counter value, there is a guarantee of no duplicates or collisions. </p>\n<p>Let's see how we will use the counter. We will use a single server that will be responsible for maintaining a counter. Whenever the application servers receive a request, they will talk to the counter, which returns a unique number and increments the counter.</p>\n<h4><strong>Two potential problems</strong></h4>\n<ol>\n<li>Single Point of Failure.</li>\n<li>If requests spike, our counter host may not be able to handle it.</li>\n</ol>\n<p>In this case, we will use a <strong>distributed systems manager</strong>, such as <strong>Zookeeper.</strong> Let us see how a distributed systems manager like Zookeeper solves our problem:</p>\n<ol>\n<li>It gives our application servers unused ranges.</li>\n<li>To avoid having Zookeeper be a single point of failure, we will run <strong>multiple instances</strong>.</li>\n<li>If a new server is added, it will give them an unused range. If the range runs out, the existing server can go to Zookeeper and ask for a new, unused range.</li>\n<li>If one of the <strong>servers dies</strong>, <strong>1 million keys</strong> are possibly <strong>wasted</strong>, which is <strong>acceptable</strong>, given the 3.5 Trillion keys we have.</li>\n<li>However, sequentially generating URLs can be a security threat. We can add another random 10–15 bits after the counter number to increase the randomness.</li>\n</ol>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*qehSoMkA65TmtCrzBttaAQ.png\" alt=\"design tiny url system design interview question image 1\" title=\"design tiny url zookeeper\"></p>\n<h4><strong>Base62 encoding algorithm</strong></h4>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*vntFMYglbeOqGNFaFRVXrw.png\" alt=\"base62 encoding algorithm for tiny url system design \" title=\"design tiny url encoding\"></p>\n<h3>System APIs</h3>\n<p>We can use <strong>REST APIs</strong> to expose the functionality of our service. Below is an example of the definitions of the APIs for creating and deleting URLs without service:</p>\n<h4>Parameters</h4>\n<ul>\n<li><code class=\"language-text\">api_dev_key</code> (string): The API developer key of a registered account. This will be used to throttle users based on their allocated quota.</li>\n<li><code class=\"language-text\">original_url</code> (string): Original URL to be shortened.</li>\n<li><code class=\"language-text\">custom_alias</code> (string): Optional custom key for the URL.</li>\n<li><code class=\"language-text\">user_name</code> (string): Optional username to be used in the encoding.</li>\n<li><code class=\"language-text\">expire_date</code> (string): Optional expiration date for the shortened URL.</li>\n</ul>\n<p>Returns (string): A successful insertion returns the shortened URL. Otherwise, it returns an error code.</p>\n<p>The <code class=\"language-text\">url_key</code> is a string that delineates the shortened URL that needs to be deleted. A successful deletion will return. <code class=\"language-text\">URL Removed</code>.</p>\n<h3>Database design</h3>\n<p>While designing systems, we have two types of databases (of course, we don’t want to back to the old days where file systems were used): <strong>Relational Databases</strong> or <strong>NoSQL.</strong> For our system, relational queries will not be a large part of it/occur rarely. So, here we will go with NoSQL. <strong>A NoSQL choice would be easier to scale.</strong></p>\n<p><strong>Database schema:</strong> We will need two main tables: 1 to store user information and 1 to store URL information.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*P1FKY9khOoVRj5_TnWHvMg.png\" alt=\"tiny url system design database design\" title=\"design tiny url database schema\"></p>\n<h3>Scaling the service</h3>\n<h4>Caching</h4>\n<p>We know that our database is going to be read heavily. We have found a way to speed up the writing process, but the reads are still slow. So we have to find some way to speed up the reading process. Let’s see.</p>\n<p>We can speed up our database reads by putting as much data in memory as possible, AKA a <strong>cache</strong>. This becomes important if we get massive load requests to a single link. If we have the redirect URL in memory, we can process those requests quickly. Our cache could store, let’s say, <strong>20% of the most used URLs</strong>. When the cache is full, we would want to replace a URL with a more popular one.</p>\n<p>A <strong>Least Recently Used (LRU)</strong> cache eviction system would be a good choice.<br>\nWe can <strong>shard</strong> the cache too. This will help us store more data in memory because of the more machines. Deciding which thing goes to which shard can be done using “<strong>Hashing”</strong> or “<strong>Consistent Hashing.”</strong></p>\n<h4>Load Balancing</h4>\n<p>With multiple access systems like this one, a web server may not handle it yet. To solve this problem, I will use many web servers. Each web server will take a partial request from the user.</p>\n<p>Since we have a large-scale system with multiple access happening simultaneously, a server may not handle the requests. To solve this problem, we can use multiple servers with a load balancer between “<strong>the client and the server”</strong> and “<strong>the server and the database”</strong> and avoid a single point of failure, and we will use multiple load balancers.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*x8eoOL8eff-CUjqidV4Fag.png\" alt=\"design tiny url system load balancing\" title=\"design tiny url load balancer\"></p>\n<p>Initially, we can use a simple <strong>Round Robin</strong> approach that distributes incoming requests equally among backend servers. This would be the easiest to implement. A Round Robin LB does not consider server load, so our LB may still forward requests to an overloaded or slow server.</p>\n<p>To distribute the load among servers, we will use the <strong>Least Connection Method.</strong> When a server is configured to use the least connection load balancing algorithm (or method), it selects the service with the fewest active connections.</p>\n<h3>Skeleton of the design</h3>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*bESPp4_BtfO1bCZAI-Sxwg.png\" alt=\"High level design of tiny url system design\" title=\"design tiny url high level design\"></p>\n<p>Hence in this way, we could design a highly scalable <strong>Tiny-URL</strong> service.</p>\n<h3>Idea to think!</h3>\n<ol>\n<li>Should we choose Consistency or Availability for our service?<br>\n(Hint: <strong>CAP Theorem</strong>)</li>\n<li>How would you shard the data if you were working with SQL DB?<br>\n(Hint: <strong>Consistent Hashing</strong>)</li>\n</ol>\n<p>Thanks to Chiranjeev and Navtosh for their contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"a7878677-4ba5-5b35-8851-21da2e54d78e","frontmatter":{"title":"Design Yelp (Nearby Friends Service)","path":"/blog/design-yelp"},"html":"<h3>What is Yelp?</h3>\n<p>Yelp is a Proximity server useful for discovering nearby places such as restaurants, theatres, temples, or recreational areas. It allows users to view and post photos and reviews for these events and places.</p>\n<h3>Key Requirements</h3>\n<p>Our proximity server will store information about various locations so that users can search for them based on their current location. When a user makes a query, the proximity server will deliver a list of places that are nearby. For example, if a user searches for \"restaurants\" near their current location, the proximity server will retrieve a list of nearby restaurants.</p>\n<p>So our system should meet the following requirements:</p>\n<h4>Functional requirements</h4>\n<ul>\n<li>A proximity server’s fundamental operation is searching. Users should be able to search all nearby locations within a specified radius for any particular location (latitude and longitude).</li>\n<li>Users can create new places or change existing ones by adding or editing basic features such as photos, descriptions, and so on.</li>\n<li>Users can give places ratings (from one to five stars) and reviews (written and photographs).</li>\n</ul>\n<h4>Non-functional requirements</h4>\n<ol>\n<li>The system should be extremely reliable, with real-time search capabilities and low latency.</li>\n<li>The application will be read-heavy due to the large number of search requests compared to the frequent inclusion of new locations.</li>\n</ol>\n<h3>Capacity Estimation</h3>\n<ul>\n<li>100K queries per second. </li>\n<li>Total number of spots in the system = 200 million.</li>\n<li>Our system should be capable of handling a population of 400 million people.</li>\n<li>A load of 200K requests per second should be no problem for our system.</li>\n</ul>\n<h3>High Level Design</h3>\n<p>We have three high-level working API's for this service: </p>\n<h4>Search API</h4>\n<p>Search API is responsible for searching any user query and returns information regarding the searched query.</p>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\">searchRequest<span class=\"token operator\">:</span> query_param <span class=\"token operator\">=</span> <span class=\"token punctuation\">{</span>\n   search_terms<span class=\"token operator\">:</span> <span class=\"token string\">\"paratha\"</span><span class=\"token punctuation\">,</span> \n   radius<span class=\"token operator\">:</span> <span class=\"token number\">10</span><span class=\"token punctuation\">,</span>                              \n   category_filter<span class=\"token operator\">:</span> <span class=\"token string\">\"indian\"</span><span class=\"token punctuation\">,</span>  \n   filter<span class=\"token operator\">:</span> <span class=\"token string\">\"5 start rated restaurants\"</span><span class=\"token punctuation\">,</span>     \n   maximun_no_of_results<span class=\"token operator\">:</span> <span class=\"token number\">50</span>              \n<span class=\"token punctuation\">}</span>\nResponse<span class=\"token operator\">:</span> A JSON containing information about a list of popular places matching the search query<span class=\"token punctuation\">.</span></code></pre></div>\n<h4>Add places</h4>\n<p>Add places API is responsible for adding places and returns the response of newly created place with its place_id.</p>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\">placesRequest<span class=\"token operator\">:</span> body <span class=\"token operator\">=</span> <span class=\"token punctuation\">{</span>\n   place_name<span class=\"token operator\">:</span> <span class=\"token string\">\"Baba ka Dhaba\"</span><span class=\"token punctuation\">,</span> \n   latitude<span class=\"token operator\">:</span> <span class=\"token number\">115</span><span class=\"token punctuation\">,</span>                              \n   longitude<span class=\"token operator\">:</span> <span class=\"token number\">100</span><span class=\"token punctuation\">,</span>  \n   category<span class=\"token operator\">:</span> <span class=\"token string\">\"Restaurant\"</span><span class=\"token punctuation\">,</span>     \n   description<span class=\"token operator\">:</span> <span class=\"token string\">\"Best local shop\"</span><span class=\"token punctuation\">,</span>\n   photos<span class=\"token operator\">:</span> <span class=\"token punctuation\">[</span><span class=\"token string\">\"url1\"</span><span class=\"token punctuation\">,</span> <span class=\"token string\">\"url2\"</span><span class=\"token punctuation\">]</span>                  \n<span class=\"token punctuation\">}</span>\nResponse<span class=\"token operator\">:</span> Response of the newly created place with place_id</code></pre></div>\n<h4>Add reviews</h4>\n<p>Add reviews API is responsible for adding reviews about the places.</p>\n<div class=\"gatsby-highlight\" data-language=\"python\"><pre class=\"language-python\"><code class=\"language-python\">reviewsRequest<span class=\"token punctuation\">:</span> body <span class=\"token operator\">=</span> <span class=\"token punctuation\">{</span>\n   user_id<span class=\"token punctuation\">:</span> <span class=\"token string\">\"user12\"</span><span class=\"token punctuation\">,</span> \n   place_id<span class=\"token punctuation\">:</span> <span class=\"token string\">\"place12\"</span><span class=\"token punctuation\">,</span>                              \n   rating<span class=\"token punctuation\">:</span> <span class=\"token number\">5</span><span class=\"token punctuation\">,</span>                               \n   review<span class=\"token punctuation\">:</span> <span class=\"token string\">\"Food was awesome\"</span><span class=\"token punctuation\">,</span>     \n   photos<span class=\"token punctuation\">:</span> <span class=\"token punctuation\">[</span><span class=\"token string\">\"url1\"</span><span class=\"token punctuation\">,</span> <span class=\"token string\">\"url2\"</span><span class=\"token punctuation\">]</span>                   \n<span class=\"token punctuation\">}</span>\nResponse<span class=\"token punctuation\">:</span> Response of the newly created review <span class=\"token keyword\">with</span> it's <span class=\"token builtin\">id</span><span class=\"token punctuation\">.</span></code></pre></div>\n<h4>Database Schema</h4>\n<p>The basic schema of the database for the system is described below.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*i3hnYyoLyFraiNddo7cKWA.png\" alt=\"Yelp design database schema\" title=\"Database schema Yelp system design\"></p>\n<p>Each dataset given above must be stored and indexed (places, reviews, etc.). Users want to see results in real time while searching for local places. So the indexing must be read-efficient to query this enormous database. We don’t need to worry about data updates because the position of a place doesn’t change too often. In contrast, if we want to construct a service where items, such as people or cabs, change their location regularly, we might develop a radically different design.</p>\n<p>Let’s look at the various options for storing this information and determine which technique is appropriate for our needs.</p>\n<h4>Simple SQL based storage</h4>\n<p>We can store places, reviews, and user details in an SQL database and index it to optimize search using latitude and longitude. To further improve search efficiency, we can use the concept of 2D grids to divide the world map into smaller squares. </p>\n<p>For example, if we assume the earth's surface area is 100 million square kilometres and the search radius is fixed at 10 kilometres, we can create 10 million squares with a grid size of 10 kilometres. By fixing the grid size to the query radius, we can limit the search to the target grid and its eight neighbouring grids. Every place with a location will belong to a specific grid. Each grid will have a unique ID that can be indexed and stored in the places table. Let’s see how our basic search flow works :) </p>\n<p>We can find the grid ID for every location and its eight nearby grids because our grids are statically created with a search radius equal to the grid size. As a result, the query’s overall runtime will be lowered and improved, as the search query execution scope has been narrowed to just nine grids instead of the brute force strategy, which requires us to search for the entire map.</p>\n<p>We can make it even faster by storing the grid’s number and a list of its locations in memory. As previously stated, we will have 10 million grids, with each grid ID being 5 bytes and the place ID being 10 bytes, similar to the gigantic scale of 100 million locations. As a result, the total amount of memory required to cache grid and place IDs is 2 GB. (10M  <em>5) + (100M</em>  10) ~ 2GB.</p>\n<p><strong>Problems with this approach</strong></p>\n<ul>\n<li>For popular locations, this approach may be slow to implement due to an imbalanced distribution of locations within the grids. This can lead to some grids being heavily populated and others being sparsely populated, such as in coastal regions or on islands.</li>\n<li>An alternative approach is to dynamically adjust grid sizes by maintaining a maximum number of locations in each grid. However, this strategy can be challenging to implement and adds complexity to the system.</li>\n</ul>\n<p>However, we can solve this issue using QuadTrees.</p>\n<h4>QuadTrees</h4>\n<p>A quadtree is a tree data structure that has zero or four child nodes for each parent node. It is designed to efficiently divide a flat two-dimensional space and store location data in its nodes. This makes it useful for spatial indexing and collision detection in applications such as games.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/0*A-sgkLNX3OCDwW_j\" alt=\"Quadtree tree data structure visualization\" title=\" Visualization of Quadtree tree data structure\"></p>\n<p>In our example, we can use each node in the quadtree to represent a grid and store information about the locations within that grid. When a node reaches a maximum capacity, or \"bucket size,\" of X locations, it is divided into four child nodes and the data is distributed among them recursively.</p>\n<p>Initially, all of the locations are stored in a single root node. However, as our system is designed to handle a scale of 100 million locations over five years, the root node will not be able to hold them all. The root node will therefore be recursively split into child nodes until no nodes contain more than 100 locations. This results in a quadtree with leaf nodes that store all of the locations. Let’s see how our basic search flow works.</p>\n<p>To search for a location in our quadtree, we start at the root node and work our way down the tree until we reach the appropriate leaf node. This is because locations are only stored in leaf nodes. Our quadtree creation approach ensures that locations in neighbouring nodes are geographically close to each other. As a result, when searching for nearby locations, we also consider the data in neighbouring nodes.</p>\n<p>To improve search performance, we can cache the quadtree information. With a total of 1 million nodes (100 million locations divided by a bucket size of 100), we can estimate that the node id will be 5 bytes long and each node will contain four child pointers. In addition, we need 10 bytes each for the location id, latitude, and longitude. This means that the total storage requirement is approximately 4GB (3* 10 * 100 M + 4*5*1 M ~= 4 GB).</p>\n<h3>Database sharding</h3>\n<p>Given the scale of our system, we cannot rely on a single server to handle all of the traffic. This could create a single point of failure, which is unacceptable in today's distributed systems. One solution to this issue is to partition the quadtree data using a method such as region-based sharding or sharding based on place id.</p>\n<ol>\n<li><strong>Region-based sharding:</strong> This approach involves partitioning the data based on regions, but this can lead to a non-uniform distribution of data because some regions are more densely populated than others. This can make it difficult to achieve a uniform distribution of data.</li>\n<li><strong>Sharding based on Place ID:</strong> An alternative approach is to shard the data based on the place id. We can use a hash function to calculate the hash of each place id as we build the quadtree and map each place id to a specific server where the location information is stored. This can help to evenly distribute the data across servers.</li>\n</ol>\n<p>The second method of sharding based on place id appears to be simpler and more effective. Even if we end up with multiple quadtrees, this is not a major issue because the data will be uniformly distributed across all servers. This ensures that the system can handle the expected traffic and maintain high availability.</p>\n<h3>Data Replication</h3>\n<p>In a distributed system of this scale, it is essential to have the ability to operate correctly even in the event of failures. To ensure high availability, we cannot rely on a single machine as this creates a single point of failure.</p>\n<p>One solution is to use a master-slave architecture, where only masters can write data and slaves can only read. If a master server goes down, any slave servers can take over as the master and handle writes. There may be a small delay of a few milliseconds in updating newly changed data, resulting in eventual consistency, but this should not be a significant issue for this application.</p>\n<p>To further improve fault tolerance, we can also have a replica of the quadtree index server. If a quadtree server fails, it can be quickly rebuilt by querying the index server instead of the database, which can help to reduce the load on the database.</p>\n<h3>Load Balancing</h3>\n<p>To improve the efficiency of our system, we can use load balancers in two places:</p>\n<ul>\n<li>Between clients and application servers</li>\n<li>Between application servers and backend servers</li>\n</ul>\n<p>A simple round robin technique can evenly distribute incoming requests among the backend servers. This type of load balancer is easy to set up and does not add significant overhead. It also has the advantage of automatically removing a server from the rotation if it goes down, which helps to prevent traffic from being sent to an unavailable server. However, the round robin method does not take server load into account, so a more intelligent load balancer that regularly polls the backend servers and adjusts traffic based on their load may be necessary in some cases.</p>\n<h3>Conclusion</h3>\n<p>In this blog, we discussed the design and scalability of a system similar to Yelp. Designing and scaling such a system is a challenging task that requires careful planning and consideration of various factors. If you have any thoughts or feedback on the topic, please feel free to share them.</p>\n<p>Thanks to Suyash Namdeo for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"4625fe9b-c83f-5c04-ae96-1cd4d6f5d07d","frontmatter":{"title":"Youtube System Design","path":"/blog/design-youtube-system-design-interview-question"},"html":"<h3>What is YouTube?</h3>\n<p>YouTube is one of the most popular video-streaming platforms that allows users to upload, watch, search, and share video-based content. It also provides features to like, dislike, add comments to videos, etc. So, YouTube is a huge system! However, Have you ever thought about how the YouTube system works and the underlying principles behind its design? Let’s move forward to understand this in detail.</p>\n<h3>Key requirement of the system</h3>\n<p>In this blog, we will focus on creating the small version of Youtube with the following features.</p>\n<h4><strong>Functional Requirements</strong></h4>\n<ul>\n<li>Users should be able to upload videos.</li>\n<li>Users should be able to view videos.</li>\n<li>Users should be able to change video quality.</li>\n<li>The system should keep the count of likes, dislikes, comments, and views.</li>\n</ul>\n<h4><strong>Non-Functional Requirements</strong></h4>\n<ul>\n<li>Video uploading should be fast and users should have a smooth streaming experience.</li>\n<li>The system should be highly available, scalable, and reliable. We can compromise with consistency to ensure high availability i.e. It should be fine if a user sees a slight lag in the video data.</li>\n<li>System should offer low latency and high throughput. </li>\n<li>System should be cost-efficient.</li>\n</ul>\n<h3>Capacity Estimation</h3>\n<ul>\n<li>Suppose total users = 2 billion.</li>\n<li>Suppose daily active users = 400 million.</li>\n<li>Suppose the number of videos watched/day/user = 5.</li>\n<li>Total video views/day = 400 million * 5 = 2 billion views/day.</li>\n<li>Youtube would be view-heavy (read-heavy) system. Suppose view to upload ratio (read-to-write ratio) is 1:100, then total video upload/day = 2 billion/100 = 20 million videos upload/day.</li>\n<li>Suppose the average video size is 100 MB. Total storage needed/day = 20 million * 100 MB = 2000 TB/day = 2 PB/day. Here we ignore video compression and replication, which would change our estimates.</li>\n<li>If we use existing CDN cloud services to serve videos, then it would cost money for data transfer. Suppose we use Amazon’s CDN CloudFront, which costs $0.01 per GB of data transfer. So the total cost for video streaming/day = Total video views/day * avg video size in GB * $0.01 = 2 billion * 0.1 * 0.01 = 2 million $/day. As we observe, serving videos from the CDN would cost lots of money.</li>\n</ul>\n<h2>High-Level Design</h2>\n<p><img src=\"https://cdn-images-1.medium.com/max/960/1*FpMN2QaHD6Os2XBBSoKAJw.jpeg\" alt=\"Youtube high-level design\" title=\"High level design of designing video sharing platforms like Youtube or Netflix\"></p>\n<p><strong>Client</strong>: A computer, mobile phone, etc.</p>\n<p><strong>Video Storage:</strong> A BLOB storage system for storing transcoded videos. Binary Large Object (BLOB) is a collection of binary data stored as a single entity in a database management system.</p>\n<p><strong>Transcoding Server:</strong> We use this for video transcoding or encoding purposes, which converts a video into multiple formats and resolutions (144p, 240p, 360p, 480p, 720p, 1080p &#x26; 4K) to provide the best video streaming for different devices and bandwidth requirements.</p>\n<p><strong>API Server</strong>: We use this to handle all requests except video streaming. This includes feed recommendations, generating video upload URL, updating metadata database and cache, user signup, etc. This server talks to various databases to get all the data.</p>\n<p><strong>Web Server:</strong> Used for handling incoming requests from the client. Based on the type of request, it routes the request to the API server or transcoding server.</p>\n<p><strong>CDN</strong>: Encoded videos are also stored in CDN for fast streaming purposes. When we play a video, a popular video is streamed from the CDN.</p>\n<p><strong>Load Balancer:</strong> We use this to evenly distributes requests among API servers.</p>\n<p><strong>Metadata Storage:</strong> We use this to store all the video metadata like title, video URL, thumbnails, user information, view count, likes, comment, size, resolution, format, etc. It is sharded and replicated to meet performance and high availability requirements.</p>\n<p><strong>Metadata Cache:</strong> We use this to cache metadata, user info, and thumbnails for better read performance.</p>\n<p>As YouTube is a heavily loaded service, it has various APIs to perform operations smoothly. We can design APIs for features like video uploading, video streaming, video search, adding comments, videos, recommendations, etc.</p>\n<h3><strong>Video Uploading Process</strong></h3>\n<p>We use the uploadVideo API for uploading the video content, which returns an HTTP response that demonstrates video is uploaded successfully or not.</p>\n<p>string <strong>uploadVideo</strong>(string apiKey, stream videoData, string videoTitle, string videoDescription, string videoCategory, string videoTags[], string videoLanguage, string videoLocation)</p>\n<ul>\n<li><strong>apiKey:</strong> An identification of the registered account.</li>\n<li><strong>videoData:</strong> Uploaded video data.</li>\n<li><strong>videoTitle:</strong> The title of the video.</li>\n<li><strong>videoDescription:</strong> The description text of the video.</li>\n<li><strong>videoCategory:</strong> Video category data like sports, education, etc.</li>\n<li><strong>videoTags[]:</strong> A list of tags for the video.</li>\n<li><strong>videoLanguage:</strong> The language of the content like English, Hindi, etc.</li>\n<li><strong>videoLocation:</strong> The location where the video was recorded.</li>\n</ul>\n<p>The video upload flow is divided into two processes running in parallel: 1) Uploading the video content and 2) Updating the video metadata.</p>\n<h4>1) Uploading the video content</h4>\n<p>First, videos are uploaded by the user, and then transcoding servers start the video encoding process. The encoding process converts video into multiple formats and resolutions set by the platform user. For increasing the throughput, we can parallelize the process by spreading this task across several machines. If a video will be popular, it can do another level of compression to ensure the same visual quality of the video at a much smaller size. Overall, video is processed by a batch job that runs several automated processes like generating thumbnails, metadata, video transcripts, encoding, etc.</p>\n<p>Video encoding is possible in two ways: <strong>lossless</strong> and <strong>lossy</strong>. In lossless encoding, there is no data loss between the original format to a new format. In the lossy encoding, some data is dropped from the original video to reduce the size of the new format. We might have experienced this when uploading a high-resolution image on a social network. After the upload, the image doesn’t look as good as the original image.</p>\n<p>After the completion of encoding process, two things get executed in parallel: 1) Storing the encoded videos in a transcoded database and CDN 2) Updating the metadata database and cache. Finally, API servers inform the client that the video is successfully uploaded and ready for streaming.</p>\n<h4>2) Updating the video metadata</h4>\n<p>While content is being uploaded to the video storage, the client, in parallel, sends a request to update the video metadata. This data includes title, video URL, thumbnails, user information, resolution, etc.</p>\n<h3>Video Streaming Process</h3>\n<p>We use the uploadVideo API for uploading the video content, which continuously sends the small pieces of the video media stream from the given offset.</p>\n<p>stream <strong>viewVideo</strong>(string apiKey, string videoId, int videoOffset, string codec, string videoResolution)</p>\n<ul>\n<li><strong>apiKey:</strong> An identification of the registered account.</li>\n<li><strong>videoId:</strong> An identifier for the video.</li>\n<li><strong>videoOffset:</strong> This is the time from the start of the video, which enables users to watch a video on any device from the same point where they left off.</li>\n<li><strong>codec:</strong> Codec is the video compression standard to compress large video content into smaller sizes. It uses efficient video compression algorithms to facilitate this process. We send the client’s codec info in the API to support play/pause from multiple devices.</li>\n<li><strong>videoResolution:</strong> We also send the client’s resolution details because different devices may have different resolutions.</li>\n</ul>\n<p>Whenever a user sends a request to watch the video, the platform checks the viewer’s device type, screen size, processing capability, the network bandwidth. Based on that, it delivers the best video version from the nearest edge server in real-time. After this, our device immediately loads a little bit of data at a time and continuously receives video streams from CDN or Video Storage. Here are two critical observations:</p>\n<ul>\n<li>CDN ensures fast video delivery on a global scale.</li>\n<li>There will be two types of video in our system: 1) Popular videos that are accessed frequently 2) Unpopular videos which have few viewers.</li>\n</ul>\n<p>Based on the above observation, we can make cost-effective decisions and implement a few optimizations. For example, We can only stream the most popular videos from CDN. Other videos from the high-capacity video storage. If a video will be popular later, it will move from video storage to CDN.</p>\n<p>We need to use a standard streaming protocol to control data transfer during video streaming. Some popular streaming protocols are MPEG–DASH, Apple HLS, Microsoft Smooth Streaming, Adobe HTTP Dynamic Streaming (HDS), etc. The different streaming protocols support different video encodings and playback players, so we need to choose the proper streaming protocol. Suppose we use Dynamic Adaptive Streaming (MPEG–DASH)protocol for video streaming, which can help us in two ways: </p>\n<ul>\n<li>Make content available to the users at different bit rates and reduce the buffering as much as possible. </li>\n<li>Deliver quality content based on network bandwidth and device type of the end-user.</li>\n</ul>\n<h3>Metadata Schema</h3>\n<p>We can use MySQL to store metadata like user and video information. For this, we can maintain two separate tables: one table to store user metadata and another table to store video metadata.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/960/1*8JgTOunDc9ZpJsm5YXVd5w.jpeg\" alt=\"Youtube metadata design\" title=\"Youtube database design\"></p>\n<h3><strong>Metadata Replication: Master-Slave Architecture</strong></h3>\n<p>In this architecture, we need two data sources to scale out the application: one to handle the write query and the other one to handle the read query. Here writes request will go to the master first and then apply to all the slaves. Read requests will be routed to slave replicas parallelly to reduce the load on the master. This could help us to increase the read throughput.</p>\n<p>Such design may cause staleness in data from the read replica. How? Suppose we performed a write by adding a new video, then its metadata would be first updated in the master. Now before this new data gets updated to the slave, a new read request came. At this point, slaves would not be able to see it and return stale data. This inconsistency may create a difference in view counts for a video between the master and the replica. But this can be okay if there is a slight inconsistency (for a short duration) in the view count.</p>\n<p>Since we have a huge number of new videos uploaded every day and read operation is extremely high, so the master-slave architecture will suffer from replication lag. On another side, update operation causes cache misses, which go to disk. Now the critical question is: how can we improve the performance of the read write operations further?</p>\n<h3><strong>Metadata Sharding</strong></h3>\n<p>Sharding is one of the ways of scaling a relational database besides the replication. In this process, we distribute our data across multiple machines so that we can perform read/write operations efficiently. The idea is: Instead of a single master handling the write requests, we distribute write requests to various sharded machines to increase the write performance. We can also create separate replicas to improve redundancy.</p>\n<p>Sharding can increase the system complexity and we need an abstract system to handle the scalability challenges. This led to the requirement of Vitess!</p>\n<h3><strong>Vitess:</strong> A database clustering system for horizontal scaling of MySQL</h3>\n<p>Vitess is a database clustering system that runs on top of MySQL. It has several built-in features that allow us to scale horizontally similar to the NoSQL database.</p>\n<p>Vitess Architecture. Source: <a href=\"https://vitess.io/\">https://vitess.io/</a></p>\n<p><img src=\"https://cdn-images-1.medium.com/max/960/1*I-mGXpWGn051W4f9D8iGHg.jpeg\" alt=\"Architecture of Vitess database system\" title=\"Vitess database system architecture\"></p>\n<p>Here are some important features of Vitess:</p>\n<p><strong>Scalability:</strong> Its built-in sharding features let you grow database without adding sharding logic to application.</p>\n<p><strong>Performance:</strong> It automatically rewrites bad queries to improve database performance. It also uses caching mechanisms and prevents duplicate queries.</p>\n<p><strong>Manageability:</strong> It improves manageability by automatically handling failovers and backups functionalities.</p>\n<p><strong>Sharding management:</strong> MySQL doesn’t natively support sharding, but we will need it as your database grows. It helps us to enable live resharding with minimal read-only downtime.</p>\n<h3>Caching</h3>\n<p>For building such a scalable system, we need to use different caching strategies. We can use distributed cache like Redis or Memcache to store the metadata. To make the caching service efficiently perform all its operations, we can use LRU (Least Recently Used) cache algorithm. We can also use CDN as a video content cache. CDN is useful in fetching media content directly from AWS S3. If the service is on AWS, it is convenient to use Cloudfront as a content cache and elastic cache service for metadata cache.</p>\n<h3>Disaster Management Strategy</h3>\n<p>We need to avoid data loss or service unavailability due to power outages, machine failures, natural disasters like earthquakes, etc. For this, we can back up data on data centers located in different geographical locations. The nearest data center serves the user request, which could help to fetch data faster and reduce the system latency. </p>\n<h3>Conclusion</h3>\n<p>YouTube is a very complex and highly scalable service to design. In this blog, we have covered only the fundamental concepts necessary for building such Video Sharing services. However, we have limited our discussion to the system’s generic and critical requirements, but this could be expanded further by including the various other functionalities like personalized recommendations, etc.</p>\n<h3>References and Additional Reading</h3>\n<ul>\n<li><a href=\"http://highscalability.com/blog/2012/3/26/7-years-of-youtube-scalability-lessons-in-30-minutes.html\">7 Years Of YouTube Scalability Lessons In 30 Minutes</a></li>\n<li><a href=\"http://highscalability.com/blog/2008/3/12/youtube-architecture.html\">YouTube Architecture</a></li>\n<li><a href=\"https://www.youtube.com/watch?v=w5WVu624fY8/\">YouTube Scalability Talk</a></li>\n<li><a href=\"https://vitess.io/docs/13.0/overview/architecture/\">Vitess Architecture</a></li>\n<li><a href=\"https://www.youtube.com/watch?v=OqQk7kLuaK4\">How YouTube Works?</a></li>\n<li><a href=\"https://www.youtube.com/watch?v=CZ3wIuvmHeM\">A Netflix Guide to Microservices</a></li>\n<li><a href=\"https://www.youtube.com/watch?v=JCtV7TmLTqQ\">YouTube Search &#x26; Discovery</a></li>\n<li><a href=\"https://youtube-eng.googleblog.com/2018/04/making-high-quality-video-efficient.html\" title=\"Making high quality video efficient\">Making high-quality video efficient</a></li>\n<li><a href=\"https://youtube-eng.googleblog.com/2016/05/machine-learning-for-video-transcoding.html\" title=\"Machine learning for video transcoding\">Machine learning for video transcoding</a></li>\n</ul>\n<p><strong>Enjoy learning, Enjoy system design!</strong></p>"},{"id":"07a887ce-2cf4-5880-ad5a-6dc168c7733d","frontmatter":{"title":"Uber System Design","path":"/blog/design-uber"},"html":"<p>Uber is a ride-sharing service that uses a mobile application to connect passengers with drivers. The company is committed to improving its operations and services by introducing new features and technologies. The Uber system has several key goals, which include:</p>\n<ul>\n<li>Providing efficient and reliable service to meet market demand.</li>\n<li>Helping drivers find the most efficient routes.</li>\n<li>Detecting and preventing fraud to ensure safety for all users.</li>\n<li>Updating data in real-time to provide efficient service.</li>\n</ul>\n<p>In this blog, we will explore various components to design Uber system.</p>\n<h3>Key Requirements</h3>\n<ul>\n<li>To use the ride-sharing service, riders simply need to request a trip from their starting point to their destination. Once the ride request is made, nearby drivers will be notified and one of them will confirm the ride.</li>\n<li>To ensure efficient service, drivers are required to update their current position and availability frequently, so the service can match them with nearby riders.</li>\n<li>Once a ride is confirmed, both the driver and rider can track each other's current positions until the ride is completed. After the ride, the rider will receive a receipt that includes trip information such as the route map and pricing.</li>\n<li>To ensure riders are informed about their pickup, the service displays a pickup ETA for the rider when the ride is dispatched.</li>\n</ul>\n<h3>Capacity Estimation</h3>\n<ul>\n<li>Let's assume we have 100M customers and 1M drivers, with 1M daily active customers and 100K daily active drivers.</li>\n<li>Let's also assume that there are 1 million daily ride requests, and all active drivers update their current location every 4 seconds.</li>\n<li>To ensure a smooth experience, the system must be able to instantly contact nearby drivers when a customer requests a ride.</li>\n</ul>\n<h3>High Level Design</h3>\n<p>The Uber app uses mobile devices to connect riders with drivers. Once a rider requests a ride through the app, the driver is directed to their location for pick-up. Behind the scenes, the service is supported by a large number of servers that handle and process significant amounts of data during each ride.</p>\n<p>Over time, Uber's system design has evolved from a simpler monolithic architecture to a more complex, service-oriented architecture. The architecture consists of two main components: the Supply Service, which manages the availability of cars and drivers, and the Demand Service, which handles rider requests.</p>\n<p>So the Real-time matching of riders and drivers is a crucial aspect of the Uber service. This is achieved through the Dispatch system, which uses data from the Supply and Demand Services to match riders with the nearest and most appropriate driver.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*DOXPEeTUI9Wg4B6vvP7k9w.png\" alt=\"Uber high-level design\" title=\"High-level design of Uber system\"></p>\n<h4>Data Storage</h4>\n<p>In its early days, Uber used a traditional relational database management system (RDBMS) to store data such as user profiles, GPS locations, and other information. However, as the number of users increased, the system struggled to process the vast amount of data in real time. To address this issue, Uber switched to using NoSQL databases, which provided better scalability and performance.</p>\n<p>By switching to NoSQL, Uber's databases became horizontally scalable i.e. they could handle an increasing amount of data and users without compromising on performance. Additionally, NoSQL databases offer better write and read availability, allowing for faster and more efficient data processing. This technology change allowed Uber to continue to grow and serve an increasing number of users.</p>\n<h3>Detailed Component Design</h3>\n<p>In this section, we will dive deep into the design of components involved in Uber system.</p>\n<h4><strong>Demand Service</strong></h4>\n<p>The Demand Service in the Uber system is responsible for managing the requests made by riders through the app. When a rider requests a ride, the demand service receives the request through a web socket and begins to monitor the rider's GPS location. It also processes other types of requests, such as the number of seats required, preferred car type, and whether the rider is requesting a shared ride. </p>\n<p>Using this information, the Demand Service matches the rider with the appropriate driver and vehicle, taking into account the rider's location and other preferences. It also tracks the supply and demand for rides in a specific geographic area to optimize the matching process.</p>\n<h4><strong>Supply Service</strong></h4>\n<p>The Uber system uses a combination of technologies to ensure an efficient supply of services to its customers. When a driver is on the road, their cab's location is constantly updated and communicated to Uber's servers every 4 seconds. This information is sent through a web application firewall and load balancer, which helps to ensure that the data is secure and properly distributed. </p>\n<p>Once the location data reaches the server, it is passed through Kafka's Rest APIs. This technology allows for the efficient transfer of large amounts of data and ensures that the server always has the most up-to-date location information for each cab. </p>\n<p>Now location data is stored in the main memory of the relevant worker nodes. This allows for quick and easy access to information, which is essential for providing accurate and timely service to customers. Additionally, a copy of the location data is also transmitted to a database, which is used for dispatch optimization. This helps to ensure that customers are always matched with the closest and most available driver.</p>\n<h4><strong>Dispatch System</strong> (DISCO)</h4>\n<p>The dispatch mechanism of the Uber system is based on map and location data, which requires correct modelling and mapping. Using latitude and longitude data to summarise and estimate places is tricky. However, summarizing and estimating places using latitude and longitude data can be challenging. To solve this issue, Uber use the Google S2 library. <strong>Note:</strong> DISCO had to meet several objectives like reducing extra driving, Minimising waiting time and Minimising overall ETA.</p>\n<p>Let’s have a look at how a dispatch service works: </p>\n<ul>\n<li>The dispatch system begins by dividing the map data into small cells and assigning a unique ID to each cell using the S2 library. This is a simple technique that allows for efficient storage and distribution of data in a distributed system.</li>\n<li>The S2 library also allows for the creation of specific areas or \"coverage\" based on a given location. For example, if you want to find all the drivers within a 5 kilometers radius of a city, you can create a 5km radius circle using the S2 libraries and it will filter out any cells with IDs that fall inside that radius.</li>\n<li>Next, the list of available cabs is given to the ETA (estimated time of arrival) algorithm, which uses the road system to compute the distance between the cab and the rider. The sorted ETA is then transmitted back to the supply system and offered to the driver.</li>\n<li>This process helps to minimize additional driving, reduce waiting time, and accurately predict the total ETA for the customer.</li>\n</ul>\n<h4>How is ETA Calculated?</h4>\n<p>The Estimated Time of Arrival (ETA) is a feature in the Uber app that allows riders to see approximately how long it will take for a driver to reach their pickup location before a trip begins. The app also displays an estimated time of arrival at the destination after the trip starts.</p>\n<p>To calculate the ETA, the location data of nearby drivers are used to estimate the time it will take for them to arrive at the rider's pickup location. Once the trip begins, the app regularly updates the ETA for the destination. This provides the rider with a better idea of when they can expect to arrive at their destination. The ETA calculation also considers real-time traffic, weather, and other factors that may affect the trip duration.</p>\n<h4>Overall Summary: How Uber system works?</h4>\n<ul>\n<li>When a user requests a ride, the request is sent through a web socket to the Demand Service. The Demand Service becomes aware of the need for a ride and submits a request to the Supply Service using the information from the ride. The critical question to think: Why Uber system uses web sockets?</li>\n<li>The Supply Service uses the information provided by the Demand Service, such as the rider's location, to determine which drivers are closest to the rider. It then sends a request to one of the servers in the server ring. These servers use ETA values to find out which cabs are closest to the rider.</li>\n<li>After the Supply Service computes the ETA values, it notifies the drivers through Web Sockets. If the driver accepts the request, the ride is allocated to the rider and driver. The driver will then pick up the rider and take them to their destination.</li>\n</ul>\n<h4>Analytics</h4>\n<p>Uber uses analytics to gain insight into the needs and behaviours of its customers and drivers, optimize its operations costs, and improve customer satisfaction. To achieve this, Uber employs a range of technologies and frameworks.</p>\n<ul>\n<li>Driver and rider location data are stored in databases such as NoSQL, RDBMS, or HDFS to enable analytics. For real-time analytics, the Hadoop platform provides several analytics-related tools.</li>\n<li>With HDFS, Uber can extract raw data from a NoSQL database and access HDFS data using query tools like HIVE. This enables Uber to analyze and interpret data to enhance its services and operations.</li>\n</ul>\n<h4>Surge Pricing</h4>\n<p>Surge pricing is a dynamic pricing strategy employed by Uber and other ride-hailing companies to increase the cost of rides during times of high demand. The aim of surge pricing is to incentivize more drivers to hit the road and provide more rides to meet the increased demand.</p>\n<p>The algorithm used by Uber to determine surge pricing is based on the principle of supply and demand. The system employs real-time data on the number of drivers and riders in a particular area, as well as other factors such as traffic and weather, to determine the level of demand for rides.</p>\n<p>When demand surges and the supply of drivers is low, the algorithm increases the price of rides to encourage more drivers to hit the road. The price may also change depending on the time of day, day of the week, and other factors.</p>\n<p>Here are some general steps and insights behind this algorithm:</p>\n<ol>\n<li><strong>Defining the base price:</strong> Uber set the base price for the ride based on distance, time, and any other relevant factors.</li>\n<li><strong>Determining the level of demand:</strong> The system estimates the demand for rides in a particular area by analyzing historical data or real-time data.</li>\n<li><strong>Determining the level of supply:</strong> Now system estimates the supply of available drivers in the same area at that time.</li>\n<li><strong>Calculating the surge multiplier:</strong> Now system calculates the surge multiplier by dividing the demand by the supply. For example, if the demand is twice the supply, the surge multiplier might be 2.0x.</li>\n<li><strong>Adjusting the price:</strong> Finally, system adjusts the price of the ride by multiplying the base price by the surge multiplier.</li>\n</ol>\n<h3>System Optimization</h3>\n<h4>Fast Searching</h4>\n<p>To improve the efficiency of the Uber system, it is necessary to reduce the number of locations it searches in the background. To accomplish this, we can implement a grid system where each grid has a maximum limit of 1000 locations (Or some other number). When a grid exceeds this limit, it is divided into four smaller grids of similar size using the <strong>QuadTree</strong> data structure.</p>\n<p>The QuadTree is a tree-based structure that stores data about all the locations within its corresponding grid. When a node in the QuadTree reaches its 1000-location limit, it is divided into four child nodes, each storing a subset of the locations. This process is repeated until we reach the leaf nodes, which represent grids that cannot be divided further. In this way, we can efficiently search through the locations while keeping the number of locations in each grid to a manageable limit.</p>\n<ul>\n<li>To ensure that the positions of all current drivers are accurately reflected in our data structures, we need to update them in real time as the drivers move. However, this can be time-consuming and resource-intensive. To achieve this, we must first locate the correct grid based on the driver's previous position and then update it with their new location. If the driver's new position is not in the same grid, we must remove them from the old grid and add them to the new one. If the new grid reaches its maximum driver limit after this transfer, we must repartition it to maintain the grid's size limit.</li>\n<li>Our system needs a fast and efficient way to communicate the current positions of all surrounding drivers to any active client in a specific region. Additionally, while a ride is in progress, the system must keep both the driver and the passenger informed about the vehicle's current location.</li>\n<li>To ensure that the information is up-to-date, we must continuously update QuadTree with the most recent driver positions. As all drivers report their positions every 4 seconds, we will receive many updates to our tree. So as soon as the server receives an update on a driver's position, it will notify the appropriate QuadTree server and all interested clients.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/0*A-sgkLNX3OCDwW_j\" alt=\"Quadtree Visualization\" title=\"Visualization of Quadtree\"></p>\n<p>To efficiently share a driver's location with customers, we use a Push Model, where the server pushes the location updates to all relevant users. When a customer opens the Uber app on their phone and requests nearby drivers, we subscribe to all updates from those drivers on the server-side before sending the list of available drivers to the client. This way, we can keep track of all clients interested in knowing a driver's location at any given time.</p>\n<p>Whenever there is an update to a driver's position in the QuadTree, we broadcast their current location to all subscribing clients. This ensures that customers always see the most up-to-date information on a driver's location and helps make the search for nearby drivers more efficient and quick.</p>\n<h4>Fault Tolerance and Replication</h4>\n<p>To ensure high availability, we would need duplicates of our servers so that if the primary server fails, the backup server can take over. Additionally, we can store this critical data in permanent storage, such as SSDs with rapid I/O speeds, so that if both the primary and secondary servers fail, we can still recover the data from the persistent storage.</p>\n<p>While data center failures are rare, Uber maintains a second data center to ensure that trips run smoothly. However, Uber does not replicate existing data into the backup data center, even though it has all the necessary components. To mitigate the effects of a data center failure, Uber uses driver phones as a source of trip data. When the driver's phone app connects with the dispatch system or an API call is made between them, the dispatch system transmits the encrypted data to the driver's phone app, which receives the data every time. </p>\n<p>In the event of a data center failure, the backup data center will not have information on ongoing trips, so it will request the data from the driver's phone app. The data obtained from the driver's phone app will then be used to update the backup data center.</p>\n<h3>Conclusion</h3>\n<p>In this blog, we discussed how to design an Uber-like system. However, we have just covered only a few components in this blog. In the coming future, we will add more insights.</p>\n<p>Additional blogs to explore:</p>\n<ul>\n<li><a href=\"https://eng.uber.com/observability-at-scale/\" title=\"Observability at Scale: Building Uber’s Alerting Ecosystem\">Building Uber’s Alerting Ecosystem</a></li>\n<li><a href=\"https://eng.uber.com/service-oriented-architecture/\" title=\"Service-Oriented Architecture: Scaling the Uber Engineering Codebase As We Grow\">Scaling the Uber Engineering Codebase As We Grow</a></li>\n<li><a href=\"https://eng.uber.com/tech-stack-part-one-foundation/\" title=\"The Uber Engineering Tech Stack, Part I: The Foundation\">The Uber Engineering Tech Stack, Part I</a></li>\n<li><a href=\"https://eng.uber.com/uber-tech-stack-part-two/\" title=\"The Uber Engineering Tech Stack, Part II: The Edge and Beyond\">The Uber Engineering Tech Stack, Part II</a></li>\n</ul>\n<p>Thanks to Suyash for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"db6c53b1-e83d-570a-820f-b0677cb315c0","frontmatter":{"title":"Basic System Design Concepts for Tech Interview","path":"/blog/basics-of-system-design"},"html":"<p>For distributed applications, system design provides a high-level understanding of the components to ensure that the application is well-structured, scalable, maintainable and meets the needs of its users. It helps us to define architecture by breaking down the system into small components. </p>\n<p>So understanding system design is important for the success of a software product and make decisions about various tradeoffs. On the other hand, system design is also one of the important concepts that large tech companies ask during interviews.</p>\n<h3>What is System Design?</h3>\n<p>System design deals with designing the structure of a distributed system to ensure that it meets desired functionality, performance, scalability, reliability, etc. It is an iterative process of defining the architecture, components, data, etc., to satisfy functional and non-functional requirements.</p>\n<p>When designing such systems, we need to have a good understanding of key terms like scalability, caching, load balancing, data partitioning, types of databases, network protocols, database sharding, etc. Similarly, we should have a good understanding of various trade-offs like latency vs throughput, performance vs scalability, consistency vs availability, etc. Overall, system design is an open-ended discussion topic. That's why most tech firms prefer to include system design interview rounds.</p>\n<p>Some key steps in designing a system:</p>\n<ul>\n<li><strong>Requirement analysis</strong>: Understanding the functional and non-functional requirements.</li>\n<li><strong>High-level design</strong>: Defining components and their interactions.</li>\n<li><strong>Database design</strong>: Designing the structure of the database(s) like defining tables, relationships, indexing strategies, data access patterns, etc.</li>\n<li><strong>API design</strong>: Designing APIs through which users or other systems interact with the system.</li>\n<li><strong>Component design</strong>: Designing individual components, their responsibilities and interactions.</li>\n<li><strong>Scalability and performance</strong>: Ensuring that the system can handle its expected workload efficiently and can scale to accommodate increasing workload demand.</li>\n<li><strong>Reliability and fault tolerance</strong>: Designing the system to minimise the impact of failures and to recover gracefully from errors or faults.</li>\n<li><strong>Security</strong>: Protecting the system from unauthorised access, data breaches, and other security threats.</li>\n<li><strong>Testing</strong>: Designing tests to verify that the system meets its requirements and performs as expected.</li>\n<li><strong>Documentation</strong>: Documenting the design decisions, architecture and other relevant information to facilitate understanding, maintenance, and future development of the system.</li>\n</ul>\n<p>Now let’s dive in to get familiar with the critical concepts used in system design.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/availability-system-design-concept\">Availability</a></h3>\n<p>Availability means: System should always remain up and return the response of a client request. In other words, whenever any user want to use the service, system must be available and satisfy user request.</p>\n<p>Availability can be quantified by measuring the percentage of time system remain operational in a given time window. We usually define availability of a system in terms of number of 9s (as shown in the table below).</p>\n<p><img src=\"https://ucarecdn.com/c27ec4b7-10ed-4e64-999d-f1d85407df0c/\" alt=\"nines of availability to measure availability percentages of a system\" title=\"nines of availability\"></p>\n<p>For designing a highly available system, we need to consider several factors like replication, load balancing, failover mechanisms, disaster recovery strategies, etc. Our goal should be to minimise downtime and ensure that system is always accessible and functioning as intended.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/throughput-in-system-design\">Throughput</a></h3>\n<p>Throughput is one of the key metrics used to measure the performance and capacity of a system in handling tasks, transactions or operations within a given time interval. In the context of a network, it represents the amount of data transferred successfully over a network within a certain timeframe.</p>\n<p>High throughput is crucial when handling a large data or a high number of concurrent requests. One excellent method to achieve high throughput is to split the requests and distribute them across various machines.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/latency-in-system-design\">Latency</a></h3>\n<p>Latency is another important metric related to the system performance. It is a measure of the time duration to produce the result. In other words, it is the time taken by a request to receive a response from a server.</p>\n<ul>\n<li>Latency can be affected by various factors like speed of processing, network congestion, distance between the source and destination and the efficiency of the system components. Lower latency means quick response times, higher system performance and a smooth user experience.</li>\n<li>Reducing latency is one of the key goals in optimizing performance in applications where real-time responsiveness is critical like online gaming, video streaming, financial trading, etc. </li>\n<li>Strategies for reducing latency: Optimizing algorithms, improving network infrastructure, using faster hardware components, minimizing unnecessary processing steps, etc.</li>\n</ul>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/network-protocols-concept\">Network Protocols</a></h3>\n<p>Almost every system has dependency on networks. It acts as a platform for communication between users and servers or different servers. On another side, protocols are the rules governing how servers or machines communicate over the network. Some of the common network protocols are HTTP, TCP/IP, etc.</p>\n<p><img src=\"https://ucarecdn.com/99f14a25-c0ec-438a-9675-aa00587e6f00/\" alt=\"network protocols layers\" title=\"different layers in computer networking\"></p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/load-balancers-in-system-design\">Load Balancing</a></h3>\n<p>Load balancers are machines that balance the load among various servers. To scale the system, we can add more and more servers. So there must be a way to direct requests to these servers so that there is no heavy load on one server. In other words, load balancers are used to to prevent a single point of failure.</p>\n<ul>\n<li>Load balancers distribute traffic, prevent service from breakdown and contribute to the system reliability. </li>\n<li>Load balancers act as traffic managers and help us to maintain system throughput.</li>\n</ul>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/proxies-in-system-design\">Proxies</a></h3>\n<p>Proxy is present between the client and server. When a client sends a request, it passes through the proxy and then reaches the server. Proxies are of two types: Forward Proxy and Reverse Proxy. The forward proxy acts as a mask for clients and hides the client identity from the server. Similarly, Reverse proxy acts as a mask for servers and hides server identity from the response.</p>\n<h3>Databases</h3>\n<p>Systems are associated with data that needs to be stored for later retrieval. One way to classify databases is based on their structure, which can be either relational or non-relational. Relational databases like MySQL and PostgreSQL, enforce strict relationships among data and have a highly structured organization. Non-relational databases like Cassandra and Redis have flexible structures and can store both structured and unstructured data. They are often used in systems that are highly distributed and require high speed.</p>\n<p><strong><a href=\"https://www.enjoyalgorithms.com/blog/data-partitioning-system-design-concept\">Database Partition</a></strong> is the way of dividing database into smaller chunks to increase performance. It is another critical concept used to improve latency and throughput so that more and more requests can be served. </p>\n<p>Another key concept is <a href=\"https://www.enjoyalgorithms.com/blog/introduction-to-database-replication-system-design/\">data replication</a>, where we maintain copies of data across different locations or servers to improve system availability and reduce latency. Here we need to explore various replication strategies like single-leader, multi-leader, no-leader, synchronous vs asynchronous, etc.</p>\n<h4>ACID vs BASE</h4>\n<p>Relational and non-relational databases ensure different types of compliance. Relational databases are associated with ACID compliance, while non-relational associated with BASE compliance.</p>\n<p><strong>ACID: Atomicity, Consistency, Isolation, Durability</strong></p>\n<p>ACID compliance ensures that transactions are executed in a controlled manner. A transaction is an interaction with the database.</p>\n<ul>\n<li><strong>Atomicity:</strong> A group of operations are treated as a single unit of work. If any of these operations fail, entire transaction will fail. This ensures that all or nothing is achieved!</li>\n<li><strong>Consistency:</strong> The database state changes do not corrupt the data and that transactions can move from one valid state to another.</li>\n<li><strong>Isolation:</strong> Transactions are executed independently of each other, without affecting other transactions.</li>\n<li><strong>Durability:</strong> Data written to the database is persisted and will remain there, even in the event of a failure.</li>\n</ul>\n<p><strong>BASE: Basically Available Soft State Eventual Consistency</strong></p>\n<p>BASE compliance maintains integrity of NoSQL databases and ensures their proper functioning. It is a key factor in the scalability of NoSQL databases.</p>\n<ul>\n<li><strong>Basically Available</strong> ensures that the system is always available.</li>\n<li><strong>Soft state</strong> provides flexibility to the system and allows it to change over time for faster access.</li>\n<li><strong>Eventual Consistency</strong> ensures that the data in the system takes some time to reach a consistent state and eventually becomes consistent.</li>\n</ul>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/sql-vs-nosql\">SQL vs NoSQL</a></h3>\n<p>While designing any application, it is important to be clear about the type of storage that best suits the system requirements. If the system is distributed in nature and scalability is essential, NoSQL databases are often the best choice. NoSQL databases are also preferred when the volume of data is large.</p>\n<p>SQL databases are favourable when the data is structured (e.g., rows and columns in tables) and complex querying capabilities are important. But there is always a trade-off when choosing between NoSQL and SQL databases. Sometimes, depending on business requirements, a polyglot architecture comprising both SQL and NoSQL databases is used to optimize application performance.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/distributed-system-in-system-design/\">Scalability in distributed systems</a></h3>\n<p>As services grow and number of requests increases, system can become slow. Scaling is the best way to address this issue, which involves increasing the capacity of system to handle more requests. There are two main ways to scale: horizontal scaling and vertical scaling.</p>\n<p><strong>Horizontal scaling:</strong> Adding more servers to distribute requests. This allows system to handle more traffic by spreading it across multiple servers. </p>\n<p><strong>Vertical scaling:</strong> Increasing the capacity of a single machine by upgrading its resources like CPU, RAM, etc. This allows the system to handle more traffic by increasing the capacity of a single machine.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/caching-system-design-concept\">Caching</a></h3>\n<p>Caching helps to improve the performance of a system by reducing its latency. It stores frequently used data in a cache so that it can be accessed quickly, instead of querying the database. But implementing a cache also adds complexity because it is important to maintain consistency between data in the cache and data in the main database.</p>\n<p>Cache is relatively expensive, so the size of cache will be limited. To ensure the best performance, various cache eviction algorithms like LIFO, FIFO, LRU, and LFU, are used to manage the cache. These algorithms determine which data should be removed from cache when space is needed.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/consistent-hashing-in-system-design\">Consistent Hashing</a></h3>\n<p>Consistent hashing is a widely used concept in distributed systems because it offers flexibility in scaling. It is an improvement over traditional hashing, which are ineffective for handling requests over a network.</p>\n<p>In consistent hashing, users and servers are located in a virtual circular ring structure called a hash ring. The ring is considered infinite and can accommodate any number of servers. Servers are assigned random locations based on a hash function. This concept allows for efficient distribution of requests or data among servers. It helps us achieve horizontal scaling, which increases throughput and reduces latency.</p>\n<h3><a href=\"https://www.enjoyalgorithms.com/blog/cap-theorem-in-system-design\">CAP Theorem</a></h3>\n<p>CAP Theorem is a concept for designing networked shared data systems. It states that a distributed database system can only provide two of these three properties: consistency, availability and partition tolerance. We can make trade-offs between three properties based on the use cases of our dbms system.</p>\n<ul>\n<li><strong>Consistency:</strong> Consistency means that everything should go on in a very well-coordinated manner and with proper synchronisation. It ensures that system returns the results such that any read operation should give value of the most recent write operation.</li>\n<li><strong>Availability:</strong> Availability means that the system is always available whenever any request is made to it. In other words, whenever any client requests the server, system should remain available and give the response irrespective of the failure of one or more nodes.</li>\n<li><strong>Partition Tolerance:</strong> Partition tolerance is necessary for any distributed system, so we always need to choose between availability and consistency. It corresponds to the condition that the system should work irrespective of any node failure or network failures.</li>\n</ul>\n<h3>Some other important concepts to explore</h3>\n<ul>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/server-sent-events\">Server-Sent Events</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/long-polling-in-system-design\">Long Polling</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/client-server-architecture/\">Client Server Architecture</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/web-sockets-in-system-design\">Web Sockets</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/types-of-load-balancing-algorithms/\">Load Balancing Algorithms</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/peer-to-peer-networks\">Peer to Peer Networks</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/master-slave-replication-databases/\">Master Slave Replication</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/throttling-and-rate-limiting\">Throttling and Rate Limiting</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/how-to-choose-right-database\">How to Choose the Right Database?</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/leader-election-system-design\">Leader Election in Distributed Systems</a></li>\n</ul>\n<h3>Conclusion</h3>\n<p>System design is an essential skill to have and is equally important from the interview point of view. One needs to be well aware of all the trade-offs while designing any system. In this blog, we tried to cover some basic concepts necessary for getting started.</p>\n<p>We are looking forward to your feedback in the message below. Enjoy learning, Enjoy system design!</p>"},{"id":"b0bbc366-ff77-58a3-849c-9a30cfa46481","frontmatter":{"title":"Introduction to Distributed System Concepts","path":"/blog/distributed-system-in-system-design"},"html":"<h3>What is a distributed system?</h3>\n<p>A distributed system is a network of software components or machines that function as a unified system for the end user. Here several computers communicate with one another and share resources to complete tasks that can be complex or time-consuming for a single machine to handle. On other side, due to the decentralised nature, distributed systems can handle a high volume of requests and serve millions of users.</p>\n<p>For example, traditional databases stored on a single machine may experience difficulties in performing read and write operations as the amount of data grows. One way to resolve this issue is to split the database system across multiple machines. If read traffic is much higher than write traffic, we can also implement master-slave replication, where read and write requests are processed on separate machines.</p>\n<p>Overall: scalability, performance, reliability and availability are four key characteristics for designing distributed systems. Let's move forward to discuss each one of them!</p>\n<h3>Scalability</h3>\n<p>Distributed systems can continuously evolve to support growing workloads like handling a large number of requests or a large number of database transactions. So one of the key characteristics of a distributed system is to achieve high scalability without decrease in performance.</p>\n<p>Traditionally, we can scale using vertical scaling. Here we increase system capacity by adding more hardware resources (CPU, RAM, etc.) to an existing server. With an increase in scale, this can be expensive and prone to a single point of failure. The idea is simple: It is limited by the capacity of a single server, and scaling beyond that capacity often requires downtime.</p>\n<p>On the other hand, we can easily increase the capacity of an existing system by using horizontal scaling. Here idea is to add more servers to the pool. So, if there is any performance degradation due to a large number of requests, we can simply add more machines. Horizontal scaling is a common approach for distributed systems because it makes the system faster with minimal cost compared to vertical scaling.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/800/0*M-ldUX-FP95GJb66.png\" alt=\"Comparison between vertical and horizontal scaling\" title=\"Vertical vs. Horizontal Scaling\"></p>\n<h3>Performance</h3>\n<p>There are two standard parameters to measure the performance of a distributed system:</p>\n<ol>\n<li>Latency or response time: Delay in obtaining the response to a request.</li>\n<li>Throughput: Number of requests served in a given time.</li>\n</ol>\n<p>These factors are related to the volume of responses sent by nodes and the size of responses. On the other side, performance also depends on factors like network load, architecture of software and hardware components, etc.</p>\n<p>Many scalable services are read-heavy, which can decrease system performance. To address this issue, one solution is to use replication. To further increase performance, distributed systems can use sharding by dividing the central database server into smaller servers called shards. This will improve performance by distributing the load across multiple servers.</p>\n<h4>Scalability vs Performance</h4>\n<p>Scalability and performance are related, but not the same thing. Performance measures how quickly a system can process a request, while scalability measures how much a system can grow or shrink. For example, consider a system with 200 concurrent users, each sending a request every 5 seconds (on average). In this situation, the system would need to handle a throughput of 40 requests/sec. So the performance determine how much time it takes to serve 40 requests/sec, while the scalability determine how many additional users the system can handle and how many more requests it can serve without degrading the user experience.</p>\n<h3>Reliability</h3>\n<p>A distributed system is reliable if it can continue delivering its services even when one or more components fail. In such systems, another healthy machine can always replace a failing machine. For example, an e-commerce store needs to ensure that user transactions are never cancelled due to a machine failure. If a user adds an item to their shopping cart, the system must not lose that data. Here reliability can be achieved through redundancy i.e. if the server hosting the user's shopping cart fails, another server with an exact replica should take its place. So there is a cost associated with high reliable systems.</p>\n<h4><strong>How important is reliability?</strong></h4>\n<p>Bugs or outages of critical applications can lead to lost productivity and significant costs in terms of lost revenue. Even in noncritical applications, businesses have a responsibility towards their users. Suppose a customer stored their important data in an application. After some time, that database suddenly got corrupted and there is no mechanism to restore it from a backup. What would happen? This would likely result in a loss of trust. So it is important to ensure the reliability of the application to avoid these issues.</p>\n<h3>Availability</h3>\n<p>Availability is the percentage of time that a system or service is operational under normal conditions. It is a measure of how often a system is available for use. In other words, an application that can continue to function for several months without experiencing downtime can be considered highly available.</p>\n<p>For example, in a company like Amazon, internal services are described in terms of the 99.9th percentile for availability. This means that 99.9% of requests are expected to be served without issue. While this only affects 1 in 1,000 requests, it is important to prioritise these requests because they are likely from the company's most valuable customers, who have made many purchases and have a lot of data on their accounts.</p>\n<h4>Reliability vs Availability</h4>\n<p>If a system is reliable, it is available. However, if it is available, it is not necessarily reliable. In other words, high reliability contributes to high availability. Still, achieving high availability even with an unreliable system is possible by minimising maintenance time and ensuring that machines are always available when needed.</p>\n<p>For example, suppose a system has 99.99% availability for the first two years after its launch. Unfortunately, the system was launched without any security testing. Customers are happy, but they never realise that system is vulnerable to security risks. In the third year, the system experiences a series of security problems that suddenly result in low availability for long periods. This may result in financial loss to the customers.</p>\n<h3>Other key characteristics of a distributed system</h3>\n<ul>\n<li><strong>Manageability:</strong> How it is easy to operate and maintain the system. If it takes a lot of time to fix system failures, availability will decrease. So early detection of faults can reduce or prevent downtime.</li>\n<li><strong>Concurrency:</strong> Ability of multiple components to access and update shared resources concurrently without interference. Concurrency reduces latency and increases the throughput.</li>\n<li><strong>Transparency:</strong> Allows users to view a distributed system as a single, logical device, without needing to be aware of the system architecture. This is an abstraction where a distributed system consisting multiple components, appears as a single system to the end user.</li>\n<li><strong>Openness:</strong> Ability to update and scale a distributed system independently. This is about how easy it is to integrate new components or replace existing ones without affecting the overall environment.</li>\n<li><strong>Security:</strong> It is crucial because users often send requests to access sensitive data managed by servers. For example, doctors may request records from hospitals, and users may purchase items through an e-commerce website. To prevent denial of service attacks and ensure security, distributed systems must use secure authentication processes.</li>\n<li><strong>Heterogeneity:</strong> Distributed system components may have a variety of differences in terms of networks, hardware, operating systems, programming languages, and implementations by different developers.</li>\n</ul>\n<h3>Types of Distributed Systems</h3>\n<p>One way to classify distributed systems is based on their architecture, which describes the way the components are organized and interact with each other. Some of the common architectures are:</p>\n<p><strong><a href=\"https://www.enjoyalgorithms.com/blog/client-server-architecture/\">Client-Server Architecture</a>:</strong> Most traditional and widely used architecture. In this model, clients send requests to a central server, which processes and returns the requested information. An example is an email system where client is an email software (Gmail) and the server is Google Mail Server.</p>\n<p><strong><a href=\"https://www.enjoyalgorithms.com/blog/peer-to-peer-networks/\">Peer-to-Peer (P2P) Architecture</a></strong>: In P2P, there are no centralized machines. Each component acts as an independent server and carries out its assigned tasks. Responsibilities are shared among multiple servers called peers, who collaborate to achieve a common goal. An example is file-sharing networks like BitTorrent, where every user acts as both a client and a server. Users can upload and download files directly from each other's computers, without the need for a central server.</p>\n<h3>Advantages of distributed systems</h3>\n<ul>\n<li><strong>Reliability:</strong> They remain available most of the time, irrespective of the failure of any particular system server. If one server fails, the service remains operational.</li>\n<li><strong>Scalability:</strong> Using horizontal scalability, we can add a large number of servers independently.</li>\n<li><strong>Low latency:</strong> We can use replication and place servers close to the user location to reduce the latency.</li>\n<li><strong>Cost-effective:</strong> Compared to a single-machine system, the distributed system is made up of several machines together. Although such systems have high implementation costs, they are far more cost-effective when working on a large scale.</li>\n<li><strong>Efficiency:</strong> Distributed systems are efficient in every aspect since they possess multiple machines. Each of these computers could work independently to solve problems.</li>\n</ul>\n<h3>Disadvantages of Distributed Systems</h3>\n<ul>\n<li><strong>Complexity:</strong> Distributed systems are highly complex. Although using a large number of machines, the system can become scalable, but it increases the system’s complexity. There will be more messages, network calls, devices, user requests, etc.</li>\n<li><strong>Network failure:</strong> Distributed systems heavily depend on network calls for communications and transferring data. In case of network failure, message mismatch or incorrect ordering of segments leads to communication failure and eventually deteriorates its application’s overall performance.</li>\n<li><strong>Consistency:</strong> Because of its complex nature, it becomes challenging to synchronise the application states and manage the data integrity in the service.</li>\n<li><strong>Management:</strong> Many functions, such as load balancing, monitoring, increased intelligence, logging, etc., need to be added to prevent the distributed systems’ failures.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>Distributed systems are a necessity of the modern world as new machines need to be added, and applications need to scale to deal with technological advancements and better services. It enables modern systems to offer highly scalable, reliable, and fast services.</p>\n<p>If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"7d53234d-9cb0-528c-8c15-cb30e26511ed","frontmatter":{"title":"Caching Strategy: Write-Through Pattern","path":"/blog/write-through-caching-strategy"},"html":"<p>The idea of write-through pattern is similar to <a href=\"https://www.enjoyalgorithms.com/blog/read-through-caching-strategy/\">read-through pattern</a> but with one key difference: Here cache is also responsible for handling write operations. So, whenever an application wants to write some data, it will first write directly to the cache. At the same time, cache system will update data synchronously to the main database.</p>\n<p>So cache is present between the application and database as an abstraction, and write operation always goes through the cache to main database. When both cache and database write are complete, write operation will be considered complete. This will ensure data consistency between cache and database.</p>\n<p><img src=\"https://ucarecdn.com/6367232b-63e9-4074-99fc-b4593cff6d25/\" alt=\"How write-through caching works?\" title=\"What is write-through caching?\"></p>\n<p>Now, what about reading the data? As architecture is similar to read through pattern, we can easily use read-through strategy. Application will first look for data in the cache, and if it exists, cache will return it. If it does not exist, cache will fetch it from database, update its entry and return it to the application.</p>\n<p>This combination adopts the fast read operation benefits of read-through strategy and data consistency benefits of write-through strategy. However, there is a significant disadvantage: This will introduce extra write latency because write must go to the cache and then to the database (two write operations). Is there a way to handle this latency issue? Can we use write-through pattern with <a href=\"https://www.enjoyalgorithms.com/blog/cache-aside-caching-strategy/\">cache-aside pattern</a>? Explore and think! </p>\n<h3>Pros and cons of write-through caching</h3>\n<ul>\n<li>Since all writes are immediately reflected in the database, read requests of the same data will always see the up-to-date version. This helps to avoid stale data in cache, and any subsequent read operations will be very fast. So, write-through strategy is the best choice when data freshness and performance of read operations are essential.</li>\n<li>As we have seen above, latency of write operation will be very high. If there are frequent writes, cache will have to access database frequently. On the other side, continuous write operations can evict data from cache which could be useful. Let's understand this! Every write is forced to go through cache. Due to this, cache may get occupied by frequently written data or data that is not frequently accessed. There can be little room left for other data that could benefit from caching. So write-through caching is not a good choice when write operations are frequent.</li>\n<li>If system uses a write-through cache for all data access (both reads and writes), a cache failure can cause severe issues. For example, if cache becomes unavailable, it can impact application performance because every read and write operation has to go through cache. So rather than using this strategy for all data access, application can selectively cache only certain types of data, such as frequently accessed or read-intensive data that doesn’t change frequently.</li>\n<li>Write-through caching reduces the risk of data loss in case of system failures like power outages or crashes. The idea is simple: When system recovers, it can pick up from where it left off because data in the cache and database were always kept synchronized.</li>\n</ul>\n<h3>Why there is a need for cache eviction strategy?</h3>\n<p>In one way, it looks like there is no need for a cache eviction strategy in a write-through cache because cache is always consistent with the database. But practically, this is not the scenario! We still need to define TTL or other eviction strategies (LRU or LFU). Why? There are several reasons:</p>\n<ul>\n<li>Even though cache is consistent with the database, it doesn't mean that every single piece of data should be kept in cache indefinitely. Storing all data indefinitely could lead to inefficient cache usage.</li>\n<li>As we have seen above, write-through can populate cache with unnecessary data. By adding a time-to-live (TTL) value to each write, we can avoid cluttering up cache with extra data and ensure that cache contains the most frequently accessed data.</li>\n<li>What will happen when some data in the database is updated by some external service? This will create inconsistency between cache and the database. So, we need to define a proper eviction strategy to ensure updated data in the cache.</li>\n</ul>\n<h3>Critical idea to explore!</h3>\n<ul>\n<li>If write succeeds on the cache but fails on the database. How can you handle this inconsistency?</li>\n<li>How does the system handle cache failures or database failures?</li>\n<li>Is there a way to optimize performance of the write operation?</li>\n<li>If system uses distributed caching with multiple cache nodes, how is cache coherency maintained across these nodes to ensure consistency?</li>\n<li>How does the write-through cache handle cache synchronization when there are multiple instances of the application running concurrently?</li>\n<li>Write behind vs Write through cache.</li>\n<li>Code implementation and real-life systems that use write-through caching.</li>\n</ul>\n<p>Please write in the message below if you want to share some feedback or if you want to share more insight. Enjoy learning, Enjoy system design!</p>"},{"id":"f4823599-0aad-513e-8354-140b91c2a003","frontmatter":{"title":"Design Google Docs","path":"/blog/design-google-docs"},"html":"<p>Google Docs is an online word processing tool that is part of Google's free, web-based Google Docs Editor package. It has many features and is a complex system. If you take a closer look at how it is built and how it works, you may realize that it is much more intricate than it appears at first glance. Without any much delay let’s see how Google Docs work :)</p>\n<h3>Key requirements</h3>\n<p>Google Docs is a large system with many features, including document storage, sharing, formatting, and editing. The system can be broken down into these main components:</p>\n<ul>\n<li>File organization: users can create and remove folders to organize their documents.</li>\n<li>Online editing: users can edit and format documents in real-time.</li>\n<li>Collaboration: multiple users can work on the same document simultaneously.</li>\n<li>Sharing and permissions: users can share documents and assign different permissions (e.g., owner, read-only, allow comments).</li>\n</ul>\n<h3>High Level Design</h3>\n<p>It’s always convenient to have a high-level overview of the system we’re trying to build. One way to simplify a complex system is to divide it into smaller components. Let's take a look at these components:</p>\n<h4>Storage and Formatting</h4>\n<p>Google Docs has a storage mechanism similar to an operating system, with concepts such as directories, files, and owners. A file object is a fundamental component of the system, and includes content, parent, owner, and additional metadata like creation date. The root directory has an empty parent, which represents the folder hierarchy.</p>\n<h4>Concurrency</h4>\n<p>Google Docs allows multiple users to collaborate on the same document, but how does it achieve this? There are two types of algorithms used for collaborative editing: <strong>Conflict-free Replicated Data Types (CRDTs</strong>) and <strong>Operational Transformation (OT)</strong>. </p>\n<p>Google Docs uses an event passing mechanism called operational transformation to maintain and synchronize a consistent state between users working on a shared document.</p>\n<p>For example, if two users, Sam and Aditi, are working on the same document, which has an initial state \"BC\", and they both make changes to it. Their changes will be sent to the server along with their last revision. Specifically, if Aditi inserts the character \"D\" at position 2 and Sam inserts the character \"A\" at the beginning, the server will transform Aditi's change to be inserted at the 3rd index. The server keeps track of the complete revision history of each document, allowing each collaborator to edit the document. </p>\n<p>The figure below shows how operational transformation works:</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/0*E_l8kCVAsP_q_LvA\" alt=\"How operational transformation works in google doc?\" title=\"Operational transformation in google doc system design\"></p>\n<h4>Access Control</h4>\n<p>Google Docs allows you to invite collaborators and assign them various levels of permission, such as read-only or owner. However, for more precise access control, you can use Role-Based Access Control (RBAC). RBAC ensures that employees only have access to the information and resources needed to fulfill their specific job responsibilities, and cannot access information that is not relevant to their role. This helps to protect sensitive data and maintain network security.</p>\n<p>In the RBAC model, access can be restricted to specific actions, such as reading, creating, or editing files, based on various variables including authorization, responsibility, and job expertise. This ensures that only those with the appropriate permissions can perform certain actions, which helps to protect sensitive data and critical applications. RBAC has several advantages, including:</p>\n<ul>\n<li>Increasing operational effectiveness</li>\n<li>Increased visibility of the owner</li>\n<li>Cost reduction</li>\n<li>Reducing the risk of data breaches</li>\n</ul>\n<h3>How Google Docs work?</h3>\n<p>Designing a collaborative document editing platform can be challenging due to several factors. One difficulty is in allowing multiple users to edit the same document without conflicting changes. Another challenge is ensuring that simultaneous edits are effectively merged without causing issues. Both of these issues require careful planning and design in order to provide a seamless and efficient collaboration experience for users.</p>\n<p>Operational transformation is a technique used in Google Docs to solve the first problem of allowing multiple users to edit a document simultaneously. The second problem is solved by the collaborative protocol. When you open a Google document, code is running in two places: your web browser and the Google servers. The code that runs in your browser is called the client. The client processes all of your changes to the document before sending them to the server, and then processes changes made by other editors after receiving them from the server.</p>\n<p>To effectively collaborate in Google Docs, each client must keep track of four pieces of information:</p>\n<ul>\n<li>The latest revision received from the server.</li>\n<li>Any local modifications that have not yet been sent to the server.</li>\n<li>Any local modifications that have been sent to the server, but have not yet been acknowledged.</li>\n<li>The current state of the document as it is visible to the editor.</li>\n</ul>\n<p>The server maintains three pieces of information:</p>\n<ul>\n<li>A list of all modifications it has received but has not yet processed.</li>\n<li>The complete history of all changes made to the document.</li>\n<li>The current state of the document as of the latest processed update.</li>\n</ul>\n<p>By effectively using the information available to them, it is possible to design the client-server communication in a way that allows all editors to quickly process each other's updates in real-time. Let's examine how this is done in a document with a simple example.</p>\n<h3>Workflow</h3>\n<p>Let’s look into actual flow by considering an example:</p>\n<ol>\n<li>Let’s say Bob begins the text by entering the word Hello at the top.</li>\n<li>Bob keeps typing and adds the term world to his paper. John types an ‘!’ in his empty version of the document at the exact moment.</li>\n<li>The edit was added to Bob’s client’s list of pending revisions. He then sent the update to the server and added it to his sent changes list.</li>\n<li>John got Bob’s edit from the server and transformed it against his pending (‘!’)update using operational transformation (OT). As a result of the transition, John’s pending change was moved up five spaces to create a place for Bob’s Hello at the top of the document. When Bob and John received the notifications from the server, they both changed their last synced revision numbers to 1.</li>\n<li>Following that, both Bob and John will send their pending changes to the server.</li>\n<li>Because Bob’s change arrived before John’s, the server processed it first. Bob received a confirmation of the adjustment. The modification was delivered to John, who had it changed against his (‘!) change, which was still waiting.</li>\n<li>The server received John’s pending modification, and John believes it should be Revision 2. The server, however, has already added Revision 2 to the revision log. The server will apply OT to John’s patch and save it as Revision 3.</li>\n<li>The server started by comparing John’s transmitted change to all the other modifications committed since the last time John synced with the server. It turned John’s change against Bob’s in this example (‘Google’ at 6). As a result, John’s change over-index was pushed by 6. This shift is identical to John’s client’s metamorphosis when he first received Bob’s (‘Hello’ at 1).</li>\n</ol>\n<p>The above example concludes with Bob and John receiving and acknowledging John’s change, respectively. The server and both editors are staring at the same page at this point —Hello Google!</p>\n<p>So this is how Google Docs work :)</p>\n<h3>Monitoring and Observability</h3>\n<p>To ensure the smooth functioning and high availability of our system, it is important to monitor and troubleshoot it by providing warnings when performance or availability decline. One way to do this is by integrating Google Cloud into our observability and alerting procedures.</p>\n<p>Google Cloud services automatically provide observability data, such as metrics, logs, and trace data, which allows for a comprehensive view of the system. We can also use third-party monitoring, alerting, and notification solutions to transmit observability and event data for our Google Cloud services. This data can be extracted and integrated into our current reliability and alerting systems. </p>\n<p>By combining these approaches, we can effectively monitor and troubleshoot our system to ensure it is running smoothly and efficiently.</p>\n<h3>Conclusion</h3>\n<p>In this blog post, we attempted to explain how Google Docs works. It can be challenging to design real-time products like Google Docs. We hope that by reading this article, you gained some insight into the inner workings of Google Docs and the considerations that must be made to ensure a seamless collaborative experience. If you have any thoughts or comments, please share them in the message section below.</p>\n<p>Thanks to Suyash Namdeo for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"4f5c8a4b-554b-5973-bdb4-fe3b192ae93b","frontmatter":{"title":"How to Approach System Design Questions during Interview?","path":"/blog/how-to-approach-system-design-interview"},"html":"<p>System design questions are popular during interview at top tech companies and large-scale tech startups. These interviews are open-ended conversations where candidates are expected to take the lead and guide the discussion with the interviewer. In this blog, we will discuss critical steps that candidates can follow to solve system design questions during interviews.</p>\n<h3>Step 1: Requirements clarifications</h3>\n<p>Since system design questions are often open-ended and do not have a single correct answer, it is important to ask questions and clarify any ambiguities early in the interview process. Candidates who take time to understand the end goals of the system are more likely to be successful. To gather requirements and understand the scope of the problem, one can ask such questions:</p>\n<ul>\n<li>Who will be using the system?</li>\n<li>How will they be using it?</li>\n<li>How many users are there?</li>\n<li>How much data is expected to be handled?</li>\n<li>How quickly might the number of users and amount of data grow?</li>\n<li>What availability and uptime are expected?</li>\n<li>How long should data be retained and does it need to be archived?</li>\n<li>How many requests per second are expected?</li>\n<li>What response times are acceptable for key operations?</li>\n<li>What is the expected read-to-write ratio?</li>\n</ul>\n<h3>Step 2: Back-of-the-envelope estimation</h3>\n<p>When designing a system, it is important to understand the system scale. We need to consider factors like read-to-write ratio, number of concurrent requests, number of monthly users, number of daily active users, required storage capacity, bandwidth requirements, etc. Once discussing these factors with the interviewer, you can start to think about the ways to design a system that can work well at scale.</p>\n<h3>Step 3: Create a high-level design</h3>\n<p>To initiate the high-level design process, we need to identify all critical components and attempt to draw a diagram representing the interaction of these components to fulfil functional and non-functional requirements. Here components can include different clients, backend services, data stores, cache systems, load balancers, CDNs, message queues, batch processing systems, etc.</p>\n<p>By outlining the interaction of key components, you can better explain the overall structure of the system and how various components will collaborate to achieve the desired results. This can help ensure that your design aligns with all the requirements and constraints.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*F9QnGqmAMcXEr1lQeSiqVA.png\" alt=\"High level design of web crawler\" title=\"how to approach system design Interview high level design\"></p>\n<h3>Step 4: Database Design</h3>\n<p>Defining how data will be processed is a crucial step in solving system design questions, especially when building scalable systems. We need to understand the type of data that needs to be stored, the relationships between different types of data, the expected workload (read-heavy, write-heavy, or balanced), how data will flow through the system, etc.</p>\n<p>Based on the various properties and data access patterns, we need to identify the appropriate database systems (relational or non-relational) that best fit the given requirements. You can try to explore this blog: <a href=\"https://www.enjoyalgorithms.com/blog/sql-vs-nosql/\">SQL vs NoSQL Comparison</a>.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*BhLIJHb0bkM4gjajxcXxFQ.png\" alt=\"Types of databases used in system design process\" title=\"how to approach system design Interview database design\"></p>\n<h3>Step 5: Design core components</h3>\n<p>Once you have outlined the high-level structure of the system, then it's time to discuss the major components in more detail. At this stage, interviewer input can often help guide you to parts of the system that need more attention. It is important to present various approaches, benefits and drawbacks of each option and justify why you have chosen a particular approach. For example, if you were asked to <a href=\"https://www.enjoyalgorithms.com/blog/design-a-url-shortening-service-like-tiny-url/\">design Tiny-URL</a>, then discuss these things:</p>\n<p><strong>Generating and storing a hash of the full URL</strong></p>\n<ul>\n<li>MD5 and Base62</li>\n<li>Hash collisions</li>\n<li>SQL or NoSQL</li>\n<li>Database schema</li>\n</ul>\n<p><strong>Translating a hashed URL to the full URL</strong></p>\n<ul>\n<li>Database lookup</li>\n<li>API and object-oriented design</li>\n</ul>\n<h3>Step 6: Scale the design</h3>\n<p>To ensure that your system can meet the scale requirements of the problem, we can use ideas like <a href=\"https://www.enjoyalgorithms.com/blog/load-balancers-in-system-design/\">load balancing</a>, horizontal scaling, <a href=\"https://www.enjoyalgorithms.com/blog/caching-system-design-concept/\">caching</a>, <a href=\"https://www.enjoyalgorithms.com/blog/data-partitioning-system-design-concept/\">database sharding</a>, <a href=\"https://www.enjoyalgorithms.com/blog/introduction-to-database-replication-system-design/\">replication</a>, etc. By considering these options and determining which are most appropriate for your system, you can ensure that your design can handle the expected workload and meet the needs of your users.</p>\n<h3>Step 7: Identifying and resolving bottlenecks</h3>\n<p>To ensure that system is robust and able to handle unexpected challenges, it is important to discuss as many potential bottlenecks as possible and consider different approaches to mitigate them.</p>\n<ul>\n<li>Is there any single point of failure? What steps can be taken to mitigate this?</li>\n<li>Are there enough replicas of data to ensure that users can still be served if a few servers are down?</li>\n<li>Are there enough copies of different services running to ensure that a few failures will not cause a total system shutdown?</li>\n<li>How is the performance of the service being monitored? Are there alerts in place to notify the team when critical components fail or their performance degrades?</li>\n</ul>\n<h3>System design questions to practice the steps</h3>\n<ul>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/design-key-value-store/\">Design Key-Value Store</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/design-typeahead-system/\">Typeahead System Design</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/design-api-rate-limiter/\">Designing API Rate Limiter</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/web-crawler/\">Designing Web Crawler</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/design-youtube-system-design-interview-question/\">Youtube System Design</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/design-google-docs/\">Designing Google Docs</a></li>\n</ul>\n<p>If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"1376b606-f7cf-5ca6-9389-fba3578b1a26","frontmatter":{"title":"Types of Load Balancing Algorithms","path":"/blog/types-of-load-balancing-algorithms"},"html":"<p>In distributed systems, a <a href=\"https://www.enjoyalgorithms.com/blog/load-balancers-in-system-design/\">load balancer</a> uses various load-balancing algorithms to efficiently distribute incoming traffic across multiple servers. So based on the load-balancing algorithms, load balancer chooses one of the servers based on factors like the nature of the workload, configuration of the servers, etc. The primary goal is to maximise throughput, minimize response time and avoid server overloading.</p>\n<p>At the high level, we can divide load balancing algorithms into two categories: static load balancing and dynamic load balancing. Let's understand each one of them.</p>\n<p><strong>Static load balancing algorithms:</strong> These algorithms rely on fixed configurations and work the same way regardless of the state of the backend server (server load, server health, etc). They are simpler to set up, but they can lead to an uneven distribution of requests. Examples of static load balancing algorithms are <strong>round robin, weighted round robin, source IP hash, URL hash, randomized algorithm</strong>, etc.</p>\n<p><strong>Dynamic load balancing algorithms:</strong> These algorithms take into account the state of the backend server (server load, server health or some other factors) when distributing requests. They require communication between the load balancers and servers, so they are slightly more complex but can distribute requests efficiently. Examples of dynamic load balancing algorithms are <strong>least connection method, weighted least connections method, least response time method</strong>, etc.</p>\n<p>Now let's discuss various static and dynamic load balancing algorithms one by one.</p>\n<h3>Round robin algorithm</h3>\n<p>Round robin is a simple load-balancing algorithm that directs client requests to different servers based on a rotating list. Load balancer maintains a list of available servers and directs incoming requests in a round-robin fashion: the first request to the first server, the second request to the second server, and so on. When the load balancer reaches the end of the list, it goes back to the beginning and starts from the first server again.</p>\n<p><img src=\"https://ucarecdn.com/6e2ee9b9-30d7-4d6f-9d94-080942cbee01/\" alt=\"Round robin load balancing explained with example\" title=\"Round robin algorithm of load balancing\"></p>\n<p>Benefits and drawbacks of the round-robin algorithm:</p>\n<ul>\n<li>Easy to implement.</li>\n<li>Balances traffic evenly between servers.</li>\n<li>Does not consider server load or specifications, so there is a risk that a server with low capacity will receive a large number of requests.</li>\n<li>Works best if every server in the load balancer list has roughly the same specifications. Otherwise, a low-processing server may have the same load as a high-processing server.</li>\n</ul>\n<h3>Weighted round robin algorithm</h3>\n<p>The weighted round-robin load-balancing algorithm is an advanced version of the simple round-robin. It distributes incoming requests based on the weighted score of the servers. The weight can be an integer that reflects the server's processing power or specifications.</p>\n<p>Benefits and drawbacks of the weighted round-robin algorithm:</p>\n<ul>\n<li>Slightly more complex than the simple round-robin.</li>\n<li>Based on the weight, some servers may receive a higher proportion of the overall request count.</li>\n<li>Works well with servers with different specifications.</li>\n<li>Does not consider the current load of each server.</li>\n</ul>\n<h3>Random load balancing algorithm</h3>\n<p>The randomized load-balancing algorithm randomly distributes requests to servers using a random number generator. When a load balancer receives a request, randomized algorithm evenly distributes it to the servers. Like the round-robin, this algorithm works well for a group of servers with similar configurations.</p>\n<ul>\n<li>Does not maintain specific knowledge about server capacities or states.</li>\n<li>Simple to implement.</li>\n<li>Due to random distribution, always a chance that some servers may receive more requests.</li>\n</ul>\n<h3>Source IP hash algorithm</h3>\n<p>The source IP hash load-balancing algorithm generates the hash value based on the source IP addresses and allocates the request to a specific server. In other words, when a request arrives, the load balancer directs the request to a specific server based on the calculated hash value.</p>\n<ul>\n<li>This mapping is deterministic i.e. requests from the same source IP address will always be directed to the same destination server. This can be useful for maintaining session state in certain applications.</li>\n<li>If the session is broken, the key can be regenerated, and the client request can be directed back to the same server that was originally handling it. In other words, this algorithm is useful when a dropped connection needs to be returned to the same server.</li>\n<li>If a server goes down or needs maintenance, clients associated with that server may experience disruption until the server is back online.</li>\n</ul>\n<h3>URL hash algorithm</h3>\n<p>Similar to the source IP hash, the URL hash load-balancing algorithm generates a hash value based on the URL present in the client request. So request will be forwarded to one of the servers based on the hash value.</p>\n<ul>\n<li>Here load balancer can cache the hashed value of the URL, so subsequent requests that use the same URL result in a cache hit and are forwarded to the same server.</li>\n<li>URL hash algorithm is useful when load-balanced servers serve mostly unique content per server. In other words, all requests related to one process (e.g., running code) will go to one server, and all requests related to another process (e.g., payments) will go to another server, and so on.</li>\n</ul>\n<h3>Least connection method</h3>\n<p>The least connection load-balancing algorithm considers the current load on a server. It does this by sending requests to the server with the least number of active connections.</p>\n<ul>\n<li>Useful when there are many persistent connections in the traffic that are unevenly distributed between the servers. If the servers are busy with long computations, the connections between client and server stay alive for a long period.</li>\n<li>Load balancer does additional calculations to determine the server with the least connections. So this can increase the load on the load balancer itself. Even this overhead can increase with the number of servers and the frequency of connection updates.</li>\n<li>This is stateful i.e. it relies on maintaining state information about each server's active connections.</li>\n</ul>\n<h3>Weighted least connections method</h3>\n<p>The weighted least connections load-balancing algorithm distributes load based on both the number of active connections to each server and the relative capacity of the server.</p>\n<ul>\n<li>Some servers can handle more connections than others.</li>\n<li>Servers are rated based on their processing capabilities.</li>\n</ul>\n<h3>Least response time method</h3>\n<p>The least response time load-balancing algorithm is a more advanced version of the least connection method. It forwards requests to the server with the fewest active connections and the least average response time seen so far. So it dynamically adapts to changes in server load and response times.</p>\n<ul>\n<li>Relies on the time taken by a server to respond to a health monitoring request.</li>\n<li>Server that responds the fastest receives the subsequent request.</li>\n<li>Considers the number of active connections on each server.</li>\n<li>May require additional calculations to determine the server with the least average response time.</li>\n</ul>\n<p><strong>Note:</strong> Our system might have multiple load balancers that use different server selection strategies. Enjoy learning, enjoy system design!</p>"},{"id":"065f9f75-7819-5ae8-a2a8-3f3aab144db8","frontmatter":{"title":"How to Choose Correct Database in System Design?","path":"/blog/how-to-choose-right-database"},"html":"<p>Databases are a critical component of the world’s most complex technology systems, and how they are used has a significant impact on their performance, scalability, and consistency. Because this is an essential topic with many moving components, hence in this article, we’ve outlined the most crucial database topics that you’ll need to know during a system design interview. If you want to do well in system design interviews for a technical position, you’ll need to know about databases and how to use them in a more extensive system.</p>\n<h3>What is Database?</h3>\n<p>A database is responsible for storing and retrieving data for an application at its most basic level. Because it is a program that handles access to physical data storage, a database is sometimes known as a Database Management System (DBMS). A database’s primary functions are storing, updating, and deleting data, returning data in response to a query, and administering the database.</p>\n<p>A database must be dependable, efficient, and accurate. It turns out that accomplishing all of these tasks at the same time is quite tricky. You must be familiar with several ideas to construct a database that is both efficient and successful. Let’s start by looking at the CAP theorem to understand the tradeoff at the very heart of database design.</p>\n<h4>CAP Theorem</h4>\n<p>According to the CAP Theorem, any distributed database can only satisfy two of the three requirements:</p>\n<ol>\n<li><strong>Consistency</strong>: Every node answers with the most recent version of the data, ensuring consistency.</li>\n<li><strong>Availability</strong>: any node can send a response.</li>\n<li><strong>Partition Tolerance</strong>: Even if connectivity between any node is disrupted, the system continues to function.</li>\n</ol>\n<p>Suppose a system provides Consistency and Partition Tolerance. In that case, it is referred to as a CP database, and if it provides Availability and Partition Tolerance, it is referred to as an AP database.</p>\n<h4>Scalability</h4>\n<p>As dataset sizes grow more significantly, scalability becomes increasingly critical in distributed clusters. Because of the functionality they give, different databases scale better or worse. Scaling can be divided into two categories:</p>\n<ol>\n<li>Vertical scaling is the process of increasing the amount of computing and memory available on a single computer.</li>\n<li>Horizontal scaling is the process of increasing the number of computers in a cluster.</li>\n</ol>\n<p>Vertical scaling is simple, but the overall memory capacity is substantially lower. On the other hand, horizontal scaling has far more overall computing and storage capacity and can be scaled dynamically without causing downtime. The main disadvantage is that relational databases, the most widely used database model, have difficulty scaling horizontally.</p>\n<h4>Transactions</h4>\n<p>A transaction is a “single unit of work” that consists of a sequence of database actions. A transaction’s operations are either successful or unsuccessful. In this sense, the concept of a transaction helps to maintain data integrity when a system component fails. The “ACID” attributes are a formalization of this:</p>\n<ol>\n<li>Atomicity — the transaction is an “atomic” unit if all operations succeed or fail simultaneously.</li>\n<li>Consistency — when a transaction is completed successfully, the database is legitimate, with no schema violations.</li>\n<li>Isolation — several transactions can run at the same time.</li>\n<li>Durability — when a transaction is “committed,” it is saved in memory.</li>\n</ol>\n<p>Till now, you have an understanding of some essential components related to database design. Let’s dive deep into exploring different types of databases.</p>\n<h3>Relational Databases</h3>\n<p>A data model is required for all databases to specify the data’s logical organization and principles. A relational database is essentially a database that uses the relational data model to organize data in tables with rows of data entries and columns of predefined data types. Foreign key columns that relate to the primary key columns of other tables represent table relationships.</p>\n<p>Above all, relational data models impose strong constraints on data values and relationships to ensure that they are always valid against the schema. To ensure schema conformance, ACID transactions are almost always used. Databases support special ordering data structures on indices to make access faster than scanning row by row. However, because each index must be updated in addition to the primary table, this performance boost on reads is offset by slower writes.</p>\n<p>Relational databases are the best choice when there are many-to-many relationships between entries, data must follow a predetermined schema, or data relationships must always be precise. On the other hand, they have the major drawback of being difficult to scale over distributed clusters. Database processes can be slowed down by slower network communication. Oracle, MySQL, and PostgreSQL are the most widely used relational database technologies.</p>\n<p>We’ll give you an overview of the many non-relational database solutions accessible in the next part so that you can pick the optimal database method for your system architecture.</p>\n<h3>Non-Relational databases</h3>\n<p>What model do you want to use to store your data based on its structure? This is the key question facing system designers today when choosing a database. Non-relational databases are designed to meet unique scalability, schema flexibility, or specialized query functionality. There are many different types of non-relational databases, as you’ll discover in this section. We’ll go through a few of the most crucial things to keep in mind when creating systems.</p>\n<p>Because they’re addressing unique use cases with variable priorities between availability and consistency, non-relational databases are classified as either AP or CP databases. The eventual consistency concept is used in AP non-relational databases to ensure that consistency occurs over time, even if it isn’t guaranteed suitable when a transaction completes.</p>\n<p>There are many different types of non-relational databases, as you’ll discover in this section. We’ll go through a few of the most crucial things to keep in mind when creating systems. Let’s get started!</p>\n<h4>Graph Database</h4>\n<p>Nodes and edges are used to model data in graph databases. Because they represent data with many relationships, they are the most similar non-relational databases to a relational data model. Because data isn’t stored in tables, the fundamental advantage of a graph database is that queries don’t require joins to follow relationship edges. As a result, queries that traverse numerous edges of a graph, such as social network analytics, are ideally suited to them. Neo4J and CosmosDB are the industry’s top graph databases.</p>\n<h4>Document Store</h4>\n<p>Simple JSON objects with a key identifier are typically stored in document stores. Documents collect data about a single topic that might be scattered across multiple tables in a relational database. A document store, for example, might archive medical documents for a particular patient such that only one document needs to be retrieved at a time and all essential information is available.</p>\n<p>MongoDB, Couchbase, Firebase, and DynamoDB are the industry’s top document store databases.</p>\n<h4>Column Database</h4>\n<p>The term “column databases” refers to a group of columns that are frequently retrieved together. A column-family database represents data in tables like a relational database but maintains column families in files rather than rows and does not enforce relational constraints. This model improves performance by minimizing the amount of data that has to be read for data with strong column-family access patterns. Columns can also be compressed to conserve space because they tend to store repetitive information, especially if the data is sparse.</p>\n<p>Cassandra, HBase is the industry’s top column-family database.</p>\n<h4>Time Series Database</h4>\n<p>A time-series database is optimized for data entries that need to be strictly organized by time. The key use case is storing real-time data streams from system monitors. Because time series databases need a lot of writing, they typically include services for sorting streams as they arrive to ensure that they are appended in the correct order. These datasets can readily be divided into periods. InfluxDB and Prometheus are the industry’s top time-series databases.</p>\n<h4>Key-Value Store</h4>\n<p>A key-value store is somewhat similar to a document store. Because the key-value store has no idea what’s in the value, it only supports read, overwrite, and delete actions. There are no schemas, joins, or indexes; it’s essentially a massive hash table. They are simple to scale due to their low overhead. Caching implementations benefit greatly from key-value storage. Redis and Memcached are the industry’s top key-value stores.</p>\n<h3>Some requirements oriented use cases</h3>\n<p>We may be looking for the following storage solutions based on our needs and how we want to utilize or access our data:</p>\n<p><strong>Caching</strong> — When developing a read-heavy system like Twitter or Facebook, we may end up catching a lot of data, even entire timelines, to achieve the low latency requirement. Redis or Memcached are two alternatives here.</p>\n<p><strong>File system storage</strong> — If we’re creating an asset delivery service and storing images or audio/video files, we’ll need to employ blob storage. Amazon S3 is a prevalent example.</p>\n<p>We’ve now seen various storage solutions and how to choose between different databases based on our needs and the type of data we need to store. Is that, however, sufficient? There are times when a single database is insufficient to meet our needs. For example, in the case of Flipkart, order data must adhere to ACID principles while also being infinitely expandable in the manner of a Columnar DB. In this situation, we’ll utilize a database combination like MySQL + Cassandra. All information about ongoing orders that must adhere to ACID properties will now be saved in a MySQL database. After they are completed, they will be moved to Cassandra, which can be used as a permanent store. As long as the ACID qualities are required, the data is stored in a relational database before being migrated to a columnar database to expand to the data. The issue has been resolved!</p>\n<h3>Summary</h3>\n<p>Database systems are not created equal, and each has its own set of benefits, drawbacks, strengths, and flaws. At a high level, the following items should be carefully evaluated and determined in the following order:</p>\n<ul>\n<li>Determine the type of data to be stored, queried, and updated (structured, unstructured, or semi-structured) and, if relevant, the best technique to model the data (e.g., schema vs. schemaless, relational, document, key-value, etc.)</li>\n<li>Determine if transactional and ACID assurances are required.</li>\n<li>Determine whether replication and horizontal partitioning are essential.</li>\n<li>Decide on any other database requirements and priorities, such as those mentioned earlier in this article.</li>\n<li>Security, scalability, performance, vendor establishment, and stability, and availability of necessary expertise are all critical considerations.</li>\n<li>Determine any cost limits and budgets that may exist.</li>\n</ul>\n<p>As new technologies emerge regularly, the objective of this essay is not to make specific recommendations. Everything discussed here and the most significant available options at the time will heavily influence the optimal solution. Also, software and data solutions can and frequently use multiple data storage systems, so think about if that applies to your specific application. If so, select the proper database system for each, as mentioned above.</p>\n<h3>Conclusion</h3>\n<p>When selecting a database, keep in mind the size of the data, the structure of the data, the links between the data, and how critical it is to enforce schemas and assure consistency. Remember that one database may not perform everything you need, especially in large systems; thus, it’s OK to use more than one. Relational databases become too expensive to maintain as scale becomes more critical; therefore, start looking at non-relational options for sections of the system that don’t require solid schemas or consistency guarantees.</p>\n<p><strong>Enjoy learning, Enjoy system design!</strong></p>"},{"id":"18bb856f-9863-58f2-a5f7-be7619b06701","frontmatter":{"title":"Implement Least Recently Used (LRU) Cache","path":"/blog/implement-least-recently-used-cache"},"html":"<p><strong>Difficulty:</strong> Hard, <strong>Asked-in:</strong> Amazon, Microsoft, Adobe, Google.</p>\n<p><strong>Key takeaway:</strong> an excellent algorithm to learn data structure design and problem-solving using hash tables and doubly-linked lists.</p>\n<h3>Understanding LRU cache problem</h3>\n<p>The Least Recently Used (LRU) cache is a popular caching strategy that discards the least recently used items first to make room for new elements when the cache is full. It organizes items in the order of their use, allowing us to easily identify items that have not been used for a long time. This strategy is useful for optimizing the use of limited cache space and improving the performance of caching systems.</p>\n<p>So our goal is to design a data structure that follows the constraints of a <strong>Least Recently Used (LRU) cache</strong>. We need to implement LRUCache class with the following operations:</p>\n<ul>\n<li><strong>LRUCache(int capacity):</strong> Initialize LRU cache with positive size capacity.</li>\n<li><strong>int get(int key):</strong> Return the value of key if key exists, otherwise, return -1.</li>\n<li><strong>void put(int key, int value):</strong> Update the value of key if key exists. Otherwise, add key-value pair to the cache. If number of keys exceeds the capacity of lru cache, evict the least recently used key.</li>\n</ul>\n<p>We want the following specifications from our LRU cache:</p>\n<ul>\n<li>Access time for any item should be O(1).</li>\n<li>Time required to get the least recently used element should be O(1).</li>\n<li>Time required to put any item should be O(1).</li>\n<li>The space required should be O(n).</li>\n</ul>\n<h3>Let's understand the problem via an example</h3>\n<p>Suppose we have five elements in the main memory, A1 to A5. And let the size of our cache be 3.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*1GVYMj7w8h-BzUQvOVp32w.png\" alt=\"LRU cache example: step 1\" title=\"Least Recently Used cache example: step 1\"></p>\n<p>Initially, the cache is empty, and all the elements are stored in memory. We want to get A1 first. We get the value of A1 from memory and store it in the cache.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*nnTlJm5xHLbD9l2r22P9qw.png\" alt=\"LRU cache example: step 2\" title=\"Least Recently Used cache example: step 2\"></p>\n<p>Next, we want to get A2. A2 gets stored at the topmost level, and A1 is moved down as it is no longer the most recently used element.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*fsd0CYU0sBHNSGZYgeOlUw.png\" alt=\"LRU cache example: step 3\" title=\"Least Recently Used cache example: step 3\"></p>\n<p>Next, we want to get A3.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*E0qLxbPrekavTeDUfl8gKA.png\" alt=\"LRU cache example: step 4\" title=\"Least Recently Used cache example: step 4\"></p>\n<p>Now suppose we want to get A2 again. Instead of getting this from memory, we can get this from our cache. Notice that the position of A2 is at the top again, as A2 is the most recently used element now.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*6oOl323d9hHfF9Sdeonzdw.png\" alt=\"LRU cache example: step 5\" title=\"Least Recently Used cache example: step 5\"></p>\n<p>Now we want to get A4. We have to get it from memory. But where will we store it in our cache? We have to remove some elements so that we can keep A4. So, we remove the least recently used element, A1, in this case.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/640/1*nFE99uv-6y4zeH8uv9Lf4g.png\" alt=\"LRU cache example: step 6\" title=\"Least Recently Used cache example: step 6\"></p>\n<p>Since our cache can store only three elements, we need to discard the least recently used element from our cache.</p>\n<p>Before designing the implementation of the LRU cache, we will look at the need for a cache. Generally, retrieving data from a computer’s memory is an expensive task. A high-speed memory known as cache memory is used to avoid accessing data from memory repeatedly. A cache holds frequently requested data and instructions to be immediately available to the CPU. Thus, cache memory reduces the average time for accessing data from the main memory.</p>\n<p>The cache memory size is generally much smaller than the main memory. So we cannot fit everything from the main memory into the cache. There are different ways of handling this; LRU cache is one such way. The main idea of the LRU cache is to store the <strong>n</strong> recently accessed elements (assume that the size of the cache is n). </p>\n<h3>Brute force approach: LRU cache using simple array</h3>\n<p>We initialize an array of sizes equal to that of our cache. Here each data element stores extra information to mark with an access time stamp. It shows the time at which the key is stored. We will use the timeStamp to find out the least recently used element in the LRU) cache.</p>\n<div class=\"gatsby-highlight\" data-language=\"java\"><pre class=\"language-java\"><code class=\"language-java\"><span class=\"token keyword\">class</span> <span class=\"token class-name\">DataElement</span> <span class=\"token punctuation\">{</span>\n    <span class=\"token keyword\">int</span> key<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">int</span> timeStamp<span class=\"token punctuation\">;</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token class-name\">DataElement</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> k<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> data<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> time<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        key <span class=\"token operator\">=</span> k<span class=\"token punctuation\">;</span>\n        value <span class=\"token operator\">=</span> data<span class=\"token punctuation\">;</span>\n        timeStamp <span class=\"token operator\">=</span> time<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n<span class=\"token punctuation\">}</span></code></pre></div>\n<p><strong>int get(int key):</strong> We need to traverse the array and search for the data element with the desired key. If the element is found, we set its timestamp to 0 and return its value. If the element is not found, we return -1. The time complexity for this operation is O(n), as we may need to traverse the entire array to find the desired element.</p>\n<p><strong>void put(int key, int value):</strong> If the array is not full, we insert the new data element by increasing the timestamp of the existing elements by 1, setting the timestamp of the new element to 0, and inserting it into the array. However, if the array is full and we need to make room for the new element, we must delete the least recently used element or the element with the largest timestamp. To do this, we iterate through the array and find the element with the largest timestamp. Then, we replace it with the new element and insert it into the array. The time complexity for this operation is O(n), as we may need to traverse the entire array to find the element with the largest timestamp.</p>\n<h3>Brute force approach: LRU cache <strong>using singly linked list</strong></h3>\n<p><strong>int get(int key):</strong> To retrieve a value from the LRU cache implemented with a linked list, we need to traverse the list and search for the data element with the desired key. If it is found, we move the element to the head of the list and return its value. If the element is not found, we return -1. The time complexity for this operation is O(n), as we may need to traverse the entire linked list to find the desired element.</p>\n<p><strong>void put(int key, int value):</strong> If the length of the linked list is less than the size of the cache, we simply insert the new data element at the head of the list. However, if the cache is full, we need to delete the least recently used element, which is located at the tail of the linked list. To do this, we must traverse the linked list until we reach the end, resulting in a worst-case time complexity of O(n).</p>\n<p>The critical question is: the time complexity of both get and put operations in both approaches is O(n), which is inefficient and does not satisfy the specification given in the problem. It's an exercise for you to write the implementation code for both approaches!</p>\n<h3>Efficient approach using hash map and doubly linked list</h3>\n<h4>Solution idea</h4>\n<p>To implement a Least Recently Used (LRU) cache that meets our requirements, we need to use a combination of a doubly-linked list and a hash map. The doubly-linked list allows us to store the elements in a specific order (with the least recently used element at the bottom) and move any element to the top in constant time. However, accessing an element in the doubly-linked list would take O(n) time. To address this, we use a hash map to map the elements to doubly-linked list nodes, allowing us to access an element in the doubly-linked list in O(1) time.</p>\n<p>When inserting new data into the cache, we insert it into the head of the linked list and record it in the map. After each cache hit, we move the accessed data (nodes) to the head of the linked list. When the linked list is full, we discard the tail of the linked list and delete the corresponding map key to make room for the new element.</p>\n<p><img src=\"https://ucarecdn.com/80434174-2689-4352-9445-2b92592954ca/\" alt=\"How LRU cache works?\" title=\"What is LRU in cache?\"></p>\n<h4>Solution steps</h4>\n<p>To access and update an element in the LRU cache implemented with a doubly-linked list and a hash map, we need to follow these steps:</p>\n<ol>\n<li>Look up the element in hash map.</li>\n<li>If the element is present in the hash map, it is stored in our cache and we do not need to retrieve it from memory (This is known as a cache hit). To mark the element as the most recently used, we retrieve its corresponding linked list node from the hash map and move it to the top of the list.</li>\n<li>If the element is not present in the hash map, it is not stored in our cache and we need to retrieve its value from memory. This is known as a cache miss.</li>\n<li>To make room for the new element, we may need to remove the least recently used element from the cache. This will be the last node of the linked list.</li>\n<li>We delete this element by removing the last node from the linked list and removing it from the hash map.</li>\n<li>Then, we create a new node for the element and insert it at the top of the cache (i.e. the head of the linked list).</li>\n<li>We also record the element in the hash map.</li>\n</ol>\n<p>If we follow all these steps, we can update our cache in O(1) time as all these steps take O(1) time. Following is the implementation code of the LRU-cache.</p>\n<h3>LRU Cache Java Implementation</h3>\n<div class=\"gatsby-highlight\" data-language=\"java\"><pre class=\"language-java\"><code class=\"language-java\"><span class=\"token keyword\">import</span> <span class=\"token import\"><span class=\"token namespace\">java<span class=\"token punctuation\">.</span>util<span class=\"token punctuation\">.</span></span><span class=\"token class-name\">Hashtable</span></span><span class=\"token punctuation\">;</span>\n\n<span class=\"token keyword\">public</span> <span class=\"token keyword\">class</span> <span class=\"token class-name\">LRUCache</span> <span class=\"token punctuation\">{</span>\n\n    <span class=\"token keyword\">class</span> <span class=\"token class-name\">Node</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">int</span> key<span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">;</span>\n        <span class=\"token class-name\">Node</span> pre<span class=\"token punctuation\">;</span>\n        <span class=\"token class-name\">Node</span> post<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">private</span> <span class=\"token class-name\">Hashtable</span><span class=\"token generics\"><span class=\"token punctuation\">&lt;</span><span class=\"token class-name\">Integer</span><span class=\"token punctuation\">,</span> <span class=\"token class-name\">Node</span><span class=\"token punctuation\">></span></span> cache <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token class-name\">Hashtable</span><span class=\"token generics\"><span class=\"token punctuation\">&lt;</span><span class=\"token class-name\">Integer</span><span class=\"token punctuation\">,</span> <span class=\"token class-name\">Node</span><span class=\"token punctuation\">></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">int</span> count<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">int</span> capacity<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> <span class=\"token class-name\">Node</span> head<span class=\"token punctuation\">,</span> tail<span class=\"token punctuation\">;</span>\n\n    <span class=\"token comment\">// Add the new node right after head</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">void</span> <span class=\"token function\">addNode</span><span class=\"token punctuation\">(</span><span class=\"token class-name\">Node</span> node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        node<span class=\"token punctuation\">.</span>pre <span class=\"token operator\">=</span> head<span class=\"token punctuation\">;</span>\n        node<span class=\"token punctuation\">.</span>post <span class=\"token operator\">=</span> head<span class=\"token punctuation\">.</span>post<span class=\"token punctuation\">;</span>\n\n        head<span class=\"token punctuation\">.</span>post<span class=\"token punctuation\">.</span>pre <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n        head<span class=\"token punctuation\">.</span>post <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token comment\">// Remove an existing node from the linked list</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">void</span> <span class=\"token function\">removeNode</span><span class=\"token punctuation\">(</span><span class=\"token class-name\">Node</span> node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token class-name\">Node</span> pre <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>pre<span class=\"token punctuation\">;</span>\n        <span class=\"token class-name\">Node</span> post <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>post<span class=\"token punctuation\">;</span>\n\n        pre<span class=\"token punctuation\">.</span>post <span class=\"token operator\">=</span> post<span class=\"token punctuation\">;</span>\n        post<span class=\"token punctuation\">.</span>pre <span class=\"token operator\">=</span> pre<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token comment\">// Move node in between to the head</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">void</span> <span class=\"token function\">moveToHead</span><span class=\"token punctuation\">(</span><span class=\"token class-name\">Node</span> node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token function\">removeNode</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token function\">addNode</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token comment\">// Pop the current tail</span>\n    <span class=\"token keyword\">private</span> <span class=\"token class-name\">Node</span> <span class=\"token function\">popTail</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token class-name\">Node</span> res <span class=\"token operator\">=</span> tail<span class=\"token punctuation\">.</span>pre<span class=\"token punctuation\">;</span>\n        <span class=\"token function\">removeNode</span><span class=\"token punctuation\">(</span>res<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">return</span> res<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token class-name\">LRUCache</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> capacity<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">this</span><span class=\"token punctuation\">.</span>count <span class=\"token operator\">=</span> <span class=\"token number\">0</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">this</span><span class=\"token punctuation\">.</span>capacity <span class=\"token operator\">=</span> capacity<span class=\"token punctuation\">;</span>\n\n        head <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token class-name\">Node</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        head<span class=\"token punctuation\">.</span>pre <span class=\"token operator\">=</span> <span class=\"token keyword\">null</span><span class=\"token punctuation\">;</span>\n\n        tail <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token class-name\">Node</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        tail<span class=\"token punctuation\">.</span>post <span class=\"token operator\">=</span> <span class=\"token keyword\">null</span><span class=\"token punctuation\">;</span>\n\n        head<span class=\"token punctuation\">.</span>post <span class=\"token operator\">=</span> tail<span class=\"token punctuation\">;</span>\n        tail<span class=\"token punctuation\">.</span>pre <span class=\"token operator\">=</span> head<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">get</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token class-name\">Node</span> node <span class=\"token operator\">=</span> cache<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>node <span class=\"token operator\">==</span> <span class=\"token keyword\">null</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> <span class=\"token operator\">-</span><span class=\"token number\">1</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        <span class=\"token function\">moveToHead</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">return</span> node<span class=\"token punctuation\">.</span>value<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">void</span> <span class=\"token function\">put</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token class-name\">Node</span> node <span class=\"token operator\">=</span> cache<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>node <span class=\"token operator\">==</span> <span class=\"token keyword\">null</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token class-name\">Node</span> newNode <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token class-name\">Node</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            newNode<span class=\"token punctuation\">.</span>key <span class=\"token operator\">=</span> key<span class=\"token punctuation\">;</span>\n            newNode<span class=\"token punctuation\">.</span>value <span class=\"token operator\">=</span> value<span class=\"token punctuation\">;</span>\n\n            cache<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> newNode<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            <span class=\"token function\">addNode</span><span class=\"token punctuation\">(</span>newNode<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\n            <span class=\"token operator\">++</span>count<span class=\"token punctuation\">;</span>\n\n            <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>count <span class=\"token operator\">></span> capacity<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n                <span class=\"token comment\">// Pop the tail</span>\n                <span class=\"token class-name\">Node</span> tailNode <span class=\"token operator\">=</span> <span class=\"token function\">popTail</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                cache<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>tailNode<span class=\"token punctuation\">.</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                <span class=\"token operator\">--</span>count<span class=\"token punctuation\">;</span>\n            <span class=\"token punctuation\">}</span>\n\n        <span class=\"token punctuation\">}</span> <span class=\"token keyword\">else</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token comment\">// Update the value.</span>\n            node<span class=\"token punctuation\">.</span>value <span class=\"token operator\">=</span> value<span class=\"token punctuation\">;</span>\n            <span class=\"token function\">moveToHead</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n<span class=\"token punctuation\">}</span></code></pre></div>\n<h3>LRU Cache Python Implementation</h3>\n<div class=\"gatsby-highlight\" data-language=\"python\"><pre class=\"language-python\"><code class=\"language-python\"><span class=\"token keyword\">class</span> <span class=\"token class-name\">Node</span><span class=\"token punctuation\">:</span>\n    <span class=\"token keyword\">def</span> <span class=\"token function\">__init__</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> k<span class=\"token punctuation\">,</span> v<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n        self<span class=\"token punctuation\">.</span>key <span class=\"token operator\">=</span> k\n        self<span class=\"token punctuation\">.</span>val <span class=\"token operator\">=</span> v\n        self<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> <span class=\"token boolean\">None</span>\n        self<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span> <span class=\"token operator\">=</span> <span class=\"token boolean\">None</span>\n        \n<span class=\"token keyword\">class</span> <span class=\"token class-name\">LRUCache</span><span class=\"token punctuation\">:</span>\n    <span class=\"token keyword\">def</span> <span class=\"token function\">__init__</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> capacity<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n        self<span class=\"token punctuation\">.</span>capacity <span class=\"token operator\">=</span> capacity\n        self<span class=\"token punctuation\">.</span>hashMap <span class=\"token operator\">=</span> <span class=\"token builtin\">dict</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>\n        self<span class=\"token punctuation\">.</span>head <span class=\"token operator\">=</span> Node<span class=\"token punctuation\">(</span><span class=\"token string\">\"#\"</span><span class=\"token punctuation\">,</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n        self<span class=\"token punctuation\">.</span>tail <span class=\"token operator\">=</span> Node<span class=\"token punctuation\">(</span><span class=\"token string\">\"-\"</span><span class=\"token punctuation\">,</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n        self<span class=\"token punctuation\">.</span>head<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span> <span class=\"token operator\">=</span> self<span class=\"token punctuation\">.</span>tail\n        self<span class=\"token punctuation\">.</span>tail<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> self<span class=\"token punctuation\">.</span>head\n\n    <span class=\"token keyword\">def</span> <span class=\"token function\">get</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n        <span class=\"token keyword\">if</span> key <span class=\"token keyword\">in</span> self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">:</span>\n            node <span class=\"token operator\">=</span> self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">[</span>key<span class=\"token punctuation\">]</span>\n            self<span class=\"token punctuation\">.</span>_remove<span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span>\n            self<span class=\"token punctuation\">.</span>_add<span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span>\n            <span class=\"token keyword\">return</span> node<span class=\"token punctuation\">.</span>val\n        <span class=\"token keyword\">return</span> <span class=\"token operator\">-</span><span class=\"token number\">1</span>\n\n    <span class=\"token keyword\">def</span> <span class=\"token function\">put</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n        <span class=\"token keyword\">if</span> key <span class=\"token keyword\">in</span> self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">:</span>\n            self<span class=\"token punctuation\">.</span>_remove<span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">[</span>key<span class=\"token punctuation\">]</span><span class=\"token punctuation\">)</span>\n        newNode <span class=\"token operator\">=</span> Node<span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span>\n        self<span class=\"token punctuation\">.</span>_add<span class=\"token punctuation\">(</span>newNode<span class=\"token punctuation\">)</span>\n        self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">[</span>key<span class=\"token punctuation\">]</span> <span class=\"token operator\">=</span> newNode\n        <span class=\"token keyword\">if</span> <span class=\"token builtin\">len</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">)</span> <span class=\"token operator\">></span> self<span class=\"token punctuation\">.</span>capacity<span class=\"token punctuation\">:</span>\n            nodeToRemove <span class=\"token operator\">=</span> self<span class=\"token punctuation\">.</span>tail<span class=\"token punctuation\">.</span>prev\n            self<span class=\"token punctuation\">.</span>_remove<span class=\"token punctuation\">(</span>nodeToRemove<span class=\"token punctuation\">)</span>\n            <span class=\"token keyword\">del</span> self<span class=\"token punctuation\">.</span>hashMap<span class=\"token punctuation\">[</span>nodeToRemove<span class=\"token punctuation\">.</span>key<span class=\"token punctuation\">]</span>\n\n    <span class=\"token keyword\">def</span> <span class=\"token function\">remove_node</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n        previousNode <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>prev\n        nextNode <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span>\n        previousNode<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span> <span class=\"token operator\">=</span> nextNode\n        nextNode<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> previousNode\n\n    <span class=\"token keyword\">def</span> <span class=\"token function\">add_node</span><span class=\"token punctuation\">(</span>self<span class=\"token punctuation\">,</span> node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">:</span>\n        nextNode <span class=\"token operator\">=</span> self<span class=\"token punctuation\">.</span>head<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span>\n        previousNode <span class=\"token operator\">=</span> self<span class=\"token punctuation\">.</span>head\n        previousNode<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span> <span class=\"token operator\">=</span> node\n        nextNode<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> node\n        node<span class=\"token punctuation\">.</span><span class=\"token builtin\">next</span> <span class=\"token operator\">=</span> nextNode\n        node<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> previousNode</code></pre></div>\n<h3>LRU Cache C++ Implementation</h3>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\"><span class=\"token macro property\"><span class=\"token directive-hash\">#</span><span class=\"token directive keyword\">include</span> <span class=\"token string\">&lt;unordered_map></span></span>\n<span class=\"token keyword\">class</span> <span class=\"token class-name\">LRUCache</span> <span class=\"token punctuation\">{</span>\n<span class=\"token keyword\">private</span><span class=\"token operator\">:</span>\n    <span class=\"token keyword\">struct</span> <span class=\"token class-name\">Node</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">int</span> key<span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">;</span>\n        Node<span class=\"token operator\">*</span> pre<span class=\"token punctuation\">;</span>\n        Node<span class=\"token operator\">*</span> post<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span><span class=\"token punctuation\">;</span>\n\n    unordered_map<span class=\"token operator\">&lt;</span><span class=\"token keyword\">int</span><span class=\"token punctuation\">,</span> Node<span class=\"token operator\">*</span><span class=\"token operator\">></span> cache<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">int</span> count<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">int</span> capacity<span class=\"token punctuation\">;</span>\n    Node<span class=\"token operator\">*</span> head<span class=\"token punctuation\">;</span>\n    Node<span class=\"token operator\">*</span> tail<span class=\"token punctuation\">;</span>\n\n    <span class=\"token comment\">// Add the new node right after head</span>\n    <span class=\"token keyword\">void</span> <span class=\"token function\">addNode</span><span class=\"token punctuation\">(</span>Node<span class=\"token operator\">*</span> node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        node<span class=\"token operator\">-></span>pre <span class=\"token operator\">=</span> head<span class=\"token punctuation\">;</span>\n        node<span class=\"token operator\">-></span>post <span class=\"token operator\">=</span> head<span class=\"token operator\">-></span>post<span class=\"token punctuation\">;</span>\n\n        head<span class=\"token operator\">-></span>post<span class=\"token operator\">-></span>pre <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n        head<span class=\"token operator\">-></span>post <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token comment\">// Remove an existing node from the linked list</span>\n    <span class=\"token keyword\">void</span> <span class=\"token function\">removeNode</span><span class=\"token punctuation\">(</span>Node<span class=\"token operator\">*</span> node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        Node<span class=\"token operator\">*</span> pre <span class=\"token operator\">=</span> node<span class=\"token operator\">-></span>pre<span class=\"token punctuation\">;</span>\n        Node<span class=\"token operator\">*</span> post <span class=\"token operator\">=</span> node<span class=\"token operator\">-></span>post<span class=\"token punctuation\">;</span>\n\n        pre<span class=\"token operator\">-></span>post <span class=\"token operator\">=</span> post<span class=\"token punctuation\">;</span>\n        post<span class=\"token operator\">-></span>pre <span class=\"token operator\">=</span> pre<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token comment\">// Move node in between to the head</span>\n    <span class=\"token keyword\">void</span> <span class=\"token function\">moveToHead</span><span class=\"token punctuation\">(</span>Node<span class=\"token operator\">*</span> node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token function\">removeNode</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token function\">addNode</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token comment\">// Pop the current tail</span>\n    Node<span class=\"token operator\">*</span> <span class=\"token function\">popTail</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        Node<span class=\"token operator\">*</span> res <span class=\"token operator\">=</span> tail<span class=\"token operator\">-></span>pre<span class=\"token punctuation\">;</span>\n        <span class=\"token function\">removeNode</span><span class=\"token punctuation\">(</span>res<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">return</span> res<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n<span class=\"token keyword\">public</span><span class=\"token operator\">:</span>\n    <span class=\"token function\">LRUCache</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> capacity<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">this</span><span class=\"token operator\">-></span>count <span class=\"token operator\">=</span> <span class=\"token number\">0</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">this</span><span class=\"token operator\">-></span>capacity <span class=\"token operator\">=</span> capacity<span class=\"token punctuation\">;</span>\n\n        head <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        head<span class=\"token operator\">-></span>pre <span class=\"token operator\">=</span> <span class=\"token keyword\">nullptr</span><span class=\"token punctuation\">;</span>\n\n        tail <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        tail<span class=\"token operator\">-></span>post <span class=\"token operator\">=</span> <span class=\"token keyword\">nullptr</span><span class=\"token punctuation\">;</span>\n\n        head<span class=\"token operator\">-></span>post <span class=\"token operator\">=</span> tail<span class=\"token punctuation\">;</span>\n        tail<span class=\"token operator\">-></span>pre <span class=\"token operator\">=</span> head<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">int</span> <span class=\"token function\">get</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>cache<span class=\"token punctuation\">.</span><span class=\"token function\">find</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> cache<span class=\"token punctuation\">.</span><span class=\"token function\">end</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> <span class=\"token operator\">-</span><span class=\"token number\">1</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        Node<span class=\"token operator\">*</span> node <span class=\"token operator\">=</span> cache<span class=\"token punctuation\">[</span>key<span class=\"token punctuation\">]</span><span class=\"token punctuation\">;</span>\n        <span class=\"token function\">moveToHead</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">return</span> node<span class=\"token operator\">-></span>value<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">void</span> <span class=\"token function\">put</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>cache<span class=\"token punctuation\">.</span><span class=\"token function\">find</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> cache<span class=\"token punctuation\">.</span><span class=\"token function\">end</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            Node<span class=\"token operator\">*</span> newNode <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            newNode<span class=\"token operator\">-></span>key <span class=\"token operator\">=</span> key<span class=\"token punctuation\">;</span>\n            newNode<span class=\"token operator\">-></span>value <span class=\"token operator\">=</span> value<span class=\"token punctuation\">;</span>\n\n            cache<span class=\"token punctuation\">[</span>key<span class=\"token punctuation\">]</span> <span class=\"token operator\">=</span> newNode<span class=\"token punctuation\">;</span>\n            <span class=\"token function\">addNode</span><span class=\"token punctuation\">(</span>newNode<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\n            <span class=\"token operator\">++</span>count<span class=\"token punctuation\">;</span>\n\n            <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>count <span class=\"token operator\">></span> capacity<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n                <span class=\"token comment\">// Pop the tail</span>\n                Node<span class=\"token operator\">*</span> tailNode <span class=\"token operator\">=</span> <span class=\"token function\">popTail</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                cache<span class=\"token punctuation\">.</span><span class=\"token function\">erase</span><span class=\"token punctuation\">(</span>tailNode<span class=\"token operator\">-></span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                <span class=\"token operator\">--</span>count<span class=\"token punctuation\">;</span>\n            <span class=\"token punctuation\">}</span>\n        <span class=\"token punctuation\">}</span> \n        <span class=\"token keyword\">else</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token comment\">// Update the value.</span>\n            Node<span class=\"token operator\">*</span> node <span class=\"token operator\">=</span> cache<span class=\"token punctuation\">[</span>key<span class=\"token punctuation\">]</span><span class=\"token punctuation\">;</span>\n            node<span class=\"token operator\">-></span>value <span class=\"token operator\">=</span> value<span class=\"token punctuation\">;</span>\n            <span class=\"token function\">moveToHead</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n<span class=\"token punctuation\">}</span><span class=\"token punctuation\">;</span></code></pre></div>\n<p>The Least Recently Used (LRU) cache has several advantages and disadvantages:</p>\n<p>Advantages:</p>\n<ul>\n<li>Fast access: The cache stores items in the order of most recently used to least recently used, allowing us to access both the least recently used element and the most recently used element in O(1) time.</li>\n<li>Fast updates: We can find any element in the cache in O(1) time, so updating the cache also takes O(1) time.</li>\n</ul>\n<p>Disadvantages:</p>\n<ul>\n<li>Space complexity: To implement the LRU cache, we need to use both a linked list and a hash map, each of which takes O(n) space.</li>\n</ul>\n<h3>Similar coding questions to practice</h3>\n<ul>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/least-frequently-used-cache/\">Design LFU Cache</a></li>\n<li><a href=\"https://www.enjoyalgorithms.com/blog/bloom-filter/\">Design Bloom Filter</a></li>\n<li><a href=\"http://localhost:8000/blog/min-stack-problem/\">Design Min Stack</a></li>\n<li>Design add and search words data structure</li>\n</ul>\n<p>Enjoy learning! Enjoy coding!</p>"},{"id":"fefe3e5c-e53f-59ae-b768-3d4ce1820f0c","frontmatter":{"title":"Introduction to Key-Value Database","path":"/blog/key-value-stores-in-system-design"},"html":"<p>A key-value database is a NoSQL database that stores data in the form of key-value pairs. These key-value pairs are similar to the way data is stored in a map or dictionary, where each key is associated with a single value. Here key serves as a primary key and values can be simple data types like strings or numbers, or complex data structures like JSON. So there are two important points to keep in mind:</p>\n<ul>\n<li>There is no connection between the values stored in a key-value database.</li>\n<li>Keys are unique to ensure that there is no ambiguity when searching for a specific value.</li>\n<li>Key-value databases provide faster read and write performance compared to relational databases, which may require complex aggregations to retrieve data.</li>\n<li>There are several implementations of key-value databases available. Each one of them has some unique features and capabilities. Here are some examples: Aerospike, Apache Cassandra, Amazon DynamoDB, Berkeley DB, Memcached, Riak, Redis, etc.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/911580ef-0a9c-4dbf-919b-df3d530e9a2a/\" alt=\"Key value database example\" title=\"Key value database\"></p>\n<h3>Key-value database operations</h3>\n<p>A key-value database helps users to store, access, and update data using simple commands like get, put, and delete. This simplicity makes key-value stores fast, easy to use, portable, and flexible. They are also partitionable and can scale horizontally i.e. they can handle large data and traffic without slowing down.</p>\n<p>Some common operations that can be performed on a key-value databases:</p>\n<ul>\n<li>get(key): Retrieve the value associated with a key.</li>\n<li>put(key, value): Insert a value into the database associated with a key.</li>\n<li>delete(key): Remove the value associated with a key.</li>\n<li>update(key, value): Update the value associated with a key.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/f2c47162-ffa2-444f-b0f0-5fb4f60547f2/\" alt=\"Key-value database operations\" title=\"Key operations on key-value store\"></p>\n<p>Some key-value databases also provide some of these operations:</p>\n<ul>\n<li>Retrieving a range of keys or all the keys in the database.</li>\n<li>Retrieving key-value pairs within a specified range of keys.</li>\n<li>Some operations can execute atomically without interference from concurrent operations.</li>\n<li>Performing multiple put, get, or delete operations together as a single atomic operation.</li>\n<li>Associating additional keys with specific values. These secondary keys can be used for efficient retrieval of data based on criteria other than the primary key.</li>\n<li>Setting an expiration time for a key-value pair. Once the TTL expires, the key-value pair is automatically deleted. This can be useful for implementing cache invalidation mechanisms.</li>\n</ul>\n<h3>Characteristics of key-value databases</h3>\n<ul>\n<li>Key-value databases use replication to maintain multiple copies of data across different nodes. This ensures that users have access to data even if some nodes become unavailable.</li>\n<li>They are designed to be partition-tolerant i.e. they can continue to operate even in the presence of network partitions or communication failures.</li>\n<li>Most key-value databases prioritize availability and partition tolerance over strong consistency. So after a write operation, it may take some time for the data to become consistent across all nodes.</li>\n<li>They can handle large data by distributing read and write across multiple servers.</li>\n<li>Key-value databases use in-memory storage and optimized data structures. So they are highly optimized for fast data retrieval based on the key, which can provide low latency and high throughput.</li>\n<li>Each key-value pair is independent and the structure of the values can vary across different records. So it does not require a predefined schema or data model.</li>\n<li>Key-value databases lack complex querying capabilities compared to relational databases. The idea is simple: They are designed for simple key-based lookups for retrieving values. But some key-value databases extend their query capability by providing features like secondary indexes or range queries.</li>\n</ul>\n<h3>Use cases of key-value database</h3>\n<ul>\n<li>We can use it to store session attributes in online applications. For example, it can manage the sessions of individual players in a multiplayer online game.</li>\n<li>We can use it as an effective cache mechanism for frequently accessed but rarely updated data.</li>\n<li>We can use it to provide product recommendations by storing personalized items for customers.</li>\n<li>Key-value databases offer fast in-memory access. This makes them useful in situations where an application needs to handle many small continuous reads and writes.</li>\n<li>We can easily store some basic data based on key-value pairs in key-value database. For example, we can use it to store URLs as keys and web pages as values.</li>\n<li>We can use it to store large web objects (videos, images) in a system like content delivery network.</li>\n</ul>\n<h3>Key-value store Vs Cache</h3>\n<p>As we have seen above, a key-value database can be used as a cache to store frequently accessed data to improve performance. But there are some differences between a cache and a key-value store:</p>\n<ul>\n<li>While a key-value database can be used to persist data, caches are used to increase read performance.</li>\n<li>Caches are not designed to enhance write performance, whereas key-value databases can be very effective for the write operations.</li>\n<li>While key-value databases are designed to be resilient to server failure, caches are typically stored in RAM and do not offer transactional guarantees if the server crashes.</li>\n</ul>\n<h3>Are key-value databases similar to tables in RDBMS?</h3>\n<p>Key-value databases and tables in RDBMS have some similarities, but they are different in terms of their data models and structures. Here are some of the key differences:</p>\n<ul>\n<li>In an RDBMS, data is organized into structured tables of rows and columns. It requires a predefined schema that specifies the structure of tables like column names, data types, relationships, etc. But key-value databases do not have a fixed schema. Here each key-value pair can have a different structure.</li>\n<li>RDBMS typically use SQL, which allows complex queries like joins, aggregations, and filtering across multiple tables. On the other side, key-value databases often provide a simpler interface with basic operations like get, put, and delete based on the key.</li>\n<li>RDBMS can establish relationships between tables using primary and foreign keys. But key-value databases do not have built-in mechanisms for managing relationships between data. So it is up to the application developer to handle any relationships.</li>\n<li>RDBMS provide ACID properties and ensure data consistency and integrity. On the other side, key-value databases often prioritize availability and partition tolerance over strong consistency.</li>\n</ul>\n<p>If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"25a26c12-79a6-54e1-a3d7-0b0cdb71f600","frontmatter":{"title":"Latency in System Design","path":"/blog/latency-in-system-design"},"html":"<p>Latency determines the speed at which data can be transferred from one end of a system to another. In this blog, we will focus on the concept of latency and how it affects the performance of a system. We will also discuss measures that can be taken to improve the latency.</p>\n<h3>What is Latency?</h3>\n<p>Latency is a measure of how quickly data can be transferred between a client and a server. It is directly related to performance of the system: Lower latency means better performance. </p>\n<p>Let's understand this from another perspective! Browser (client) sends a request to the server. Server process the request, retrieve the information and send response back to the client. So, latency is the time interval between start of a request from the client to delivering the result back to the client by the server i.e. the round trip time between client and server.</p>\n<p>Our ultimate goal would be to develop a latency-free system, but various bottlenecks prevent us in developing such an ideal system. The idea is simple: We can try to minimize it. Lower the system latency, the less time it takes to process our requests. </p>\n<p><img src=\"https://ucarecdn.com/64e791cb-c61d-4ffa-b512-2c586439a756/\" alt=\"What is network latency?\" title=\"What is latency in system design?\"></p>\n<h3>How does latency work?</h3>\n<p>Latency is nothing other than the estimated time the client has to wait after starting the request to receive the result. Let’s take an example and look into how it works. Suppose you interact with an e-commerce website, you liked something and added it to the cart. Now when you press the “Add to Cart” button, following events will happen:</p>\n<ul>\n<li>“Add to Cart” button is pressed, the clock for latency starts and browser initiate a request to the server.</li>\n<li>Server acknowledges the request and processes it.</li>\n<li>Server response to the request, the response reaches your browser, and product gets added to your Cart.</li>\n</ul>\n<p>You can start the stopwatch in the first step and stop the stopwatch in the last step. This difference in time would be the latency.</p>\n<h3>What causes Latency?</h3>\n<p>Latency depends on various parameters. One of the major factors contributing to latency is outbound calls. In the previous example of adding cart exercise, when you click the button on browser, the request goes to some server in the backend, which again calls multiple services internally for computation (parallel or sequentially) and then waits for a response. All this adds to the latency. </p>\n<p>Latency is mainly caused by following factors:</p>\n<p><strong>Transmission mediums:</strong> Transmission medium is the physical path between the start and endpoints. So system latency depends on the type of medium used to transmit requests. Transmission mediums like WAN and Fiber Optic Cables are widely used, but each medium has limitations.</p>\n<p><strong>Propagation:</strong> System latency highly depends on the distance between communicating nodes. Farther the nodes are located, more the latency.</p>\n<p><strong>Routers:</strong> Routers take some time to analyse the header information of a packet. So latency depends on how efficiently router processes the request. Router to router hop increases the system latency.</p>\n<p><strong>Storage delays:</strong> System latency also depends on the type of storage system used, as it may take some time to process and return data. So accessing stored data can increase latency of the system.</p>\n<h3>How to measure Latency?</h3>\n<p>There are many methods used to quantify latency.</p>\n<p><strong>Ping:</strong> Ping is the most common utility used to measure latency. It sends packets to an address and sees how fast response is coming. It measures how long it takes for the data to travel from source to destination and back to the source. A faster ping corresponds to a more responsive connection.</p>\n<p><strong>Traceroute:</strong> Traceroute is another utility used to test latency. It also uses packets to calculate the time taken for each hop when routed to the destination.</p>\n<p><strong>MTR:</strong> MTR is a combination of both ping and Traceroute. MTR gives a list of reports on how each hop in a network is required for a packet to travel from one end to the other. The report generally includes details such as percentage Loss, Average Latency, etc.</p>\n<h3>Latency optimisation</h3>\n<p>Latency restricts performance of the system, so it is necessary to optimize it. We can reduce latency by adopting following measures:</p>\n<p><strong>HTTP/2:</strong> We can reduce latency by the use of HTTP/2. It allows parallelized transfers and minimizes round trips from the sender to the receiver.</p>\n<p><strong>Less external HTTP requests:</strong> Latency increases because of third-party services. By reducing number of external HTTP requests, system latency gets optimized as third-party services affect speed of the application.</p>\n<p><strong>CDN:</strong> CDN stores resources in multiple locations and reduces the request and response travel time. So instead of going back to the origin server, request can be fetched using the cached resources closer to the clients.</p>\n<p><strong>Browser Caching:</strong> Browser caching can also help to reduce the latency by caching specific resources locally to decrease the number of requests made to the server.</p>\n<p><strong>Disk I/O:</strong> Instead of often writing to disk, we can use write-through caches or in-memory databases or combine writes where possible or use fast storage systems, such as SSDs.</p>\n<p>Latency can also be optimised by making smarter choices regarding storage layer, data modelling, outbound call fanout, etc. Here are some ways to optimize it at an application level:</p>\n<ul>\n<li>Inefficient algorithms are the most apparent sources of latency. It is necessary to avoid unnecessary loops or expensive operations.</li>\n<li>Use design patterns that avoid locking because multithreaded locks introduce latency.</li>\n<li>Use an asynchronous programming model for better usage of hardware resources because blocking operations can cause long wait times.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>Latency is a concept associated with the design of every system. One can never make a fully latency-free system, but one can easily optimize it. With modern hardware and heavy computationally efficient machines, latency is no longer a bottleneck of the system.</p>\n<p>Thanks to Chiranjeev and Suyash for their contribution in creating the first draft of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"8a6cd9e3-60eb-558e-be86-48073f768a0f","frontmatter":{"title":"What is Leader Election in Distributed Systems?","path":"/blog/leader-election-system-design"},"html":"<p>The goal of leader election is to give a specific entity (such as a process, host, thread, object, or person) special powers within a distributed system. These powers may include the ability to delegate tasks, the ability to modify data, or the responsibility for handling all system requests. Leader election can be a useful tool for improving efficiency, minimizing coordination, simplifying architectures, and reducing overhead, but it can also introduce additional failure modes and scaling challenges and make it more difficult to assess the validity of the system.</p>\n<p>To achieve this goal, a leader election algorithm selects a processor to coordinate the actions of a distributed system. The leader is typically chosen based on a criterion, such as selecting the processor with the highest identifier. The other processors enter a terminated state when the leader is chosen. In a leader election algorithm, the terminated states are divided into <strong>elected</strong> and <strong>non-elected</strong> states. Once a processor enters an elected or non-elected state, it remains in that state at all times.</p>\n<p>The safety and liveness conditions must be met for a leader election algorithm to be effective. The <strong>liveness condition</strong> states that every processor will eventually enter either an elected or non-elected state. The <strong>safety condition</strong> states that only one processor is allowed to enter the elected state and become the leader of the distributed network.</p>\n<h3>Use cases of leader election</h3>\n<p>There are three scenarios to consider when determining if a leader election is appropriate:</p>\n<ol>\n<li><strong>When each node is roughly the same and there is no clear contender for a permanently assigned leader:</strong> In this case, any node can be voted as the leader of the system, and there is no single point of failure.</li>\n<li><strong>When the cluster is performing a complex task that requires close collaboration:</strong> Coordination can involve dividing the work, assigning work to specific nodes, and combining the results from different nodes. For example, in a scientific computation to determine how a protein folds, a leader node may be needed to allocate each node to a specific part of the computation and then combine the results to obtain the full folded protein configuration.</li>\n<li><strong>When the system executes many distributed writes to data and requires good consistency:</strong> Consistency means that no matter which node processes a request, the user will always have the most recent version of the data. In this case, a leader ensures consistency by being the source of truth about the system's current state, and the leader election algorithm must maintain this properly. For example, a bank may require strong consistency to ensure that a user's bank account balance is accurate regardless of which server responds to their online banking request and that multiple transactions on the same bank account do not conflict with each other.</li>\n</ol>\n<h3>Advantages and disadvantages of leader election</h3>\n<p>Leader election is a common pattern in distributed systems because it has several benefits:</p>\n<ul>\n<li>It is easier for humans to think about systems with a single leader, as it consolidates all of the system's concurrency, reduces the risk of partial failures, and provides a single location for logs and metrics.</li>\n<li>A single leader can be more efficient, as it can simply inform other systems about changes rather than needing to form consensus on them.</li>\n<li>A single leader can easily provide consistency to clients by observing and controlling all changes to the system's state.</li>\n<li>A single leader can improve performance or reduce costs by providing a single consistent cache of data that can be used simultaneously.</li>\n<li>It may be simpler to write software for a single leader than for a quorum, as the single leader does not need to consider other systems that may be working in the same state simultaneously.</li>\n</ul>\n<p>There are also some significant drawbacks to leader election:</p>\n<ul>\n<li>A single leader creates a single point of failure, as the entire system may become unavailable if the system fails to detect or correct a faulty leader.</li>\n<li>A single leader limits the scale of the system in terms of data size and request rate, and a complete re-architecture may be required if the system needs to expand beyond a single leader.</li>\n<li>A leader represents a single point of trust, as a faulty leader can quickly spread problems throughout the system and have a large impact (referred to as a \"blast radius\").</li>\n<li>Leader-elected systems may be difficult to implement partial deployments, which are used in many Amazon software safety policies, such as one-box, A/B testing, blue/green deployment, and incremental deployment with automatic rollback.</li>\n</ul>\n<h3>Leader Election Algorithms</h3>\n<p>A leader election algorithm guides a cluster to jointly agree on one node to serve as a leader with as few back-and-forth interactions as possible. Generally, the algorithm assigns each node one of three states: Leader, Follower, or Candidate. The leader is also required to pass a \"health check\" or \"heartbeat\" regularly so that follower nodes can determine if the leader has become unavailable or failed, and a new leader can be elected.</p>\n<p>The type of leader election mechanism used depends on whether the cluster is synchronous or asynchronous. In a synchronous cluster, nodes are synchronized to the same clock and send messages in a predictable order. In an asynchronous cluster, messages are not reliably sent within a specific timeframe or in any particular order.</p>\n<p><strong>Asynchronous algorithms</strong> cannot guarantee both safety (ensuring that only one leader is elected) and liveness (ensuring that every node completes the election) because any number of nodes in an asynchronous cluster can lag indefinitely. In practice, implementations prioritize safety because it has more serious consequences for the service.</p>\n<p><strong>Synchronous algorithms</strong> are easier to understand and may be preferable because they can guarantee both safety and liveness. However, synchronizing a cluster requires imposing additional constraints on how it operates, which may not be possible or scalable in practice.</p>\n<h4>Bully Algorithm</h4>\n<p>The Bully Algorithm is a simple synchronous leader election technique that requires each node to have a unique numeric id and for all nodes in the cluster to know each other's ids. When a node starts up or the current leader fails the health check, the election process begins.</p>\n<p>There are two possible outcomes:</p>\n<ol>\n<li>If a node has the highest id, it declares itself the winner and sends this message to the other nodes.</li>\n<li>If a node has a lower id, it sends messages to all nodes with higher ids. If it does not receive a response, it assumes that these nodes have failed or are unavailable and declares itself the winner.</li>\n</ol>\n<h4>Paxos</h4>\n<p>Paxos is a general consensus protocol that can be used for asynchronous leader elections. The idea behind the Paxos algorithm is to reach consensus among the nodes in the network about which node should be the leader. In this algorithm, a node proposes itself as the leader and other nodes in the network vote for or against it. If majority of nodes vote for the node, it becomes the leader. If not, the node that failed to become the leader tries again. The process continues until a node receives a majority vote and becomes the leader.</p>\n<h4>RAFT</h4>\n<p>Raft is a popular alternative to Paxos because it is easier to understand, implement, and use. It is a non-blocking algorithm that involves each node in the Raft consensus keeping track of the current \"election term.\" When the leader election begins, each node increments its copy of the term number and listens for messages from other nodes. If the node does not receive any messages after a random interval, it becomes a candidate leader and asks other nodes for votes.</p>\n<p>If the candidate wins a majority of votes, it becomes the leader. If another candidate with a higher term number sends a message, the original candidate concedes. If the election is split or the timer runs out without a consensus, the algorithm restarts. Restarts are rare due to random timeouts, which prevent nodes from colliding.</p>\n<h4>Apache ZooKeeper (ZAB)</h4>\n<p>Apache Zookeeper is a centralized coordination service that is \"self-distributed and highly dependable.\" It is designed to help distributed systems handle coordination tasks. Apache Zookeeper philosophy is that coordination is difficult, so it is better to have a shared, open-source implementation with all the necessary components so that services do not have to reinvent everything from scratch. This can be especially useful in large distributed systems.</p>\n<p>Apache Zookeeper uses the ZAB (ZooKeeper Atomic Broadcast) protocol to manage leader election, replication order guarantees, and node recovery. ZAB is an asynchronous algorithm that ensures that writes are consistent and propagated to all nodes by \"broadcasting\" state changes from the leader to followers.</p>\n<h3>Conclusion</h3>\n<p>Leader election is a strong technique that may be employed in systems to help make them more fault-tolerant and easier to use. However, when we employ leader election, we pay close attention to the guarantees that each protocol gives and, more critically, does not give.</p>\n<h3>References</h3>\n<p><a href=\"https://aws.amazon.com/builders-library/leader-election-in-distributed-systems/\">https://aws.amazon.com/builders-library/leader-election-in-distributed-systems</a>/</p>\n<p>Thanks to Navtosh for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"7aae3c51-c9f6-525c-a609-62dd2f970db9","frontmatter":{"title":"Design and Implement Least Frequently Used (LFU) Cache","path":"/blog/least-frequently-used-cache"},"html":"<h3>What is LFU Cache?</h3>\n<p>Least Frequently Used is a cache algorithm used to manage memory within a computer. In this method, system keeps track of number of times a block is referenced in memory. The system removes the item with the lowest reference frequency when the cache is full.</p>\n<h3>Problem Statement of LFU Cache</h3>\n<p>Our goal is to design a data structure that follows the constraints of a Least Frequently Used(LFU) cache. Our LFU cache should support the following operations:</p>\n<ul>\n<li><strong>LFUCache(int capacity)</strong>: Initialises the object with the capacity of the data structure.</li>\n<li><strong>int get(int key)</strong>: Returns the value of the key if the key exists in the cache; otherwise, returns -1.</li>\n<li><strong>void put(int key, int value)</strong>: Update the value of the key if present or insert the key if not already present. When the cache reaches its capacity, it should invalidate and remove the least frequently used key before inserting a new item. If there is a tie, i.e. two or more keys with the same frequency, the least recently used key would be invalidated.</li>\n</ul>\n<p>The frequency for a key in the cache is incremented when either a get or put operation is called on it. The operations get and put must each run in O(1) average time complexity.</p>\n<h3>Brute force approach to implement LFU Cache</h3>\n<p>We initialize an array of sizes equal to that of our cache. Each element stores the key and value along with the frequency and the time at which this key is accessed. We use this frequency and the timestamp to determine the least frequently used element.</p>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\"><span class=\"token keyword\">class</span> <span class=\"token class-name\">Element</span> \n<span class=\"token punctuation\">{</span>\n    <span class=\"token keyword\">int</span> key<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">int</span> val<span class=\"token punctuation\">,</span> \n    <span class=\"token keyword\">int</span> frequency<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">int</span> timeStamp\n    \n    <span class=\"token keyword\">public</span> <span class=\"token function\">Element</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> k<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> v<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> time<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        key <span class=\"token operator\">=</span> k<span class=\"token punctuation\">;</span>\n        val <span class=\"token operator\">=</span> v<span class=\"token punctuation\">;</span>\n        frequency <span class=\"token operator\">=</span> <span class=\"token number\">1</span><span class=\"token punctuation\">;</span>\n        timeStamp <span class=\"token operator\">=</span> time<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n    \n<span class=\"token punctuation\">}</span></code></pre></div>\n<p><strong>int get(int key):</strong> We traverse the array and compare the key of each element in our cache with the given key. If we find the key of an element equal to the key, we increment the frequency of this element and update the timestamp. If we do not find any element, we return -1. Time complexity = O(n).</p>\n<p><strong>void put(int key, int value):</strong> If the array is not full, we create a new element with frequency 1 and timestamp 0. We increase the timestamp of the existing data in the array by 1. If the array is full, we must delete the least frequently used element. For this, we iterate through the array and find the element with the least frequency. In the case of frequency ties, we consider the least recently used element (oldest timestamp). Now we insert the new element.</p>\n<h3>An efficient approach to implement LFU Cache</h3>\n<p>First, we will implement the insert and access operations in O(1) time. We need two maps to achieve this, one to store the key-value pairs and the second one to store counts/frequency of access.</p>\n<p><img src=\"https://ucarecdn.com/0ffd98e0-a59a-4321-9d98-5d5c882582cc/\" alt=\"LFU cache example get and put operation\" title=\"Example of LFU cache\"></p>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\"><span class=\"token keyword\">public</span> <span class=\"token keyword\">class</span> <span class=\"token class-name\">LFUCache</span>\n<span class=\"token punctuation\">{</span>\n    <span class=\"token keyword\">private</span> Map<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> Integer<span class=\"token operator\">></span> valueMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">HashMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> Map<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> Integer<span class=\"token operator\">></span> frequencyMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">HashMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">final</span> <span class=\"token keyword\">int</span> size<span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">public</span> <span class=\"token function\">LFUCache</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> capacity<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        size <span class=\"token operator\">=</span> capacity<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n    \n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">get</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">containsKey</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token boolean\">false</span><span class=\"token punctuation\">)</span>\n            <span class=\"token keyword\">return</span> <span class=\"token operator\">-</span><span class=\"token number\">1</span><span class=\"token punctuation\">;</span>\n        frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n        <span class=\"token keyword\">return</span> valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        \n    <span class=\"token punctuation\">}</span>\n    \n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">void</span> <span class=\"token function\">put</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">containsKey</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token boolean\">false</span><span class=\"token punctuation\">)</span>\n        <span class=\"token punctuation\">{</span>\n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n        <span class=\"token punctuation\">}</span>\n        <span class=\"token keyword\">else</span>\n        <span class=\"token punctuation\">{</span>\n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n<span class=\"token punctuation\">}</span></code></pre></div>\n<p>Now in the above code, we need to implement the code for the eviction strategy. When the map size reaches the max capacity, we need to find the item with the lowest frequency count. In our current implementation, we have to iterate through all the values of the frequencyMap and find the lowest count and remove the corresponding key from both the maps. This takes O(n) time.</p>\n<p>Also, if multiple keys have the same frequency count, we cannot find the least recently used key with our current implementation as HashMap does not store the insertion order. To handle this, we add one more data structure, a sorted map with the key as the frequency and value as a list of elements with the same frequency.</p>\n<p><img src=\"https://ucarecdn.com/96922fb8-654c-4a03-aefb-245a0a322268/\" alt=\"LFU cache example using linked list\" title=\"Example of LFU cache using linked list\"></p>\n<p>Now, new items can be added to the end of the list with frequency 1. Since the map stores the frequencies in sorted order, we can find the list with the lowest frequency in O(1) time. Also, we can delete the list’s first item (of lowest frequency) since that will be the least recently used in O(1) time.</p>\n<h3>LFU cache implementation in java: Using a singly linked list</h3>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\"><span class=\"token keyword\">public</span> <span class=\"token keyword\">class</span> <span class=\"token class-name\">LFUCache</span>\n<span class=\"token punctuation\">{</span>\n    <span class=\"token keyword\">private</span> Map<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> Integer<span class=\"token operator\">></span> valueMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">HashMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> Map<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> Integer<span class=\"token operator\">></span> countMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">HashMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> TreeMap<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> List<span class=\"token operator\">&lt;</span>Integer<span class=\"token operator\">>></span> frequencyMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">TreeMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">final</span> <span class=\"token keyword\">int</span> size<span class=\"token punctuation\">;</span>\n    \n    <span class=\"token keyword\">public</span> <span class=\"token function\">LFUCache</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> capacity<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        size <span class=\"token operator\">=</span> capacity<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n    \n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">get</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">containsKey</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token boolean\">false</span> <span class=\"token operator\">||</span> size <span class=\"token operator\">==</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n            <span class=\"token keyword\">return</span> <span class=\"token operator\">-</span><span class=\"token number\">1</span><span class=\"token punctuation\">;</span>\n        <span class=\"token comment\">//remove element from current frquency list and move it to the next containsKey</span>\n        <span class=\"token comment\">// Not O(1)</span>\n        <span class=\"token keyword\">int</span> frequency <span class=\"token operator\">=</span> countMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>    \n        frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">new</span> <span class=\"token function\">Integer</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        \n        <span class=\"token comment\">//remove frquency from map if list is empty</span>\n        <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">size</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        \n        frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">computeIfAbsent</span><span class=\"token punctuation\">(</span>frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">,</span> k <span class=\"token operator\">-></span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">LinkedList</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">add</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        countMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">return</span> valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n    \n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">void</span> <span class=\"token function\">put</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">)</span>\n    <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">containsKey</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token boolean\">false</span> <span class=\"token operator\">&amp;&amp;</span> size <span class=\"token operator\">></span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n        <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">size</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> size<span class=\"token punctuation\">)</span>\n            <span class=\"token punctuation\">{</span>\n                <span class=\"token comment\">// remove first element from the list</span>\n                <span class=\"token comment\">//element with minimum frrquency</span>\n                lowestCount <span class=\"token operator\">=</span> frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">firstKey</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                <span class=\"token keyword\">int</span> keyToDelete <span class=\"token operator\">=</span> frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>lowestCount<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span><span class=\"token number\">0</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                \n                <span class=\"token comment\">//remove frquency from map if list is empty</span>\n                <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>lowestCount<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">size</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n                    frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>lowestCount<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                \n                valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>keyToDelete<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">;</span>\n                countMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>keyToDelete<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            <span class=\"token punctuation\">}</span>\n            \n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            countMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">computeIfAbsent</span><span class=\"token punctuation\">(</span><span class=\"token number\">1</span><span class=\"token punctuation\">,</span> k <span class=\"token operator\">-></span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">LinkedList</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">add</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            \n        <span class=\"token punctuation\">}</span>\n        <span class=\"token keyword\">else</span> <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>size <span class=\"token operator\">></span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n        <span class=\"token punctuation\">{</span>\n            <span class=\"token comment\">//update value</span>\n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            <span class=\"token comment\">// update count and frequency map</span>\n            <span class=\"token keyword\">int</span> frequency <span class=\"token operator\">=</span> countMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">new</span> <span class=\"token function\">Integer</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            \n            <span class=\"token comment\">//remove frquency from map if list is empty</span>\n            <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">size</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span>\n                frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            \n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">computeIfAbsent</span><span class=\"token punctuation\">(</span>frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">,</span> k <span class=\"token operator\">-></span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">LinkedList</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">add</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            countMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n<span class=\"token punctuation\">}</span></code></pre></div>\n<p>Thus, both insert and delete operations are O(1), i.e. constant-time operations.</p>\n<p>While solving the eviction problem, we have increased the access operation time to O(n). All of the item-keys sharing the same frequency are in a list. Now, if one of these items is accessed, we need to move it to the list of the following frequency. We will have to iterate through the list first to find the item, which in worst-case will take O(n) operations.</p>\n<p>We somehow need to access that item directly in the list without iteration to solve this problem. If we can do this, we can delete this item in O(1) time from the current frequency list and move it at the end of the list of the following frequency list in O(1) time.</p>\n<p>For this, we need a doubly linked list. We will create a node that stores an item’s key, value, and position in the list. We will convert the list to a doubly-linked list.</p>\n<p><img src=\"https://ucarecdn.com/f05d7a32-24e2-4c54-8abb-8edd23ea8b18/\" alt=\"LFU cache implementation using doubly linked list\" title=\"LFU cache visualisation\"></p>\n<h3>LFU Cache Implementation Using Doubly Linked List (Java)</h3>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\"><span class=\"token keyword\">public</span> <span class=\"token keyword\">class</span> <span class=\"token class-name\">LFUCache</span> <span class=\"token punctuation\">{</span>\n\n    <span class=\"token keyword\">private</span> Map<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> Node<span class=\"token operator\">></span> valueMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">HashMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> Map<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> Integer<span class=\"token operator\">></span> countMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">HashMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> TreeMap<span class=\"token operator\">&lt;</span>Integer<span class=\"token punctuation\">,</span> DoubleLinkedList<span class=\"token operator\">></span> frequencyMap <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token generic-function\"><span class=\"token function\">TreeMap</span><span class=\"token generic class-name\"><span class=\"token operator\">&lt;</span><span class=\"token operator\">></span></span></span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">final</span> <span class=\"token keyword\">int</span> size<span class=\"token punctuation\">;</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token function\">LFUCache</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> n<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        size <span class=\"token operator\">=</span> n<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">get</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span><span class=\"token operator\">!</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">containsKey</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">||</span> size <span class=\"token operator\">==</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> <span class=\"token operator\">-</span><span class=\"token number\">1</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n        \n        <span class=\"token comment\">// Remove element from current frequency list and move it to the next one in O(1) time</span>\n        Node nodeTodelete <span class=\"token operator\">=</span> valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n        Node node <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> nodeTodelete<span class=\"token punctuation\">.</span><span class=\"token function\">value</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>                    \n        <span class=\"token keyword\">int</span> frequency <span class=\"token operator\">=</span> countMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>nodeTodelete<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        <span class=\"token function\">removeIfListEmpty</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>  \n        valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        countMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        countMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n        frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">computeIfAbsent</span><span class=\"token punctuation\">(</span>frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">,</span> k <span class=\"token operator\">-></span> <span class=\"token keyword\">new</span> <span class=\"token function\">DoubleLinkedList</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">add</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>         \n        <span class=\"token keyword\">return</span> valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span>value<span class=\"token punctuation\">;</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">public</span> <span class=\"token keyword\">void</span> <span class=\"token function\">put</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span><span class=\"token operator\">!</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">containsKey</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span> <span class=\"token operator\">&amp;&amp;</span> size <span class=\"token operator\">></span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">{</span>\n\n            Node node <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\n            <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">size</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> size<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n              \t<span class=\"token comment\">// remove the first node(LRU) from the list the of smallest frequency(LFU)</span>\n                <span class=\"token keyword\">int</span> lowestCount <span class=\"token operator\">=</span> frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">firstKey</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                Node nodeTodelete <span class=\"token operator\">=</span> frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>lowestCount<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">head</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n                frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>lowestCount<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>nodeTodelete<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n\t\t\t\t<span class=\"token function\">removeIfListEmpty</span><span class=\"token punctuation\">(</span>lowestCount<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n              \n                <span class=\"token keyword\">int</span> keyToDelete <span class=\"token operator\">=</span> nodeTodelete<span class=\"token punctuation\">.</span><span class=\"token function\">key</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>  \n                valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>keyToDelete<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n                countMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>keyToDelete<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            <span class=\"token punctuation\">}</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">computeIfAbsent</span><span class=\"token punctuation\">(</span><span class=\"token number\">1</span><span class=\"token punctuation\">,</span> k <span class=\"token operator\">-></span> <span class=\"token keyword\">new</span> <span class=\"token function\">DoubleLinkedList</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">add</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            countMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            \n        <span class=\"token punctuation\">}</span>\n      \t<span class=\"token keyword\">else</span> <span class=\"token keyword\">if</span><span class=\"token punctuation\">(</span>size <span class=\"token operator\">></span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">{</span>\n            Node node <span class=\"token operator\">=</span> <span class=\"token keyword\">new</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> value<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>                    \n            Node nodeTodelete <span class=\"token operator\">=</span> valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n            <span class=\"token keyword\">int</span> frequency <span class=\"token operator\">=</span> countMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>nodeTodelete<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            <span class=\"token function\">removeIfListEmpty</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>  \n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            countMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            valueMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            countMap<span class=\"token punctuation\">.</span><span class=\"token function\">put</span><span class=\"token punctuation\">(</span>key<span class=\"token punctuation\">,</span> frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">computeIfAbsent</span><span class=\"token punctuation\">(</span>frequency <span class=\"token operator\">+</span> <span class=\"token number\">1</span><span class=\"token punctuation\">,</span> k <span class=\"token operator\">-></span> <span class=\"token keyword\">new</span> <span class=\"token function\">DoubleLinkedList</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">add</span><span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span> \n            \n        \n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n    \n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">void</span> <span class=\"token function\">removeIfListEmpty</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> frequency<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token comment\">// remove from map if list is empty</span>\n        <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">get</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">.</span><span class=\"token function\">len</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token operator\">==</span> <span class=\"token number\">0</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            frequencyMap<span class=\"token punctuation\">.</span><span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>frequency<span class=\"token punctuation\">)</span><span class=\"token punctuation\">;</span>  \n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">class</span> <span class=\"token class-name\">Node</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">private</span> <span class=\"token keyword\">int</span> key<span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">private</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">;</span>\n        Node next<span class=\"token punctuation\">;</span>\n        Node prev<span class=\"token punctuation\">;</span>\n\n        <span class=\"token keyword\">public</span> <span class=\"token function\">Node</span><span class=\"token punctuation\">(</span><span class=\"token keyword\">int</span> key<span class=\"token punctuation\">,</span> <span class=\"token keyword\">int</span> value<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">this</span><span class=\"token punctuation\">.</span>key <span class=\"token operator\">=</span> key<span class=\"token punctuation\">;</span>\n            <span class=\"token keyword\">this</span><span class=\"token punctuation\">.</span>value <span class=\"token operator\">=</span> value<span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">key</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> key<span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">value</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> value<span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n\n    <span class=\"token keyword\">private</span> <span class=\"token keyword\">class</span> <span class=\"token class-name\">DoubleLinkedList</span> <span class=\"token punctuation\">{</span>\n        <span class=\"token keyword\">private</span> <span class=\"token keyword\">int</span> n<span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">private</span> Node head<span class=\"token punctuation\">;</span>\n        <span class=\"token keyword\">private</span> Node tail<span class=\"token punctuation\">;</span>\n\n        <span class=\"token keyword\">public</span> <span class=\"token keyword\">void</span> <span class=\"token function\">add</span><span class=\"token punctuation\">(</span>Node node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>head <span class=\"token operator\">==</span> null<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n                head <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n            <span class=\"token punctuation\">}</span> <span class=\"token keyword\">else</span> <span class=\"token punctuation\">{</span>\n                tail<span class=\"token punctuation\">.</span>next <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n                node<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> tail<span class=\"token punctuation\">;</span>\n            <span class=\"token punctuation\">}</span>\n            tail <span class=\"token operator\">=</span> node<span class=\"token punctuation\">;</span>\n            n<span class=\"token operator\">++</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        <span class=\"token keyword\">public</span> <span class=\"token keyword\">void</span> <span class=\"token function\">remove</span><span class=\"token punctuation\">(</span>Node node<span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n\n            <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">.</span>next <span class=\"token operator\">==</span> null<span class=\"token punctuation\">)</span> tail <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>prev<span class=\"token punctuation\">;</span>\n            <span class=\"token keyword\">else</span> node<span class=\"token punctuation\">.</span>next<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>prev<span class=\"token punctuation\">;</span>\n\n            <span class=\"token keyword\">if</span> <span class=\"token punctuation\">(</span>node<span class=\"token punctuation\">.</span>prev <span class=\"token operator\">==</span> null<span class=\"token punctuation\">)</span> head <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>next<span class=\"token punctuation\">;</span>\n            <span class=\"token keyword\">else</span> node<span class=\"token punctuation\">.</span>prev<span class=\"token punctuation\">.</span>next <span class=\"token operator\">=</span> node<span class=\"token punctuation\">.</span>next<span class=\"token punctuation\">;</span>\n\n            n<span class=\"token operator\">--</span><span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        <span class=\"token keyword\">public</span> Node <span class=\"token function\">head</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> head<span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n\n        <span class=\"token keyword\">public</span> <span class=\"token keyword\">int</span> <span class=\"token function\">len</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span> <span class=\"token punctuation\">{</span>\n            <span class=\"token keyword\">return</span> n<span class=\"token punctuation\">;</span>\n        <span class=\"token punctuation\">}</span>\n    <span class=\"token punctuation\">}</span>\n    \n<span class=\"token punctuation\">}</span></code></pre></div>\n<h3>Real-World Application of LFU Cache</h3>\n<p>Consider a caching network proxy application for the HTTP protocol. This proxy typically sits between the internet and the user or a set of users. It ensures that all the users can access the internet and enables sharing of all the shareable resources for optimum network utilisation and improved responsiveness. Such a caching proxy should maximize the amount of data it can cache in the limited amount of storage or memory at its disposal. Typically, many static resources such as images, CSS style sheets, and javascript code can quickly be cached for a reasonably long time before newer versions replace them. These static resources are included in pretty much every page, so it is most beneficial to cache them since pretty much every request will require them. Furthermore, since a network proxy must serve thousands of requests per second, the overhead needed to do so should be kept to a minimum.</p>\n<p>It should evict only those resources that are not used very frequently to that effect. Hence, the frequently used resources should be kept at the expense of the not so frequently used ones since the former has proved helpful over a while. Thus, these caching proxies can employ the LFU cache replacement strategy to evict the least frequently used items in its cache when there is a lack of memory. <a href=\"https://www.enjoyalgorithms.com/blog/implement-least-recently-used-cache/\">LRU cache</a> might also be an appropriate strategy here, but it would fail when the request pattern is such that all requested items do not fit into the cache and the items are requested in a round-robin fashion. In the case of LRU, items will constantly enter and leave the cache, with no user request ever hitting the cache. Under the same conditions, however, the LFU algorithm will perform much better, with most cached items resulting in a cache hit.</p>\n<p><strong>Enjoy learning, Enjoy Algorithms!</strong></p>"},{"id":"33d1952f-f89b-51a1-b264-bfe61d0bad4b","frontmatter":{"title":"What is Long Polling in System Design?","path":"/blog/long-polling-in-system-design"},"html":"<p>When developing a web application that handles real-time data, it's important to consider how to efficiently deliver the data to the client. One option for this is long polling. In this blog, we'll explore how long polling works and the features it offers.</p>\n<h3>What is long polling?</h3>\n<p>Let’s first look at what polling is and how it extends to Long polling. Polling is a technique that allows servers to send information to clients. On another side, Long polling is a variation of traditional polling that enables the server to send data to the client whenever it becomes available. The client sends a request for information to the server, but the server may not immediately respond. Instead, it waits until the data is available and then sends a complete response to the client.</p>\n<p>Long Polling is an efficient way to send information to a client because it reduces the number of HTTP requests required to transmit the same amount of data. However, it can be a challenge for the server to manage unfulfilled client requests and handle new information that becomes available before the client has made a new request. </p>\n<p>One advantage of Long Polling is that it is part of the HTTP protocol, which makes it widely accepted and generates less bandwidth than short polling. Overall, Long Polling is a reliable way to keep clients updated with new information in real-time. Here is the basic process of an application that uses HTTP Long Polling:</p>\n<ol>\n<li>The client sends an HTTP request and waits for a response from the server.</li>\n<li>When an update is available, the server provides the client with a complete response.</li>\n<li>The client then sends a new long-poll request immediately or after a pause to allow for an appropriate latency period.</li>\n<li>A timeout is set for each long-poll request. If the connection is lost due to a timeout, the client must re-establish the connection regularly.</li>\n</ol>\n<p><img src=\"https://ucarecdn.com/879ceb85-40ee-42a7-a5a1-7353d3259a1f/\" alt=\"HTTP long polling\" title=\"What is HTTP long polling?\"></p>\n<h3>How does long polling work?</h3>\n<p>Long Polling reduces the number of requests that need to be made to the server. In traditional polling, each request requires establishing a new connection, parsing HTTP headers, issuing a new data query, and generating and delivering a response. The connection is then terminated and resources are cleaned up.</p>\n<p>Long Polling allows the server to keep a client's connection open for as long as possible, only responding when data becomes available or a timeout threshold is reached. This avoids the need to repeat the process for each client until new data becomes available, which helps to conserve resources.</p>\n<p>In Long Polling, most of the work is done on the server side. On the client side, only one request to the server needs to be managed. When the client receives a response, it can make a new request. The main difference between polling and Long Polling from the client's perspective is that a client performing polling may intentionally leave a small time window between each request to reduce server load. It may also handle timeouts differently than a server that does not support Long Polling.</p>\n<p>With Long Polling, the client can be configured to allow a longer timeout when waiting for a response. This is done to avoid problems with communication with the server, which may be identified by a timeout period.</p>\n<p>Apart from this, there isn't much else that a client needs to do that isn't already covered by basic polling. The server on the other hand, must manage the state of multiple unresolved connections. When multiple servers and load balancers are used, it can be necessary to create solutions to preserve session state. The server must also gracefully handle connection timeout issues, which are more common with Long Polling.</p>\n<h3>Considerations and challenges when using long-polling</h3>\n<p>Using HTTP long Polling to build real-time interactivity in an application comes with various considerations, both in terms of development and scaling.</p>\n<ul>\n<li>How will you manage the real-time backend as usage grows?</li>\n<li>Will long Polling automatically re-establish connections when mobile devices switch between networks or lose connections, or when the IP address changes?</li>\n<li>Can you manage the message queue and catch up on missed messages with long Polling?</li>\n<li>Does long Polling provide load balancing and failover support across multiple servers?</li>\n</ul>\n<p>Keep in mind that long Polling is simply an improvised version of the underlying request-response mechanism, which adds complexity to its implementation. As a result, these are important factors to consider when using long Polling in your application. </p>\n<p>You'll have to develop your communication management system when building a real-time application with HTTP long polling for server push. This means that you'll be responsible for updating, maintaining, and scaling your backend infrastructure.</p>\n<h4>Message ordering and delivery</h4>\n<p>Long Polling can pose challenges in terms of reliable message ordering. This is because it's possible for multiple HTTP requests from the same client simultaneously. For example, if a client has two browser tabs open consuming the same server resource or if the client implementation uses more than one connection at a time, there may be no guarantee that duplicate data will be written only once. </p>\n<p>Another issue is that a server may send a response, but network or browser issues may prevent the message from being successfully received. If a message receipt confirmation process is not implemented, a subsequent call to the server may result in missed messages.</p>\n<h4>Performance and scaling</h4>\n<p>One challenge of using Long Polling is that it can be difficult to scale due to its complexity. Maintaining the session state for a given client requires either sharing it among all servers behind a load balancer or routing subsequent client requests within the same session to the same server that processed the original request. This can result in uneven distribution of load and potentially overload a single server in a cluster. </p>\n<h3>Conclusion</h3>\n<p>HTTP Long Polling is a way for the server to send data to the client independently, which allows the server to push data to the client in real-time as it becomes available. However, it is most effective when the messages from the server are rare and not too frequent.</p>\n<p>Thanks Suyash Namdeo, for his contribution in creating the first version of this content. Please write in the message below if you want to share more insight. Enjoy learning. Enjoy algorithms!</p>"},{"id":"27783ec4-13c1-5ac7-9246-a9614fd9ec39","frontmatter":{"title":"Load Balancers in System Design","path":"/blog/load-balancers-in-system-design"},"html":"<p>Load balancing is a technique to distribute incoming traffic or workloads across multiple servers. For this, we use load balancers. A load balancer is a device or software component that acts as an intermediary between clients and a group of servers. Its goal is to evenly distribute workloads to prevent any server from becoming overloaded. So load balancer is one of the critical components in distributed systems to optimize resource usage, maximise performance and ensure high scalability and reliability.</p>\n<h3><strong>Why do we need a load balancer</strong>?</h3>\n<p>Suppose we have a single-server setup where several clients are sending requests. When the number of requests increases, two critical issues arise:</p>\n<ul>\n<li>Server overloading: There is a limit to the number of requests a single server can handle. If the number of requests exceeds this limit, the server may become overloaded and unable to function properly.</li>\n<li>Single point of failure: If the single server goes down for any reason, the entire application will become unavailable to users. This can result in a poor user experience and impact the system's reliability.</li>\n</ul>\n<p>We can solve the above problems in two ways:</p>\n<ul>\n<li>Vertical scaling: We can increase the capacity of the single server by adding more hardware resources. However, there are limits to how much we can increase the capabilities of a single machine.</li>\n<li>Horizontal scaling: To increase the system capacity on a large scale, we can add more servers to the pool. However, this will bring a new challenge: How to distribute requests evenly across these servers? The answer is: We should use load balancers!</li>\n</ul>\n<p>The load balancer will not only help us distribute requests across servers but also increase system capacity by adding more servers if the number of requests increases. On the other hand, the load balancer can continuously check the health of each server. If one of the servers goes offline, it will redirect the traffic to another available server. So load balancers will help us ensure that the service remains available, even in the case of the failure of one or multiple servers.</p>\n<p><img src=\"https://ucarecdn.com/2e87ec83-832b-46ca-ad33-de342a3165ee/\" alt=\"How load balancing works?\" title=\"What is load balancing?\"></p>\n<h3>How a system with a load balancer works?</h3>\n<p>Now there are some critical questions: How user requests are served when there is a load balancer in the system? What are the key steps? Let's understand the process starting from the DNS request.</p>\n<ol>\n<li>The user device sends a request to a DNS server. </li>\n<li>The DNS server maps the domain name in the request to an IP address.</li>\n<li>The DNS server responds to the device with the IP address of the requested domain. Here is one important thing: This IP address is the IP address of the load balancer! Due to this, web servers are not reachable directly by the end user and only the load-balancer layer is visible.</li>\n<li>Now user device establishes a connection with the load balancer using the provided IP address. This connection is initiated over the HTTP(S) protocol.</li>\n<li>The load balancer receives the incoming request from the user device. At this point, the load balancer acts as the entry point for the traffic. Now load balancer determines which web server should handle the request based on a load-balancing algorithm. This decision depends on factors like server availability, server load and the logic of the load-balancing algorithm.</li>\n<li>The load balancer forwards the incoming request to the selected web server through private IPs. Here private IP address is unreachable for the end device and reachable only between servers in the same network. They are used for communication between servers.</li>\n<li>The selected web server processes the request and generates a response. </li>\n<li>Now web server sends back the response to the load balancer.</li>\n<li>The load balancer receives the response from the web server and forwards it back to the user's device.</li>\n<li>Finally, the user device receives the response from the load balancer. They treat it as if it originated directly from the requested server.</li>\n</ol>\n<p>These steps may be different based on the load balancer configuration. Most of the time, load balancers also perform additional functions like SSL termination, session persistence, traffic monitoring, caching, etc.</p>\n<h3><strong>Where do we add a load balancer?</strong></h3>\n<p>Load balancers can be placed at different points in a system to distribute the workload.</p>\n<ul>\n<li><strong>Between clients and frontend web servers:</strong> This is often the first point of contact between the client and the system. So load balancer receives incoming requests from clients and distributes them across frontend web servers.</li>\n<li><strong>Between frontend web servers and backend application servers:</strong> In a system with multiple frontend web servers, a load balancer can be used to distribute incoming requests from the web servers to the backend application servers.</li>\n<li><strong>Between backend application servers and cache servers:</strong> Load balancers can be used to distribute requests from the application servers to cache servers, which store frequently accessed data in memory to reduce response times.</li>\n<li><strong>Between cache servers and database servers:</strong> In systems with multiple cache servers, a load balancer can be used to distribute requests from the cache servers to the database servers, which store the actual data. This helps to ensure that the database servers are not overwhelmed with requests.</li>\n</ul>\n<h3>Types of load balancers</h3>\n<p>There are two main types of load balancers: software load balancers and hardware load balancers. The main difference between them is the level of customization and scalability they offer.</p>\n<p><strong>Software load balancers</strong> are implemented as software applications installed on physical servers or virtual machines. They are configured to meet the specific needs and provide greater flexibility in terms of customization. Scaling with software load balancers is also easier because we can add more capacity by adding more servers.</p>\n<p><strong>Hardware load balancers</strong> are physical devices installed in a network or in data centres. They are generally less flexible and offer fewer options for customization. But they are faster and more reliable than software load balancers because they are dedicated hardware devices designed specifically for load balancing.</p>\n<p>Overall, the choice between a software or hardware load balancer depends on the specific requirements of a system. Software load balancers are more suitable for systems that require a high level of customization and scalability, while hardware load balancers are better for systems that require high performance and reliability.</p>\n<h4><strong>Pros and cons of software load balancers</strong></h4>\n<p><strong>Pros</strong></p>\n<ul>\n<li>Provide more options for customization and configuration.</li>\n<li>Scale horizontally to handle more traffic by adding more instances.</li>\n<li>Cheaper than hardware load balancers because they can be installed on commodity hardware. The best thing is: They do not require the purchase and maintenance of physical hardware.</li>\n<li>They can be deployed in the cloud for easy scaling and cost savings.</li>\n<li>Provide a simple interface or API to configure, monitor, and manage multiple load balancer instances from a single control point. This simplifies administration and reduces operational overhead.</li>\n</ul>\n<p><strong>Cons</strong></p>\n<ul>\n<li>There can be some delay when scaling beyond the initial capacity because the software needs to be configured and set up.</li>\n<li>They introduce an additional layer of processing in the networking stack. Due to this, there can be some performance overhead in terms of throughput, latency and concurrent connections. This performance can also be constrained by the processing capabilities of the host machine.</li>\n<li>Require regular maintenance and updates to ensure good performance and security.</li>\n</ul>\n<h4><strong>Examples of software load balancers</strong></h4>\n<ul>\n<li>HAProxy: A TCP load balancer.</li>\n<li>NGINX: An HTTP load balancer with SSL termination support.</li>\n<li>Varnish: A reverse proxy-based load balancer.</li>\n<li>Balance: Open-source TCP load balancer.</li>\n<li>LVS: Linux virtual server offering layer 4 load balancing.</li>\n</ul>\n<h4><strong>Pros and cons of hardware load balancers</strong></h4>\n<p><strong>Pros</strong></p>\n<ul>\n<li>Offer consistent performance because load-balancing logic runs on specialized hardware. They can handle large concurrent connections, and provide fast throughput and low latency.</li>\n<li>Built on well-optimized and tested hardware, with an underlying operating system that is optimized for performance and stability. This makes them less prone to failure compared to software load balancers.</li>\n<li>Increase security, as only authorized personnel can physically access the servers.</li>\n</ul>\n<p><strong>Cons</strong></p>\n<ul>\n<li>Require a higher upfront cost for purchase and maintenance.</li>\n<li>Struggle to scale beyond a certain number of requests because they are limited by the hardware.</li>\n<li>Require more human resources to configure and manage, compared to software load balancers.</li>\n</ul>\n<h4><strong>Examples of hardware load balancers</strong></h4>\n<ul>\n<li>F5 BIG-IP load balancer</li>\n<li>CISCO system catalyst</li>\n<li>Barracuda load balancer</li>\n<li>Coytepoint load balancer</li>\n<li>Citrix NetScaler</li>\n</ul>\n<h3>Load balancing algorithms</h3>\n<p>Load balancers use various load-balancing algorithms to distribute incoming network traffic across multiple servers. So, it is the responsibility of these algorithms to select a server from a pool of available servers to direct each incoming request.</p>\n<p>Here are some popular load-balancing algorithms. If you want to learn more about these algorithms, you can explore this blog: <a href=\"https://www.enjoyalgorithms.com/blog/types-of-load-balancing-algorithms/\">load balancing algorithms</a>.</p>\n<ul>\n<li><strong>Round Robin Method:</strong> Each request is sequentially distributed to the next server in a circular manner.</li>\n<li><strong>Least Connections Method</strong>: Directs new requests to the server with the fewest active connections.</li>\n<li><strong>Weighted Round Robin Method:</strong> Similar to the Round Robin but there is a weight associated with each server (weight represents the server capacity). So the servers with higher weights receive a larger proportion of the load.</li>\n<li><strong>Least Response Time Method:</strong> Considers the response times of the servers and directs the request to the server with the lowest response time.</li>\n<li><strong>IP Hash Method:</strong> Calculates a hash value based on the client's IP address to select the server from the pool. This will ensure that requests from the same client will be directed to the same server.</li>\n</ul>\n<p>Different algorithms have different properties. So the correct choice of load-balancing algorithm depends on the characteristics of the workload and the goals of the load-balancing strategy. On the other side, load balancers also provide various configuration options to choose the load-balancing algorithm.</p>\n<h3><strong>Advantages of load balancing</strong></h3>\n<ul>\n<li>Ensure that the application is always available and can scale as needed. Servers can be added or removed based on the number of requests.</li>\n<li>Prevent a single server from becoming overloaded with requests.</li>\n<li>Provide encryption, authentication, and other types of additional support.</li>\n<li>End users only need to know the address of the load balancer, rather than the addresses of every server in the cluster, providing a layer of abstraction.</li>\n<li>Minimize server response time and maximize throughput.</li>\n<li>Load balancers can perform health checks and monitor the servers' request-handling capability to ensure proper functioning. They can also be used to roll out software updates without taking the entire service down, by removing one server at a time.</li>\n</ul>\n<h3>Critical concepts to explore further</h3>\n<ul>\n<li>What is the difference between Load Balancer and Reverse Proxy?</li>\n<li>Different Categories of Load Balancing: 1) Layer 4 (L4) load balancer 2) Layer 7 (L7) load balancer 3) Global server load balancing (GSLB)</li>\n<li>Health check feature of the load balancer.</li>\n<li>DNS load balancing vs Hardware load balancing</li>\n<li>The application load balancer in designing several systems</li>\n<li>Cloud load balancing</li>\n</ul>\n<p>Thanks to Navtosh for his contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"2ddb3d88-2c72-50cb-a057-6ff33bc6e164","frontmatter":{"title":"Fundamentals of MapReduce","path":"/blog/map-reduce"},"html":"<h3>What is MapReduce?</h3>\n<p>MapReduce is a batch processing programming paradigm that enables massive scalability across a large number of servers in a Hadoop cluster. It was developed by Google in 2004 and is known for its ability to handle large amounts of data on commodity hardware.</p>\n<ul>\n<li>MapReduce processes large amounts of data in a parallel and distributed manner by breaking it down into smaller pieces that can be processed concurrently. The results of these individual processing tasks are then combined to produce the final result.</li>\n<li>So MapReduce consists of two main phases: the map phase and the reduce phase. In the map phase, the input data is split into smaller chunks and processed in parallel by different nodes in a cluster. The output of the map phase is a set of intermediate key/value pairs. In the reduce phase, these intermediate key/value pairs are processed and combined to produce the final output.</li>\n</ul>\n<h3>MapReduce Dataflow</h3>\n<p>By dividing the data processing workload into smaller pieces and distributing it across multiple nodes, MapReduce is able to handle large amounts of data efficiently and quickly. This approach is particularly useful for analyzing and processing data sets that are too large to be handled by a single machine.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*ac_pZvj0HgnMV6q9yFtopw.png\" alt=\"Phases of MapReduce dataflow\" title=\"MapReduce dataflow\"></p>\n<p>MapReduce consists of a series of steps that are performed to transform and analyze data. These steps include:</p>\n<ul>\n<li><strong>Input reader:</strong> This component reads the incoming data, splits it into data blocks, and associates each data block with a Map function. It reads files stored in Hadoop Distributed File System (HDFS) and generates corresponding key-value pairs.</li>\n<li><strong>Map function:</strong> This function takes a series of key/value pairs, processes each pair, and generates zero or more output key/value pairs. The input and output types may be different from each other.</li>\n<li><strong>Partition function:</strong> This function assigns the output of each Map function to the appropriate reducer for sharding purposes and returns the reducer's index.</li>\n<li><strong>Shuffling and sorting:</strong> The data produced by the Map function is shuffled between and within nodes, and then sorted using a comparison function.</li>\n<li><strong>Reduce function:</strong> For each unique key stored in sorted order, the framework calls the Reduce function. The values associated with the keys can be iterated by the Reduce function and used to generate the corresponding output.</li>\n<li><strong>Output writer:</strong> This component writes the output of the Reduce function to storage.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*3SOTouuw4FayEIntxGVRYw.png\" alt=\"Data flow in a Hadoop MapReduce\" title=\"How MapReduce works?\"></p>\n<p>The above figure shows the data flow in a Hadoop MapReduce job. In a Hadoop MapReduce job, the input data is partitioned into separate files stored in a directory in HDFS. Each of these files is processed by a separate map task, labeled M1, M2, and M3 in the figure. The MapReduce scheduler attempts to run each mapper on a machine that has a replica of the input file, which can be hundreds of megabytes in size. The mapper reads the input file, one record at a time, and produces key-value pairs as output.</p>\n<p>The number of map tasks is determined by the number of input file blocks, while the number of reduce tasks is decided by the job author. The MapReduce framework uses a hash of the key to ensure that all key-value pairs with the same key are sent to the same reducer. The key-value pairs must be sorted, and this is done in stages where each map task partitions its output based on the key's hash and writes it to a sorted file on the mapper's local disk.</p>\n<p>When a mapper has completed its job, the MapReduce scheduler informs the reducers that they can begin fetching the output files from the mapper. The reducers connect to the mappers, retrieve the files, and merge them while preserving the sort order. The reducer is called with a key and an iterator that progressively reads all records with the same key. The reducer processes these records and generates output records that are written to a file on the distributed filesystem.</p>\n<h3>How does MapReduce Work?</h3>\n<p>MapReduce operates by dividing a job into two tasks, which are managed by two types of entities: the <strong>Jobtracker</strong> and <strong>multiple Task Trackers</strong>. The Jobtracker acts as a master and resides on the Namenode, while the Task Trackers act as slaves and reside on the Datanode.</p>\n<p>A job is divided into multiple tasks, which are then run on multiple data nodes in a cluster. The Task Tracker is responsible for the execution of individual tasks and sends progress reports to the Jobtracker. It also periodically sends a heartbeat signal to the Jobtracker to update it on the current state of the system. If a task fails, the Jobtracker can reschedule it on a different Task Tracker.</p>\n<h4>Example</h4>\n<p>MapReduce can be used to count the number of words in a text file. It does this by reading the text file and counting the frequency of each word. The input and output for this process are both text files. Each mapper takes a line of the input file as input and breaks it into words. It produces a key/value pair for each word, with the word itself being the key. The reducer receives a list of all the key-value pairs for each word and sums the counts for each word. It then outputs a single key-value pair for each word, with the word itself as the key and the sum of its counts as the value.</p>\n<p>The code for the mapper is stored in <strong>mapper.py</strong></p>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\">mapper<span class=\"token punctuation\">.</span>py\n<span class=\"token keyword\">import</span> <span class=\"token module\">sys</span> \n<span class=\"token keyword\">for</span> line in sys<span class=\"token punctuation\">.</span><span class=\"token constant\">stdin</span><span class=\"token operator\">:</span> \n    line <span class=\"token operator\">=</span> line<span class=\"token punctuation\">.</span><span class=\"token function\">strip</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>    # remove <span class=\"token function\">whitespace</span><span class=\"token punctuation\">(</span>leading<span class=\"token operator\">/</span>trailing<span class=\"token punctuation\">)</span>\n    words <span class=\"token operator\">=</span> line<span class=\"token punctuation\">.</span><span class=\"token function\">split</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>   # split the line into words\n    <span class=\"token keyword\">for</span> word in words<span class=\"token operator\">:</span> \n        <span class=\"token function\">print</span> <span class=\"token punctuation\">(</span><span class=\"token char\">'%s\\t%s'</span> <span class=\"token operator\">%</span> <span class=\"token punctuation\">(</span>word<span class=\"token punctuation\">,</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span></code></pre></div>\n<p>The code for the reducer is stored in <strong>reducer.py</strong></p>\n<div class=\"gatsby-highlight\" data-language=\"cpp\"><pre class=\"language-cpp\"><code class=\"language-cpp\">reducer<span class=\"token punctuation\">.</span>py\n<span class=\"token keyword\">import</span> <span class=\"token module\">sys</span>\ncur_word <span class=\"token operator\">=</span> None\ncur_count <span class=\"token operator\">=</span> <span class=\"token number\">0</span>\nword <span class=\"token operator\">=</span> None\n<span class=\"token keyword\">for</span> line in sys<span class=\"token punctuation\">.</span><span class=\"token constant\">stdin</span><span class=\"token operator\">:</span>\n    line <span class=\"token operator\">=</span> line<span class=\"token punctuation\">.</span><span class=\"token function\">strip</span><span class=\"token punctuation\">(</span><span class=\"token punctuation\">)</span>       # remove <span class=\"token function\">whitespace</span><span class=\"token punctuation\">(</span>leading<span class=\"token operator\">/</span>trailing<span class=\"token punctuation\">)</span>\n    word<span class=\"token punctuation\">,</span> count <span class=\"token operator\">=</span> line<span class=\"token punctuation\">.</span><span class=\"token function\">split</span><span class=\"token punctuation\">(</span><span class=\"token char\">'\\t'</span><span class=\"token punctuation\">,</span> <span class=\"token number\">1</span><span class=\"token punctuation\">)</span>        # parse the mapper<span class=\"token punctuation\">.</span>py\n\n    <span class=\"token keyword\">if</span> cur_word <span class=\"token operator\">==</span> word<span class=\"token operator\">:</span>\n        cur_count <span class=\"token operator\">+=</span> count\n    <span class=\"token keyword\">else</span><span class=\"token operator\">:</span>\n        <span class=\"token keyword\">if</span> cur_word<span class=\"token operator\">:</span>\n            <span class=\"token function\">print</span> <span class=\"token punctuation\">(</span><span class=\"token char\">'%s\\t%s'</span> <span class=\"token operator\">%</span> <span class=\"token punctuation\">(</span>cur_word<span class=\"token punctuation\">,</span> cur_count<span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span>\n        cur_count <span class=\"token operator\">=</span> count\n        cur_word <span class=\"token operator\">=</span> word\n<span class=\"token keyword\">if</span> cur_word <span class=\"token operator\">==</span> word<span class=\"token operator\">:</span>\n    <span class=\"token function\">print</span> <span class=\"token punctuation\">(</span><span class=\"token char\">'%s\\t%s'</span> <span class=\"token operator\">%</span> <span class=\"token punctuation\">(</span>cur_word<span class=\"token punctuation\">,</span> cur_count<span class=\"token punctuation\">)</span><span class=\"token punctuation\">)</span></code></pre></div>\n<p>The above program can be run using <code class=\"language-text\">cat word_count.txt | python mapper.py | sort -k1,1 | python reducer.py</code></p>\n<h3>Use cases of MapReduce</h3>\n<p>MapReduce is a powerful tool that is used in a variety of applications, including distributed pattern-based searching, distributed sorting, and web link-graph reversal. It is also used in many different computing environments, such as multi-core systems, desktop grids, multi-cluster systems, volunteer computing environments, dynamic cloud environments, mobile environments, and high-performance computing environments.</p>\n<p>One notable use of MapReduce is in regenerating Google's index of the World Wide Web. It replaced the old programs that were used to update the index and run various analyses.</p>\n<h3>Conclusion</h3>\n<p>MapReduce is a powerful tool for processing large amounts of data in the big data paradigm. It allows businesses to efficiently process petabytes of data stored in HDFS, providing more accessible access to multiple data sources and data types. It also enables fast processing of massive amounts of data through parallel processing and minimal data movement.</p>\n<p>In addition, MapReduce programming offers several other benefits. It is easy to use and allows developers to write code in a variety of languages, including Java, C++, and Python. This makes it a popular choice for many businesses and organizations.</p>\n<p>Enjoy learning, Enjoy algorithms, Enjoy system design!</p>"},{"id":"659c0a52-e5c2-57be-89ea-f683f8c7210a","frontmatter":{"title":"Network Protocol Fundamentals","path":"/blog/network-protocols-concept"},"html":"<p>In computer networking, a <strong>network</strong> is a group of devices (computers, servers, etc.) connected through communication channels (wired or wireless) for resource sharing, data exchange and various types of communication. Here devices communicate with each other by following certain rules known as <strong>network protocols</strong>. These protocols are standardised methods for reliable and efficient data transmission.</p>\n<h3>Why network protocols?</h3>\n<ul>\n<li>Protocols define the format of data packets so that devices can transmit data correctly.</li>\n<li>Specify how devices can exchange messages.</li>\n<li>Provide a way to route data packets: how devices are identified, how addresses are assigned, how routing decisions are made to send data to the desired destinations, etc.</li>\n<li>Provide features to detect and correct errors during data transmission</li>\n<li>Implement flow control mechanisms to manage the rate of data transmission.</li>\n<li>Some protocols include security features to protect data integrity.</li>\n</ul>\n<h3>The OSI model of networking</h3>\n<p>The OSI (Open Systems Interconnection) model is a structured framework to understand the system of computer networking. It divides the whole system into seven different layers.</p>\n<ul>\n<li>Each layer has specific responsibilities.</li>\n<li>Each layer interacts with adjacent layers to facilitate communication.</li>\n</ul>\n<h4>Seven layers of the OSI model (from top to bottom)</h4>\n<ol>\n<li><strong>Application layer</strong> (Top layer): Facilitate communication between software applications or services running on different devices over a network. Examples: HTTP (Hypertext Transfer Protocol), SMTP (Simple Mail Transfer Protocol), and FTP (File Transfer Protocol).</li>\n<li><strong>Presentation layer:</strong> Responsible for data formatting, encryption, and compression. </li>\n<li><strong>Session layer:</strong> Establish, manage, and terminate sessions between applications.</li>\n<li><strong>Transport layer:</strong> Ensure reliable and efficient data transfer. It segments data received from the upper layers into smaller units (data packets) and provides error recovery, flow control, and congestion control. Examples: TCP (Transmission Control Protocol) and UDP (User Datagram Protocol).</li>\n<li><strong>Network layer:</strong> Determine the best path for data packets to reach their destination across different networks, using routing protocols and IP addressing.</li>\n<li><strong>Data link layer:</strong> Provide reliable and error-free transmission of data over the physical layer. It is responsible for establishing and terminating connections between network nodes, performing error detection and correction, and handling the flow of data frames between adjacent nodes.</li>\n<li><strong>Physical layer</strong> (Lowest layer): Deals with the physical transmission of data. It defines the electrical, mechanical, and physical aspects of the network like cables, connectors, and signalling methods.</li>\n</ol>\n<h3>Types of network protocols</h3>\n<p>Various types of network protocols serve different purposes and operate at different layers of the OSI model. So each protocol has its own set of rules, formats, and functions. Some commonly used network protocols:</p>\n<ul>\n<li>Internet Protocol (IP)</li>\n<li>Transmission Control Protocol (TCP)</li>\n<li>User Datagram Protocol (UDP)</li>\n<li>Hypertext Transfer Protocol (HTTP)</li>\n<li>Hypertext Transfer Protocol Secure (HTTPS)</li>\n<li>File Transfer Protocol (FTP)</li>\n<li>Simple Mail Transfer Protocol (SMTP)</li>\n<li>Domain Name System (DNS)</li>\n<li>Remote Procedure Call (RPC)</li>\n<li>Dynamic Host Configuration Protocol (DHCP)</li>\n</ul>\n<p>Let's move forward to discuss some of them in detail.</p>\n<h3>Internet Protocol (IP)</h3>\n<p>Internet Protocol (IP) is a network layer protocol, which helps data packets to reach their destination. When one machine wants to send data to another machine, it sends it in the form of IP packets. </p>\n<p>An IP packet is the fundamental data unit that is transmitted over the Internet or any other network that uses the IP. The IP packet has two key parts: IP header and data (payload). Here size of the packet includes the size of both the IP header and data section.</p>\n<ul>\n<li>The IP header section contains information about the packet like source and destination IP addresses, size of the packet, version of the IP, and other control information.</li>\n<li>Payload contains the actual data being transmitted i.e. web pages, images, or any other content.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*Ew_eNrZF0IIor3D5wVNtDA.png\" alt=\"Internet protocol packet visualization\" title=\"ip packet visualization\"></p>\n<p>When a device sends a packet, it first determines the IP address of the destination device. Packet is then sent to the first hop on the network i.e. a router. The router examines the destination IP address in the header and determines the next hop on the network to which the packet should be forwarded. This process continues until the packet reaches its final destination. Here routing algorithms are used to determine the best path for the packets. We will cover various routing algorithms in a separate blog.</p>\n<h4><strong>What is an IP address?</strong></h4>\n<ul>\n<li>IP addresses are like digital addresses that helps us to identify and locate devices on a network. </li>\n<li>They are unique and assigned to each device or domain that connects to the Internet.</li>\n<li>IP addresses are written as four numbers separated by periods, such as 192.168.1.1. These numbers represent the numerical value of the address. </li>\n<li>There are two main types of IP addresses: IPv4 and IPv6.</li>\n</ul>\n<h4><strong>IPv4 vs. IPv6</strong></h4>\n<ul>\n<li>IPv4 is the first version of IP. It uses a 32-bit address scheme, so it can accommodate a maximum of 4,294,967,296 unique IP addresses. Despite its limitations, IPv4 still carries a significant chunk of internet traffic today. IPv4 also provides benefits like encrypted data transmission and cost-effective data routing.</li>\n<li>With the increasing demand for IP addresses, IPv6 was developed to overcome the limitations of IPv4. It uses 128-bit address space, so it can accommodate a maximum of 3.4 x 10^38 unique IP addresses! IPv6 provides improved routing, packet processing and security features compared to IP4. However IPv6 is not compatible with IPv4, and upgrading can present a challenge.</li>\n</ul>\n<h3>Transmission Control Protocol (TCP)</h3>\n<p>For sending files like images or emails, a single IP packet may not be sufficient. So there is a need to divide data into multiple packets. Unfortunately, increase in packets increases the risk of packets not reaching destination or arriving in the wrong order. That's where transport layer protocol comes into the picture.</p>\n<p>TCP is a transport layer protocol built on top of the IP. It is used in conjunction with the IP to provide a complete suite of protocols for transmitting data.</p>\n<ul>\n<li>TCP breaks data into small packets and numbers them in sequence to ensure that they are received in the correct order at the receiver. When there is a loss of data packets, TCP can also retransmit it. So IP is responsible for delivering the packets, while TCP helps to put them back in the correct order.</li>\n<li>When a device wants to communicate with another device using TCP, it establishes a connection with the destination device through a sequenced acknowledgement (handshake process). Once the connection is established, both devices can communicate freely. So TCP is a connection-oriented protocol, where a connection must be established between devices before data transfer.</li>\n<li>TCP provides error-checking mechanisms to ensure that the data is received without any errors.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*95_hwrT21l9xpZwEI4nfbw.png\" alt=\"Transmission control protocols (TCP) packet visualization\" title=\"Transmission control protocols (TCP)\"></p>\n<p>Here is how TCP works:</p>\n<ol>\n<li>The sender and receiver establish a connection using a three-way handshake. The sender sends a SYN (synchronize) packet to the receiver, the receiver responds with a SYN-ACK (synchronise-acknowledgement) packet, and the sender sends an ACK (acknowledgement) packet to complete the connection.</li>\n<li>Now sender divides data into TCP segments, where each segment contains a sequence number so that data is reassembled correctly on the receiver. These segments contain other information like source port number, destination port number, checksums for error detection, etc.</li>\n<li>Now TCP segments are sent over the network to the destination IP address and port number. Here TCP uses flow control to ensure that the sender does not overwhelm the receiver with too much data at once.</li>\n<li>Now destination receives the TCP segments and reassembles them into the original data. If a segment is lost or corrupted, receiver sends a request for retransmission to the sender.</li>\n<li>Once data transmission is complete, sender and receiver initiate a connection termination using a four-way handshake. The sender sends a FIN (finish) packet to the receiver, and the receiver sends an ACK packet to acknowledge the FIN and then sends its own FIN packet. The sender responds with an ACK packet to acknowledge the receiver's FIN packet.</li>\n</ol>\n<h4>Use cases of TCP in real life applications</h4>\n<ul>\n<li>Email applications like Outlook or Gmail use the Simple Mail Transfer Protocol (SMTP) over TCP to send and receive email messages.</li>\n<li>When you transfer files using protocols like FTP or SFTP, these protocols use TCP.</li>\n<li>Remote desktop software uses TCP to transmit data and user inputs between the client and server.</li>\n</ul>\n<h3>User Datagram Protocol (UDP)</h3>\n<p>UDP is a connectionless transport layer protocol. Unlike TCP, UDP does not provide any reliability, flow control, or error recovery. As a result, UDP is useful in situations where reliability mechanisms of TCP are not necessary and faster transmission is desired. In other words, UDP is useful in applications where speed and low latency are more important. Here dropped packets can be tolerated without significant impact on the application performance.</p>\n<p>Here is how UDP works:</p>\n<ol>\n<li>Sender encapsulates the data into a UDP datagram, which includes source and destination ports, as well as the length of the datagram.</li>\n<li>UDP datagram is then sent over the network to the destination IP address and port number. UDP does not establish a connection before transmitting data, so the data is sent to the destination without any handshaking.</li>\n<li>Destination receives UDP datagram and processes the data within the datagram. Since UDP is an unreliable protocol, it does not provide any mechanism for detecting lost or corrupted packets. So the receiving application must implement its own error detection and recovery mechanisms if required.</li>\n<li>If the receiving application requires acknowledgement of receipt, it can send a response packet back to the sender, indicating that the data was received successfully.</li>\n</ol>\n<h4>Use cases of UDP in real-life applications</h4>\n<p>There are some situations where UDP is preferred over TCP.</p>\n<ul>\n<li>In applications like online gaming, video streaming (YouTube or Netflix) or video conferencing (Skype or Zoom), low latency is crucial for a good user experience. So in such cases, UDP can be a better choice.</li>\n<li>Low-power devices like IoT sensors have limited processing power and battery life, and the overhead of establishing a TCP connection can be too high. So in such cases, one can use UDP.</li>\n<li>DNS primarily uses UDP for communication between DNS servers and clients. But if a DNS query or response packet is too large to fit within a single UDP datagram, then DNS can use TCP to transmit the packet in multiple parts. In such cases, DNS uses TCP as a fallback protocol.</li>\n</ul>\n<h3>HyperText Transfer Protocol (HTTP)</h3>\n<p>Hypertext Transfer Protocol (HTTP) is an application layer protocol built on top of TCP and provides a higher level of abstraction that allow developers to focus on their applications rather than the details of TCP and IP packets. Some common HTTP methods are PUT, GET, POST, and DELETE.</p>\n<ul>\n<li>HTTP protocol operates on a request-response model i.e. one machine sends a request and the other machine sends back a response with the requested information. This is how text, images, and other files are shared all across the World Wide Web!</li>\n<li>HTTP is a stateless protocol i.e. the client and server only maintain awareness of each other during the active connection. Once the connection is terminated, both parties forget about each other's existence.</li>\n</ul>\n<p>HTTP has several benefits that make it a popular choice for transmitting information. It requires low memory and CPU usage. Additionally, it has the ability to report errors without closing connections, which can reduce network congestion. But HTTP is not equipped with encryption capabilities, so it may not be the most secure option for transmitting sensitive information.</p>\n<h4>Hypertext Transfer Protocol Secure (HTTPS)</h4>\n<p>Hypertext Transfer Protocol Secure (HTTPS) is a secured version of HTTP. Unlike HTTP, which transfers data in a plain text format, HTTPS transfers data in an encrypted format. It is often used when sensitive information like passwords and financial transactions is transmitted. But HTTPS may be slower than HTTP due to the overhead of the encryption and decryption process.</p>\n<h3>File Transfer Protocol (FTP)</h3>\n<p>File Transfer Protocol (FTP) is used for transferring files between hosts, both local and remote. It runs on top of TCP and creates two TCP connections: control connection and data connection. Control connection is used to transfer control information like passwords and commands to retrieve and store files, while data connection is used to transfer the actual file. Both connections run in parallel during the entire file transfer process.</p>\n<ul>\n<li>FTP has several advantages: ability to share large files at the same time, resume file sharing if it is interrupted, recover lost data and schedule file transfers. </li>\n<li>FTP has some disadvantages as well. It lacks security, as data, usernames, and passwords are transferred in plain text, which makes them vulnerable to malicious actors. FTP also lacks encryption capabilities, which makes it non-compliant with industry standards.</li>\n</ul>\n<h3>Remote Procedure Call (RPC)</h3>\n<p>Remote Procedure Call (RPC) allows a program on one device to request a service from a program on another device, without the need to understand details of the network. It is used for interprocess communication.</p>\n<ul>\n<li>RPC works on a client-server model, where requesting program is the client and service providing program is the server. It uses either TCP or UDP to carry messages between communicating programs.</li>\n<li>RPC can improve performance by omitting many protocol layers and minimize code rewriting or redevelopment efforts. However, RPC has not yet been proven to work effectively over wide-area networks and does not support other transport protocols besides TCP/IP. </li>\n</ul>\n<h3>Challenges with Network Protocols</h3>\n<ul>\n<li>Handling large amounts of data, traffic, and devices can affect protocol performance.</li>\n<li>Some network protocols designed for specialized applications can add significant overhead. Due to this, they can consume network resources and impact performance.</li>\n<li>Achieving fault tolerance and reliable data transfer in different network conditions can be challenging.</li>\n<li>Hackers may exploit protocol weaknesses to gain unauthorized access or disrupt network operations.</li>\n<li>Sometimes, different systems may implement protocols differently or support different versions of protocols. This can lead to compatibility issues.</li>\n<li>With advancements in technology, network protocols need to evolve.</li>\n</ul>\n<p>Thanks to Chiranjeev and Navtosh for their contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"0c47964d-5fd9-5c59-ac12-aa0df9d5cf76","frontmatter":{"title":"Design Notification Service","path":"/blog/notification-service"},"html":"<p>Notification services are widely used in various products today to alert users of important updates or changes. For example, you might receive a notification about a price change for a product you're interested in or a new job specification that matches your search criteria. With the widespread use of mobile applications, receiving push notifications on our mobile devices and traditional email notifications has become more convenient.</p>\n<p>In this blog, we will explore the design of a notification service, a critical component of many products and a popular topic in interviews. Let's delve into the process of building such a service.</p>\n<h3>Key Requirements</h3>\n<p>Before designing a notification service, it is important to clearly define the key requirements that it should meet. These requirements may vary depending on the specific use case, but some common ones include:</p>\n<ul>\n<li>Users should be able to \"watch\" or \"follow\" a product to receive notifications about it.</li>\n<li>The service should be able to send users email or push notifications when relevant events occur.</li>\n<li>The notification service should be able to operate seamlessly with third-party integrations.</li>\n</ul>\n<p>It is also worth considering whether the notification service should be integrated into an existing service or separated into its own set of components.</p>\n<h3>High Level Design</h3>\n<p>Having well-designed APIs is an essential part of the notification service. These APIs should allow us to create and delete notification entries as needed.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*wsJ29gEdTyhjeIEz1Rmt7Q.png\" alt=\"Notification Service high level design\" title=\"High level design of Notification System\"></p>\n<p>In order to determine when a user should be notified, the notification service needs to save criteria for each product or set of search parameters selected by the user. These criteria can be stored in a database. The following APIs would be important for managing and accessing these notification criteria:</p>\n<ul>\n<li><strong>create<em>notification(key, product</em>id, user_id)</strong>: This API would create a new notification event based on the given product and user.</li>\n<li><strong>delete<em>notification(key, notification</em>id)</strong>: This API would delete an existing notification event with the given id.</li>\n<li><strong>get<em>notifications(key, user</em>id)</strong>: This API would retrieve the notifications for a particular user.</li>\n</ul>\n<p>APIs for the notification service can be implemented using RESTful or gRPC architectures. Depending on the specific requirements of the service, one of these approaches may be more suitable. For example, if the service allows consumers to view a list of generated notifications, it may be expected to have more read operations. On the other hand, if the service frequently create and delete notifications, the write performance of the API may be more important.</p>\n<h3>Detailed Components Design</h3>\n<p>We may use a variety of ways to create our notification service. We have the option of using batch jobs or the API internal response logs. However, in this article, we’ll construct our service utilizing the batch jobs technique.</p>\n<p>A simple method for alerting would be to run a batch process regularly to see if any user changes are “watched.” Let’s suppose users want to be notified if the price changes; thus, the batch job has to call an internal API that returns the product’s pricing to verify the lowest price.</p>\n<p>This method will provide adequate coverage and will cover all of the database’s search criteria. However, as the number of notification items grows, so does the number of calls to the internal API. Scheduler, Data Pipeline, Message Queue, Batch Job, and Communications service is required for the batch jobs-based strategy.</p>\n<h4>Scheduler</h4>\n<p>This component may be used to perform our data pipeline, batch processing, and other services regularly, such as daily or weekly.</p>\n<p>The following factors influence the start time of the data pipeline and batch processing:</p>\n<ul>\n<li>Off-peak hours: When there is less traffic, we want to check the database for the Notification service.</li>\n<li>User’s time zone: The scheduler should be set to receive the notice when the user is awake. This will undoubtedly result in a higher probability of receiving the notice, especially if it is a push notification.</li>\n</ul>\n<h4>Data pipeline</h4>\n<p>The data pipeline is told when to start by the scheduler. It searches the database for alerts, organizes them according to criteria, and adds them to a queue. This component reads the Notification API Database. When reading from the Notification API Database, consider the following strategies:</p>\n<ul>\n<li>Full Database scan: The batch job will read all the rows in the Notification API database.</li>\n<li>Partial database scan grouped by regions: The batch job reads the entries based on their region.</li>\n</ul>\n<p>The data pipeline will publish the notification items to a message queue after reading them from the database.</p>\n<h4>Batch Jobs</h4>\n<p>Batch jobs are subscribed to the message queue’s notification entries subject. These jobs will pick up the messages from the notification queue. The batch job will use the Internal Pricing or Availability API to retrieve the most up-to-date pricing depending on the database’s notification criteria.</p>\n<ul>\n<li>Compare the most current API pricing to what was previously stored in the database.</li>\n<li>Decide whether or not to alert the user based on the price difference or whether or not the product is available. Sending a request to the Communications component to notify the user.</li>\n<li>We’ll update the notification record in the database when we’ve finished the request to the Communication component. At this time, the last update and Prices fields will need to be changed.</li>\n</ul>\n<h4>Notification Service Use Case</h4>\n<p>This part will discuss how notification systems and real-time feeds are implemented using message queues using the Facebook newsfeed notification system.</p>\n<p>Assume we’re creating a social network similar to Facebook. To add asynchronous behavior to our program, we’d utilize a message queue. A user will have a large number of followers and friends. One person can have a lot of friends and be friends with a lot of people. We must store a post created by a user in the database. As a result, we’ll need a User table and a Post table. Because a single user can create many posts, the user and posts database will have a one-to-many connection. We must display the user’s post on his friends’ home page while saving the post in the database. This new post necessitates sending a notification to our friends.</p>\n<p>Now the question arises, how to send notifications? Let’s find out :)</p>\n<p>We can utilize a notification system that is based on push notifications. When data becomes available, the available server will deliver it to the client. As a result, instead of polling the database regularly, we’ll use a message queue to transmit data to the clients.</p>\n<p>When a user makes a new post in this method, the system does two distributed activities. One action is to update the database to save the post, which is necessary; otherwise, the data may be lost. Another option is to submit the post’s content to a message queue.</p>\n<p>As a result, after the data has been received, the message queue will asynchronously push the post to the user’s connections. As a result, there’s no need to query the database regularly to see if a buddy has made a post. In this approach, we may avoid polling procedures. We can utilize message queue storage with an expiry period to wait for offline users’ connections to come online in the event of offline users. Then send the changes to those individuals.</p>\n<p>As a result, combining a push-based technique with message queue implementation may improve application performance while lowering resource consumption.</p>\n<p>The database storage procedure must be handled with caution. We can’t push the message to the message queue if it fails since it would generate a system discrepancy. Before sending the message to the message queue, we need to ensure that the database persistence was successful. Otherwise, we’ll have to inform the user that the post could not be created due to server issues.</p>\n<h3>Conclusion</h3>\n<p>In this blog, we tried to explain the internal working and design of the notification service. They are used in almost every product, and as a result, it is essential to know about them from the perspective of any system design interview. I hope you liked this blog. Please do share your views.</p>\n<p>Thanks to Suyash for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"0be5eae1-100b-536b-a8fc-c9c3794851ab","frontmatter":{"title":"Peer-to-Peer (P2P) Architecture","path":"/blog/peer-to-peer-networks"},"html":"<h3>W﻿hat is Peer-to-Peer Architecture?</h3>\n<p>Peer-to-peer (P2P) architecture is a distributed system in which each node acts as both a client and a server so that nodes can share the workload or tasks among themselves.</p>\n<ul>\n<li>Each node (or peer) has a similar functional capability and holds some parts of the system resources.</li>\n<li>Some peers may have more resources and may be able to contribute more to the network, while others may not contribute any resources and only consume from the network.</li>\n<li>Peers are visible to each other and can connect with as many other peers as they wish. They are free to join or leave the network at any time and can transmit and receive data from other peers simultaneously.</li>\n<li>Tasks are distributed among all the peers rather than being handled by a single server. This makes P2P architecture more resilient because there is no single point of failure. In some cases, one can use a central tracking server to manage the network and help the peers find each other, but it should not have control over the network itself.</li>\n<li>P2P systems are often used in applications where decentralised control is desired like file sharing or distributed computing. It provide high performance and scalability due to the decentralised nature and the ability to share resources among multiple nodes.</li>\n</ul>\n<p><img src=\"https://ucarecdn.com/2a4b0f58-1087-4010-b972-f011928b49cc/\" alt=\"Client-server vs peer-to-peer (P2P) architecture\" title=\"Peer-to-peer network\"></p>\n<h3>How does P2P network work?</h3>\n<ul>\n<li>Peer-to-peer networks do not have a dedicated server for user authentication. So each computer is responsible for its protection and may require a separate user account for each machine that a user wants to access. Users save files on their machines and are responsible for backing them properly.</li>\n<li>P2P architecture is most effective when there are many active peers in the network. If there are more active peers, new peers can easily find other peers to connect to and ensure that there are enough remaining peers to take up the slack if some peers leave the network. When there are only a few active peers, there are fewer resources available overall.</li>\n<li>In a P2P file-sharing, the more common a file is, the faster it can be downloaded because many peers are sharing it. To make P2P more efficient, one can divide the workload into small pieces that can be reassembled later. This will help many peers to work on the same task simultaneously and reduce the amount of work that each peer has to do.</li>\n<li>Security is a concern in P2P networks, as access to data and resources is managed on each machine. Each computer can allow or deny access to other computers, so users must be careful about what they share. For example, a user might share a folder containing sensitive payroll information. Because there is no central administrator, users must manage access to their files and resources. So P2P networks are often used in small deployments or situations where security is not a concern.</li>\n</ul>\n<h3><strong>Applications of P2P architecture</strong></h3>\n<ul>\n<li>Used for file sharing: Users can share files directly with each other without the need for a central server.</li>\n<li>Used for instant messaging or real-time communication between users.</li>\n<li>Used for voice communication, such as through VoIP (Voice over Internet Protocol) technology.</li>\n</ul>\n<h3><strong>Use cases of P2P architecture</strong></h3>\n<ul>\n<li><strong>Napster:</strong> Napster was a file-sharing service that was shut down in 2001 due to copyright issues. It used a central tracking server to facilitate file sharing among users.</li>\n<li><strong>BitTorrent:</strong> BitTorrent is a P2P file-sharing protocol that allows users to share large files directly with each other without the need for a central server.</li>\n<li><strong>Skype:</strong> Skype originally used a hybrid P2P protocol for voice and video communication, but after being acquired by Microsoft, it now uses a client-server model.</li>\n<li><strong>Bitcoin:</strong> Bitcoin is a P2P cryptocurrency that operates without a central monetary authority. Transactions are verified and recorded on a decentralised network of computers, rather than being centrally managed.</li>\n</ul>\n<h3>Types of peer-to-peer network</h3>\n<p>There are three different types of P2P networks.</p>\n<ul>\n<li>Unstructured P2P networks</li>\n<li>Structured P2P networks</li>\n<li>Hybrid P2P networks</li>\n</ul>\n<h4>Unstructured (or Decentralised) P2P networks</h4>\n<p>In unstructured peer-to-peer networks, nodes are not arranged in any particular order i.e. node-to-node communication is random. Such type of P2P networks is well-suited for high-activity use cases like social platforms where users may regularly join or leave the network. </p>\n<ul>\n<li>No central control and nodes communicate directly.</li>\n<li>Resilience with no single point of failure.</li>\n<li>One of the use cases is file-sharing applications.</li>\n</ul>\n<p>However unstructured P2P networks have a disadvantage. They require a significant amount of CPU and memory to function properly. The hardware must be able to support the maximum number of network transactions so that all nodes can communicate with each other at any given time. This can be a challenge, especially if the network is large or has a high volume of activity.</p>\n<h4>Structured (or Centralised) P2P networks</h4>\n<p>Structured peer-to-peer (P2P) networks are the opposite of unstructured P2P networks i.e. nodes have a way to interact with each other in a more organised manner. So one can achieve this through a well-organized architecture that allows users to find and use files more efficiently rather than searching randomly.</p>\n<p>While structured P2P networks are more effective, they have some centralization due to their organised architecture. This means that they may be more expensive to maintain and set up than unstructured P2P networks. However, structured P2P networks are more stable than unstructured P2P networks.</p>\n<ul>\n<li>Central server manages communication between nodes.</li>\n<li>Greater control and security.</li>\n<li>Efficient management of tasks.</li>\n<li>Can be used in online gaming, chat rooms, etc.</li>\n</ul>\n<h4>Hybrid P2P networks</h4>\n<p>Hybrid peer-to-peer (P2P) networks combine structured and unstructured P2P networks. </p>\n<ul>\n<li>There can be some central servers with P2P capabilities like indexing servers, super peers, or trackers. Such centralized components can help us achieve higher performance and reliability by optimizing resource allocation, improving search performance, and coordinating communication.</li>\n<li>This can also use the core principle of P2P networking by decentralising components for certain functions. For example, data storage and sharing.</li>\n<li>Can be more scalable than purely centralized systems.</li>\n<li>Depending on the specific requirements, one can use different degrees of centralization. So it provides flexibility by creating a balance between the benefits of centralized and decentralized architectures. </li>\n</ul>\n<h3>Advantages of peer-to-peer networks</h3>\n<ul>\n<li>P2P networks are resistant to single points of failure. Unless the network is very small, it is unlikely that the failure of a single peer will have a significant impact on the overall network. Additionally, changes in the number of peers in the network do not affect its stability.</li>\n<li>P2P networks are decentralized and distributed, which means that as more people interact with your data, more nodes are added to the network. This allows for scaling of bandwidth and storage.</li>\n<li>All nodes have equal privileges in a P2P network, ensuring minimal corporate control or intervention. This means that you have control over your data.</li>\n<li>P2P networks can be more cost-effective due to the absence of a central server that needs to be maintained and paid for (apart from monitoring servers).</li>\n<li>P2P networks can handle large simultaneous connections without difficulty due to their decentralized structure. This also makes them more resistant to attacks since no central server could be targeted.</li>\n</ul>\n<h3>Disadvantages of peer to peer networks</h3>\n<ul>\n<li>Peer-to-peer networks have some security risks. For example, if a peer becomes infected with a virus and uploads a file containing the virus, the virus can easily spread to other peers in the network. </li>\n<li>Another concern is that it can be difficult to ensure that all peers have the correct permissions to access the network, particularly if there are a large number of peers.</li>\n<li>Some users of P2P networks take advantage of resources shared by other nodes without contributing anything themselves. These users are known as \"leachers.\" While the decentralized nature of P2P networks can make them difficult to shut down, this can also be a drawback if the network is used for illegal or unethical activities.</li>\n<li>If all the nodes in the P2P network are offline, the data becomes unavailable.</li>\n</ul>\n<p>Thanks to Navtosh Kumar for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy system design, Enjoy algorithms!</p>"},{"id":"a49356ba-c7fc-5514-9321-b7cb3455e5a7","frontmatter":{"title":"Forward and Reverse Proxy in System Design","path":"/blog/proxies-in-system-design"},"html":"<p>Proxies are one of the critical components in designing large-scale distributed systems. We use them to decouple clients from servers and provide additional functionalities like caching, authentication, logging, etc. So let's move forward and discuss two types of proxies: Forward proxy and Reverse proxy. One of the key goals here is to understand their differences.</p>\n<h3>What is Forward Proxy?</h3>\n<p>A forward proxy (also known as a proxy or proxy server) is a server that sits in front of a group of clients. When a client sends a request to a server, instead of going directly to the server, the request is routed to the forward proxy, which then forwards the request to the server.</p>\n<p>In other words, when clients send requests to websites on the Internet, the forward proxy intercepts these requests and communicates with the servers on behalf of the clients (acting as an intermediary). This can help us to perform certain actions before the request reaches the original destination.</p>\n<h3><strong>Why do we use a forward proxy?</strong></h3>\n<p>There are several reasons to use a forward proxy.</p>\n<ul>\n<li><strong>To filter network traffic based on predefined rules</strong>. For example, an organization can use a proxy server to block access to specific websites or restrict access to certain types of content. We can also use it to enforce security policies and protect against malicious activities.</li>\n<li><strong>To provide anonymity for clients by hiding their IP addresses</strong>. This can be useful for accessing content that is restricted in certain regions or for protecting clients' privacy and security.</li>\n<li><strong>To optimize network traffic</strong> by compressing data, removing unnecessary headers, and reducing the size of transferred data. This reduces bandwidth usage and improves network performance.</li>\n</ul>\n<h3>What is Reverse Proxy?</h3>\n<p>Reverse proxies are the exact opposite of forward proxies in terms of interaction pattern. A forward proxy acts on behalf of clients and a reverse proxy acts on behalf of servers. So, unlike a forward proxy, which is used to protect clients, a reverse proxy is used to protect servers. To simplify the difference: A forward proxy ensures that no origin server ever communicates directly with that specific client. On the other hand, a reverse proxy ensures that no client ever communicates directly with that origin server.</p>\n<p>When clients send requests to the server, the requests are intercepted by the reverse proxy. The reverse proxy will then send requests to and receive responses from the server. In other words, the client thinks it is directly interacting with the server and the request goes to the reverse proxy without the client knowing about it (or does not know that some other server processed its request).</p>\n<p>Let's take an example: Suppose we type \"<a href=\"https://www.enjoyalgorithm.com/\">https://www.enjoyalgorithm.com</a>\" in our browser. Our browser makes a DNS query to get the IP address of enjoyalgorithm.com. If enjoyalgorithm.com uses a reverse proxy and configures it correctly, the DNS query will return the reverse proxy's IP address.</p>\n<p><img src=\"https://ucarecdn.com/8db0c399-d1ce-473a-99bb-8a104ae326f5/\" alt=\"Forward and reverse proxy in system design\" title=\"Difference between forward and reverse proxy\"></p>\n<h3>Use cases of reverse proxy</h3>\n<ul>\n<li>By using a reverse proxy, we can hide the origin server's IP address from the clients so that malicious clients cannot access them directly. Reverse proxy can also implement features like rejecting traffic from specific client IP addresses (blacklisting) or limiting the number of requests accepted from each client.</li>\n<li>We can use a reverse proxy as a load-balancing solution to distribute traffic among servers.</li>\n<li>A reverse proxy can cache frequently requested resources like web pages, images, and videos. By serving cached content directly from the reverse proxy, we can reduce the load on the web server, speed up content delivery, save bandwidth and decrease network latency.</li>\n<li>We can configure a reverse proxy to decrypt all incoming requests and encrypt all outgoing responses i.e. your origin server won't have to handle the encryption/decryption process for each client request. This can help to reduce the load on the origin server.</li>\n</ul>\n<h3>Disadvantages of reverse proxy</h3>\n<ul>\n<li>Adding a reverse proxy to a system can increase its complexity.</li>\n<li>If the reverse proxy fails, the entire system can be affected, as all traffic goes through it. It is important to ensure redundancy and failover mechanisms to mitigate this risk.</li>\n<li>While a reverse proxy can improve performance by caching and load balancing, it can also introduce additional latency and overhead due to the additional processing required for each request.</li>\n<li>When SSL encryption is used, the reverse proxy must handle SSL certificates, which can be challenging to manage if there are many backend servers with different SSL configurations.</li>\n</ul>\n<p>If you have any query or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"cc3b4963-b38b-561d-8555-76e6b7378aed","frontmatter":{"title":"Process Management in Operating System (OS)","path":"/blog/process-management-in-operating-system"},"html":"<p>Process management is one of the critical concepts used in operating systems that involves activities related to managing and controlling processes. So before looking into process management and how it works, let's start with the definition of a process. In simple words, a program in execution is called a <strong>process</strong>. It is an instance of a program that runs, i.e., an entity that can be assigned and executed on a processor. There are two essential elements of a process: program code and data associated with that code.</p>\n<p><strong>Process memory</strong> is divided into four sections:</p>\n<ul>\n<li><strong>Program memory</strong> stores the compiled program code.</li>\n<li><strong>The data section</strong> stores global and static variables.</li>\n<li><strong>The heap section</strong> manages dynamic memory allocation inside the program. It is the portion of memory where dynamically allocated memory resides i.e., memory allocation via new or malloc and memory deallocation via delete or free, etc.</li>\n<li><strong>The stack section</strong> stores local variables defined inside the program. Stack space is created for local variables when declared, and space is freed up when they go out of scope.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*6vsoP1cWzQkN95AlEt2WoA.jpeg\" alt=\"An operating system process in a memory\" title=\"Process memory visualization in an operating system\"></p>\n<p><strong>Note:</strong> Stack and heap sections start at opposite ends of the process free space and grow towards each other. When they meet, a stack overflow error will occur or a call to new memory allocation will fail due to insufficient memory available in the heap section.</p>\n<h3>Process Control Block (PCB)</h3>\n<p>A program executing as a process is uniquely determined by various parameters. These parameters are stored in a Process Control Block (PCB) i.e. a data structure which holds the following information:</p>\n<ul>\n<li><strong>Process ID:</strong> Unique ID associated with every process.</li>\n<li><strong>Process state:</strong> A process can be in a start, ready, running, wait or terminated state.</li>\n<li><strong>CPU scheduling information:</strong> Related to priority level relative to the other processes and pointers to scheduling queues.</li>\n<li><strong>CPU registers and program counter:</strong> Need to be saved and restored when swapping processes in and out of the CPU.</li>\n<li><strong>I/O status information:</strong> Includes I/O requests, I/O devices (e.g., disk drives) assigned to the process, and a list of files used by the process.</li>\n<li><strong>Memory management information:</strong> Page tables or segment tables.</li>\n<li><strong>Accounting information:</strong> User and kernel CPU time consumed, account numbers, limits, etc.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*jG1_hpHeXoB3rToVulnzqQ.jpeg\" alt=\"Process Control Block in Operating System\" title=\"What is Process Control Block in OS?\"></p>\n<h3>States of a Process in Operating System</h3>\n<p>Now, we understood the process, where we defined one parameter of a process called <strong>Process State</strong>. Processes in the operating system can be in any of the five states: start, ready, running, wait, and terminated. </p>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*WfpVwfeQXyfQ4nlqVlhd0Q.jpeg\" alt=\"Process states in operating system\" title=\"Process states in OS\"></p>\n<p>Let's understand these states and the transition of a process from one state to another state:</p>\n<ul>\n<li><strong>Null to Start:</strong> When a new process is created, a process transitions from the NULL to the Start state.</li>\n<li><strong>Start to Ready:</strong> The OS moves a process from the Start state to the Ready state when it is prepared to execute an additional process. At this stage, the process has all the resources available that it needs to run and is waiting to receive CPU time for its execution.</li>\n<li><strong>Ready to Running:</strong> The OS chooses one of the processes from the Ready state using the process scheduler or dispatcher. After this, the CPU starts executing the selected process in the running state.</li>\n<li><strong>Running to Terminated:</strong> The running process is terminated by the OS when it has been completed.</li>\n<li><strong>Running to Ready:</strong> The scheduler sets a specific time limit for executing any active process. If the current process takes more time than specified by the scheduler, it is moved back to the ready state. The most critical reason for this transition is that it has reached its maximum permissible period for uninterrupted execution. Almost all multiprogramming operating systems enforce this time constraint.</li>\n<li><strong>Running to Wait:</strong> If a process requires anything for which it must wait, it is placed in the waiting state. Now, the process cannot run because it is waiting for some resource to become available or for some event to occur. For example, the process may be waiting for keyboard input, a disk access request, inter-process messages, a timer to go off, or a child process to finish.</li>\n<li><strong>Wait to Ready:</strong> A process in the Waiting state is transferred to the Ready state when the event for which it has been waiting is completed.</li>\n</ul>\n<p><strong>Note:</strong> Some operating systems may have other states besides the ones listed here.</p>\n<h3>Execution of a process in Operating System</h3>\n<p>The operating system uses a process control block (PCB) to track the execution status of each process.</p>\n<ul>\n<li>To keep track of all processes, it assigns a process ID (PID) to each process to identify it uniquely. As discussed above, it also stores several other critical details in the PCB.</li>\n<li>The OS updates information in the PCB as the process makes a transition from one state to another.</li>\n<li>To access PCB frequently, the operating system keeps pointers to each PCB in a process table.</li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/900/1*usQZh0BLpOhsA09QPm9dhg.jpeg\" alt=\"Execution of a process in an Operating System\" title=\"How process executes in an Operating System?\"></p>\n<h3>Process Schedulers in Operating System</h3>\n<p>Process scheduling is critical for selecting and removing running processes based on a particular algorithm. Here one of the key objectives is to keep the CPU busy and deliver \"acceptable\" response times for all programs. Note: Multiprogramming operating systems allow more than one process to be loaded into the executable memory at a time, and the loaded process shares CPU using time multiplexing.</p>\n<p>The operating system has three types of process schedulers:</p>\n<p><strong>Long-Term or Job Scheduler:</strong> This scheduler's job is to bring new processes to the Ready state. It determines which process is assigned to the CPU for processing, selects processes from the queue, and loads them into memory for execution. The primary objective of the job scheduler is to provide a balanced mix of jobs (such as I/O bound and processor bound) and control the degree of multiprogramming.</p>\n<p><strong>Short-Term or CPU Scheduler:</strong> It is in charge of selecting one process from the ready state and scheduling it to the running state. They are also known as <strong>Dispatchers</strong>.</p>\n<ul>\n<li>CPU scheduler is responsible for ensuring no starvation due to high burst time processes.</li>\n<li>It runs frequently and quickly swaps one process from the running state to the ready state.</li>\n<li><strong>Medium Term Scheduler:</strong> It is in charge of swapping processes when a particular process is performing an I/O operation. If a running process makes an I/O request, it may be suspended. The suspended process is transferred to secondary storage to remove it from memory and make room for other processes. This is known as <strong>switching.</strong></li>\n</ul>\n<p><img src=\"https://cdn-images-1.medium.com/max/600/1*WXH4XmlSyZCZzRukqOgReQ.jpeg\" alt=\"Medium-term process scheduler in Operating System\" title=\"Medium-term process scheduler in OS\"></p>\n<h3>CPU Scheduling in Operating System</h3>\n<p>Now our aim would be to understand CPU Scheduling concept and why we need it. </p>\n<ul>\n<li>Both I/O and CPU time is used in a typical procedure. Time spent waiting for I/O in an old operating system like MS-DOS is wasted, and CPU is free during this time. In multiprogramming modern operating systems, one process can use CPU while another waits for I/O.</li>\n<li>CPU scheduling determines which process will exclusively use CPU while another is paused. The goal is to ensure that whenever CPU is idle, OS chooses at least one of the programs in the ready queue to run.</li>\n</ul>\n<h4>Types of CPU Scheduling</h4>\n<p>There are two major types of CPU Scheduling:</p>\n<ul>\n<li><strong>Preemptive Scheduling:</strong> In this scheduling, processes are assigned with their priorities and currently running processes can be interrupted by the operating system so that another process with a higher priority can run. So lower priority process is put on hold for a while and resumes when the higher priority process is completed. This is used in real-time operating systems where responsiveness is critical.</li>\n<li><strong>Non-Preemptive Scheduling:</strong> In this scheduling, a running process cannot be interrupted until it has completed its execution or voluntarily relinquishes control of the CPU. So in this approach, once a process starts running, it continues to run until it finishes or blocks (e.g., waiting for I/O). Non-preemptive scheduling is simpler to implement and may lead to less overhead compared to preemptive scheduling, but it may not be good for systems where responsiveness or prioritization of tasks is crucial.</li>\n</ul>\n<h4>Differences between Preemptive and Non-Preemptive <strong>Scheduling</strong></h4>\n<ul>\n<li>In pre-emptive scheduling, memory allocated in the main memory is for a limited time. It may be assigned to any other process depending on the state of the current process or priority of the new incoming process. In non-Preemptive scheduling, memory is allocated to the same process until it is completed.</li>\n<li>In preemptive scheduling, if high-priority processes keep on coming, a low-priority process can be interrupted for an indefinite time. In non-preemptive scheduling, if any large process is processing, it will keep on processing and will not allow even a small process to process before it completes execution.</li>\n<li>In preemptive scheduling, it has to save all data of the process in a halted state so that it can continue from that point only, whereas no such requirements exist in non-preemptive scheduling.</li>\n</ul>\n<h3>Different Scheduling Algorithms in Operating System</h3>\n<p>Now, we will look at the various scheduling algorithms involved in process management one by one.</p>\n<h4><strong>First Come First Serve (FCFS) Scheduling Algorithm</strong></h4>\n<p>It is the most basic CPU scheduling algorithm, where a FIFO queue is used to manage the scheduling strategy. The idea is simple: a process that asks first to get the CPU allocation, get access to the CPU first. </p>\n<p>PCB of the process is linked to the tail of the queue as it enters the ready queue. As a result, whenever a CPU becomes available, it is assigned to the process at the front of the queue. Some important points:</p>\n<ul>\n<li>Offers both non-preemptive and pre-emptive scheduling.</li>\n<li>Jobs are always executed on a first-come first-serve basis.</li>\n<li>Easy to use and implement.</li>\n<li>Poor performance and overall wait time are quite high.</li>\n</ul>\n<p>A simple example: Suppose we have five processes p1, p2, p3, p4, and p5, and the ready queue receives them at times t1, t2, t3, t4, and t5 such that t1 &#x3C; t2 &#x3C; t3 &#x3C; t4 &#x3C; t5. So p1 arrived first in the ready queue, so it will be executed first, followed by p2, p3, p4, and p5, respectively.</p>\n<p><strong>Convoy Effect</strong></p>\n<p>In First Come First Serve algorithm, if a process with a large CPU burst time arrives before any small process, then the small process will get blocked by that large process, which is called Convoy Effect.</p>\n<h4><strong>Shortest Job First(SJF) Scheduling Algorithm</strong></h4>\n<p>SJF algorithm is a non-preemptive scheduling algorithm. This policy prioritises waiting process with the shortest execution time. Among all scheduling algorithms, Shortest Job First has the advantage of having the shortest average waiting time. It first creates a pool of processes and then chooses the process with the shortest burst times. After completing the process, it again selects the process with the shortest burst time from the pool. Some important points:</p>\n<ul>\n<li>Short processes are handled very quickly.</li>\n<li>When a new process is added, algorithm just has to compare the currently running process to the new one, ignoring any other processes waiting to run.</li>\n<li>Long processes can be postponed indefinitely if short processes are added regularly.</li>\n</ul>\n<p>It is further categorized into two types: </p>\n<ol>\n<li><strong>Preemptive priority scheduling:</strong> Due to the arrival of a smaller CPU burst time process, the current process is sent to halted state and execution of the new process proceeds.</li>\n<li><strong>Non-preemptive priority scheduling:</strong> The current process is not disturbed due to the arrival of a smaller CPU bursts time process.</li>\n</ol>\n<p><strong>A simple example:</strong> Suppose we have five processes p1, p2, p3, p4, and p5, and the ready queue receives them at times t1, t2, t3, t4, and t5 such that t1 &#x3C; t2 &#x3C; t3 &#x3C; t4 &#x3C; t5. Now, you can assume the ready queue as the priority queue, which rearranges the incoming process based on CPU bursts time. Therefore, process with the least CPU burst time is delivered first, and so on.</p>\n<h4><strong>Longest Job First Scheduling Algorithm</strong></h4>\n<p>Longest Job First (LJF) is non-preemptive scheduling. This algorithm keeps track of the burst time of all processes accessible at the moment of arrival and then assigns processor to the process with the longest burst time. In this algorithm, once a process begins to run, it cannot be halted in the middle of its execution.</p>\n<p>It organise processes in ascending order of their arrival time. Then, out of all processes that have arrived up to that point, it will choose one with the longest burst time. After that, it will process it throughout the duration of the burst. Until this process completes its execution, LJF monitors if any more processes arrive.</p>\n<p>Some important points:</p>\n<ul>\n<li>The average waiting time and turn-around time for a particular set of processes can increase.</li>\n<li>A process with a short burst time may never get executed while system continues to run large processes.</li>\n<li>Reduces processing speed, resulting in a decrease in system efficiency.</li>\n</ul>\n<p><strong>A simple example:</strong> Similar to the above example, you can assume here that the same ready queue prioritises based on a larger CPU burst time i.e. out of those five processes, the one with the largest CPU burst time will be executed first.</p>\n<h4><strong>Round Robin scheduling Algorithm</strong></h4>\n<p>In Round Robin scheduling each process is cyclically assigned a set time slot. This looks similar to FCFS scheduling, except that it includes preemption, which allows system to transition between processes. In other words, round robin strategy was created with time-sharing systems in mind. </p>\n<p>Each process has a set amount of time assigned to it, and once that time period has passed, process is preempted and another process takes its place. As a result, all processes get an equal amount of CPU time. This algorithm performs best in terms of average response time.</p>\n<p>Some important points:</p>\n<ul>\n<li>Round robin is a pre-emptive algorithm similar to FCFS with some improvements.</li>\n<li>CPU is shifted to the next process after a fixed interval time.</li>\n<li>Widely used scheduling method in traditional OS.</li>\n</ul>\n<p><strong>A simple example:</strong> Suppose there are five processes p1, p2, p3, p4, and p5, with total execution times of t1, t2, t3, t4, and t5. Now, we have one extra factor, t (a time quanta), which will ensure equal CPU time sharing for each process. Suppose the first process p1 arrives, and after t time of execution, p1 will be preempted and wait for the remaining execution of (t1 — t) time. At this stage, p1 will move to the wait state, where it can perform its I/O operation. Now CPU is released for the next process, p2. After completing the I/O operation, process p1 is pushed to the ready queue again for its next processing cycle. The same goes for all the processes.</p>\n<h4><strong>Priority Based Scheduling</strong></h4>\n<p>Priority scheduling is one of the most often used priority scheduling methods. Here priority is assigned to each process, and the highest-priority process will be executed first. The processes of the same priority will be executed based on the first-come first-serve basis. Note: Priority can be determined by memory limitations, time constraints, or other resource constraints.</p>\n<p>Some important points:</p>\n<ul>\n<li>There are chances of indefinite blocking or starvation.</li>\n<li>We can leverage the concept of aging to prevent starving of any process by increasing the priority of low-priority processes based on their waiting time.</li>\n</ul>\n<p>It is also categorized into two types: </p>\n<ol>\n<li><strong>Preemptive priority scheduling:</strong> Due to the incoming higher priority process, the current process is sent to halted state and execution of the new process proceeds. </li>\n<li><strong>Non-preemptive priority scheduling:</strong> Due to an incoming high priority process, the current process is not disturbed.</li>\n</ol>\n<p>A simple example: Similar to the example of the FCFS algorithm, the processes are inserted in the ready queue but based on a priority now, which can be CPU burst time, memory constraints, etc.</p>\n<p><strong>Reference:</strong> Operating System Concepts by Abraham Silberschatz, Greg Gagne, and Peter Baer Galvin </p>\n<p>Thanks to <strong>Vishal Das</strong> for his contribution to creating the first version of the content. Please write in the message below if you find anything incorrect, or if you want to share more insight. Enjoy learning, Enjoy algorithms!</p>"},{"id":"cb1100f5-26ea-5d5a-b82e-b1453fcad78e","frontmatter":{"title":"Publisher Subscriber (Pub-Sub) Design Pattern","path":"/blog/publisher-subscriber-pattern"},"html":"<h3><strong>What is Publisher-Subscriber Pattern?</strong></h3>\n<p>The Publish-Subscribe pattern (also known as the pub-sub pattern), is a design pattern that helps publishers and subscribers to communicate with one another in a loosely coupled manner. In this arrangement, the publisher and subscriber rely on a message broker to send messages from the publisher to the subscribers. Messages (events) are sent out by the host (publisher) to a channel, which subscribers can join.</p>\n<p>Compared to other design patterns like message queues and event brokers, Pub-Sub is more versatile and scalable. The key to this is that Pub-Sub allows messages to flow between different system components without the components knowing one other’s identities.</p>\n<h3>A real-life example of Pub-Sub Pattern</h3>\n<p>The Publish-Subscribe model is used in social networks through features like \"following.\" To better understand how this works, let's take an example of a social network for sharing recipes.</p>\n<p>In this network, users can share their recipes and follow the recipes of other users. When sharing a recipe, the user can categorize it by topic like meal or ingredient seasonality. When a user follows another user, they are subscribing to the recipes that the friend publishes.</p>\n<p>Subscribers, or followers, have the option to view all of the recipes published by the user they are following or only those that match their interests. They can also create filters to exclude certain types of recipes, such as those containing certain ingredients.</p>\n<p>Users can follow as many other users as they like, so their timeline will be filled with recipes from a variety of sources. However, each recipe is only published once, by the original user.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*TWR4ENOiSBm-UJ9PjuNfCw.png\" alt=\"Example of Pub-Sub Pattern\" title=\"Publisher-Subscriber Design Pattern\"></p>\n<p>H﻿ere Event Bus is responsible for routing messages to the appropriate subscribers. It does this by keeping track of which subjects each subscriber is subscribed to.</p>\n<p>Publishers decide what topics their messages will belong to, and the Event Bus filters the messages by topic before delivering them to the relevant subscribers. For example, if a publisher sends a message with Topic A, it will be forwarded to any subscribers who have subscribed to Topic A. Similarly, a message with Topic B will be delivered to subscribers of Topic B.</p>\n<p>It's important to note that if a publisher sends a message with an incorrect topic, it will only be delivered to subscribers of that incorrect topic. In the diagram above, if a publisher sends a message with Topic A but mistakenly labels it as Topic B, it will only be sent to Subscriber 2 and Subscriber 3.</p>\n<h3>How does Publish-Subscribe Pattern work?</h3>\n<p>Building reusable arrangements of modules and their interconnections is the foundation of software design patterns. In a UML design diagram, these modules are often classes or objects. Modern architectural patterns, on the other hand, see modules as larger, self-executing processes scattered over distributed systems.</p>\n<p>To fully comprehend the benefits of the Pub-Sub design, you must first understand the fundamental pattern upon which an information system is formed and then trace its progression into a distributed system. An information system is often made up of a generic set of software modules that are organized in this simple sequential structure.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*UXj-LbmB-wfX6te_jqqyNw.png\"></p>\n<p>Consider the diagram above as simple software made up of three parts. The input module receives user input and converts it into a message that is sent to the processing module. The data is processed by the processing module and sent to the output module as a new message. The data is shown on the user’s screen using the output module.</p>\n<p>The real world, on the other hand, is seldom so simple. For handling concurrent queries at a suitable size, the system will require several inputs and output modules.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*3XNb000sYiKZshMpNV2E6g.png\" alt=\"How Publish-Subscribe Pattern Works? Part 1\" title=\" How Pub-Sub Pattern Works?  Part 1\"></p>\n<p>The system faces the difficulty of routing messages from input modules to their appropriate output modules at this scale. The input and output modules will require an addressing mechanism to handle this challenge. The messages will be processed and routed to the relevant recipient based on an address by the processing module. In order to solve the routing problem, all three modules work together.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/533/1*yDkvZg1VG1tosPoo4BMkaA.png\" alt=\"How Publish-Subscribe Pattern Works? Part 2\" title=\"How Pub-Sub Pattern Works? Part 2\"></p>\n<p>The system will be able to manage thousands of concurrent connections at an Internet-scale. Users from all over the world will send and receive messages using the system. It must also be able to handle a large number of users from all over the world.</p>\n<p>The system modules, on the other hand, will not work as planned at such a vast scale.</p>\n<ul>\n<li>The load is too much for the processing module to handle. The load must be divided across numerous processing modules due to the high volume and geographical spread.</li>\n<li>The dynamics of input and output change at this scale. The use of pre-defined addressing between modules adds a significant amount of overhead.</li>\n</ul>\n<p>The first issue can be solved by using numerous processing units. This has the effect of horizontally separating the system. This, however, adds to the routing complexity. The messages must now be routed to the appropriate processing module by the input modules.</p>\n<p>At the internet scale, attaching module-specific routing metadata to messages becomes a bottleneck. Under these conditions, the design of message transfer from one module to the next necessitates a complete overhaul.</p>\n<h3>Advantages of Publisher-Subscriber Pattern</h3>\n<ul>\n<li><strong>Low coupling on the publisher's end:</strong> Publishers do not need to know the number, identity, or types of messages that subscribers are interested in. They simply output data in response to the correct events through their API. This allows for flexibility and scalability, as new subscribers can be easily added to the system without affecting the publisher.</li>\n<li><strong>Reduced cognitive load for subscribers:</strong> Subscribers do not need to worry about the inner workings of the publisher or have access to the publisher's source code. They can only communicate with the publisher using the publisher's public API, which simplifies their understanding of the system.</li>\n<li><strong>Separation of concerns:</strong> The simplicity of the Pub/Sub architecture (data flows one way from publishers to subscribers) allows developers to practice fine-grained separation of concerns. This means that different message types can be split into distinct categories that each fulfill a single, straightforward purpose. For example, data with the topic \"/cats\" could contain information about cats, while data with the topic \"/dogs\" could contain information about dogs.</li>\n<li><strong>Improved testability:</strong> The fine-grained control over topics makes it easy to confirm that the various event buses are transmitting the necessary messages.</li>\n<li><strong>Improved security:</strong> The Pub/Sub architecture is well-suited for the security principle of assigning minimal privileges or information. Developers can easily create modules that are subscribed to only the minimum number of message types needed to function.</li>\n</ul>\n<h3>Disadvantages of Publisher-Subscriber Pattern</h3>\n<p><strong>The inflexibility of data sent by the publisher</strong></p>\n<p>The publish/subscribe approach can introduce a high level of semantic coupling in the messages that are sent between publishers and subscribers. This means that it can be difficult to modify the data structure of these messages once it has been established. To change the format of the messages, all of the subscribers must be updated to accept the new format, which can be challenging or impossible if the subscribers are external. This is a common problem with versioned APIs.</p>\n<p>One solution to this issue is to use a versioned message format that allows subscribers to validate the format they are receiving. However, this assumes that subscribers are correctly consuming the versioning information.</p>\n<p>Another option is to use versioned endpoints, such as \"/APIV0/\" and \"/APIV1/,\" to maintain backward compatibility. This approach has the disadvantage of requiring developers to support multiple versions, which can be time-consuming.</p>\n<p><strong>Instability of Delivery</strong></p>\n<p>One disadvantage of the publish/subscribe model is that it can be difficult to determine the health of subscribers. The publisher does not have complete information about the systems that are listening to the messages, which can lead to problems.</p>\n<p>For example, logging systems often use the publish/subscribe model. If a logger subscribed to the \"Critical\" message type crashes or becomes stuck in an error state, it may miss important messages. Any services that rely on error warnings will then be unaware of issues with the publisher.</p>\n<p>This problem is not unique to the publish/subscribe model, and can occur in any client/server system. However, one advantage of pub/sub is that it allows for high levels of redundancy by allowing multiple instances of a logger to run concurrently with minimal additional system effort.</p>\n<p>To mitigate this risk, changes to the design could be implemented, such as requiring receipts of received messages. This would allow the publisher to receive feedback on the status of the subscribers.</p>\n<p><strong>Additional Reading:</strong> <a href=\"https://docs.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber\">https://docs.microsoft.com/en-us/azure/architecture/patterns/publisher-subscriber</a></p>\n<p>Thanks to Navtosh for his contribution in creating the first version of this content. If you have any queries/doubts/feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design, Enjoy algorithms!</p>"},{"id":"83d15a62-df49-527a-b1da-bbc4840b8ab2","frontmatter":{"title":"Server Sent Events: System Design Concept","path":"/blog/server-sent-events"},"html":"<h3>What is Server-Sent Events?</h3>\n<p>There are two main approaches for delivering data from servers to clients: client pull and server push. In a client pull scenario, the client (web browser) requests data from the server. In a server push scenario, the server continuously sends updates to the client without an initial request.</p>\n<p>Server-Sent Events (SSE) is a type of server push technology that establishes a long-lasting connection between the client and the server. It allows the server to automatically send updates to the client via an HTTP connection, using a single directional channel for data delivery. To receive these updates, the client uses a JavaScript API called <a href=\"https://developer.mozilla.org/en-US/docs/Web/API/EventSource\">EventSource</a>. </p>\n<p><img src=\"https://ucarecdn.com/6057a5a8-2aed-4e79-92bf-d0d5d54ec7f7/\" alt=\"What is Server-Sent Events?\" title=\"How Server-Sent Events Work?\"></p>\n<p>To understand how Server-Sent Events (SSE) work, it's important to have a basic understanding of how communication occurs over the internet using HTTP requests. HTTP is a widely used protocol for data transfer in client-server architecture.</p>\n<p>In the context of SSE, if the client loses its connection to the EventSource, it tries to reconnect by sending the ID of the last event received to the server in a new HTTP request. This is done using the \"<strong>Last-Event-ID</strong>\" header. The server listens for this request and starts sending events that have taken place since the supplied ID. This way, the client can keep receiving updates from the server, even if the connection is temporarily lost.</p>\n<h3>Let’s look into the internal working of Server-Sent Events</h3>\n<p>Server Sent Events are real-time events that are emitted by the server and received by the browser. To establish communication between the client and server, client creates a new JavaScript EventSource object and passes the URL of the endpoint that is expected to return a stream of events. The EventSource interface connects to the server over HTTP and receives events in an event-stream or text format.</p>\n<p>The client sends a regular HTTP request to the server and expects to receive a series of event messages in return. Whenever the server writes an event to the HTTP response, the client receives it and processes it in a listener callback function. The HTTP response connection remains open until it is considered stale or until the client closes it. Note: You can find a full code example of server-sent events on <a href=\"https://github.com/mdn/dom-examples/tree/main/server-sent-events\">this</a> GitHub link.</p>\n<p>Let’s look at how client and server are implemented in SSE.</p>\n<h4>Client-Side Implementation</h4>\n<p>As previously mentioned, the client creates a new EventSource object to receive events from the server. The EventSource object takes a URL as an argument, which is the location from where the events will be drawn in \"text/event-stream\" format.</p>\n<p>The client receives the events and processes them in a listener callback function. Callback functions, also known as event handlers, are registered to handle specific events. The EventSource object has a method called addEventListener, which is used to register these handlers. If the original event message contains multiple data lines, they will be concatenated together by the browser to form a single string before the callback functions are called.</p>\n<p>It is worth noting that there is a limit to the number of Server-Sent Events connections that can be active at any given time. Most browsers are limited to six SSE connections.</p>\n<h4>Server-Side Implementation</h4>\n<p>The server receives an HTTP request from the client and responds with valid Server-Sent Event messages. It instructs the client about the content type and tells the client to keep the connection alive in order to facilitate the transmission of events over the same connection.</p>\n<p>The server can only accept EventSource requests and must maintain a list of all connected users in order to emit new stream events. It also needs to keep a history of messages in case a client misses any, and should be able to remove dropped connections from the list of connected users.</p>\n<p>It's worth noting that the server-side implementation of Server-Sent Events can be coded in any programming language, such as Java, C, Python, or Go, while the client-side requires the use of JavaScript.</p>\n<h3>Stopping an Event Stream</h3>\n<p>Once the required data has been received, it is necessary to close the connection and stop the event stream. There are two ways to do this, depending on the client and server:</p>\n<p><strong>Client-side:</strong> The client has the option to stop the events using the .close() method of the EventSource object. When this method is called, the server detects it and stops sending events to the client by closing the corresponding HTTP response.</p>\n<p><strong>Server-side:</strong> The server can also stop the event stream by sending a final event with a unique ID that corresponds to the \"end of stream\" event. Alternatively, the server can stop the event stream by closing the HTTP response connection with the client.</p>\n<p>In this way, both the client and server have the ability to close the connection and stop the event stream.</p>\n<h3>Connection failure</h3>\n<p>In the real world, it is not possible to maintain a fully persistent connection. In Server-Sent Events, the connection is established via HTTP and may be dropped due to network issues, which can disrupt the transfer of events and lead to incomplete event messages. To address this issue, the client can try to reconnect to the event source by sending the ID of the last event as an HTTP header called \"Last-Event-ID\" to the server via a new HTTP request. The server listens for this request and begins sending events that have occurred since the supplied ID. This allows the client to catch up on any missed events and helps to ensure the reliability of the event stream.</p>\n<h3>Application of Server Sent Events</h3>\n<p>Server-Sent Events are widely used in the development of real-time web applications and are also used to build real-time notification services, which are common in many applications for notifying users or administrators. Companies like Uber have also relied on Server-Sent Events for updating trip information for both drivers and customers.</p>\n<p>For example, consider a scenario in which a rider requests a ride and a driver is available to provide the service. Uber's matching technology finds a match on the backend and sends a trip offer to the driver. All parties involved (rider, driver, and backend) should now be aware of each other's intentions. To keep updated on the progress of the trip, the driver app can poll the server every few seconds to check for new offers, while the rider app can also poll the server every few seconds to check if a driver has been assigned. This continuous communication is facilitated by Server-Sent Events.</p>\n<h3>Conclusion</h3>\n<p>Server-Sent Events are useful for delivering fast updates and have a low overhead in their implementation. They are well-suited for systems that require real-time, one-way data flow and are commonly used in the news feeds of social media platforms like Twitter, Instagram, and Facebook. They are also useful for updating stock price charts and providing live sports updates.</p>"},{"id":"934401fd-e466-58aa-b061-519447558289","frontmatter":{"title":"Storage and Redundancy in Distributed Systems","path":"/blog/storage-and-redundancy"},"html":"<p>A storage device is a hardware component used primarily for data storage. It provides a way for computers to save data, either temporarily or permanently. Examples of storage devices are flash drives and hard drives. These devices store various types of data like videos, documents, photos, and raw data.</p>\n<p><strong>Persistence:</strong> Persistence is the ability of an object or state to remain unchanged even after the process that created it has stopped or the computer has shut down. When data needs to persist, it is saved in a non-volatile storage location like a hard drive, instead of a temporary file or volatile memory like RAM.</p>\n<h3>Types of Storages</h3>\n<p>Storage is one of a computer system’s core components and can be categorized into many types, but there are two primary types:</p>\n<h4><strong>Volatile Storage (Memory)</strong></h4>\n<p>A storage device that requires a continuous supply of electricity for data storage/retention. This storage serves as the primary storage for temporarily storing data and managing the workloads of applications. Cache memory and random access memory are examples of nonvolatile storage (RAM).</p>\n<h4><strong>Non-Volatile Storage</strong></h4>\n<p>A storage device that preserves digital data even though it is turned off or electrical power is not supplied. This is often referred to as a secondary storage device and is used for I/O operations involving permanent data storage. A hard disc, USB storage, and optical media are examples of volatile storage.</p>\n<h3>What is Redundancy?</h3>\n<p>In distributed systems, Redundancy is the duplication of critical components (web server, database server, etc.) to increase reliability and availability. This concept provides a backup or failsafe, ensuring that the system continues to operate even if few components fail. The term redundancy is used because if everything is working correctly, the duplicate device or competent does nothing and is redundant.</p>\n<p>Redundancy comes into play when we need to prevent a single point of failure (a point that can lead to system failure). By duplicating or adding components, the system becomes more resilient and less prone to failure. For example, consider a system that includes two identical web servers that are managed by a load balancer. Traffic from clients is evenly distributed between the two servers, but if one fails, the load balancer redirects all traffic to the remaining functioning server. In this way, redundancy eliminates the single point of failure and enhances the reliability of the system.</p>\n<h3><strong>Passive Redundancy</strong></h3>\n<p>In passive redundancy, we duplicate components in a system without actively using them. When the primary component is functioning correctly, duplicated components stay in a standby or passive state. During the passive state, the redundant component monitors the primary component's status and waits for a failure event. Once a failure is detected, the redundant component will take over the task of the failed component.</p>\n<p>Note: In some distributed systems, the decision for the standby component to take over from the primary can be made by other components or depend on the system architecture.</p>\n<ul>\n<li>In case of a failure, the system must ensure that the state of the redundant component is sufficiently fresh before resuming services. So it is the responsibility of the primary to continuously inform standby components to update their state. This is important to prevent data discrepancies between the primary and standby components.</li>\n<li>Periodically forcing switchovers (once a day or once a week) can increase the availability of the system. This practice ensures that the standby components are ready to take over when needed.</li>\n<li>Sometimes, passive redundancy may not provide instant recovery because there can be delays in detecting failures and switching to the redundant component. So it may require other fault-tolerant techniques like active monitoring, regular testing of the redundant components, and fault detection mechanisms.</li>\n</ul>\n<h4>When to use passive redundancy?</h4>\n<ul>\n<li>When scalability is not a primary concern or workload is expected to remain stable.</li>\n<li>When downtime or failure can have significant consequences.</li>\n<li>When system can tolerate a slight delay in failover or recovery.</li>\n<li>When the failover process requires manual intervention.</li>\n</ul>\n<p>Some distributed systems requirements directly align well with passive redundancy. For example, when data retrieval speed is not critical (data archiving systems), the passive state of redundant storage devices can be used for backup and recovery purposes.</p>\n<h3><strong>Active Redundancy</strong></h3>\n<p>In active redundancy, multiple identical components are simultaneously active in parallel, executing tasks concurrently and sharing the workload. In other words, they all actively contribute to the system's operations at the same time and work in a coordinated manner. </p>\n<ul>\n<li>We can use load balancing among redundant components to distribute the workload. So active redundancy reduces the likelihood of a single point of failure.</li>\n<li>Redundant components can work in parallel. This can improve performance because the system can handle a larger volume of work in a given time.</li>\n<li>The redundant components are synchronized in the same state. Due to this, the switching time from a failed component to a redundant one is typically minimal (in milliseconds) in the case of a failure. In other words, active redundancy ensures smooth failover by automatically transferring the workload from the failed component to the remaining active components.</li>\n</ul>\n<p>There are some critical points to consider:</p>\n<ul>\n<li>Active redundancy requires careful coordination and fault detection mechanisms to ensure that redundant components operate in a synchronized manner. </li>\n<li>When multiple components concurrently access shared resources, active redundancy should maintain consistency and avoid conflicts. </li>\n<li>Active redundancy generally requires more resources and can be more complex to implement compared to passive redundancy. But the increased reliability and continuous operation make it a preferred approach in many critical systems.</li>\n</ul>\n<h4>When to use active redundancy?</h4>\n<ul>\n<li>When systems demand uninterrupted operation (high-availability environments).</li>\n<li>When systems require fast response and minimal latency.</li>\n<li>When significant processing power is required (high traffic situations).</li>\n<li>When seamless scalability is required.</li>\n<li>When rapid recovery is required in the event of a failure.</li>\n</ul>\n<h3>Redundancy vs. Replication</h3>\n<p>In distributed systems, the terms \"redundancy\" and \"replication\" are often used interchangeably. Both concepts involve having multiple nodes, components, or processes in a system, but they serve different purposes.</p>\n<ul>\n<li>The focus of redundancy is to provide backup components to handle failures, while the focus of replication is to create and maintain consistent copies of data across multiple components.</li>\n<li>The aim of redundancy is to enhance fault tolerance and availability i.e. minimize downtime and ensure continuous operation. But the aim of replication is to ensure data consistency, availability, and performance.</li>\n<li>Replication can be a part of redundancy because redundant components often require synchronized copies of data to operate effectively.</li>\n</ul>\n<h3>Some critical ideas to think!</h3>\n<ul>\n<li>How does the choice of storage device impact system performance, reliability, and scalability? Think about factors like access times, throughput, latency, durability, etc.</li>\n<li>What mechanisms are used to ensure data consistency in active redundancy?</li>\n<li>What are the challenges in implementing fault detection mechanisms?</li>\n<li>How can we measure and quantify the reliability of redundant systems?</li>\n<li>How automated monitoring and alerting systems work to detect faults?</li>\n<li>What are the best practices for system maintenance in redundant systems?</li>\n</ul>\n<p>Enjoy learning, Enjoy system design!</p>"},{"id":"a0bfa3e2-7482-5fbe-91dd-c95db0587931","frontmatter":{"title":"Throttling and Rate Limiting","path":"/blog/throttling-and-rate-limiting"},"html":"<h3>What is Rate Limiting?</h3>\n<p>In distributed systems, rate limiting is one of the critical techniques to control the amount of traffic going through a system within a specific time period. The goal here is to prevent overuse of resources by limiting the rate at which events can occur. In other words, rate limiting is one of the protective mechanisms to restrict excessive use of services and preserve their availability.</p>\n<p>For example, a website can use rate limiting to prevent someone from repeatedly attempting to log in to an account with incorrect credentials. If a user tries to log in too frequently within a certain time window, the website can either block their access or slow down their requests. This will help to prevent malicious actors from overburdening the system and initiating attacks like DoS (Denial of Service).</p>\n<h3>How rate limiting works?</h3>\n<p>A rate limiter tracks the IP addresses of incoming requests and the time between them. If a single IP address makes too many requests within a specified timeframe, the rate limiter will throttle the IP address and not fulfil its requests for a certain period of time. In this way, a rate limiter acts like a virtual traffic cop, telling users who are making requests rapidly to slow down. This is similar to a traffic officer pulling over a driver for speeding.</p>\n<p>Here is how it generally works:</p>\n<ol>\n<li>When a client sends a request to a server, the server identifies the client based on various factors like IP address, user account, API key, etc.</li>\n<li>Now server will evaluate the request against predefined rate limits i.e. number of requests allowed within a certain time period (e.g., 100 requests per minute).</li>\n<li>Based on the evaluation, the server will decide whether to allow, throttle, or reject the request. If the request falls within the allowed limits, it is processed as usual. If the request exceeds the limits, the server may throttle the request (delay processing) or reject it.</li>\n<li>After the end of the specified time period, the rate limits reset so that the client can make a new set of requests within the limits.</li>\n</ol>\n<h3>What is API rate limiting?</h3>\n<p>Granting unlimited access to an API can be risky because anyone can use the API as much as they want. So rate limiting can protect APIs from being overwhelmed by too many requests. </p>\n<p>An API that uses rate limiting can throttle or temporarily block any client that tries to make too many API calls. Here rate limiter sets a threshold on the number of requests allowed per API key within a specified timeframe. When a client exceeds the threshold, the API can respond with an error code or delay processing the request. Note: One can <a href=\"https://www.enjoyalgorithms.com/blog/design-api-rate-limiter/\">implement API rate limiting</a> at various points like the API gateway, load balancer, or within the application code.</p>\n<p><img src=\"https://cdn-images-1.medium.com/max/1080/1*TQTDYcf7c6p1uNyn9tgxSg.png\" alt=\"API rate limiter in system design\" title=\"How API rate limiter works?\"></p>\n<h3>Benefits of rate-limiting<a href=\"https://www.enjoyalgorithms.com/blog/design-api-rate-limiter/\"></a></h3>\n<ul>\n<li><strong>Avoid resource depletion due to a Denial of Service (DoS) attack</strong>: By limiting the number of excess calls, a rate limiter can prevent DoS attacks. For example, Twitter restricts users to 300 tweets every three hours, and Google Docs APIs have a default limit of 300 per user every 60 seconds for read queries.</li>\n<li><strong>Reduce expenses:</strong> Limiting request counts can help to reduce the number of servers needed and allocate more resources to high-priority APIs. This is important for companies that use paid third-party APIs, which may be charged on a per-call basis.</li>\n<li><strong>Ensure that servers are not overloaded:</strong> Rate limiter can reduce server load by filtering out extra requests produced by bots or users.</li>\n</ul>\n<h3>Methods to implement API Rate-Limiting</h3>\n<h4><strong>Request Queues</strong></h4>\n<p>There are many request queue libraries that make it easy to implement rate limiting in different development environments. These libraries often come with pre-written code and can be easily accessible. For example, there are request rate limiter libraries that limit number of requests per second to a specific number and queue any additional requests. This can be an easy-to-use solution for API development.</p>\n<h4><strong>Throttling</strong></h4>\n<p>Throttling is another common method for implementing rate limiting. It involves establishing a temporary state in which each request is evaluated by the API. If throttle is triggered, a user may be disconnected or have their bandwidth reduced. Throttling can be done at the application, API, or user level. </p>\n<p>There are many products available that make it easy to implement throttling like Hybrid Data Pipeline from Progress, which provides throttled API access to various databases and services including IBM DB2, Oracle, SQL Server, MySQL, PostgreSQL, SAP Sybase, Hadoop Hive, Salesforce, and Google Analytics.</p>\n<p>Another technique to make scalable rate-limited APIs is to use rate-limiting algorithms.</p>\n<h3>Rate-limiting Algorithms</h3>\n<h4><strong>Fixed window technique</strong></h4>\n<p>The fixed window technique uses an incremental counter to track the number of incoming requests over a fixed time window. If number of requests exceeds the specified limit during this time period, any additional requests will be discarded.</p>\n<p>For example, a server can implement an algorithm to allow 300 API requests per minute. This means that the server will not serve more than 300 requests between 8:00 and 8:01. Here window will reset at 8:01 so that another 300 requests will served in the next time window [8:01, 8:02].</p>\n<h4><strong>Leaky bucket technique</strong></h4>\n<p>The leaky bucket technique is easy to implement and efficient in terms of memory usage. It converts incoming requests into a First In First Out (FIFO) queue so that it can process requests at a consistent rate.</p>\n<p>So leaky bucket algorithms don't rely on specified timeframes. Instead, they focus on the fixed length of request queues, regardless of time. The server will service requests on a FIFO basis, with new requests joining the back of the queue. If a new request arrives when the queue is full, it will be dropped.</p>\n<p>This idea will help in smoothing out traffic spikes because the server receives API requests constantly (requests are forwarded to the server one by one) and there are no sudden bursts of requests. However, it is not perfect because the queue is static and there is a higher chance of starvation.</p>\n<h4><strong>Sliding window technique</strong></h4>\n<p>Sliding-window algorithm is time-based and similar to the fixed-window algorithm, but they differ in the starting point of each time window. Here the timeframe starts when a user makes a new request rather than at a predetermined time. For example, if the first request arrives at 7:00:18 am (and the rate limit is 200 per minute), the server will allow up to 200 requests until 7:01:18.</p>\n<p>This idea mitigates the starvation issue of the leaky bucket idea by starting a new time window whenever a request is made. So this is suitable for processing large requests while being lightweight and fast to run.</p>\n<p>Thanks to Navtosh for his contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"2a77b3d6-739b-5d43-97cb-4fb78d335109","frontmatter":{"title":"Throughput in System Design","path":"/blog/throughput-in-system-design"},"html":"<p>Have you ever observed the flow of water coming out of a pipe? The flow of water can vary, sometimes being less and sometimes being more, but there is a maximum capacity for the flow of water that a pipe can handle. This concept is similar to throughput in distributed systems.</p>\n<h3>Throughput meaning in System Design</h3>\n<p>Throughput is a measure of the rate at which something is processed. It is usually expressed in terms of the number of items that are processed in a given time period like the number of bits transmitted per second or the number of HTTP operations per day. In simple terms, throughput is the number of items processed over a given time interval. This calculation gives the average processing rate for that interval, but it can hide fluctuations like periods of faster or slower processing.</p>\n<p>Suppose an assembly line is manufacturing cars. Let’s consider the factory can able to produce around 100 cars per day. So the Throughput of the line is <strong>Throughput ~ 100 cars/day</strong>.</p>\n<h3>Misconceptions with Latency</h3>\n<p>It is commonly assumed that systems with high throughput should also have low latency. But, this is not always the case. For example, data processing using disks may have high throughput but also suffer from high latency. So latency can also increase with throughput!</p>\n<p>As the throughput increases, more packets are transmitted on the network, which can increase latency. On the other hand, it is also possible to have systems with low throughput and low latency. So it is important to consider both latency and throughput when designing a system and selecting the appropriate combination based on the requirements.</p>\n<h3>Factors affecting Throughput</h3>\n<p>Throughput of a system can be affected by several factors. These factors are: analog limitations, hardware processing power, service accessibility, network traffic, transmission errors, protocol overhead, etc. Protocol overhead refers to extra data that must be transmitted along with the actual message to ensure proper communication. This additional data can limit the maximum achievable throughput.</p>\n<h4>Analog limitations</h4>\n<p>The physical medium of a networked system can have a significant impact on its maximum achievable throughput. This can set an upper bound on the amount of information that can be transmitted. In other words, analog characteristics of the medium can limit amount of data that can be transmitted at a given time.</p>\n<h4>Hardware limitations</h4>\n<p>Every computing and processing system has limitations that can impact its throughput. This is because these systems have limited processing power and can only handle a certain amount of workload at a time.</p>\n<ul>\n<li>If a complex query requires a significant amount of computation, it can slow down the processing speed and ultimately affect the system throughput. </li>\n<li>When the workload becomes too much for the system to handle, its ability to process data may decrease, and its throughput will be affected.</li>\n</ul>\n<h4>High accessibility or concurrent requests</h4>\n<p>Service accessibility can also impact its throughput. When multiple users share a single communication system at the same time, they may need to share resources, which can reduce the systems ability to process and transmit data efficiently. This, in turn, can affect the throughput of the service.</p>\n<p>Increased accessibility to a service can also increase network traffic, which further decrease its throughput. For example, if demand for the service is high and multiple users are accessing it simultaneously, it can put strain on the system.</p>\n<h4>Other Factors</h4>\n<ul>\n<li>I/O operations like reading or writing to a disk can affect throughput. </li>\n<li>Throughput will get affected if a system does not have enough memory to store data. In this situation, system will need to constantly swap data in and out of disk. </li>\n<li>Use of inefficient algorithms can increase processing time and increase throughput.</li>\n<li>Due to lack of proper load balancing, some components become overloaded and affect throughput.</li>\n<li>If system relies on external services or APIs, the performance of these services can affect throughput.</li>\n</ul>\n<h3>How to increase throughput of a system?</h3>\n<ul>\n<li>Upgrade hardware components like processors, memory, and storage to increase processing speed.</li>\n<li>Use proper load-balancing techniques to evenly distributed workload among different components.</li>\n<li>Increase network bandwidth or upgrade network components to improve data transmission speed.</li>\n<li>Write efficient code and use optimised algorithms.</li>\n<li>Cache frequently used data in memory to reduce the time required for data retrieval.</li>\n<li>Break down a task into smaller sub-tasks and process them simultaneously (parallel processing).</li>\n<li>Minimize protocol overhead to increase the speed of data transmission.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>Throughput is a critical concept in the design of any system. It is used to measure the capacity and performance of a system. As such, architects and designers often strive to increase throughput as much as possible in order to improve the system's capacity.</p>\n<p>Thanks Suyash for his contribution in creating the first version of this content. If you have any queries or feedback, please write us at contact@enjoyalgorithms.com. Enjoy learning, Enjoy system design!</p>"},{"id":"7c8181d0-952f-5dda-8e30-414b0786c0bc","frontmatter":{"title":"WebSockets in System Design","path":"/blog/web-sockets-in-system-design"},"html":"<h3>What is WebSockets?</h3>\n<p>Traditional web is built around the HTTP request and response mechanism, where the client sends a request to the server and the server responds with the requested data. This happens in a separate connection for each request and response, which means HTTP is not well-suited for real-time applications that require continuous, bidirectional communication. To solve this problem, we use WebSockets.</p>\n<ul>\n<li>WebSocket provides bidirectional communication between the client and server. In other words, it provides a full-duplex communication channel over a single TCP connection and allows data transfer in both directions.</li>\n<li>Once a WebSocket connection is established between the client and server, it remains open until either the server or client terminates it. So this will help in continuous, real-time communication between the client and the server without the need for frequently re-establishing the connection.</li>\n<li>WebSocket and HTTP protocols have differences, but both rely on the TCP protocol at the 4th layer (transport layer) of the OSI model, and they exist at the 7th layer (application layer).</li>\n<li>WebSocket provides lightweight communication, which reduces the overhead associated with HTTP. </li>\n<li>By maintaining a persistent single TCP socket connection between the client and server, WebSocket provides a low-latency connection for responsive communication. It is a crucial component in developing real-time applications like multiplayer games or any application that requires quick, reliable, and bidirectional data transfer.</li>\n</ul>\n<h3>How does WebSocket works?</h3>\n<p><img src=\"https://ucarecdn.com/41eb1623-c997-4936-9602-8af30287a6df/\" alt=\"How websocket works?\" title=\"websockets visualization diagram\"></p>\n<ol>\n<li>To establish a WebSocket connection, the client first initiates a handshake with the server by sending an HTTP request with an \"Upgrade\" header to upgrade the connection to a WebSocket connection. Note: The client opens a WebSocket connection by calling the WebSocket constructor, which uses the URL schemas \"ws:\" or \"wss:\" (Just like \"http:\" or \"https:\" for HTTP).</li>\n<li>Once the client's request is received by the server, the server accepts it by sending a response with a \"101 Switching Protocols\" status code and a header indicating the use of WebSockets. </li>\n<li>With the established WebSocket connection, both the client and server can send messages to each other without the need for a new request-response cycle. </li>\n<li>The messages are sent in a binary format using the send() method on the connection object. In the latest specification, the WebSocket connection supports sending both string and binary messages using Blob or ArrayBuffer objects. </li>\n<li>The WebSocket connection remains open until either the client or server closes it.</li>\n</ol>\n<h3>Key Features of WebSockets</h3>\n<ul>\n<li><strong>Two-Way Data Exchange:</strong> WebSockets are essential in reducing network traffic by enabling the simultaneous transfer of data in both directions using a single connection.</li>\n<li><strong>HTTP Compatibility:</strong> WebSockets are highly compatible with previous versions of HTTP connections, enabling us to switch between HTTP and WebSockets.</li>\n<li><strong>Publish-Subscribe Event Pattern:</strong> WebSockets provide an efficient data transfer model by establishing communication channels. This allows messages to be sent to and from a server, as well as receiving event-driven responses without having to continuously poll the server.</li>\n</ul>\n<h3>Real-life applications of WebSockets</h3>\n<p>WebSockets are commonly used in various types of applications that require real-time, bidirectional communication between the client and server. Here are some examples:</p>\n<ul>\n<li><strong>Real-time messaging:</strong> Web Sockets are essential in real-time communication like instant chat applications. They provide fast data transfer, making it possible to build complex features like encrypted messages, typing indicators and more.</li>\n<li><strong>Multiplayer gaming:</strong> Web Sockets allow for low-latency and real-time synchronisation of game states between players. This results in interactive gaming experiences across multiple devices.</li>\n<li><strong>Live online maps:</strong> Real-time location data is crucial in building live online maps. Web Sockets allow for efficient transfer of this data, enabling the routing and navigation of moving assets in real-time.</li>\n<li><strong>Live updates:</strong> Web Sockets are valuable in providing real-time updates for various platforms like live election results, live score updates, etc.</li>\n</ul>\n<h3>Conclusion</h3>\n<p>Web Sockets have genuinely revolutionised web development. Using their stateful and bi-directional nature, they are beneficial in systems that require real-time updates or continuous data streams. However, if we only need to fetch only once, then a simple HTTP request would be considered compared to Web Sockets.</p>\n<p>Thanks Suyash Namdeo for his contribution in creating the first version of this content. Please write in the message below if you find anything incorrect or want to share more insight. Enjoy learning. Enjoy algorithms!</p>"},{"id":"7e8f272f-a923-5af6-bb32-0d9774d63946","frontmatter":{"title":"Designing Workflow in a Distributed System","path":"/blog/workflow-in-distributed-system"},"html":"<h3><strong>Introduction</strong></h3>\n<p>Ever wondered how does 1-click-buy works on Amazon? How does an e-commerce platform show the status of your order after the order is placed? What happens when you cancel your order right after you place an order, or after your item is shipped, or even delivered? How is all the activity related to an order tied to just one order Id? This blog will try to tackle such system design challenges and layout key insights on designing a workflow system.</p>\n<p>In a distributed software system, it is often required to achieve some goals with a combination of multiple operations, such as downstream calls, database transactions, and authentications. A workflow engine is an orchestration system that will help you hand out worker processes and manage the central state. It is possible to encounter failure, network glitch, or timeout, or the host running the software can fail. Therefore, a reliable workflow engine design is essential to the smooth performance of the overall software system.</p>\n<p>I will highlight the design considerations of a good workflow engine in this blog from the lessons I’ve learned in the past while working on designing highly scalable systems.</p>\n<p><strong>Simple Example: Payment workflow</strong></p>\n<p><img src=\"https://miro.medium.com/max/471/1*DS-JqehhY9nxuBcRt_PZig.png\" alt=\"Payment system workflow example\" title=\"Workflow of a payment system \"></p>\n<h3><strong>Design Challenges</strong></h3>\n<p>Given the distributed nature of the software system, the workflow engines face a series of challenges:</p>\n<ol>\n<li>The workflow state may be distributed and can be expensive to reconstruct if only part of the workflow work items has failed. A workflow restart should be started from the failed activities using the result of completed activities.</li>\n<li>Actions in a workflow are not always reliable. Because they go through network calls, the request can be lost, the remote system can be unresponsive, the response can be lost, or the latency can be high to cause a timeout. A failed workflow can produce various changes to the state, including NO-OP (action is not taken), expected operation (action is taken but the response is lost), and sometimes unexpected operation.</li>\n<li>The workflow engine can fail, so in-flight workflows must be picked up cleanly by another host running the same workflow instance.</li>\n<li>If more than one host runs the workflow engine, workflows must be efficiently and correctly distributed to them to ensure one and only one host works on any workflows at any time.</li>\n<li>Actions are rate-limited (throttled) by type or activities. Proper rate limiting must be done to protect the workflow engine’s dependencies.</li>\n<li>Some actions are asynchronous, and state transitions of such actions take time. The workflow engine must timely and efficiently go through state transitions.</li>\n</ol>\n<p><strong>Workflow Modelling</strong></p>\n<p>A workflow is a finite, ordered series of actions transforming the start state to the end state. It can be modeled as a directed acyclic graph with multiple paths between two special states: the initial state and the end state. Each state is a node, and each action is an edge. There can be multiple actions given some conditions for each state, but only one action can be taken on a state, and the choice of action is well-defined. The initial and end state of a workflow is well defined, and the consequence (“change set”) of each action is well-defined. Therefore, each workflow state is well-defined.</p>\n<p>The state of a workflow is the state of a collection of resources, database entries, or states in remote systems. An action in a workflow is a dependencies API call, a database transaction, etc., that changes at least one aspect of the state of the workflow, e.g., payment is completed. Item is shipped etc.</p>\n<h3><strong>Design Principles</strong></h3>\n<p>Following design principles should be considered to address the challenges:</p>\n<p><strong>Action</strong></p>\n<ol>\n<li>Each action should include only one mutating operation that, under normal circumstances, can either succeed as a whole or fail as a whole. This includes one API call, one database transaction, etc. Multiple operations can be grouped into one action if a transactional lock can be used to roll back partially succeeded operations if later operations in the same activity fail.</li>\n<li>After an action is taken successfully, the change set of the workflow state is well-defined. There should be no ambiguity of the change action.</li>\n<li>The failure mode of activity is finite and well-defined.</li>\n<li>Any API calls, database operations, etc., should be rate-limited to prevent downstream from bad clients.</li>\n<li>The result of asynchronous activities should be retrieved timely.</li>\n<li>Whenever possible, activity should be idempotent. If an action cannot be idempotent, it can either be taken multiple times without side effects, or the consequence of the extra calls does not impact the workflow state and can be efficiently cleaned up independently.</li>\n</ol>\n<p><strong>Workflow State</strong></p>\n<ol>\n<li>The workflow state should be cached in memory after an action is taken, and the next action can be taken without reconstructing the workflow state from the persistent storage.</li>\n<li>Workflow state should be persisted in the data store after each action is taken so that when it is assigned to a different host, its state can be efficiently retrieved without restarting the workflow.</li>\n<li>External conditions required to take different actions are well-defined, and their true state can be efficiently computed.</li>\n</ol>\n<h3><strong>Design Guideline</strong></h3>\n<p>A workflow engine can be divided into the following functional units.</p>\n<p><strong>Workflow Worker</strong></p>\n<p>Workflow workers perform actions and state transitions. There are a couple of ways of designing workflow workers (Let’s call them activity).</p>\n<ol>\n<li>We can build a workflow engine such that we can have individual workers dedicated to working on only one workflow item until the item completes.</li>\n<li>We can have individual workers work on only a small set of actions, and workflow items are worked by multiple workers in a defined order in a pipeline.</li>\n<li>workers are agnostic to the workflow item or the specific action; they receive the workflow state and the action, execute the action and record the state change, and move on to the next one, which can be the same workflow item, a different workflow item in a different state, or even a different type of workflow. Because the workers are completely dumb, this is called the “dumb worker” paradigm.</li>\n</ol>\n<p>There are both advantages and disadvantages to all techniques. In the dedicated worker approach, workflow state is maintained by a single worker, eliminating the need of passing state from one worker to the next. All the actions of this workflow must be defined in a single place, which can be good or bad depending on the situation. However, this process does not allow batch processing, and thus a workflow scheduler that distributes workflows to workers is required.</p>\n<p>In the assembly line approach, workflow actions are spread to multiple workers, reducing the readability of the code. It requires passing workflow state between workers but allows batch processing.</p>\n<p>It is recommended to model complicated, less frequent workflows with the dedicated worker approach and define many simple workflows using the assembly line approach.</p>\n<p><strong>Workflow Scheduling</strong></p>\n<p>You will always have some activities which run faster than other activities in a workflow. If you get n number of requests, then a particular activity from all the workflows may be completed, but the remaining activities are just piling up in the queue. To ensure workflows are organized desirably, prioritize on-the-fly tasks over new tasks, or distribute workflow tasks to multiple hosts running the workflow engine to avoid conflict, a workflow scheduler is helpful to manage workers so that workflows can be processed in the desired way.</p>\n<p>The scheduler should adjust the speed of each worker so that it is limited to the slowest one to ensure no accumulation of work items becomes operational debt. The number of concurrent workers should be limited so that no dependency is overwhelmed just because lightweight activities are completed quickly.</p>\n<p><strong>Capacity planning and Rate Limiter</strong></p>\n<p>There is usually a limit of capacity for any dependencies. This is often defined as allowed Transaction per Second, or TPS, from a given service. Thus, the workflow engine should implement a rate limiter at the client-side to efficiently use the fleet capacity and network bandwidth.</p>\n<p>The implementation of a Rate Limiter depends on the design of the worker. If workers are designed as threads, then the threads must be paused when allowed TPS is exceeded. If workers are designed as event handlers, API calls or database transactions should be registered as events scheduled with proper internal between them not to exceed the TPS requirement. The workflow engine should handle throttled requests with an appropriate back-off and retry mechanism.</p>\n<p><strong>Wait Queue</strong></p>\n<p>Activities in a Workflow blocked on state change of asynchronous operations must be handled efficiently. For example, if payment confirmation notification from the bank is delayed, the order shouldn’t be confirmed. This use case must be handled using a wait queue. Similarly, workflows that are waiting for an asynchronous operation to complete should be handled separately. The wait queue in the workflow engine should gather pending activities and perform state polling at a much slower rate than the workers. Wait queue should handle notification of completion. The wait queue should remove messages from the queue as the related activities are completed and notify the worker.</p>\n<p><strong>Questions</strong></p>\n<ol>\n<li>Design google play store app submission workflow system. A user needs to submit a binary file through the developer portal. List out a set of activities to be performed for the following workflows.</li>\n<li>a) app submission workflow,</li>\n<li>b) version update</li>\n<li>c) cancel a submission</li>\n</ol>\n<p><strong>References:</strong></p>\n<ul>\n<li><a href=\"https://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dev-about-workflows.html\">https://docs.aws.amazon.com/amazonswf/latest/developerguide/swf-dev-about-workflows.html</a></li>\n<li><a href=\"https://searchapparchitecture.techtarget.com/tip/API-workflow-design-patterns-for-microservices-scenarios\">https://searchapparchitecture.techtarget.com/tip/API-workflow-design-patterns-for-microservices-scenarios</a></li>\n<li><a href=\"https://www.sciencedirect.com/topics/computer-science/workflow-pattern\">https://www.sciencedirect.com/topics/computer-science/workflow-pattern</a></li>\n</ul>"}]}}}