What is Caching in System Design?

Have you ever experienced that when you open a website the first time, it takes more time than usual, but when you open the same website again, it takes no time to load? Do you know why this happens? Let’s find out! In this blog, we will learn about caching, an important fundamental concept in system design.

How Caching Works?

Caching is the process of storing the results of a request in a cache or a temporary storage location so they can be accessed more quickly. So critical question is: What is cache? In system design, cache is a high-speed data storage that stores a subset of data so that future requests for that data are served up faster. In other words, caching allows us to reuse previously retrieved data efficiently.

What is caching?

Caching Examples

  • Web Browsers cache the HTML, CSS, JS, and images for faster access to the website when requested again.
  • CDN store static files and help us to reduce latency.
  • DNS is used to get the IP address of a query. So, rather than requesting the IP address multiple times, it can be stored in a cache storage, allowing us not to re-perform a DNS query again, and the web pages can be accessed more quickly.

Real-world Analogy of Caching

Let us take an example of a library to understand the idea behind the cache system. Suppose there is a library with thousands of books, and a librarian behind the desk assigns the desired book from the library store. First, let’s start with a librarian without a cache.

The day starts, and the first learner arrives. They ask for some book A. The librarian goes to the storeroom, fetches the book, returns it to the desk, and issues it to the learner. After some days, the learner returns the book, and the librarian keeps it back in its place. Now, the next learner arrives and asks for the same book A. The librarian again has to go to the same place, fetch the book, and give it to the learner. In this system, the librarian must visit the store whenever a learner arrives.

Now, let’s put the librarian on the desk with a bag that can store 15 books (like a 15-book cache). In this bag, the librarian can keep the books that learners have recently returned. Now, the first learner arrives and requests book A, and the librarian has to go to the storeroom and give it to the learner. Later, when the learner returns the book, the librarian keeps it in the bag instead of going to the storeroom to return it. Another learner arrives and asks for the same book A. This time the librarian checks for the book in the bag and finds it! This time the librarian doe not have to make a round trip to the storeroom, and the learner can be served more efficiently.

What are different types of Caching?

Application server cache

We can cache the data directly in the Application Layer. Every time a request is made to the service, it will return local, cached data quickly if it exists. If it is not in the cache, it will query the data from the database.

Global caches

In Global Caches, the same single cache space is used for all the nodes. Each of the application nodes queries the cache in the same way as a local one would be.

Distributed cache

The cache is usually broken up using a consistent hashing algorithm, and each of its nodes owns part of the cached data. If a requesting node is searching for a certain piece of data, it can easily use the hashing function to locate information from the distributed cache to decide if the data is available.

Content Distribution Network (CDN)

When our pages serve huge amounts of static media, this is the best option. Suppose that the framework we are developing is not yet big enough to have a CDN of its own! “Using a lightweight HTTP server like apache, we can serve static media from a different subdomain such as “blog.enjoyalgorithms.com” and cut the DNS from your servers to a CDN layer.

Client-Side Caches

Client-side caching duplicates the data of previously requested files directly within browser applications or other clients (such as intermediate network caches).

ISP layer cache

ISP caching works in much the same way as browser caching. Once you have visited a website, your ISP may cache those pages so that they appear to load faster the next time you visit them. The main problem with this is that, unlike your browser cache, you can not delete these temporary files; instead, you have to wait until your ISP's cache expires and requests fresh copies of the files.

Cache Eviction Policy

We need to delete existing items for new resources when the cache is complete. In fact, it is just one of the most popular methods to delete the least recently used object. The solution is to optimize the probability in the cache that the requesting resource exists.

  • Random Replacement (RR) : As the term suggests, we can randomly delete an entry.
  • Least Frequently Used (LFU): We can keep a count of how frequently an item is requested and delete the least frequently used.
  • Least Recently Used (LRU): In LRU, we delete the item that has been used least recently.
  • First In First Out (FIFO): The FIFO algorithm holds an object queue in the order that the objects have been loaded into the cache. It evicts one or more objects from the head when a cache misses and inserts a new object into the queue tail. Upon a cache hit, the list does not shift.

What is Cache Invalidation?

Cache invalidation refers to when web cache proxies declare cached content as invalid, meaning it will no longer be served as the most recent piece of content when requested. The ultimate purpose, of course, is to ensure that the next time a client requests the affected content, the client receives the newest version at all times. There are primarily three kinds of systems for caching:

  1. Write through cache: The writes go through the cache, and only if writes to DB and the cache both succeed, write verified as a success. Between cache and storage, we will have full data consistency. Nothing can get lost in case of a crash, power failure, or other system disturbances. In this case, however, writing latency would be higher since two different systems are written.
  2. Write around cache: The write directly goes to the DB, bypassing the cache. Cache misses are increased because, in a cache error, the cache device reads the information from the Database. Consequently, in applications that quickly write and re-read the data, this can lead to higher reading latency. Reading must take place through slower back-end storage and higher latency.
  3. Write back cache: The write is rendered directly to the cache layer, and as soon as the write to the cache is finished, the write is verified. The cache then synchronizes this writing to the DB asynchronously. For write-intensive applications, this will result in rapid write latency and high write throughput. However, if the caching layer dies, there is a chance of losing the data since the cache is the only single copy of the written data. By having more than one replica that recognizes the writing in the cache, we can maximize this.

Implementation of a simple Cache using NodeJS

Let’s implement a simple caching system:

First, we will create a simple server and a database. We will use the database to get the HTML page and the server to host the page locally. We will create two end-points, one that uses the cache and the other that does not.


Cache Implementation Using NodeJS

If we look in the browser on the server with the endpoint having no-cache, it will take 3 seconds to load (since we used setTimeout to load the page with no cache in 3 seconds). If you refresh this page again, it will again take 3 seconds to load the page because every time we refresh the page, it goes to the database to fetch it.

Now, in the end-point, which uses caching, the first time we go to the endpoint with cache, it takes 3 seconds to load the page since the cache is empty and has to go to the database to fetch the data. But, when we refresh the page, it loads instantly. As soon as we loaded the page the first time, we cached the results for future requests.

Advantages of Caching

  • Improve Application Performance : Caching can be used to improve system performance and API latency.
  • Reduce Database Cost : Caching can take up additional traffic to its cache server and reduce database traffic, eventually reducing database cost.
  • Reduce the Load on the Backend:  Offloading the same request traffic from the main server to caching server would reduce the backend load.
  • Increase Read Throughput (IOPS):  Caching server responds much faster than the main server for the cached key, which increases read throughput.

Thanks to Navtosh Kumar for his contribution to creating the first version of the content. Please write in the message below if you find anything incorrect, or if you want to share more insight. Enjoy learning, Enjoy algorithms!

Share feedback with us

More blogs to explore

Our weekly newsletter

Subscribe to get weekly content on data structure and algorithms, machine learning, system design and oops.

© 2022 Code Algorithms Pvt. Ltd.

All rights reserved.