• Single Server setup

  

• With the growth of the user base, one server is not enough.

  

  • Which database to use?

• Vertical scaling VS Horizontal scaling

  Vertical scaling: adding more power (CPU, RAM, etc.) to your servers.

  • Impossible to add unlimited CPU and memory to a single server.
  • Does not have failover and redundancy.

  Horizontal scaling: adding more servers into your pool of resources.

• Load balancer

  

  • A load balancer evenly distributes incoming traffic among web servers that are defined in a load-balanced set.
  • If server 1 goes offline, all the traffic will be routed to server 2. This prevents the website from going offline. We will also add a new healthy web server to the server pool.
  • If the website traffic grows rapidly, and two servers are not enough to handle the traffic, the load balancer can handle this problem gracefully.

• Database replication

  

  • Better performance: Allows more queries to be processed in parallel.
  • Reliability: Do not need to worry about data loss because data is replicated across multiple locations.
  • High availability: Even if a database is offline as you can access data stored in another database server.

  

• Cache Tier

  

  The benefits of separate cache tier:

  • Better system performance
  • Reduce database workloads
  • Scale the cache tier independently.

• Consideration for Cache using

  • When to use

    When data is read frequently but modified infrequently. But important data should be saved in persistent data stores. (Since cached data is stored in volatile/easy-loss memory.)

  • Expiration policy

    Do not make the expiration date too short as this will cause the system to reload data from the database too frequently, which will decrease the efficiency of cache.

    Do not make the expiration date too long as the data can become stale.

  • Consistency

    Inconsistency can happen because data-modifying operations on the data store and cache are not in a single transaction.

  • Mitigating (Reducing) failures

    A single cache server represents a potential single point of failure (SPOF), multiple cache servers across different data centers are recommended.

  • Eviction Policy

    Least-recently-used (LRU), Least Frequently Used (LFU) or First in First Out (FIFO).

• Content delivery network (CDN)

  A CDN is a network of geographically dispersed servers used to deliver static content. CDN servers’ cache static content like images, videos, CSS, JavaScript files, etc.

  

  

• Considerations of using CDN

  • Cost

    CDNs are run by third-party providers.

    Caching infrequently used assets provides no significant benefits.

  • Setting an appropriate cache expiry

  • CDN fallback

    If there is a temporary CDN outage, clients should be able to detect the problem and request resources from the origin.

  • Invalidating files

    Remove a file from the CDN before it expires.

  

  1. Static assets (JS, CSS, images, etc.,) are no longer served by web servers. They are fetched from the CDN for better performance.
  2. The database load is lightened by caching data.

• Stateless web tier

  A good practice is to store session data in the persistent storage such as relational database or NoSQL. Each web server in the cluster can access state data from databases. This is called stateless web tier.

• Stateful architecture

  

  The issue is that every request from the same client must be routed to the same server. This can be done with sticky sessions in most load balancers; however, this adds the overhead(cost). Adding or removing servers is much more difficult with this approach. It is also challenging to handle server failures.

• Stateless architecture

  

  

  We move the session data out of the web tier and store them in the persistent data store. The shared data store could be a relational database. The NoSQL data store is chosen as it is easy to scale. After the state data is removed out of web servers, auto-scaling of the web tier is easily achieved by adding or removing servers based on traffic load.

• Data centers

  

  Several technical challenges must be resolved to achieve multi-data center setup:

  • Traffic redirection

    Effective tools are needed to direct traffic to the correct data center.

  • Data synchronization

    Users from different regions could use different local databases or caches. In failover cases, traffic might be routed to a data center where data is unavailable. A common strategy is to replicate data across multiple data centers.

  • Test and deployment

    It is important to test your website/application at different locations. Automated deployment tools are vital to keep services consistent through all the data centers.

• Message queue

  A message queue is a durable(persistent) component, stored in memory, that supports asynchronous communication.

  

  Decoupling makes the message queue a preferred architecture for building a scalable and reliable application.

• Logging, metrics, automation

  • Logging

    Monitoring error logs is important because it helps to identify errors and problems.

  • Metrics

    Collecting different types of metrics help us to gain business insights and understand the health status of the system.

    CPU, Memory, disk I/O, etc.

    the performance of the entire database tier, cache tier, etc.

    daily active users, retention, revenue, etc.

  • Automation

To conclude this chapter, we provide a summary of how we scale our system to support millions of users:

• Keep web tier stateless
• Build redundancy at every tier
• Cache data as much as you can