Single Server setup

With the growth of the user base, one server is not enough.

Vertical scaling VS Horizontal scaling
Vertical scaling: adding more power (CPU, RAM, etc.) to your servers.
Horizontal scaling: adding more servers into your pool of resources.
Load balancer

Database replication


Cache Tier

The benefits of separate cache tier:
Consideration for Cache using
When to use
When data is read frequently but modified infrequently. But important data should be saved in persistent data stores. (Since cached data is stored in volatile/easy-loss memory.)
Expiration policy
Do not make the expiration date too short as this will cause the system to reload data from the database too frequently, which will decrease the efficiency of cache.
Do not make the expiration date too long as the data can become stale.
Consistency
Inconsistency can happen because data-modifying operations on the data store and cache are not in a single transaction.
Mitigating (Reducing) failures
A single cache server represents a potential single point of failure (SPOF), multiple cache servers across different data centers are recommended.
Eviction Policy
Least-recently-used (LRU), Least Frequently Used (LFU) or First in First Out (FIFO).
Content delivery network (CDN)
A CDN is a network of geographically dispersed servers used to deliver static content. CDN servers’ cache static content like images, videos, CSS, JavaScript files, etc.


Considerations of using CDN
Cost
CDNs are run by third-party providers.
Caching infrequently used assets provides no significant benefits.
Setting an appropriate cache expiry
CDN fallback
If there is a temporary CDN outage, clients should be able to detect the problem and request resources from the origin.
Invalidating files
Remove a file from the CDN before it expires.

Stateless web tier
A good practice is to store session data in the persistent storage such as relational database or NoSQL. Each web server in the cluster can access state data from databases. This is called stateless web tier.
Stateful architecture

The issue is that every request from the same client must be routed to the same server. This can be done with sticky sessions in most load balancers; however, this adds the overhead(cost). Adding or removing servers is much more difficult with this approach. It is also challenging to handle server failures.
Stateless architecture


We move the session data out of the web tier and store them in the persistent data store. The shared data store could be a relational database. The NoSQL data store is chosen as it is easy to scale. After the state data is removed out of web servers, auto-scaling of the web tier is easily achieved by adding or removing servers based on traffic load.
Data centers

Several technical challenges must be resolved to achieve multi-data center setup:
Traffic redirection
Effective tools are needed to direct traffic to the correct data center.
Data synchronization
Users from different regions could use different local databases or caches. In failover cases, traffic might be routed to a data center where data is unavailable. A common strategy is to replicate data across multiple data centers.
Test and deployment
It is important to test your website/application at different locations. Automated deployment tools are vital to keep services consistent through all the data centers.
Message queue
A message queue is a durable(persistent) component, stored in memory, that supports asynchronous communication.

Decoupling makes the message queue a preferred architecture for building a scalable and reliable application.
Logging, metrics, automation
Logging
Monitoring error logs is important because it helps to identify errors and problems.
Metrics
Collecting different types of metrics help us to gain business insights and understand the health status of the system.
CPU, Memory, disk I/O, etc.
the performance of the entire database tier, cache tier, etc.
daily active users, retention, revenue, etc.
Automation

To conclude this chapter, we provide a summary of how we scale our system to support millions of users: