Load Balancing — How Do We Handle Millions of Requests?

A Load Balancer distributes incoming client requests across multiple servers instead of sending all traffic to a single server.

This improves scalability, availability, and reliability.

1. Why Do We Need a Load Balancer?

Suppose one server receives thousands of requests.

flowchart LR
Client["Users"]
Server["Single Server"]

Client --> Server

If traffic increases significantly, the server can become overloaded or unavailable.

Instead, we can add multiple application servers.

flowchart LR
Client["Users"]
LB["Load Balancer"]
Server1["Server 1"]
Server2["Server 2"]
Server3["Server 3"]

Client --> LB
LB --> Server1
LB --> Server2
LB --> Server3

2. What Does a Load Balancer Do?

The load balancer receives client traffic and decides which backend server should handle each request.

sequenceDiagram
Client->>LoadBalancer: HTTP Request
LoadBalancer->>Server1: Forward Request
Server1-->>LoadBalancer: Response
LoadBalancer-->>Client: Response

3. Load Balancing Algorithms

Common algorithms include:

• Round Robin
• Weighted Round Robin
• Least Connections
• IP Hash

Round Robin

Requests are distributed sequentially across servers.

flowchart LR
LB["Load Balancer"]
S1["Server 1"]
S2["Server 2"]
S3["Server 3"]

LB --> S1
LB --> S2
LB --> S3
LB --> S1
LB --> S2

Least Connections

The load balancer sends new requests toward a server with fewer active connections.

flowchart LR
LB["Load Balancer"]
S1["Server 1 - 100 connections"]
S2["Server 2 - 20 connections"]
S3["Server 3 - 50 connections"]

LB --> S2

4. Health Checks

A load balancer should avoid sending traffic to unhealthy servers.

flowchart LR
LB["Load Balancer"]
S1["Server 1 - Healthy"]
S2["Server 2 - Unhealthy"]
S3["Server 3 - Healthy"]

LB --> S1
LB -.-> S2
LB --> S3

The load balancer periodically performs health checks against backend servers.

5. Load Balancer and Horizontal Scaling

Load balancing is commonly combined with horizontal scaling.

flowchart LR
Users["Users"]
LB["Load Balancer"]
ASG["Auto Scaling"]
Servers["Multiple API Servers"]

Users --> LB
LB --> ASG
ASG --> Servers

When traffic increases, additional application instances can be added and the load balancer can distribute traffic across them.

6. Layer 4 vs Layer 7 Load Balancing

7. Layer 7 Routing

A Layer 7 load balancer can route requests based on information such as the hostname or URL path.

flowchart TD
Client["Client"]
LB["Layer 7 Load Balancer"]
Order["Order Service"]
Payment["Payment Service"]
User["User Service"]

Client --> LB
LB -->|/orders| Order
LB -->|/payments| Payment
LB -->|/users| User

8. Load Balancer and HTTPS

A load balancer can accept HTTPS traffic from clients and forward traffic to backend services.

flowchart LR
Client["Client"]
LB["HTTPS Load Balancer"]
API1["API Server 1"]
API2["API Server 2"]

Client -->|HTTPS :443| LB
LB --> API1
LB --> API2

In many architectures, TLS can be terminated at the load balancer before traffic is forwarded to backend servers.

9. Load Balancer and Session State

If an application stores user session state only in local server memory, requests from the same user may reach different servers.

flowchart LR
Client["Client"]
LB["Load Balancer"]
S1["Server 1"]
S2["Server 2"]

Client --> LB
LB --> S1
LB --> S2

Stateless application design is generally easier to scale horizontally. When server-side session state is required, shared session storage or another appropriate design can be used.

10. Sticky Sessions

Sticky sessions attempt to keep a client's requests associated with the same backend server.

flowchart LR
Client["Client"]
LB["Load Balancer"]
S1["Server 1"]
S2["Server 2"]

Client --> LB
LB --> S1
Client --> LB
LB --> S1

Sticky sessions can help applications that depend on local session state, but they can also make traffic distribution and failover more complicated.

11. Load Balancer Failure

The load balancer itself can become a single point of failure if it is deployed as only one instance.

Production systems therefore commonly use highly available or managed load-balancing infrastructure.

flowchart LR
Users["Users"]
LB["Highly Available Load Balancer"]
S1["API Server 1"]
S2["API Server 2"]
S3["API Server 3"]

Users --> LB
LB --> S1
LB --> S2
LB --> S3

12. Load Balancer in AWS

AWS provides several load-balancing options. For HTTP/HTTPS applications, an Application Load Balancer (ALB) is commonly used.

flowchart LR
User["Users"]
ALB["AWS Application Load Balancer"]
EC2_1["EC2 Instance 1"]
EC2_2["EC2 Instance 2"]
EC2_3["EC2 Instance 3"]

User --> ALB
ALB --> EC2_1
ALB --> EC2_2
ALB --> EC2_3

13. Load Balancer with Auto Scaling

flowchart LR
Users["Users"]
ALB["Application Load Balancer"]
ASG["Auto Scaling Group"]
Instances["EC2 Instances"]

Users --> ALB
ALB --> ASG
ASG --> Instances

The Auto Scaling Group manages the number of application instances while the load balancer distributes traffic across healthy instances.

14. Important Interview Questions

• What is a load balancer?
• Why do we need load balancing?
• What is horizontal scaling?
• What are common load-balancing algorithms?
• What is health checking?
• What is Layer 4 vs Layer 7 load balancing?
• What is sticky session?
• Why is statelessness important with load balancing?
• What happens if one backend server goes down?
• How do you prevent the load balancer from becoming a single point of failure?
• How does a load balancer work with Auto Scaling?
• What is an AWS Application Load Balancer?

Key Takeaways

• A load balancer distributes traffic across multiple backend servers.
• Health checks prevent traffic from being sent to unhealthy instances.
• Round Robin and Least Connections are common algorithms.
• Layer 4 works at the transport level, while Layer 7 understands application protocols such as HTTP.
• Load balancing enables horizontal scaling.
• Stateless applications are easier to scale behind a load balancer.
• Load balancers are commonly combined with Auto Scaling.

Conclusion

A load balancer acts as the entry point between clients and multiple application servers, distributing traffic while helping the system remain scalable and available.

Users
  ↓
Load Balancer
  ↓
Multiple Application Servers
  ↓
Database / Other Services