Health Checks 🩺 — Detecting Unhealthy Servers

A health check allows a load balancer or service to determine whether a backend instance is healthy enough to receive traffic.

flowchart LR
    LB["Load Balancer"] -->|Health Check| S1["Server 1"]
    LB -->|Health Check| S2["Server 2"]
    LB -->|Health Check| S3["Server 3"]

    S1 --> H1["Healthy"]
    S2 --> H2["Unhealthy"]
    S3 --> H3["Healthy"]

1. Why Are Health Checks Needed?

A server can be running but still unable to process requests correctly. For example, the application may be unable to connect to its database.

flowchart LR
    LB["Load Balancer"] --> S["Server"]

    S --> A["Application"]
    S --> DB["Database"]

    DB -->|Failure| S

A health check helps prevent traffic from being sent to an unhealthy instance.

2. Basic Health Check

A common approach is to expose an endpoint such as:

GET /health

If the service responds successfully, the load balancer can consider the instance healthy.

sequenceDiagram
    participant LB as Load Balancer
    participant API as API Server

    LB->>API: GET /health
    API-->>LB: 200 OK
    LB->>API: Send User Request

3. Liveness vs Readiness

This distinction is especially important in containerized and Kubernetes environments.

4. Shallow vs Deep Health Check

Shallow check: verifies that the application process is running.

flowchart LR
    LB["Load Balancer"] --> API["API"]
    API --> H["Application Alive"]
    H --> R["Healthy"]

Deep check: verifies important dependencies such as databases or required services.

flowchart LR
    LB["Load Balancer"] --> API["API"]

    API --> DB["Database"]
    API --> C["Cache"]
    API --> D["Dependency Service"]

    DB --> R["Ready"]
    C --> R
    D --> R

Deep checks provide more information but should be designed carefully so health checks do not themselves overload dependencies.

5. What Happens When a Server Becomes Unhealthy?

flowchart LR
    LB["Load Balancer"] --> S1["Server 1"]
    LB --> S2["Server 2"]
    LB --> S3["Server 3"]

    S2 --> X["Health Check Failed"]

    LB -. "Stop Sending Traffic" .-> S2

The load balancer removes or temporarily excludes the unhealthy instance from normal traffic distribution.

6. Recovery

When the server becomes healthy again, subsequent successful health checks can allow it to receive traffic again.

sequenceDiagram
    participant LB as Load Balancer
    participant API as Server

    LB->>API: Health Check
    API-->>LB: Failure
    Note over LB: Remove from traffic

    LB->>API: Health Check
    API-->>LB: Success
    Note over LB: Add back to traffic

7. Important Health Check Settings

• Interval — how frequently the check runs
• Timeout — how long to wait for a response
• Healthy threshold — successful checks required to become healthy
• Unhealthy threshold — failed checks required to become unhealthy

8. Health Checks in Auto Scaling

Health checks become especially important when instances are automatically added and removed.

flowchart LR
    ALB["Load Balancer"] --> ASG["Auto Scaling Group"]

    ASG --> S1["Instance 1"]
    ASG --> S2["Instance 2"]
    ASG --> S3["Instance 3"]

    S2 --> X["Unhealthy"]

    ASG --> R["Replace / Recover Instance"]

Important Interview Questions

• Why do we need health checks?
• What is a health-check endpoint?
• What is the difference between liveness and readiness?
• What happens when a server fails its health check?
• What is a deep health check?
• What are healthy and unhealthy thresholds?
• How does health checking work with Auto Scaling?
• Should a health check call the database?
• How does Kubernetes use liveness and readiness probes?

Key Takeaways

• Health checks prevent traffic from reaching unhealthy instances.
• Liveness checks whether the application is alive.
• Readiness checks whether it can receive traffic.
• Health checks can be shallow or dependency-aware.
• Thresholds prevent temporary failures from causing unnecessary removal.
• Healthy instances can be automatically added back into traffic.

Conclusion

In an interview, remember: Health Check → Detect Failure → Stop Traffic → Recover → Rejoin Traffic.