CAP Theorem in System Design

CAP Theorem is one of the most important concepts in distributed system design.

It helps us understand what can happen when data is distributed across multiple servers or locations and communication between those servers fails.

CAP stands for:

• C — Consistency
• A — Availability
• P — Partition Tolerance

Why Do We Need CAP Theorem?

Imagine that our application stores data in two database nodes.

flowchart LR
    API --> Database1
    API --> Database2
    Database1 --> Database2

Both databases need to communicate so that their data stays synchronized.

Now imagine the network connection between them breaks.

flowchart LR
    API --> Database1
    API --> Database2
    Database1 -. Network Failure .-> Database2

This is called a network partition.

At this point, a distributed system has to make an important decision about how it behaves.

The Three CAP Concepts

1. Consistency

Consistency means that every successful read receives the latest valid data according to the system's consistency model.

Consider a bank account.

If ₹10,000 is the current balance, different clients should not unexpectedly receive conflicting values after a successful update.

flowchart LR
    User --> API
    API --> Database1
    API --> Database2
    Database1 --> Database2

When data is replicated, consistency determines how closely the copies need to agree.

2. Availability

Availability means that the system continues responding to requests even when some components have problems.

For example, if one database node is unavailable, another node may continue serving requests.

flowchart LR
    User --> API
    API --> Database1
    API --> Database2

3. Partition Tolerance

Partition tolerance means the system continues operating despite a communication failure between parts of the distributed system.

For example:

flowchart LR
    Database1 -. Network Partition .-> Database2

The network between the two nodes has failed, but the system must decide how to behave.

The Important Part of CAP

The most important thing to understand is this:

When a network partition occurs, a distributed system has to choose between consistency and availability.

Partition tolerance is not something we simply turn off in a distributed system where network failures are possible.

During a partition:

• Choosing consistency may mean refusing or delaying some operations until the system can safely coordinate.
• Choosing availability may mean continuing to serve requests even though different nodes may temporarily have different data.

Simple Example

Imagine two servers storing the same product inventory.

flowchart LR
    User --> API1
    User --> API2
    API1 --> Database1
    API2 --> Database2
    Database1 --> Database2

Suppose the inventory is:

Product A
Quantity = 10

Now the network connection between Database1 and Database2 fails.

A customer buys one item through API1.

Database1 now has:

Quantity = 9

But Database2 may still have:

Quantity = 10

Now another customer reaches API2.

Should API2 return the old quantity of 10?

Or should it refuse the operation until the databases can communicate again?

This is the type of trade-off CAP Theorem helps us understand.

Consistency-Oriented Behavior

Suppose inventory accuracy is extremely important.

During a partition, the system may choose to reject or delay operations rather than return potentially stale data.

flowchart TD
    Request --> CheckConnection
    CheckConnection --> Available
    CheckConnection --> Partition
    Available --> Process
    Partition --> Reject

This favors consistency during the partition.

Availability-Oriented Behavior

Another system may decide that serving the request is more important, even if the returned data can temporarily be stale.

flowchart TD
    Request --> CheckDatabase
    CheckDatabase --> Primary
    CheckDatabase --> Replica
    Primary --> Response
    Replica --> Response

This approach favors availability during the partition, with the possibility of temporary inconsistency.

CAP and ASP.NET Core

CAP Theorem is not an ASP.NET Core feature.

It is a distributed-system concept that influences how we design the architecture behind our ASP.NET Core APIs.

For example:

flowchart LR
    User --> API
    API --> Service1
    API --> Service2
    Service1 --> Database1
    Service2 --> Database2

If Service1 and Service2 depend on distributed data and communication between their data stores can fail, we need to decide how the system behaves during that failure.

CAP and SQL Server

A simple single SQL Server database is not normally the example used to explain CAP trade-offs.

CAP becomes particularly relevant when data is distributed across multiple nodes and those nodes need to communicate over a network.

For example:

flowchart LR
    API --> PrimaryDatabase
    PrimaryDatabase --> SecondaryDatabase
    SecondaryDatabase --> AnotherDatabase

Once data is distributed, network communication becomes an important part of the system's behavior.

CAP Does Not Mean "Pick Any Two"

A common interview statement is:

"CAP means you can choose any two of the three."

This is an oversimplification.

In a distributed system, network partitions can occur. The important CAP trade-off is what the system does when a partition happens.

During normal operation, a system may provide both strong consistency and high availability. The trade-off becomes important when communication between distributed components is interrupted.

Real-World Example

Consider a social media application.

A user posts a comment.

The system may replicate the comment across multiple servers.

flowchart LR
    User --> API
    API --> Database1
    Database1 --> Database2
    Database1 --> Database3

If replication is temporarily delayed, one user may see the comment immediately while another user may see it slightly later.

For a social media feed, temporary differences may be acceptable depending on the product requirements.

For financial transactions, the consistency requirements can be much stricter.

CAP Is About Trade-Offs

The correct question is not:

"Is this system CP or AP?"

The better question is:

"What should this system do when distributed components cannot communicate?"

For example:

System Type	Possible Priority During Partition
Banking transaction	Protect correctness and consistency
Social media feed	Continue serving users with possible temporary staleness
Inventory system	Depends on business rules and overselling risk

The actual design depends on the requirements of the system.

CAP vs ACID

CAP and ACID are different concepts.

ACID describes properties of database transactions:

• Atomicity
• Consistency
• Isolation
• Durability

CAP describes a trade-off in distributed systems when a network partition occurs.

The word "Consistency" appears in both, but it does not mean exactly the same thing.

Key Takeaways

• CAP stands for Consistency, Availability and Partition Tolerance.
• CAP is mainly relevant to distributed systems.
• A network partition means distributed components cannot communicate normally.
• During a partition, the system has to make a trade-off between consistency and availability.
• CAP does not simply mean choosing any two characteristics at all times.
• The correct design depends on the business requirements.
• Financial systems may require stronger consistency guarantees for certain operations.
• Some systems can continue serving users with temporarily stale data.

Conclusion

CAP Theorem helps us understand an important reality of distributed systems: network failures can happen, and our architecture needs to define what happens when they do.

When distributed components cannot communicate, we need to decide whether the system should prioritize maintaining consistent data or continuing to serve requests.

There is no single answer that is correct for every application. The decision depends on what the business considers more important during a network partition.

Next, we will look at another fundamental System Design concept: Horizontal vs Vertical Scaling.