Reliability in System Design

Reliability means the ability of a system to consistently perform the correct operation and produce the expected result.

A system can be available but still not be reliable.

For example, a banking application may be running and accepting requests, but if a money transfer sometimes deducts money without completing the transfer, the system is not reliable.

Reliability is about correctness, consistency and dependable behavior.

What Is Reliability?

Consider an online payment system.

A customer pays ₹1,000.

The expected result is:

• ₹1,000 is deducted from the customer.
• Payment is recorded.
• The merchant receives the payment status.
• The customer receives confirmation.

If the system sometimes performs only some of these operations, we have a reliability problem.

flowchart LR
    User --> PaymentAPI
    PaymentAPI --> PaymentService
    PaymentService --> Database
    PaymentService --> Bank

Availability vs Reliability

Availability and reliability are related, but they are not the same.

Availability: Is the system available to receive requests?

Reliability: Does the system correctly process those requests?

For example, an API can return HTTP 200 for every request and still be unreliable if it stores incorrect data.

Availability	Reliability
System is accessible	System behaves correctly
Focuses on uptime	Focuses on correct and dependable operation
Can users reach the system?	Can users trust the result?

Real-World Example: Money Transfer

Suppose a customer transfers ₹5,000 from Account A to Account B.

The operation should be:

flowchart TD
    Request --> Debit
    Debit --> Credit
    Credit --> Record
    Record --> Success

But imagine the application deducts ₹5,000 from Account A and then crashes before crediting Account B.

Now the system has produced an incorrect result.

This is a reliability problem.

Database Transaction

SQL Server transactions can help keep related database operations together.

For example:

BEGIN TRANSACTION;
UPDATE Accounts
SET Balance = Balance - 5000
WHERE Id = 1;
UPDATE Accounts
SET Balance = Balance + 5000
WHERE Id = 2;
COMMIT TRANSACTION;

If an error occurs before the transaction is committed, we can roll back the transaction.

BEGIN TRANSACTION;
UPDATE Accounts
SET Balance = Balance - 5000
WHERE Id = 1;
UPDATE Accounts
SET Balance = Balance + 5000
WHERE Id = 2;
IF @@ERROR <> 0
BEGIN
    ROLLBACK TRANSACTION;
END
ELSE
BEGIN
    COMMIT TRANSACTION;
END

This helps ensure that the two database operations are treated as one unit.

ASP.NET Core Example

The same concept can be implemented using Entity Framework Core.

public async Task TransferMoney(
    int fromAccount,
    int toAccount,
    decimal amount)
{
    await using var transaction =
        await _context.Database.BeginTransactionAsync();
    try
    {
        var source = await _context.Accounts
            .FirstAsync(x => x.Id == fromAccount);
        var destination = await _context.Accounts
            .FirstAsync(x => x.Id == toAccount);
        if (source.Balance < amount)
            throw new Exception("Insufficient balance");
        source.Balance -= amount;
        destination.Balance += amount;
        await _context.SaveChangesAsync();
        await transaction.CommitAsync();
    }
    catch
    {
        await transaction.RollbackAsync();
        throw;
    }
}

If the operation fails before the commit, the transaction is rolled back.

What Can Make a System Unreliable?

Several problems can affect reliability.

• Application crashes
• Database failures
• Network failures
• Timeouts
• Duplicate requests
• Race conditions
• Partial failures
• Incorrect retry logic
• Data corruption
• External service failures

Handling External Service Failures

Consider an order application that calls a payment provider.

flowchart LR
    User --> OrderAPI
    OrderAPI --> PaymentService
    PaymentService --> PaymentProvider

What happens if the payment provider does not respond?

The application should not wait forever.

We can use a timeout.

using var cts =
    new CancellationTokenSource(
        TimeSpan.FromSeconds(5));
var response = await httpClient.PostAsync(
    "payment",
    content,
    cts.Token);

If the external service does not respond within the expected time, the request can fail or move into an appropriate recovery flow.

Retry and Reliability

Some failures are temporary.

For example, a network request may fail once and succeed immediately afterward.

A retry can help in such cases.

flowchart TD
    Request --> Service
    Service --> Success
    Service --> Failure
    Failure --> Retry
    Retry --> Service

But blindly retrying is dangerous.

Imagine a payment request is successfully processed by the payment provider, but the response is lost because of a network failure.

If we simply send the same payment request again, we could potentially create a duplicate payment.

This is why retries often need to be combined with idempotency.

Idempotency

Idempotency means that sending the same operation multiple times should not produce an unintended additional effect.

For example, a payment request could contain a unique idempotency key.

{
    "orderId": 1001,
    "amount": 5000,
    "idempotencyKey": "PAY-1001-ABC123"
}

The system can store the key.

If the same request arrives again, the system can recognize that the operation was already processed.

flowchart TD
    Request --> CheckKey
    CheckKey --> Existing
    CheckKey --> NewRequest
    Existing --> ReturnPreviousResult
    NewRequest --> ProcessPayment
    ProcessPayment --> StoreKey
    StoreKey --> ReturnResult

This is especially important for payment, order creation and other operations where duplicate processing can cause business problems.

Reliability and SQL Server

Reliability also depends on protecting data from incorrect operations.

For example, suppose two users try to purchase the last available product at the same time.

Both requests may read:

Available Quantity = 1

Both may then attempt to reduce the quantity.

This can create incorrect inventory.

Database concurrency control and appropriate transaction strategies are important for such scenarios.

Monitoring Reliability

We cannot improve reliability if we don't know when failures are happening.

Production systems should monitor things such as:

• Failed requests
• Database errors
• Timeouts
• Exception rates
• Failed background jobs
• Payment failures
• Message processing failures

flowchart LR
    Application --> Logs
    Application --> Metrics
    Application --> Alerts
    Metrics --> Dashboard

For example, if payment failures suddenly increase, monitoring should help the engineering team identify the problem quickly.

Reliability in Distributed Systems

As systems become larger, they contain more components.

flowchart LR
    User --> API
    API --> OrderService
    OrderService --> PaymentService
    OrderService --> InventoryService
    OrderService --> NotificationService
    PaymentService --> Database
    InventoryService --> Database

Each component can fail independently.

This is called a partial failure.

For example, the order service may be working while the notification service is unavailable.

A reliable system should handle such failures gracefully instead of unnecessarily bringing down the entire application.

Key Takeaways

• Reliability means consistently producing the correct and expected result.
• Availability means the system is accessible.
• A system can be available but unreliable.
• Database transactions help keep related operations consistent.
• Timeouts prevent the application from waiting indefinitely for external services.
• Retries can help with temporary failures but must be designed carefully.
• Idempotency helps prevent duplicate operations.
• Monitoring and alerting are important for detecting reliability problems.
• Distributed systems must handle partial failures.

Conclusion

Reliability is about making users confident that the system will do what it is supposed to do.

A reliable system should handle failures without producing incorrect results, duplicate operations or corrupted data.

Transactions, idempotency, timeouts, controlled retries, monitoring and proper failure handling are some of the important techniques used to improve reliability.

Once a system is reliable, we also need to ask another important question:

How quickly can the system process requests?

That brings us to the next concept: Performance.

Reliability means the ability of a system to consistently perform the correct operation and produce the expected result.

A system can be available but still not be reliable.

For example, a banking application may be running and accepting requests, but if a money transfer sometimes deducts money without completing the transfer, the system is not reliable.

Reliability is about correctness, consistency and dependable behavior.

What Is Reliability?

Consider an online payment system.

A customer pays ₹1,000.

The expected result is:

• ₹1,000 is deducted from the customer.
• Payment is recorded.
• The merchant receives the payment status.
• The customer receives confirmation.

If the system sometimes performs only some of these operations, we have a reliability problem.

flowchart LR
    User --> PaymentAPI
    PaymentAPI --> PaymentService
    PaymentService --> Database
    PaymentService --> Bank

Availability vs Reliability

Availability and reliability are related, but they are not the same.

Availability: Is the system available to receive requests?

Reliability: Does the system correctly process those requests?

For example, an API can return HTTP 200 for every request and still be unreliable if it stores incorrect data.

Availability	Reliability
System is accessible	System behaves correctly
Focuses on uptime	Focuses on correct and dependable operation
Can users reach the system?	Can users trust the result?

Real-World Example: Money Transfer

Suppose a customer transfers ₹5,000 from Account A to Account B.

The operation should be:

flowchart TD
    Request --> Debit
    Debit --> Credit
    Credit --> Record
    Record --> Success

But imagine the application deducts ₹5,000 from Account A and then crashes before crediting Account B.

Now the system has produced an incorrect result.

This is a reliability problem.

Database Transaction

SQL Server transactions can help keep related database operations together.

For example:

BEGIN TRANSACTION;
UPDATE Accounts
SET Balance = Balance - 5000
WHERE Id = 1;
UPDATE Accounts
SET Balance = Balance + 5000
WHERE Id = 2;
COMMIT TRANSACTION;

If an error occurs before the transaction is committed, we can roll back the transaction.

BEGIN TRANSACTION;
UPDATE Accounts
SET Balance = Balance - 5000
WHERE Id = 1;
UPDATE Accounts
SET Balance = Balance + 5000
WHERE Id = 2;
IF @@ERROR <> 0
BEGIN
    ROLLBACK TRANSACTION;
END
ELSE
BEGIN
    COMMIT TRANSACTION;
END

This helps ensure that the two database operations are treated as one unit.

ASP.NET Core Example

The same concept can be implemented using Entity Framework Core.

public async Task TransferMoney(
    int fromAccount,
    int toAccount,
    decimal amount)
{
    await using var transaction =
        await _context.Database.BeginTransactionAsync();
    try
    {
        var source = await _context.Accounts
            .FirstAsync(x => x.Id == fromAccount);
        var destination = await _context.Accounts
            .FirstAsync(x => x.Id == toAccount);
        if (source.Balance < amount)
            throw new Exception("Insufficient balance");
        source.Balance -= amount;
        destination.Balance += amount;
        await _context.SaveChangesAsync();
        await transaction.CommitAsync();
    }
    catch
    {
        await transaction.RollbackAsync();
        throw;
    }
}

If the operation fails before the commit, the transaction is rolled back.

What Can Make a System Unreliable?

Several problems can affect reliability.

• Application crashes
• Database failures
• Network failures
• Timeouts
• Duplicate requests
• Race conditions
• Partial failures
• Incorrect retry logic
• Data corruption
• External service failures

Handling External Service Failures

Consider an order application that calls a payment provider.

flowchart LR
    User --> OrderAPI
    OrderAPI --> PaymentService
    PaymentService --> PaymentProvider

What happens if the payment provider does not respond?

The application should not wait forever.

We can use a timeout.

using var cts =
    new CancellationTokenSource(
        TimeSpan.FromSeconds(5));
var response = await httpClient.PostAsync(
    "payment",
    content,
    cts.Token);

If the external service does not respond within the expected time, the request can fail or move into an appropriate recovery flow.

Retry and Reliability

Some failures are temporary.

For example, a network request may fail once and succeed immediately afterward.

A retry can help in such cases.

flowchart TD
    Request --> Service
    Service --> Success
    Service --> Failure
    Failure --> Retry
    Retry --> Service

But blindly retrying is dangerous.

Imagine a payment request is successfully processed by the payment provider, but the response is lost because of a network failure.

If we simply send the same payment request again, we could potentially create a duplicate payment.

This is why retries often need to be combined with idempotency.

Idempotency

Idempotency means that sending the same operation multiple times should not produce an unintended additional effect.

For example, a payment request could contain a unique idempotency key.

{
    "orderId": 1001,
    "amount": 5000,
    "idempotencyKey": "PAY-1001-ABC123"
}

The system can store the key.

If the same request arrives again, the system can recognize that the operation was already processed.

flowchart TD
    Request --> CheckKey
    CheckKey --> Existing
    CheckKey --> NewRequest
    Existing --> ReturnPreviousResult
    NewRequest --> ProcessPayment
    ProcessPayment --> StoreKey
    StoreKey --> ReturnResult

This is especially important for payment, order creation and other operations where duplicate processing can cause business problems.

Reliability and SQL Server

Reliability also depends on protecting data from incorrect operations.

For example, suppose two users try to purchase the last available product at the same time.

Both requests may read:

Available Quantity = 1

Both may then attempt to reduce the quantity.

This can create incorrect inventory.

Database concurrency control and appropriate transaction strategies are important for such scenarios.

Monitoring Reliability

We cannot improve reliability if we don't know when failures are happening.

Production systems should monitor things such as:

• Failed requests
• Database errors
• Timeouts
• Exception rates
• Failed background jobs
• Payment failures
• Message processing failures

flowchart LR
    Application --> Logs
    Application --> Metrics
    Application --> Alerts
    Metrics --> Dashboard

For example, if payment failures suddenly increase, monitoring should help the engineering team identify the problem quickly.

Reliability in Distributed Systems

As systems become larger, they contain more components.

flowchart LR
    User --> API
    API --> OrderService
    OrderService --> PaymentService
    OrderService --> InventoryService
    OrderService --> NotificationService
    PaymentService --> Database
    InventoryService --> Database

Each component can fail independently.

This is called a partial failure.

For example, the order service may be working while the notification service is unavailable.

A reliable system should handle such failures gracefully instead of unnecessarily bringing down the entire application.

Key Takeaways

• Reliability means consistently producing the correct and expected result.
• Availability means the system is accessible.
• A system can be available but unreliable.
• Database transactions help keep related operations consistent.
• Timeouts prevent the application from waiting indefinitely for external services.
• Retries can help with temporary failures but must be designed carefully.
• Idempotency helps prevent duplicate operations.
• Monitoring and alerting are important for detecting reliability problems.
• Distributed systems must handle partial failures.

Conclusion

Reliability is about making users confident that the system will do what it is supposed to do.

A reliable system should handle failures without producing incorrect results, duplicate operations or corrupted data.

Transactions, idempotency, timeouts, controlled retries, monitoring and proper failure handling are some of the important techniques used to improve reliability.

Once a system is reliable, we also need to ask another important question:

How quickly can the system process requests?

That brings us to the next concept: Performance.