Designing Data-Intensive Applications

I recently studied Designing Data-Intensive Applications by Martin Kleppmann.

This book is widely recommended for engineers working on distributed systems, data platforms, and large-scale applications.

Rather than teaching specific tools or frameworks, the book focuses on fundamental system design principles and the trade-offs behind real-world data systems.

Below is a detailed summary of the key learnings that shaped my understanding of system design.

1. Core Goals of Data-Intensive Systems

The book defines three foundational goals that every data-intensive system must balance:

Reliability

A system is reliable if it continues to produce correct results, even when failures occur.

Failures include hardware crashes, network issues, retries, and malformed data.

Key insight:

Failures are normal, not exceptional. Systems must be designed assuming things will break.

Scalability

Scalability is not about how fast a system is today, but how it behaves as load increases.

Load can grow in different dimensions:

• Data volume

• Traffic (QPS)

• Number of users

• Complexity of queries

A scalable system degrades predictably and can grow horizontally

Maintainability

Maintainability ensures that systems can be:

• Debugged

• Operated

• Extended over time

Good abstractions, clear data models, automation, and documentation all contribute to maintainability.

2. Data Models Shape System Architecture

The book emphasizes that data models influence everything: query patterns, performance, scalability, and correctness.

Common data models discussed include:

• Relational model (tables, joins, ACID)

• Document model (schema flexibility, nested data)

• Key-value and wide-column stores (high throughput)

• Graph model (relationship-heavy queries)

Key insight:

There is no universally best data model. The right choice depends entirely on access patterns and consistency requirements.

3. Storage and Retrieval Internals

Understanding how databases store and retrieve data is critical for system design.

The book explains:

• B-Trees: optimized for reads and in-place updates

• LSM Trees: optimized for writes using append-only logs and background compaction

Many modern systems favor write-optimized designs, which explains:

• Cheap writes

• Background compaction

• Performance dependence on data layout

System designers benefit greatly from understanding these internals rather than relying only on query tuning.

4. Schema Evolution and Encoding

Schemas change over time, especially in distributed systems where producers and consumers evolve independently.

The book highlights:

• Backward compatibility

• Forward compatibility

• Use of structured formats such as Avro and Protobuf

Key insight:

Schema evolution is inevitable. Systems must be designed to handle it safely without breaking consumers.

5. Replication and Its Trade-offs

Replication is used to improve:

• Availability

• Fault tolerance

• Read performance

Replication strategies include-

• Single-leader replication

• Multi-leader replication

• Leaderless replication

Each approach introduces trade-offs between:

• Consistency

• Latency

• Availability

  Strong consistency increases coordination cost, while eventual consistency improves availability but allows temporary inconsistency.

6. Partitioning for Horizontal Scale

To scale beyond a single machine, data must be partitioned.

Common strategies include:

• Hash partitioning for even distribution

• Range partitioning for efficient range queries

Poor partitioning can lead to hotspots and uneven load distribution, which limits scalability.

7. Distributed Transactions Are Hard

The book explains why distributed transactions, especially Two-Phase Commit (2PC), are problematic:

• High coordination overhead

• Blocking behavior during failures

• Reduced system availability

As a result, many modern architectures avoid distributed transactions and instead rely on:

• Idempotent operations

• Eventual consistency

• Compensation mechanisms

8. Failure Is the Default State

Distributed systems fail in complex ways:

• Partial failures

• Network partitions

• Clock skew

• Duplicate message delivery

Key insight:

A system that works only on the happy path is not production-ready.

Designs must assume retries, duplicates, and out-of-order data as normal conditions.

9. Time Is a Fundamental Challenge

The book clearly distinguishes between:

• Event time: when an action actually occurred

• Processing time: when the system processed it

Using processing time for analytics can lead to incorrect results due to delays, retries, and reordering.

Event-time–based processing is harder but necessary for correctness in real-world systems.

10. Batch and Stream Processing

Batch processing handles bounded datasets with high accuracy and high latency.

Stream processing handles unbounded datasets with low latency and complex state management.

Modern systems often combine both approaches using:

• Immutable event logs

• Replayable data

• Incremental computation

Logs act as the source of truth, enabling reprocessing and recovery.

Key System Design Takeaway

The most important lesson from this book is that system design is about trade-offs, not perfection.

Every decision affects:

• Reliability

• Consistency

• Availability

• Latency

• Cost

• Operational complexity

Strong system designers understand these trade-offs and make deliberate choices based on requirements.

Why This Book Matters

This book is valuable for engineers working on:

• Distributed systems

• Data pipelines

• Streaming platforms

• Analytics and ML systems

It builds a mental model for designing systems that survive scale, failure, and change, rather than focusing on specific technologies.

Final Thought

Good systems work when everything goes right.

Well-designed systems continue to work when things inevitably go wrong.