System Design Basics – The Parts of a Production System

Every system design diagram is drawn from the same small parts list. There are about a dozen boxes, they mean the same thing every time, and once you know what each one is for — and, more usefully, what problem forces it to appear — most diagrams stop being intimidating.

This post builds the whole picture from one server. Each box arrives because something specific broke, because a part added for no reason is a part you cannot defend when asked.

Stage 0 — one box

Everything starts here, and this is a real architecture, not a strawman. A single virtual machine running your application and its database serves a surprising amount of traffic.

   browser ──HTTP──> [ server + database ]
                          one machine

What actually happens when someone types your domain: the browser asks DNS for an IP address, opens a TCP connection to it, negotiates TLS, and sends an HTTP request. DNS is the only piece of internet infrastructure in every design, and it is worth remembering that its answers are cached for as long as the TTL says — which is why a DNS change “does not take effect” for minutes or hours after you make it.

This stage fails in three ways, and they arrive in a predictable order: the machine dies and everything is gone; the application and the database start competing for the same CPU and memory; and there is nowhere to put a second server even if you wanted one.

Stage 1 — split the database out

The first thing to separate is the application from its data, because the two scale differently and one of them is much harder to replace.

   browser ──> [ app server ] ──> [ database ]

An application server is mostly CPU: it parses requests, runs logic, serialises JSON. A database is mostly memory and disk, and it cares enormously about being interrupted. Sharing a box means a traffic spike starves the database of the memory its cache was using, and the whole system falls over in a way that looks like a database problem but is not.

The split also buys the thing everything else depends on: you can now run more than one application server.

Choosing the database

The first genuine fork in the road. It is asked in every interview and the answer is more boring than the debate suggests.

The useful heuristic: start relational unless you can name the specific reason not to. Money, bookings, orders and inventory all involve invariants across several rows, and a transaction is the cheapest way to enforce one. StayHub’s entire booking correctness rests on a Postgres constraint; there is no application-level equivalent that works.

Reach for non-relational when the access pattern is genuinely one key at a time at enormous volume — a session store, event logs, a feed cache, telemetry.

Stage 2 — more than one app server

One server is a single point of failure and a hard ceiling. The fix is several, with something in front deciding where each request goes.

                          ┌──> [ app 1 ]
   browser ──> [ LB ] ────┼──> [ app 2 ] ────> [ database ]
                          └──> [ app 3 ]
                    health checks: is each one still answering?

A load balancer spreads requests across the pool and stops sending traffic to instances that fail their health check. That second job is the more important one: without it, losing a server means a third of requests fail instead of none.

This only works if the application is stateless — if any server can handle any request. The moment a server keeps something in memory that a request needs (a session, an upload in progress, a cached user object), requests are no longer interchangeable and you are back to one server wearing a costume.

StayHub is stateless because authentication is a signed token rather than a server-side session. Any instance can verify it with the shared secret and nothing is stored between requests:

def create_access_token(user: User) -> str:
    now = datetime.now(UTC)
    expires = now + timedelta(minutes=settings.access_token_expire_minutes)

    claims = {
        "sub": str(user.public_id),
        "email": user.email,
        "role": user.role.value,
        "iat": int(now.timestamp()),
        "exp": int(expires.timestamp()),

The cost is real and worth stating: a token cannot be revoked before it expires, because nothing is looked up. StayHub accepts that by re-reading the user row on every request — the token proves who, the database says what they currently are. Load balancing gets a post of its own.

Two ways to get bigger

Worth naming explicitly, because the choice recurs at every stage.

Vertical scaling is a bigger machine — more cores, more memory. It is enormously underrated: it requires no code changes, no distributed anything, and modern hardware goes a very long way. A database server with 128 cores and a terabyte of RAM handles more than most companies will ever need.

Horizontal scaling is more machines. It has no ceiling and it is how you get redundancy, but it demands that the work be divisible — which for application servers means statelessness, and for databases means the much harder problem of sharding.

  VERTICAL                          HORIZONTAL
    [ 8 cpu ] -> [ 64 cpu ]           [ x ] -> [ x ][ x ][ x ][ x ]

    no code changes                   needs the work to be divisible
    hard ceiling                      no ceiling
    still ONE machine to lose         survives losing one
    seconds of downtime to resize     add capacity with none

The sensible order is vertical first for the database, horizontal first for the application tier. That is exactly what the stages below do, and reversing it — sharding a database that would fit comfortably on one larger box — is the single most expensive premature decision in this field.

Stage 3 — a cache

Now the app tier scales and the database is the bottleneck, which it always becomes, because it is the one part you cannot simply run more of.

   [ app ] ──1──> [ cache ]  hit? return it        ~0.3ms
      │              ▲
      └──2──> [ database ]  miss: read, then       ~8.8ms
                            populate the cache

A cache is an in-memory key-value store — Redis or Memcached — holding the results of expensive reads. The pattern above is cache-aside: the reader checks the cache, falls back to the database on a miss, and stores what it found.

On StayHub’s listing page, that read goes from 8.8ms to 0.3ms at the service layer and 15.2ms to 2.0ms measured over HTTP. Those are real numbers from this application, not a rule of thumb; the caching post shows how they were taken.

The rule that matters more than the speedup: a cache must be optional. If the site goes down when Redis does, you have not added an optimisation, you have added a second database and made availability worse. StayHub’s cache treats an outage as a permanent miss:

def get_json(key: str) -> Any | None:
    """Read a cached value. Returns None on a miss, on a failure, and on unreadable JSON."""
    client = _client()
    if client is None:
        return None
    try:
        raw = client.get(key)
    except Exception as exc:  # noqa: BLE001
        _warn_once(exc)
        return None

Every path returns “not cached” rather than raising, so a dead Redis makes the site slower and nothing else.

Stage 4 — read replicas

Caches help repeated reads. They do nothing for reads that are all different, and they do nothing for writes.

                 writes
   [ app ] ─────────────> [ PRIMARY ]
      │                        │  replication (asynchronous)
      │                        ├──> [ replica 1 ]
      └──── reads ─────────────┴──> [ replica 2 ]

The primary takes every write and streams its changes to replicas that serve reads. Since most systems read far more than they write — ten to one is ordinary, a hundred to one is common — this buys a lot.

It also introduces replication lag, and that lag is a correctness problem rather than a performance one. A user updates their profile, the write goes to the primary, the next page load reads a replica that is 200 milliseconds behind, and their change has vanished. They will try again, and now you have a support ticket about a bug that does not reproduce.

The usual fix is to route a user’s reads to the primary for a short window after they write. It is not free, and knowing that this problem exists at all is most of what an interviewer is checking. Scaling the database goes through the full ladder.

Stage 5 — CDN and object storage

Images, video, CSS and JavaScript should never touch an application server. They are large, they never change, and serving them occupies a worker that could be answering a real request.

   browser ──> [ CDN edge ]  hit  ──> done, ~10ms, never reached you
                    │ miss
                    v
              [ object store ]  S3 / GCS — cheap, durable, not a database

A CDN is a global network of caches sitting near users. An object store is where the files actually live: cheap per gigabyte, effectively unlimited, and deliberately not a filesystem — you cannot append to an object or seek within it, you replace it.

The rule is that a database row holds the URL, never the bytes. Storing images as blobs bloats every backup, wrecks the caching the database does of its own pages, and makes replication crawl. This site is itself an example: it is a static build in an object store behind a CDN, and nothing is computed when you load this page.

Stage 6 — a queue

Some work does not belong in the request. Sending an email, generating a thumbnail, rebuilding a search index — the user does not need to wait, and if the provider is slow they should not have to.

   request ──> [ app ] ──> respond immediately  ✓
                  │
                  └──> [ queue ] ──> [ worker ] ──> email provider
                        durable        separate process,
                        survives       restarted / scaled
                        a restart      independently

Two benefits, and the second is the one that matters. The obvious one is latency: the response does not wait for a 400ms provider call. The subtle one is failure isolation — if the email provider is down for ten minutes, the queue holds the work and the site keeps taking orders. Without a queue, that outage is your outage.

A queue also changes what “done” means, and this is where most systems get it wrong. Writing to the database and then sending to a queue is two operations that can half-happen. The transactional outbox fixes it by making the message part of the same database transaction as the change that caused it — StayHub does exactly this, and the queues post is about why.

What a queue is not

Almost every web framework ships something called background tasks, and it is not a queue. The distinction costs people real incidents, so it is worth being precise. StayHub’s notification module states it plainly, because it used to rely on exactly this:

It is NOT a job queue, and the difference is not academic:

  * no retry           — a provider blip loses the email
  * no persistence     — a deploy or a crash mid-task loses it too
  * no back-pressure   — a burst of requests is a burst of concurrent tasks
  * no visibility      — nothing anywhere records that it was meant to happen

In-process background tasks run after the response, in the same process, in the same event loop. That buys one thing — the user does not wait — and buys none of the others. So the rule is: in-process for work that is genuinely nice to have, a real queue for work that must happen.

Stage 7 — a search index

“Find me somewhere in San Francisco under $200 that sleeps four and has wifi” is not a query a relational database is good at. Full-text matching, relevance ranking and a dozen optional filters are what a search engine does, and doing it in SQL means a query that cannot use an index.

   writes ──> [ Postgres ]  source of truth
                   │
                   │  sunk from application code on every write
                   v
              [ Elasticsearch ]  derived, disposable, rebuildable
                   ▲
   search ─────────┘

The critical property is the one written on the diagram: the index is derived data. Postgres is the truth and the index is a copy that can be thrown away and rebuilt. That is what makes it safe for the two to disagree briefly, and they always will — two datastores cannot be updated atomically.

StayHub sinks the index from application code, and the ordering is load-bearing:

def _sync(self, prop: Property) -> None:
    indexed = indexer.index_property(prop)
    cache.invalidate(cache.property_key(prop.public_id))

Called after the commit, never before. Indexing first means a transaction that then rolls back leaves a listing in the search results that does not exist in the database, and the guest who clicks it gets a 404 from a page that just offered it to them.

Stage 8 — knowing which part is broken

The last box is the one candidates forget, and it is the one that makes the others operable. With eight components, “the site is slow” is not a diagnosis.

• Logs — what happened, per request. Structured (JSON) rather than prose, and carrying a request id that follows one request through every service.
• Metrics — numbers over time. Request rate, error rate, latency percentiles, queue depth, cache hit rate. These are what you alert on.
• Traces — where one request spent its time across services.
• Health checks — what the load balancer polls to decide whether an instance should receive traffic.

A health check that returns a bare “ok” answers the wrong question. StayHub’s reports each dependency separately, and deliberately does not let an optional one mark the instance unfit:

    es_ok = es_available()
    cache_ok = cache.available()

    return Health(
        status="ok" if db_ok else "degraded",
        database=db_ok,
        elasticsearch=es_ok,
        cache=cache_ok,
    )

Only the database decides status. If Redis being down marked the instance unhealthy, a cache outage would pull every server out of the load balancer and turn a slow site into no site at all.

Every box is also a failure mode

Adding a component adds a way to fail. That is not an argument against adding them, but it is the thing to say out loud when you do — and it is what an interviewer means when they point at a box and ask “what if that dies?”

Two rows are worth dwelling on. The primary database is the outage — everything else on that list degrades, and this one stops the system doing its job. It is why so much of this track is about the database.

And the cache row has a condition attached. “Slower, not broken” is only true if it was built that way. StayHub’s test suite asserts it in both directions: 165 tests pass with Redis running, and with Redis stopped 142 pass and 23 skip, with nothing failing. An availability claim you have not tested is an availability claim you do not have.

The whole picture

                              [ DNS ]
                                 │
   browser ──> [ CDN ] ──────────┴──> [ load balancer ]
                  │                          │
            [ object store ]        ┌────────┼────────┐
             images, static         v        v        v
                                 [app 1]  [app 2]  [app 3]     stateless
                                    │        │        │
                    ┌───────────────┼────────┴────────┘
                    v               v               │
               [ cache ]      [ PRIMARY DB ]        └──> [ search index ]
                                    │
                                    ├──> [ replica ] ──> reads
                                    │
                                    └──> [ queue ] ──> [ worker ] ──> email, etc.

                    everything above emits ──> [ logs · metrics · traces ]

What the numbers look like at each stage

Rough figures, but the orders of magnitude are what matter — they are how you decide which stage you are actually designing for.

The gap between stage 2 and stage 8 is enormous, and almost every system lives inside it. If your estimate lands at 500 requests a second, the design is stage 2 with a cache, and saying so confidently — with the arithmetic to back it — is a better answer than a sharded, multi-region system that solves a problem you do not have.

Monolith or services?

The question that always follows, and the honest answer disappoints people who expect “microservices”.

Nothing above requires splitting the application into separate deployable services. Every box on that diagram can be one codebase that scales horizontally, and for the overwhelming majority of systems that is the right answer. A monolith gives you a transaction across your whole domain, one deployment, one place to look, and a function call where services need a network round trip that can fail.

Split when a specific pressure forces it: one component needs to scale on a completely different axis, teams are blocking each other on releases, or a piece genuinely needs a different runtime. Those are real reasons. “Microservices are modern” is not, and the cost is a distributed system, which means every function call becomes a thing that can time out, retry and arrive twice.

StayHub is a useful middle case: it is one application, but it does talk to four separate datastores — Postgres, Elasticsearch, Redis and a GraphQL layer. That is enough to have every distributed-systems problem this track covers, without a single microservice.

What to take from this

The parts list is short and it does not change. What changes is which parts a given problem needs, and the skill is being able to say what forced each one:

One habit is worth carrying out of this post. When you add a box, say the sentence that forced it — “the numbers say 40,000 reads a second and one database cannot serve that, so a cache goes here” — and then say what it costs, which for a cache is stale data and one more thing to operate. Two sentences per box. A diagram annotated that way is a design; the same diagram without them is a picture someone memorised.

Next: back-of-the-envelope estimation, which is how you work out which row of that table you are actually in.