Designing a Notification System

Notifications look like a feature and are a system. One service, three or four channels, dozens of message types, and a set of delivery guarantees that are harder than they appear — plus a legal obligation, which is unusual for a piece of infrastructure.

The failure that defines it: a bug in a notification system does not produce an error page, it produces ten thousand emails to real people, and there is no rollback.

Step 1 — Scope

IN SCOPE                         OUT OF SCOPE (say so)
  email, SMS, push, in-app         marketing campaign management
  user preferences and opt-out     A/B testing of copy
  templating and localisation      an analytics UI
  retries and deduplication        rich interactive notifications

WORTH ASKING
  transactional or marketing?     -> completely different compliance rules
  how fast must it be?            -> "2FA code" and "weekly digest" differ by hours
  who triggers it?                -> one service, or every service in the company?

NON-FUNCTIONAL
  at-least-once, never zero        a missing password reset is a support ticket
  no duplicate storms              worse than late — it is what users complain about
  one bad channel must not
    block the others               SMS down should not stop email
  a bug must not send 10,000       rate control is a SAFETY feature here

The first question is the one that shapes everything. Transactional notifications — a receipt, a password reset, a booking confirmation — are things the user asked for by acting. Marketing notifications are things you decided to send. The distinction is not editorial: in most jurisdictions marketing requires consent and a working unsubscribe, and transactional does not. Mixing them in one pipeline means applying the strictest rules to everything, or breaking the law.

Step 2 — Estimate

ASSUME  10M users · ~3 notifications/user/day across all channels

VOLUME    30M/day / 100k    = 300/sec        peak x5 = 1,500/sec
          (x5 not x3 — batch sends are spiky by nature)

MIX       email  60%   ~180/sec    cheap, slow, high volume
          push   30%    ~90/sec    cheap, fast
          SMS    10%    ~30/sec    EXPENSIVE — ~$0.01 each
                                   = ~$26,000/day if unchecked

STORAGE   30M events x ~500 B = 15 GB/day of delivery records
          (retain 30-90 days, then aggregate)

The SMS line is why this system needs spend limits as a first-class feature rather than a nice-to-have. A retry loop with no cap on an expensive channel is a bug that bills you.

1,500 per second is otherwise not a throughput problem. Like the booking system in the estimation post, the difficulty here is correctness, not scale.

Step 3 — The shape

   any service ──> POST /notifications  {userId, type, data}
                        │
                        v
                  [ ingestion ]  validate, deduplicate, enqueue
                        │
                        v
                  [ preference check ]  is this user opted in, on this
                        │               channel, at this hour?
                        v
                  [ template render ]  copy + locale, NOT in code
                        │
                        v
                  [ QUEUE, one per channel ]
                    email │ sms │ push │ in-app
                        │
                        v
                  [ per-channel workers ] ──> provider APIs
                        │
                        v
                  [ delivery log ]  what was sent, when, and what happened

The single most important structural decision is on the queue line: one queue per channel. Channels have wildly different latencies, failure modes and costs, and a shared queue means a slow SMS provider blocks email delivery for everybody. Separate queues let each channel fail, retry and scale on its own.

Step 4 — Getting the event in reliably

Before any of that, the triggering service has to record its intent without losing it. This is the transactional outbox again, and notifications are its canonical use.

            outbox_service.enqueue(
                self.db,
                notification_service.TOPIC_BOOKING_CREATED,
                {
                    "bookingId": str(booking.public_id),
                    "guestEmail": guest.email,
                    "propertyTitle": prop.title,
                    "checkIn": req.check_in,
                    "checkOut": req.check_out,
                    "total": breakdown.total,
                    "idempotencyKey": f"booking-created:{booking.public_id}",
                },
            )

            self.db.commit()

The enqueue is inside the transaction that creates the booking, so the booking and the instruction to notify commit together or not at all. A crash between them is impossible, and a booking rejected by the database takes its notification with it — nobody is emailed about a reservation that lost the race.

Two things in that payload are deliberate. A snapshot, not just an id: by the time the worker runs, the booking may have changed, and the email should describe what happened rather than what is now true. And an idempotency key, because delivery is at-least-once and the provider needs a way to recognise a redelivery.

Where the outbox stops and this system starts

Worth drawing the boundary, because the two are easy to conflate.

   BOOKING SERVICE                    NOTIFICATION SERVICE
   owns: the booking                  owns: who gets told, how, and whether
         the outbox row                     preferences, templates, providers

   "booking.created happened"   ──>   "that means 5 notifications,
    a fact, stated once                 subject to these rules"

The producing service’s only obligation is to record the fact durably. Everything about delivery — channels, retries, quiet hours, suppression — belongs on the other side of that line.

Getting this boundary wrong is the most common structural mistake in notification systems, and it shows up as domain services that import an email client. Once the booking service knows what an SMTP failure is, it has acquired a dependency on a third party for a job that has nothing to do with bookings — and it will eventually fail a booking because of it.

Step 5 — Preferences, which are not optional

The check that runs before every send, and the one with legal weight behind it.

CREATE TABLE notification_preferences (
    user_id     BIGINT      NOT NULL,
    type        VARCHAR(64) NOT NULL,   -- 'booking.confirmed', 'marketing.weekly'
    channel     VARCHAR(16) NOT NULL,   -- 'email' | 'sms' | 'push' | 'in_app'
    enabled     BOOLEAN     NOT NULL,
    PRIMARY KEY (user_id, type, channel)
);

CREATE TABLE notification_suppressions (
    address     VARCHAR(320) PRIMARY KEY,  -- email or phone, NOT user_id
    reason      VARCHAR(32)  NOT NULL,     -- 'unsubscribed'|'bounced'|'complained'
    created_at  TIMESTAMPTZ  NOT NULL DEFAULT now()
);

The second table is the one people forget, and it is keyed by address rather than by user for a specific reason: a hard bounce or a spam complaint attaches to the address, and must survive the user deleting and recreating their account, or the same address appearing on a second account. Continuing to send to a complained address is how a sending domain gets blacklisted, at which point none of your email arrives anywhere.

The rule to state clearly: suppression beats preference beats default. A suppressed address is never sent to, whatever any other table says — and, importantly, a suppression list applies to marketing but never to genuinely transactional messages like a password reset, which is the other half of why the two must be distinguishable.

Two more that belong here: quiet hours in the user’s timezone, because a 3am push is a deleted app; and digesting, so twenty events in ten minutes become one message.

Quiet hours have a subtlety worth catching. Deferring a notification until morning means it must be held, and held notifications for a whole timezone all become due at the same instant — so 9am local time produces a spike shaped exactly like the thundering herd from the caching post. Spreading the release over a window, with jitter, is the same fix as everywhere else.

And the check must be against the recipient’s timezone, not the server’s. That sounds obvious and is the single most common bug in this area, because the server’s clock is the one that is conveniently available.

Step 6 — Fan-out: one event, several notifications

A single business event usually produces more than one notification, to more than one person, on more than one channel — and the expansion has to happen somewhere.

   one event: booking.created

   ├── guest   email  "your stay is confirmed"
   ├── guest   push   "confirmed — 4 Jan to 7 Jan"
   ├── host    email  "you have a new reservation"
   ├── host    push   "new booking for Sunlit Loft"
   └── ops     in-app (only if the booking value is over a threshold)

   1 event -> 5 notifications -> 5 independent delivery attempts

The rule that keeps this manageable: the producing service emits one event; the notification service decides who hears about it. If the booking service had to know that hosts get push notifications and that operations wants alerts above a value threshold, then every change to notification policy would be a change to the booking service.

That is the events-not-commands distinction from the queues post, and this is where it pays off: adding a sixth recipient is a routing rule, not a deployment of the domain service.

Each of the five is then independently queued, retried and logged. One failing does not affect the others — the host’s email is not lost because the guest’s push token was stale.

Step 7 — Providers

Every channel goes through a third party, and every third party has bad days. The abstraction to build is thin and its purpose is failover.

   interface Sender:
       send(recipient, rendered, idempotency_key) -> ProviderResult

   email:  Postmark  ─fails─>  SES         (secondary)
   sms:    Twilio    ─fails─>  MessageBird
   push:   APNs (iOS) + FCM (Android)      — not interchangeable

The value of the interface is that switching providers is a configuration change rather than a project. That matters because it will happen: providers have outages, get acquired, and change pricing.

Three things the wrapper must normalise, because providers disagree about all of them:

• Which failures are retryable. A 500 or a timeout is worth retrying; an invalid address is not, and retrying it eight times wastes eight attempts on something that will never succeed. This distinction is the difference between a retry policy and a loop.
• Rate limits. Providers impose their own, and exceeding them gets you throttled or suspended. The worker has to respect them, which means the token bucket appears here too — this time protecting somebody else’s service from you.
• Asynchronous outcomes. Accepting a message is not delivering it. Bounces and complaints arrive minutes later by webhook, which means the delivery log has states that are written long after the send.

Step 8 — At-least-once, and its consequence

The guarantee is the same one from the queues post: a worker can hand a message to a provider and die before recording that it did.

Building StayHub’s outbox, that duplicate appeared immediately — two delivery paths were briefly live at once, and one booking produced two files:

$ ls notifications/
20260822T190807315928-guest_at_stayhub.test.json
20260822T190807858799-guest_at_stayhub.test.json    # same email, twice

That had an obvious fix. But it is exactly what a redelivery looks like, and the same shape arrives legitimately whenever a worker dies at the wrong moment. It is worth having seen.

The defences, in order of preference:

   1  pass an idempotency key to the provider   they deduplicate for you
   2  record (idempotency_key) as UNIQUE        a second attempt is rejected
      in the same transaction as the send        by the database
   3  suppress by content hash within a window  catches near-duplicates from
                                                 different code paths

The second is the one you control. A unique constraint on the key means a duplicate send is impossible rather than unlikely — the concurrency post’s principle applied to delivery.

Step 9 — Rate control as a safety feature

In most systems rate limiting protects a server. Here it protects users from you, and it is the control that prevents the worst possible incident.

   PER USER      max 5 notifications/hour, 20/day
                 -> a loop cannot spam one person

   PER TYPE      max N/minute globally
                 -> a bad deploy cannot mail everybody

   GLOBAL        a hard ceiling and a SPEND cap
                 -> stop sending, page a human, do not "retry harder"

   CIRCUIT       provider failing? stop calling it for a while

The global ceiling deserves emphasis. The classic notification incident is a scheduled job that re-sends because its cursor was not persisted, and the outcome is measured in the hundreds of thousands. A cap that halts sending and alerts, rather than one that queues everything for later delivery, is what turns that into an embarrassing hour instead of a company-wide apology.

Note that halting is the right behaviour and it is failing closed — the opposite of the rate limiter in the rate limiting post, which fails open. The difference is what the failure costs: refusing to send is recoverable, and sending wrongly is not.

Step 10 — Retries that respect the channel

The backoff schedule from the queues post — 2, 4, 8, 16, 32, 64, 128, 256 seconds — is a sensible default, and notifications need it tuned per channel because the value of a late message differs enormously.

The first row is the interesting one, because it inverts the usual instinct. Most retry policies try harder for the most important message; here the most important message has the shortest retry window, because its value decays to zero in about a minute. Giving up quickly and letting the user request another is better than delivering a stale code.

A related rule: attach a deadline to the message, not just an attempt count. “Do not send this after 14:35” is more meaningful than “try eight times”, because it expresses the thing you actually care about.

Step 11 — Templates

Copy does not belong in code. Changing “Your stay is confirmed” should not require a deploy, and it certainly should not require one per language.

   template: booking.confirmed
   locale:   en-GB
   channel:  email

   subject: "Your stay at {{propertyTitle}} is confirmed"
   body:    "Hi {{firstName}}, we are holding {{propertyTitle}} from
             {{checkIn}} to {{checkOut}}."

   The same template id renders differently per channel:
   SMS gets 160 characters; push gets a title and a line.

Two rules. Escape by channel — the same variable needs HTML escaping in an email and none in an SMS, and getting that backwards is either broken text or an injection vulnerability. And version templates, so a delivery log written three months ago can still be rendered as it was actually sent.

A template also needs a fallback locale, because the alternative to a missing translation is a blank email.

Step 12 — Testing something that emails real people

A category of risk this system has and most do not: a mistake in a test run reaches actual inboxes, and there is no undo.

The defences are unglamorous and all of them are worth having before the first send:

• A hard environment gate. Non-production sends to a local capture directory, a mail-catcher, or a provider’s sandbox — never to a real provider. StayHub writes files to notifications/ for exactly this reason, and the file-per-send behaviour also makes duplicates countable.
• An allowlist in staging. If staging must use a real provider — to test rendering across mail clients — restrict recipients to a handful of internal addresses, enforced in the sender rather than in configuration.
• A dry-run mode. Render, log the resolved recipient and body, do not send. This is what you run before a large campaign, and reading its output is what catches the template variable that resolves to None.
• Reserved test addresses. RFC 6761 reserves .test precisely so it can never be a real domain, which makes it the correct TLD for fixtures — and strict email validators reject it for that same reason, so the validator needs a test-mode flag rather than a looser rule.

The general principle: the blast radius of a bug here is external and permanent, so the guard rails belong in the code path rather than in configuration or process. A configuration mistake should not be able to send, and “remember to check the environment variable” is not a control.

Step 13 — Knowing what happened

An asynchronous system fails silently. The request succeeded, the notification did not arrive, and nobody is holding an error.

CREATE TABLE notification_log (
    id              BIGSERIAL PRIMARY KEY,
    idempotency_key VARCHAR(128) NOT NULL UNIQUE,   -- the duplicate guard
    user_id         BIGINT       NOT NULL,
    type            VARCHAR(64)  NOT NULL,
    channel         VARCHAR(16)  NOT NULL,
    status          VARCHAR(16)  NOT NULL,  -- QUEUED|SENT|DELIVERED|BOUNCED|FAILED
    provider_id     VARCHAR(128),           -- to correlate with their support
    attempts        INTEGER      NOT NULL DEFAULT 0,
    last_error      TEXT,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT now()
);

This table answers the question every support ticket asks — “did they get it?” — and it is worth having before you need it, because reconstructing it from provider dashboards after the fact is miserable.

What to alert on: queue depth and the age of the oldest pending message; bounce rate, which rising sharply means a list problem or a domain reputation problem; dead-letter count; and spend on the expensive channel.

In-app notifications are a different problem

Three of the four channels push outward to a provider. The fourth — the bell icon in your own product — is a read model you own, and it behaves nothing like the others.

CREATE TABLE in_app_notifications (
    user_id     BIGINT      NOT NULL,
    id          BIGINT      NOT NULL,   -- time-sortable, so no ORDER BY sort
    type        VARCHAR(64) NOT NULL,
    payload     JSONB       NOT NULL,   -- rendered client-side, so it can be localised late
    read_at     TIMESTAMPTZ,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (user_id, id)
);

The composite key is the same trick as the chat message table: one user’s notifications are stored together and already in order, so “the latest 20” is a range scan and pagination is by cursor.

Two differences from the outbound channels are worth stating. There is no delivery guarantee to worry about — the row exists or it does not, and the client reads it when it looks; there is no provider to fail. And the unread count is the expensive part, because it is requested on every page load by every user.

Counting unread rows on each request is the counter problem in disguise. The usual answers are to keep a denormalised count updated on write, or to store a last-read marker and count the range — the latter being preferable for the reason the chat post gives: a derived value that can be recomputed beats a stored one that can drift.

Delivery here is also a natural fit for the connection tier from that post: if the user has a live WebSocket, push the notification to it immediately; otherwise it simply waits in the table.

What an interviewer will push on

• “How do you avoid sending twice?” — you cannot avoid at-least-once delivery, so you make duplicates harmless: an idempotency key, unique in your database and passed to the provider.
• “A provider is down for an hour.” — the channel’s queue absorbs it, backoff stops you hammering them, the circuit breaker stops you trying at all, and other channels are unaffected because the queues are separate.
• “How do you stop a bug mailing everyone?” — per-user, per-type and global caps, plus a spend limit, and the behaviour on breach is to halt and page rather than to queue.
• “How does the triggering service avoid losing events?” — transactional outbox: the notification intent commits with the business change.
• “How do you handle unsubscribes?” — a suppression list keyed by address, not by user, checked before every marketing send and never applied to transactional messages.
• “Priority?” — separate queues by urgency, not a priority field. A 2FA code must never wait behind a weekly digest, and the cleanest way to guarantee that is for them not to share a queue.

The framing that makes this system click: it is a queue with policy on both ends. Reliable ingestion at the front (the outbox), reliable delivery at the back (retries, idempotency, dead letters), and in the middle a set of rules — preferences, suppressions, quiet hours, rate caps — that decide whether the message should exist at all. Most of the hard parts are in the middle, and none of them are about scale.

Next: designing Airbnb — the first of three whole-system case studies, and the one where every claim is backed by an application that actually runs.