Design a Notification System

1. The problemCopy Anchor Link

Turn "something happened to user 42" into a message arriving on one of user 42's devices, through a channel they agreed to, once, even when everything in between is flaky. Channels: mobile push (iOS, Android), SMS, email (often also in-app / web push).

Three properties make it a distributed system instead of a send() helper:

• Downstream of everyone. Every service is a producer. A retry-loop bug in the order service becomes our volume problem → idempotency and rate limiting are this system's responsibility, not the callers'.
• Upstream of nobody we control. Every channel terminates at a third party (APNs, FCM, Twilio, SES) with its own rate limits, error codes and outages.
• Trust cost per mistake. A duplicate OTP or a 3 a.m. promo is user-visible in a way a duplicated log line never is.

The trap: this is NOT "design a chat system." Nothing is bidirectional or connection-oriented. It is a fan-out and dispatch pipeline with reliability guarantees. Say so early.

2. Requirements (extract these)Copy Anchor Link

Senior questions to ask unprompted:

• Priority classes? An OTP and a "you might like…" cannot share a queue.
• Delivery semantics? Queues are at-least-once; "exactly once" must be manufactured with dedup — and it has a residual gap (§6.2).
• What counts as success? Sent (gateway accepted) ≠ delivered (receipt confirmed). Push never confirms; SMS/email do, via webhooks.
• Expiry? A promo for a sale that ended is worse than nothing → per-message TTL.

3. EstimationCopy Anchor Link

  16M/day ÷ 10⁵ s                     ≈ 160/sec average   → trivial, one box
  marketing blast 10M users in 1 hour  ≈ 3,000/sec         → 20× average, past provider limits
  → design point: bursts of 10³–10⁴/sec, NOT the average

  notification log: 16M/day × ~1 KB   ≈ 16 GB/day ≈ 6 TB/yr → time-partitioned store, 90-day TTL
  device tokens: 100M users × 2 × 200 B ≈ 40 GB            → fits a cache cluster

Thesis (say it): steady-state load is trivial; the architecture exists to absorb bursts and survive third parties. Every box is justified by one of those two.

4. The channels — what physically deliversCopy Anchor Link

Your servers never touch the device. Every channel ends at a gateway you don't own.

iOS push — APNs (Apple Push Notification service)Copy Anchor Link

The only path to an iPhone. Apple holds one persistent encrypted connection per device; only Apple can ring the doorbell. Three-party handshake:

1. App registers → APNs issues a device token (opaque; means "this app on this device" — a capability, not a serial number).
2. App posts the token to your backend (POST /devices) → stored against the user.
3. To send: HTTPS/2 request to APNs with token + small JSON payload (aps: alert, badge, sound, custom keys) + a JWT signed with your .p8 key (token-based auth). Payload limit 4 KB (2 KB on the old certificate-based path). Rich content → put an ID in the push, let the app fetch.

Android push — FCM (Firebase Cloud Messaging)Copy Anchor Link

Same shape. Say the 2026-correct version: the legacy HTTP/XMPP APIs were shut down (June 2024); the HTTP v1 API authenticates with OAuth 2.0 service-account tokens, not a static server key. Payload 4 KB. Error codes drive worker logic directly:

SMS / email — gatewaysCopy Anchor Link

Twilio / SNS; SES / SendGrid. Don't run SMTP (deliverability, SPF/DKIM, reputation is a full-time job). They report asynchronously via webhooks (delivered / bounced / complained) — no webhook endpoint = you never learn what arrived. SMS is 160 chars GSM-7 / 70 chars UCS-2 per segment.

What this teaches: every channel has a stored address, its own payload format, its own error set, its own gateway with its own limits → one worker pool per channel (§5). Tokens rot → provider "dead token" errors are an instruction to delete; retrying them burns quota on ghosts.

5. High-level designCopy Anchor Link

5.1 Contact-info gatheringCopy Anchor Link

Client registers on login through the LB → API servers → user DB.

• One user, many devices → fan-out is per device (≈2 per user).
• last_logged_in + provider UNREGISTERED = how you prune.
• The log's PK is the idempotency key, not an auto-increment — that's what makes dedup a single insert-if-absent (§6.2).

5.2 Naive design — one synchronous server — break it out loudCopy Anchor Link

Common cause: the caller's request and the provider's work are welded together. One idea fixes all four:

The API accepts the job. It does not do the job. Validate, record, enqueue, 202 in milliseconds. Workers drain at the providers' pace.

5.3 The queue-decoupled designCopy Anchor Link

Notification servers — stateless, behind LB. POST /v1/notifications {idempotency_key, user_id | audience, type, channels[], params, priority, ttl}. In order:

1. Authenticate + rate-limit the caller (appKey/secret, per-caller limit — the funnel protecting itself from what's above it).
2. Idempotency check — insert-if-absent into the log; exists → return existing status, stop.
3. Enrich — devices, email, timezone, settings from cache (DB on miss).
4. Persist — one log row per (device, channel), status = queued.
5. Enqueue one message per (device, channel) → 202 {notification_id}.

Never talks to a provider.

Cache + DB — users/devices/settings: small, relational (Postgres). The log is append-heavy, 16M rows/day, queried by ID and time → wide-column or time-partitioned table with TTL. Say the tension ("one Postgres table" = 6 TB in a year).

Queues — one per channel. Isolation by queue: APNs down for an hour → the iOS queue backs up (that's its job); Android/SMS/email don't notice. Blast radius of a provider outage = exactly one queue. Kafka if you need replay + partition ordering; RabbitMQ/SQS for native delay + dead-lettering. Name what you're buying.

Workers — one pool per channel. Each speaks one gateway's protocol and error dialect. Autoscale on queue depth and oldest-message age, never on API request rate (the API is fast no matter what).

One-liner: "Callers POST to stateless servers that dedup, enrich and persist, then fan out to one queue per channel; per-channel workers drain them into APNs/FCM/SMS/email with retries and provider-aware rate limiting."

6. Deep dive: reliability (where the interview is decided)Copy Anchor Link

6.1 "Can a notification be lost?"Copy Anchor Link

Trace where the message lives: caller memory → API memory → log → queue → worker memory → provider. Only the two "memory" spots can lose it.

• API crashes before writing the log → caller gets no 202 → caller retries (same idempotency key).
• Worker crashes before acking → queue redelivers to another worker.

Rule: persist before you acknowledge; acknowledge only after the work is logged. The remaining gap — log row written, crash before enqueue — is closed by a reaper (sweep stale queued rows) or, more principled, a transactional outbox: log row + outbox row in one DB transaction; a relay publishes outbox rows to the queue and can crash/restart freely.

6.2 "Can a user get it twice?" — yes; make the second harmlessCopy Anchor Link

Nothing in that timeline is broken. The worker cannot distinguish "provider never got it" from "provider got it, ack died" — the ambiguous timeout — so retrying is correct, and the user gets two OTPs. Exactly-once delivery across a network is impossible. Duplicates enter at three layers:

1. Caller retries after a lost 202.
2. Queue redelivers after a worker crash mid-send (at-least-once).
3. Worker retries after a lost provider ack.

One mechanism for all three: a key that identifies the logical event + a store that remembers keys.

• Caller-supplied key derived from the event, never the request: order-987-shipped. A fresh UUID per request makes retries look new → defenceless. This is why the API requires the key: only the caller knows which requests are the same event.
• Boundary check: insert-if-absent on the log (PK = key).
• Dispatch check in the worker: SET dispatch:{id}:{device}:{channel} NX EX 86400 before the provider call; mark "done" after success.

The honest sentence (say it unprompted): the worker marks after the provider says OK; a crash in that sub-second window means the redelivery sends again. You can't close it from your side — only provider-side dedup would (APNs apns-collapse-id coalesces on device; that's the closest). Correct claim: "at-least-once delivery, exactly-once effect except for a crash inside the post-ack window." "Exactly once" is the junior answer.

6.3 "What happens when a send fails?"Copy Anchor Link

Worker decision order:

1. Classify. Permanent recipient error (UNREGISTERED, BadDeviceToken, hard bounce) → drop, prune token, never retry. Transient (429 / 5xx / timeout) → retry.
2. Backoff + jitter, never a tight loop. 2,000 workers retrying at the same instant = a synchronised battering ram against your own provider — indistinguishable from a DoS. delay = min(cap, base × 2^attempt) × rand(0.5, 1.5), base ~1 s, cap ~5 min, N ≈ 5. Implement as re-enqueue with delay, not sleep() — a sleeping worker isn't working.
3. Dead-letter after N. Parking lot: unclogs the main queue, alerted on, human replay. Transactional DLQ growth pages someone; marketing = a report.
4. Circuit breaker per provider — failure rate > X% → stop calling for 30 s, queue holds the backlog, probe, reopen. Paired with a shared per-provider token bucket so even the good case never exceeds the provider's rate. Queue holds the backlog; limiter shapes the drain.
5. TTL — check before every attempt; expired promos drop silently.
6. Fallback vendor for SMS/email (Twilio→SNS, SendGrid→SES). Impossible for push — APNs/FCM are monopolies.

Sent ≠ Delivered. Provider 200 = "accepted." Push never confirms display (infer from app-open telemetry); SMS/email confirm later by webhook, matched on provider_message_id. Separate states or you overstate delivery.

7. Deep dive: the five gates inside a workerCopy Anchor Link

Order is deliberate: cheap rejections first, provider call last.

Workers are dumb by design; everything clever lives in a store they consult.

8. Deep dive: priority lanes and head-of-line blockingCopy Anchor Link

Scenario: OTP enqueued 30 s after marketing dropped 10M promos into the same iOS queue → OTP arrives in 40 minutes; user hits "resend" six times; Gate 1 correctly dedups them all. Head-of-line blocking.

Fix = isolation by queue again, per priority class: channel × class matrix of topics.

Lanes vs one priority queue (RabbitMQ x-max-priority): a priority queue only reorders the queue — the flood still shares the same worker pool and the same provider token bucket, so it still starves OTPs of provider quota. Lanes give separate pools and budgets; cost = more queues to operate. Name both, pick lanes.

Broadcast fan-out ("send to 10M users"): the API must not resolve 10M recipients inline. Enqueue one campaign message → a fan-out worker pages the audience by cursor (~5k at a time), emits per-device messages into the marketing lane, checkpoints the cursor per page so a crash resumes at page 400 rather than re-sending page 1. Batch provider endpoints where offered (FCM v1 accepts multiple messages per request).

9. Settings, templates, analytics, securityCopy Anchor Link

• Settings — (user, channel, type, opt_in); cached; checked at enqueue and in the worker; transactional exempt.
• Templates — versioned per locale; message carries template_id + version for deterministic late rendering.
• Analytics — emit every transition (queued, sent, delivered, opened, clicked, unsubscribed). Product funnel and your best outage detector: delivered/sent for one provider dropping 98% → 60% shows before their status page does. Ops metric #1: queue depth + oldest-message age per lane.
• Security — callers authenticate (appKey / mTLS) and are rate-limited per key; provider credentials (.p8, FCM service account) in a secrets manager, never in a payload; no PII in push payloads (Apple/Google see them in the clear) — send an ID, let the app fetch.

10. End-to-end trace (narrate it, then kill something)Copy Anchor Link

  ORDER 987 SHIPPED  (transactional, push + email)
  order-svc ─► POST /v1/notifications  key=order-987-shipped  user=42
            ─► auth appKey → log insert-if-absent (new) → cache: 2 devices (ios, android), email,
               push=on, email=on → 3 rows queued → enqueue ios-txn, android-txn, email-txn
            ─► 202  (~8 ms; caller moves on)

  iOS worker     dedup SETNX ok → prefs ok → txn: no cap → provider bucket ok → render
                 → APNs 200 → log sent → ack.  Phone buzzes.

  Android worker → FCM 404 UNREGISTERED → log dropped, DELETE device row, ack.  No retry.

  Email worker   → SES 503 → log attempt=1 → re-enqueue +2 s×jitter → ack original
                 → retry → SES 200 → log sent
                 → 40 s later SES webhook "delivered" → match provider_message_id → log delivered

  KILL: iOS worker gets APNs 200, dies before mark/log/ack → queue redelivers
        mark present  → Gate 1 drops it. No duplicate.
        died in the post-ack gap → mark absent → sends again. Duplicate.  ← say it unprompted

11. Interview scriptCopy Anchor Link

  1. clarify      channels · soft real-time · triggers (client + scheduler) · opt-out ·
                  10M / 1M / 5M per day · PRIORITY CLASSES? sent vs delivered?   ← unprompted
  2. estimate     ~160/sec avg, bursts 10³–10⁴/sec → design for bursts + third parties
  3. channels     APNs (token, HTTP/2, 4 KB) · FCM v1 (OAuth; legacy dead) · SMS/email gateways + webhooks
  4. contact info devices table, many per user, tokens rot → prune on UNREGISTERED
  5. naive        one sync server → caller latency, no retry, skyscraper, SPOF
  6. decoupled    "API accepts the job, doesn't do it" → persist + dedup → queue per channel → pool per channel
  7. reliability  persist-before-ack (+ outbox) · ambiguous timeout → idempotency key, 3 layers,
                  residual window · classify 4xx vs 5xx · backoff+jitter · DLQ · breaker + bucket · TTL
  8. gates        dedup → prefs (re-check!) → two rate limiters → late render → send+log
  9. lanes        head-of-line blocking → lanes per class, own pools + budgets, TTL; fan-out worker w/ cursor
 10. extras       settings, templates, funnel analytics, depth-based autoscaling, no PII in payload
 11. walk it      order-shipped end to end; kill the iOS worker after APNs 200 yourself

Steps 7 and 9 separate a pass from a strong pass.

12. The lines that earn the mostCopy Anchor Link

1. "Steady load is trivial — the design exists to absorb bursts and survive third parties; that's why there's a queue between every pair of tiers."
2. "The API accepts the job, it doesn't do the job — persist, enqueue, 202. Provider latency never reaches the caller."
3. "An ambiguous timeout makes retrying correct and makes duplicates inevitable — so I make the second attempt harmless with a caller-supplied idempotency key derived from the logical event, checked at the boundary and again in the worker."
4. "At-least-once delivery, exactly-once effect — except a crash in the window after the provider ack, which I can't close from my side."
5. "Classify before you retry: UNREGISTERED prunes the token and never retries; 429/5xx back off with jitter; N failures dead-letter. A tight retry loop is a DoS on my own provider."
6. "Sent is not delivered. Push never confirms; SMS/email confirm by webhook. Separate states or I lie to the business."
7. "An OTP must never sit behind a 10M-recipient campaign — lanes per class with their own pools and provider budgets; a priority queue alone still shares the quota."
8. "I autoscale on queue depth and oldest-message age, not request rate."

13. Numbers to rememberCopy Anchor Link

14. Common follow-upsCopy Anchor Link

15. When NOT to go deepCopy Anchor Link

Product interview ("design Instagram") → one sentence:

"Events go to a notification service — stateless API that dedups and persists, one queue per channel and priority class, per-channel workers with retries and provider rate limiting into APNs/FCM/SMS/email."

Go deep only when the notification system is the question, or the interviewer pushes on duplicates, ordering or provider outages.

16. Glossary (A–Z, one line each)Copy Anchor Link