Design YouTube
1. The map: three pipes and one rule
YouTube is three systems connected by storage, plus a normal web app on the side:
| Pipe | Job | Character |
|---|---|---|
| Upload | get a creator's file (up to 1 GB, shaky network) into storage | reliable, resumable |
| Processing factory | turn one upload into many playable versions + thumbnails + checks | CPU-heavy, slow, asynchronous |
| Delivery network | stream versions from near the viewer, adapting quality live | bandwidth-heavy, cost-dominant |
| Side app | metadata, views, comments, search, recommendations | normal web service |
Video bytes never flow through your API servers. API servers hand out permissions and metadata. Uploads go straight to storage; playback comes straight from the CDN.
What makes it different from everything so far
| Chat | News feed | YouTube | |
|---|---|---|---|
| Unit of data | 200 B message | 1 KB post | 300 MB video |
| Main cost | servers | servers, cache | network bandwidth |
| Main work | routing | fan-out | transcoding (CPU) |
| Read:write | ~1:1 | ~100:1 | ~50:1 by bytes |
| Latency need | ms | ~100 ms | upload: minutes OK; playback start ~1–2 s |
- Cost drives the design — the bill is bytes sent to viewers.
- Processing is a pipeline, not a request — minutes to hours, parallel, retryable.
- Popularity is extremely skewed — a long tail shapes storage, encoding, caching.
2. Requirements (extract these)
Book's scoped answers:
| Question | Answer | Consequence |
|---|---|---|
| Features | upload + watch | comments/search/recs out of scope |
| Clients | mobile, web, smart TV | many formats → transcoding |
| Scale | 5M DAU, ~30 min/day | |
| Users | international | global CDN |
| Inputs | most resolutions and formats | robust inspection step |
| Encryption | required | |
| Max file | 1 GB | |
| Cloud services | allowed (blob storage, CDN) | don't build object storage |
Focus areas: fast uploads, smooth streaming, change quality, low infrastructure cost, HA, scalability, reliability.
Senior questions to ask unprompted:
| Question | Why it matters |
|---|---|
| VOD or live? | very different systems; scope to VOD, cover live at the end (§12) |
| DRM for premium content? | changes packaging (§10) |
| Copyright matching / moderation? | extra pipeline stages → pipeline must make adding stages easy (DAG, §6) |
| How soon watchable after upload? | "low quality in minutes" → publish the low rung first (§5, §6) |
3. Estimation
Book assumptions: 5M DAU · 5 videos watched/day · 10% upload 1 video/day · 300 MB avg.
Storage (originals) 5M × 10% × 300 MB = 150 TB/day (renditions multiply this)
Upload ingest 150 TB/day ÷ 10⁵ s ≈ 1.5 GB/s ≈ 12 Gbps
Egress (viewing) 5M × 5 × 300 MB = 7.5 PB/day
7.5 PB ÷ 10⁵ s ≈ 75 GB/s ≈ 600 Gbps average
CDN cost 7.5 PB/day × $0.02/GB = $150,000/day ≈ $55M/year($0.02/GB = book's CloudFront average; big-customer discounts still leave it substantial.)
Each number → a decision:
- 150 TB/day → blob storage, not a DB; storage tiers over time.
- 600 Gbps egress → must come from a globally spread CDN.
- $55M/yr at only 5M DAU → cost is the headline problem; why YouTube/Netflix built their own delivery networks.
- Egress ≈ 50× ingest → optimise the read path's cost first.
4. High-level design — three boxes
- CDN — press play → video streams from the CDN.
- API servers — everything else: feed, recommendations, video page, upload permission, metadata.
- Metadata DB + cache — behind the API.
5. The upload pipe
5.1 Naive version — POST the file to an API server — breaks
- API fleet becomes a file-transfer fleet (12 Gbps ingest, minutes per connection).
- Dropped connection restarts from zero (1 GB fails at 90%).
- Slow for distant users (~200 ms RTT per TCP window to a far data center).
5.2 The real flow
Flow (a) — upload the video:
- Creator asks API → server creates video row
status=uploading, returns a pre-signed URL (signature + expiry; write exactly one object, no credentials). - Client uploads directly to original storage (S3/GCS) — API never touches bytes.
- "Object created" event → transcoding fetches the original (§6).
- On finish, two outputs:
- renditions → transcoded storage → distributed to the CDN,
- "transcoding complete" event → completion queue.
- Completion handler workers pull events → update metadata DB + cache
(
ready, renditions list) → API tells the client.
Flow (b) — metadata, in parallel: title, description, file name, size, format → API → metadata DB + cache. Don't make users wait for a 1 GB upload to save a title.
Why a completion queue? Decouples the compute fleet from the DB: transcoders finish in bursts; if the DB is briefly down, events wait instead of being lost.
5.3 Video state machine
- Visible to others only in
ready. - Partially ready (senior): mark playable when the lowest rung (360p) finishes; add higher rungs to the manifest as they complete.
failedcarries a reason: corrupt file, unsupported codec, policy.
5.4 Fast, resilient uploads
| Technique | Detail |
|---|---|
| Parallel, resumable, chunked | chunks upload independently; failed chunk retried alone; dropped connection resumes from last confirmed chunk (S3 multipart, GCS resumable). Book: client splits at GOP boundaries so chunks transcode without re-splitting |
| Upload centers near users | CDN edges as upload entry points; short RTT to edge, provider backbone to central storage |
| Idempotent chunks | key by (upload_id, chunk_number) → retries overwrite |
| Dedup identical files | hash the file; byte-identical re-uploads reuse existing transcoded output |
6. The processing factory — transcoding
6.1 Why transcode (book's four reasons)
- Raw video is enormous — an hour of HD at 60 fps can be hundreds of GB.
- Devices support different formats.
- Viewers have different bandwidth → several qualities.
- Bandwidth changes mid-video → switch smoothly (ABR, §7).
6.2 Container vs codec — keep them apart
| Term | Meaning | Examples |
|---|---|---|
| Container | the box holding video, audio, subtitles, metadata; usually the file extension | .mp4, .webm, .mov |
| Codec | the compression algorithm for the video track | H.264 (plays everywhere, least efficient) · VP9 / HEVC · AV1 (smallest, most expensive to encode) |
Analogy: container = shipping box; codec = how tightly the contents are packed.
Codec as a cost lever (senior): AV1 = smaller files = less egress (the biggest bill), but much more CPU. Watched 100M times → pay CPU once, save on every view. Watched 12 times → not worth it. → Encode everything cheap first; re-encode proven-popular videos in expensive codecs later.
YouTube Argos VCU: custom transcoding chip; each upload → more than a dozen versions; 20–33× compute efficiency vs previous software encoders; 4K from days to hours; later generations target AV1. Shows how central transcoding cost is (don't propose custom silicon, mention it).
6.3 Bitrate ladder and per-title encoding
- Bitrate ladder = the renditions produced (240p … 4K, each at a target bitrate). More rungs → smoother switching, more encode time + storage.
- Fixed ladders waste bits: flat cartoon looks perfect at low bitrate; sports needs far more at the same resolution.
- Per-title encoding (Netflix, 2015): a ladder per video. Quality measured with VMAF (Netflix's perceptual metric). Senior "next step" after the book's fixed ladder.
6.4 The DAG model
Different jobs need different processing → model it as a directed acyclic graph of tasks with dependencies, configured per job (borrowed from Facebook's streaming video engine).
| Task | Purpose |
|---|---|
| Inspection | quality + malformed check — fail corrupt files before spending encode compute |
| Encodings | every rung × codec × container served |
| Thumbnail | generated or creator-uploaded |
| Watermark | overlay, if configured |
| Audio encoding | compress audio track |
| Extract info | duration, resolution, frame rate, codec |
Why a DAG, not a fixed script:
- Parallelism — independent tasks run at once on different machines.
- Flexibility — new step = new node. Copyright matching / moderation
are nodes producing a verdict;
readycan require them to pass. - Isolated retries — 1080p fails → retry that task only.
6.5 GOP — the trick that makes encoding parallel
- Most frames are stored as differences from earlier frames → can't cut at an arbitrary frame.
- GOP (Group of Pictures) = run of frames starting with a keyframe (complete picture), all others depending only on frames inside the group → each GOP decodes on its own; typically a few seconds.
- Split at GOP boundaries → every chunk encodes independently, in parallel. 10-min video → ~200 chunks → ~200 workers → ≈ one chunk's time, then stitch.
- Bonus: if every rendition is cut at the same boundaries, the player can switch quality at any boundary without a glitch (§7).
6.6 Transcoding architecture
| Component | Role |
|---|---|
| Preprocessor | split into GOP chunks (if client didn't) · generate DAG from config files · cache chunks + metadata in temporary storage (retry from cache, not from scratch) |
| DAG scheduler | split the DAG into stages (1: split video/audio/metadata · 2: inspection, audio encode, extract · 3: encodings, thumbnail, watermark); a stage starts when the previous finishes |
| Resource manager | task queue (priority queue of tasks) · worker queue (workers by availability) · running queue (task → worker) · task scheduler pops best task + best worker, records running, removes when done |
| Task workers | stateless; run whatever task they're given; different hardware per task type (GPU/VCU for encode, cheap CPU for thumbnails) |
| Temporary storage | metadata in memory; chunks in blob storage; deleted when done |
| Encoded video | every rendition, segmented, plus the manifest → transcoded storage → CDN |
Priority (senior): low-resolution rendition of every new upload = highest priority → everything watchable fast; 4K and expensive codecs = low priority / off-peak when compute is cheap.
Message queues between stages → slow stage doesn't block the previous one; each stage scales independently.
7. The delivery network — streaming
7.1 Downloading vs streaming
Downloading = fetch whole file first. Streaming = play while fetching → start in ~1–2 s, seek without downloading the beginning. Protocols (book): MPEG-DASH, Apple HLS, Microsoft Smooth Streaming, Adobe HDS — HLS and DASH dominate today.
7.2 Adaptive bitrate streaming (ABR)
- Transcoding cut every rung into segments of a few seconds, boundaries aligned across rungs.
- It wrote a manifest listing every rendition and segment URL:
.m3u8(HLS),.mpd(DASH). - Player: download manifest → start at a low rung → measure download speed + buffer → pick each next segment's rung (up if fast and buffer healthy, down if slowing).
Switching quality is a client-side decision about which segment URL to request next. The server transcodes nothing on the fly and doesn't even know the player switched.
- "Change video quality" requirement = user overriding the player's row choice.
- Every cell is a small, static, immutable file with its own URL → ideal CDN object: cache forever, serve to everyone; CDN needn't understand video.
7.3 Streaming flow end to end
- Open video page → API: metadata + signed manifest URL.
- Fetch manifest from the CDN.
- Request segments from nearest edge.
- Edge hit → serve. Miss → regional tier or origin (transcoded storage) → cache → serve.
- Seek to minute 7 = request the segment covering minute 7.
One small API request per view; hundreds of segment requests to the CDN.
8. Cost — what the design is really about
Root observation: popularity is a long tail — few videos get most views; most are rarely watched.
8.1 Book's four optimizations
| Optimization | Idea |
|---|---|
| Only popular videos on the CDN | long tail served from your own high-capacity storage-backed video servers (cheaper per byte stored) |
| Encode less for unpopular / short videos | fewer rungs, or encode on demand when actually requested |
| Distribute by region | regionally popular videos don't need copies in every region |
| Build your own CDN with ISPs | at scale, stop renting; put your caches inside/next to ISP networks |
8.2 Senior additions
| Optimization | Idea |
|---|---|
| Codec by popularity | re-encode proven-popular videos in VP9/AV1: CPU once, save bandwidth every view |
| Per-title ladders | don't spend 1080p bits on a slideshow |
| Storage tiers | originals + rarely watched renditions → cold storage; keep originals to re-encode with better codecs later |
| Dedup identical uploads | reuse transcoded output |
⚠️ Cold originals vs encode-on-demand. On-demand encoding reads the original; from an archive tier that retrieval takes minutes to hours → the first viewer waits. Fix: always keep the low rung of every video warm (cheap, playable immediately); encode higher rungs on demand while the viewer watches the low one.
8.3 What "own CDN" looks like — Netflix Open Connect
- Netflix's own CDN carries 100% of its video traffic; ~95% delivered via direct connections to residential ISPs.
- Open Connect Appliances inside ISP networks, offered to ISPs free — ISP saves upstream transit, Netflix saves egress.
- Content copied to appliances off-peak when capacity is idle.
- YouTube does the equivalent via Google's own edge caching.
At scale, the cheapest byte never leaves the viewer's ISP — own cache boxes inside ISPs, filled at night, stocked with what's popular in that region.
9. Metadata and the rest of the app
| Area | Design |
|---|---|
Metadata (video_id, owner_id, title, description, status, duration, renditions, manifest_url, created_at) | relational, read-heavy, heavily cached. Start replicated MySQL/Postgres → shard by video_id/channel_id when writes outgrow one primary → routing in a proxy layer, not the app |
| YouTube's story | metadata on MySQL → primary + replicas → sharding → built Vitess (2010), a proxy that routes queries to the right shard. YouTube later moved off it; it became widely used open source |
| View counts | counters problem (news-feed notes): stream (Kafka) → aggregate per video in memory → batched increments every few seconds; dedup + fraud filtering before accepting (why counts freeze then jump) |
| Watch progress | KV keyed (user_id, video_id), written every few seconds while playing; eventually consistent — losing seconds is harmless |
| Thumbnails | small images from the CDN, like segments |
| Comments, likes, search, recs | separate services; recs build the home page — one sentence unless asked |
10. Safety
| Measure | Purpose |
|---|---|
| Pre-signed upload URLs | only authorised clients, only their object, limited time |
| DRM (FairPlay, Widevine, PlayReady) | segments encrypted; license only if allowed to watch; premium content |
| AES encryption | decrypted only during playback |
| Visual watermark | owner overlay, as a DAG task |
| Signed playback URLs (senior) | short-expiry tokens on manifests/segments → private/paid videos can't be shared by URL |
| Policy + copyright checks (senior) | DAG nodes; publishing gated on them |
11. Error handling
Recoverable (segment fails, worker crashes) → retry a few times, then mark failed + tell user. Non-recoverable (malformed / unsupported) → stop all tasks for the video, return a clear error.
| Failure | Response |
|---|---|
| Upload error | retry the failed chunk (resumable) |
| Video splitting error | old client can't split by GOP → server splits |
| Transcoding error | retry the task |
| Preprocessor error | regenerate the DAG |
| DAG scheduler error | reschedule the task |
| Resource manager queue down | use a replica |
| Task worker down | retry on another worker |
| API server down | stateless → LB routes elsewhere |
| Metadata cache node down | replicated; read another, replace |
| Metadata DB primary down | promote a replica |
| Metadata DB replica down | read another replica, replace |
Two things make the table work:
- Every task is idempotent — output to a deterministic path,
video_id/720p/h264/chunk_0042; retry overwrites, never duplicates (same idea asZADD). - Cached chunks make retries cheap — re-encode from temporary storage; nothing re-uploaded or re-split.
12. Live streaming — what changes
| Aspect | VOD | Live |
|---|---|---|
| Latency | minutes to process is fine | seconds; LL-HLS / LL-DASH partial segments → ~2–4 s |
| Encoding | queued, prioritised | must keep up in real time; dedicated capacity per stream |
| Parallelism | split into GOP chunks, 200 workers | future frames don't exist → little chunk parallelism |
| Errors | retry in the background | a 30 s retry is a visible outage |
| Ingest | upload to blob storage | stream protocol (traditionally RTMP) to an ingest server near the broadcaster |
13. End-to-end trace (narrate, then break)
UPLOAD creator (Jaipur), 6 min, 400 MB
API create v9 status=uploading → resumable pre-signed URL at a nearby upload edge
client chunks, 4 in parallel; title/description saved via API meanwhile
storage "object created" → preprocessor: check, ~120 GOP chunks → temp storage, build DAG
(inspection · 240p–1080p H.264 · audio · thumbnail · copyright + policy)
RM 360p tasks top priority → ~2 min: 360p + checks done → completion queue
handler v9 = ready, manifest lists 360p → creator notified → higher rungs appear over minutes
WATCH viewer (Mumbai)
API metadata + signed manifest URL
CDN manifest → start 360p → climb to 1080p in a few segments
first viewer: edge misses → origin; everyone after in the region: hits
POPULAR 20M views in a week → low-priority overnight VP9/AV1 re-encode → added to manifest
→ modern players switch → egress per view drops → pushed to ISP caches where popular
BREAK 1 upload drops at 70% → resume from last confirmed chunk; nothing re-sent
BREAK 2 worker dies mid chunk 42 at 1080p → RM reassigns → new worker reads chunk 42
from temp storage, writes same path (overwrite) → no duplicate, no re-upload
BREAK 3 corrupt file → inspection fails, non-recoverable → no encode spent,
status=failed with reason → creator told to re-export
BREAK 4 CDN edge region down → segments fail over to another edge / regional tier;
ABR may dip a few segments, playback doesn't stop14. Interview script
1. clarify upload + watch · clients · 5M DAU · international · ≤1 GB · encryption · cloud OK
VOD or LIVE? DRM? copyright/moderation? how soon watchable? ← unprompted
2. estimate 150 TB/day in · 7.5 PB/day out (~600 Gbps) · $150K/day CDN → COST drives design
3. framing three pipes (upload · processing · delivery) + "bytes never touch API servers"
4. high level client · CDN (bytes) · API servers (everything else) · metadata DB
5. upload pre-signed URL · direct, chunked, resumable · upload edges · metadata in parallel
· completion queue + handler · state machine with partial ready
6. transcode why (4 reasons) · container vs codec · codec-by-popularity · ladder / per-title
7. DAG video/audio/metadata branches · parallel, flexible, isolated retries · moderation node
8. GOP split at keyframes → parallel encode + aligned segments ← senior
9. architect. preprocessor · DAG scheduler · resource manager (3 queues) · workers · temp storage
· low rung first priority
10. streaming ABR: segments + manifest, client picks each segment; static files → perfect CDN objects
11. cost long tail: CDN for head, origin + lazy encode for tail, regional, own CDN in ISPs
12. side app metadata sharding via proxy (Vitess) · view counters · watch progress
13. safety signed upload + playback URLs · DRM · AES · watermark · policy gates
14. errors recoverable vs not · idempotent task outputs · cached chunks
15. live what changes
16. walk it upload → watch → goes viral, then break it four waysSteps 2, 8, 10 and 11 separate a pass from a strong pass.
15. The lines that earn the most
- "YouTube is three pipes — upload, processing, delivery — and video bytes never pass through the API servers."
- "At 5M DAU the CDN bill is about $150K a day, so cost drives every decision on the read path."
- "Uploads go directly to storage via a pre-signed, resumable, chunked URL; metadata is saved in parallel."
- "A container is the box, a codec is the compression — and codec choice is a cost lever: pay CPU once on popular videos to save bandwidth on every view."
- "Processing is a DAG: parallel, configurable per job, retried per task — moderation and copyright are just more nodes."
- "Splitting at GOP boundaries makes encoding embarrassingly parallel and aligns segments so quality switches are seamless."
- "The resource manager prioritises the lowest rung of every new upload, so videos are watchable in minutes."
- "ABR: the manifest lists every rendition's segments and the player picks each next segment — quality switching is a client decision about which URL to fetch."
- "Popularity is a long tail: head in ISP-embedded caches with efficient codecs, tail from origin with fewer rungs or encode-on-demand."
- "Every task writes to a deterministic path, so retries overwrite instead of duplicating."
16. Numbers to remember
| Thing | Value |
|---|---|
| Book scope | 5M DAU · 30 min/day · 5 videos watched/day · 10% upload 1/day · 300 MB avg · ≤1 GB |
| New originals | 150 TB/day |
| Upload ingest | ~1.5 GB/s ≈ 12 Gbps |
| Egress | 7.5 PB/day ≈ 75 GB/s ≈ 600 Gbps avg |
| CDN price (book) | $0.02/GB → $150K/day ≈ $55M/yr |
| Egress : ingest | ~50:1 |
| Raw HD video | an hour at 60 fps can be hundreds of GB |
| Segment length | a few seconds (GOP-aligned) |
| Low-latency live | LL-HLS / LL-DASH ≈ 2–4 s |
| Argos VCU | >12 versions per upload · 20–33× efficiency · 4K days → hours |
| Netflix Open Connect | 100% of video · ~95% via direct ISP connections · appliances free to ISPs · filled off-peak |
| Vitess | built at YouTube, 2010, MySQL sharding proxy |
17. Common follow-ups
| Follow-up | Answer skeleton |
|---|---|
| Upload fails at 90% | resumable chunked upload; retry only failed chunks |
| How to make videos watchable fast | low-rung-first priority; partial ready; add rungs to manifest later |
| Change quality / smooth playback | ABR: aligned segments + manifest, client picks per segment |
| Reduce CDN cost | long-tail split, encode-on-demand, regional distribution, codec by popularity, own CDN in ISPs |
| Add copyright check / moderation | new DAG node; gate ready on it |
| Transcoding too slow | GOP-split parallel encode, priority queues, specialised hardware (GPU/VCU) |
| Worker crashes mid-task | reassign; deterministic output path + cached chunk → safe retry |
| Corrupt upload | inspection fails non-recoverable → failed with reason, no compute wasted |
| Private / paid videos | signed short-expiry playback URLs, DRM |
| View counts at scale | stream + batched increments + fraud filtering; lagging display OK |
| Metadata DB too big | shard by video/channel ID behind a routing proxy (Vitess) |
| Live streaming | real-time encode, short/partial segments, RTMP ingest, little chunk parallelism |
| Duplicate uploads | content hash → reuse transcoded output |
18. When NOT to go deep
Inside a bigger question (e.g. "design TikTok's backend" focused on feed):
"Uploads go straight to blob storage via pre-signed URLs; a DAG-based transcoding pipeline produces GOP-aligned renditions and manifests; viewers stream adaptively from a CDN, with popular content pushed to edge caches and the long tail served from origin."
Go deep only when video IS the question, or the interviewer pushes on transcoding, streaming or cost.
19. Glossary (A–Z, one line each)
| Term | Meaning |
|---|---|
| ABR (adaptive bitrate) | player chooses each next segment's quality from bandwidth + buffer |
| AES encryption | symmetric encryption of video, decrypted at playback |
| Argos VCU | Google's custom video transcoding chip for YouTube |
| AV1 / VP9 / H.264 / HEVC | video codecs; AV1 most efficient + most compute, H.264 most compatible |
| bitrate ladder | the set of renditions (resolution + bitrate) produced per video |
| CDN | edge servers caching and serving content near viewers |
| codec | compression algorithm for the video track |
| completion handler / queue | workers + queue that mark videos ready after transcoding |
| container | file format holding video, audio, metadata (.mp4, .webm) |
| DAG | directed acyclic graph of processing tasks with dependencies |
| DAG scheduler | splits the DAG into stages of runnable tasks |
| DRM | license-based playback protection (FairPlay, Widevine, PlayReady) |
| encode on demand | transcoding a rarely watched video only when requested |
| GOP | group of pictures: keyframe + dependent frames; independently decodable unit |
| HLS / MPEG-DASH | dominant HTTP adaptive streaming protocols |
| keyframe | a complete frame others depend on; starts each GOP |
| long tail | most videos get few views; drives tiered cost strategy |
| manifest | file listing renditions + segment URLs (.m3u8 HLS, .mpd DASH) |
| non-recoverable error | retry can't help (malformed/unsupported) → stop and report |
| Open Connect | Netflix's own CDN with appliances inside ISP networks |
| original / transcoded storage | blob storage for raw uploads / processed renditions |
| per-title encoding | a bitrate ladder tailored to each video's complexity |
| pre-signed URL | time-limited signed URL allowing one direct upload/download |
| preprocessor | splits GOPs, builds the DAG, caches chunks |
| rendition | one encoded version (resolution × bitrate × codec) |
| resource manager | task / worker / running queues + scheduler assigning tasks |
| resumable upload | chunked upload that continues from the last confirmed chunk |
| segment | few-second piece of one rendition; static, cacheable file |
| signed playback URL | short-expiry token restricting segment/manifest access |
| task worker | stateless machine executing one DAG task |
| temporary storage | cached chunks + metadata for in-flight jobs |
| transcoding | converting video to other formats/resolutions/codecs |
| upload center | edge entry point near creators for faster uploads |
| Vitess | MySQL sharding proxy created at YouTube |
| VMAF | Netflix's perceptual video-quality metric |
| VOD vs live | on-demand (process later) vs real-time (encode in seconds) |
| watermark | visual overlay identifying owner |