System Design: Video Streaming Platform Like YouTube

Video is unusual because the files are huge, processing takes a long time, and nearly all the cost lies in storage and delivery.

1. Requirements#

Functional: upload videos, watch them smoothly on any device, search, and count views.

Non-functional: videos must never be lost, playback starts quickly, and it works on slow networks. A short delay before a new video appears is acceptable.

2. Estimates#

uploads_per_day = 1_000_000
avg_minutes = 5
mb_per_minute = 50
formats_multiplier = 3

raw_tb = uploads_per_day * avg_minutes * mb_per_minute / 1_000_000
stored_tb = raw_tb * formats_multiplier
print(f"New uploads: {raw_tb:,.0f} TB/day raw, {stored_tb:,.0f} TB/day with all formats")

views_per_day = 5_000_000_000
watch_mb = 5 * 20
gbps = views_per_day * watch_mb * 8 / 86_400 / 1000
print(f"Average outgoing traffic: {gbps:,.0f} Gbps")

New uploads: 250 TB/day raw, 750 TB/day with all formats
Average outgoing traffic: 46,296 Gbps

No single data centre should serve that. A CDN is essential.

3. Upload#

Send large files straight to storage, not through your servers:

1. Client --> API: start upload, get a pre-signed URL
2. Client --> Object storage: upload in chunks
3. Storage --> Queue: "new video" event
4. Workers process it, and the video becomes available

Chunked uploads split the file into parts, so that a failed connection only repeats one part. Each chunk is uploaded separately, and they are joined at the end:

size_mb = 1_024
chunk_mb = 8
chunks = -(-size_mb // chunk_mb)
print(f"{chunks} chunks")
print(f"If the network fails at 90%, re-send 1 chunk ({chunk_mb} MB), not {size_mb} MB")

128 chunks
If the network fails at 90%, re-send 1 chunk (8 MB), not 1024 MB

4. Transcoding#

A raw upload will not play well everywhere. Transcoding converts it into several resolutions and formats. It is slow, so it runs as a pipeline of jobs:

Raw video --> Split into segments --> Encode each (parallel workers)
                                          |
    thumbnails, captions, safety checks <-+--> Combine --> Storage

Splitting makes work parallel, so that a long video needs far less waiting time:

video_minutes = 60
encode_minutes_per_video_minute = 2
for workers in (1, 10, 60):
    print(f"{workers:>2} workers: {video_minutes * encode_minutes_per_video_minute / workers:>5.0f} minutes")

 1 workers:   120 minutes
10 workers:    12 minutes
60 workers:     2 minutes

Failed segments can be retried on their own. A queue feeds the workers, and cheap spare cloud capacity often runs this kind of job.

5. Adaptive streaming#

Videos are cut into short segments of a few seconds, each at several qualities. The player measures the connection, and chooses the quality of each next segment. This is adaptive bitrate streaming, using formats such as HLS and DASH.

video/
  master.m3u8           lists the available qualities
  1080p/seg0.ts, seg1.ts, ...
  720p/seg0.ts, seg1.ts, ...
  360p/seg0.ts, seg1.ts, ...

A simple player's logic looks like this:

levels = {"360p": 1.0, "720p": 3.0, "1080p": 6.0}


def choose(mbps):
    usable = mbps * 0.8
    fits = [name for name, need in levels.items() if need <= usable]
    return fits[-1] if fits else "360p"


for speed in (10, 5, 1.5, 0.8, 8):
    print(f"{speed:>4} Mbps -> {choose(speed)}")

  10 Mbps -> 1080p
   5 Mbps -> 720p
 1.5 Mbps -> 360p
 0.8 Mbps -> 360p
   8 Mbps -> 1080p

That is why quality drops on a weak network, instead of the video stopping.

6. Serving#

Viewer --> CDN edge --hit--> segment
             |
             miss --> Regional cache --> Origin (object storage)

Video popularity is very uneven: a small share of videos gets most views. Popular videos sit on edge servers, and rarely viewed ones stay at the origin, in cheaper storage.

7. Metadata#

Titles, channels and comments live in a database, with heavy caching, and read replicas. Search uses a search engine such as Elasticsearch.

8. Counting views#

Do not update a database row on every view. Send events to a stream such as Kafka, aggregate them in batches, and update counts every so often. Counts can be slightly behind, and the pipeline can also filter fake views.

9. Cost#

Storage and CDN traffic are the main costs. Common savings are better compression such as AV1 for popular videos, moving old videos to cold storage, and creating rare formats only on request.

Trade-offs to mention#

• More formats help playback, but raise storage cost.
• Encoding before publishing delays a video, but avoids buffering.
• Eventual view counts are acceptable, and exact ones are expensive.

Key takeaway#

Next: Design a Ride-Sharing App.