System Design: How to Build a Real-time Collaborative Editor Like Google Docs — Part 2

This is the second part of a 2-part series. In Part 1 we covered — requirements, architecture, OT vs CRDT (concurrent edit), the storage layer, and real-time sync (WebSocket + Pub/Sub). In this part we'll dive into the remaining hard pieces of a production-grade system: offline editing, permissions, fault tolerance, scaling.

Part 1 ended with the real-time sync flow — meaning a character reaching one online user from another. But in the real world the user isn't always online, the network isn't always reliable, servers crash, document permissions get complex, and users are spread across millions. What it takes to handle all of this is the focus of this part.

Offline Editing — the hardest part

Say you're editing a document while sitting on a train. Suddenly the train enters a tunnel — the internet drops. But you don't even know it, you keep typing. After 30 minutes the tunnel ends and the internet comes back. During those 30 minutes the people editing the same document alongside you were online — they changed plenty. Now what?

This scenario is the hardest challenge of a collaborative editor. Real-time conflict (when two people are online at the same time) is easy — the divergence is small, just milliseconds. But offline conflict — the divergence is huge, sometimes hours on end. Filling this gap takes two steps:

1. Local-first mode — so the user can keep working normally while offline
2. Reconciliation — merging the two versions back into one on reconnect (detail in the next subsection)

Local-first mode — work continues even without internet

The core idea — the primary copy of the document lives on the user's device, not with the server. When there's internet it syncs with the server. When there isn't, the app keeps functioning as if nothing happened.

This philosophy is called "local-first software". Google Docs supports it to a large extent, Linear/Notion do it more seriously. It has three core components:

1. Operation queue (in IndexedDB)

The moment a user types, an operation is generated. While online it would go to the server right away. While offline — it gets stored in the browser's IndexedDB.

What is IndexedDB? The browser's built-in persistent database. Unlike ordinary in-memory state (which disappears on page reload), data in IndexedDB is written to disk. Meaning — even if the user shuts the laptop down, when they open the app tomorrow morning the queued operations are intact.

Offline op queue (stored in IndexedDB):
[
  { op_id: "abc-001", type: "insert", position: 5, text: "Hello", timestamp: ... },
  { op_id: "abc-002", type: "insert", position: 10, text: " world", timestamp: ... },
  { op_id: "abc-003", type: "delete", position: 0, length: 1, timestamp: ... },
  ...
]

Each operation has a unique op_id — needed later for deduplication during reconcile. (If the network is flaky one operation might get sent twice — with op_id the server can detect "this is already processed".)

2. Optimistic UI update

The user's experience has to stay normal. Meaning — when they type, the character should show up on screen instantly, without waiting for the server's confirmation.

This is optimistic update — "I'm assuming the operation will succeed, and updating the UI accordingly".

User keystroke 'X'
    ↓
1. Insert 'X' into local document state (instant)
2. UI re-render — 'X' shows on screen (instant)
3. Push to operation queue (instant)
4. If online — send to server (background)
   If offline — just wait in the queue

The user has no idea whether there's internet or not. At the top of the app there's usually a small indicator — "Online" / "Offline — changes saved locally" — so the user can know if they want to.

Risk — sometimes the optimistic assumption can be wrong (e.g. a conflict marker can show up during reconcile). But it works 99% of the time, and for a responsive feel that risk is worth taking.

3. Reconnection detection

Two ways to track network status in the browser:

1. navigator.onLine API

The browser has a built-in property — navigator.onLine — that returns true/false. It can be checked from JavaScript at any time:

if (navigator.onLine) {
  console.log("Online আছি মনে হচ্ছে");
} else {
  console.log("Offline");
}

// Event listener ও আছে:
window.addEventListener('online', () => { /* online হলো */ });
window.addEventListener('offline', () => { /* connection গেলো */ });

Looks nice — built-in, free, event-based. But what it actually reports is essentially the state of the OS-level network interface. Meaning — whether your laptop is connected to WiFi, or whether an ethernet cable is plugged in. But being connected to WiFi doesn't mean there's internet.

Real scenarios where it's wrong:

• You're connected to a cafe's WiFi, but their router's upstream cable is cut — navigator.onLine = true says, but actually no site will load
• A hotel's captive portal — no internet until you log in, but WiFi is connected — onLine = true
• VPN is connected but the VPN server is unreachable — onLine = true
• ISP outage — there's a link at the modem, but no internet — onLine = true

The reverse can happen too — in a virtual machine or VPN environment the OS often gets confused and says onLine = false even though there actually is internet.

Bottom line: navigator.onLine gives you only a hint, not ground truth. Depending on it alone often leaves the user frustrated — "I'm online, why isn't it working?"

2. Heartbeat ping

The way to know the real ground truth — ask the server yourself. Every few seconds (typically 10-30s) send a small request — if a response comes back, online; if not, offline.

setInterval(async () => {
  try {
    const res = await fetch('/api/ping', { method: 'HEAD' });
    if (res.ok) setStatus('online');
    else setStatus('offline');
  } catch {
    setStatus('offline');  // network error = offline
  }
}, 15000);

This is a real test — it verifies whether data is actually getting through. All captive portals, ISP outages, VPN issues get caught.

But the trade-off:

• Each ping costs bandwidth + battery (especially on mobile)
• If a disconnect happens between intervals, detection lags by a few seconds
• It adds load to the server (10K concurrent users × 4 pings/min = 40K req/min)

So in production the WebSocket connection itself is usually used as the heartbeat — no extra polling needed. WebSocket's ping/pong frame protocol is natively built in:

Client → Server: PING frame (every 30s)
Server → Client: PONG frame (instant reply)

If no PONG comes back, assume the connection is dead after a timeout.

3. Combining both — best of both

A production system usually uses a layered approach:

• Fast hint — immediately show an offline indicator in the UI on a navigator.onLine event (fast user feedback)
• Real verification — track the actual connection state with a WebSocket heartbeat
• Reconnect logic — when a connection drop is detected, start a reconnect attempt immediately, with exponential backoff (wait 1s, on fail 2s, then 4s, 8s, 16s ... max 60s) — so the server isn't flooded if the problem is long-running

With this combination the user sees an "Offline" badge in the UI fast, the internal logic knows the real truth, and reconnect retries smartly.

A combination of the two is generally used. When the WebSocket connection drops, offline mode triggers immediately; reconnect attempts keep running with exponential backoff (1s, 2s, 4s, 8s...).

The moment reconnect happens — the queued operations from IndexedDB start getting pushed to the server one by one. After that comes reconciliation (detail in the next subsection).

The user's journey, all together

[Online, typing normally]
   ↓ Internet drops
[Offline indicator shows — "Changes saved locally"]
   ↓ User keeps typing (UI optimistic, queue piles up)
[30 minutes later — internet comes back]
   ↓ "Syncing..." indicator
[Queued ops get pushed to the server]
   ↓ Reconciliation (next subsection)
[Sync complete — normal collaboration mode]

This whole flow feels seamless to the user. Underneath, IndexedDB persistence, optimistic update, heartbeat, exponential backoff, queue replay — all work silently.

Reconciliation — the job of merging two versions into one

Say Alice was editing a document at a cafe, and the WiFi suddenly dropped. She didn't notice, she kept typing. After 30 minutes the WiFi came back. Two things happened in those 30 minutes:

• In Alice's browser — Alice herself generated 50 operations (sitting in the offline queue)
• On the server — Bob, Carol, Dave and others together applied another 200 operations. The document has changed a lot now.

The two worlds have drifted apart. Alice wrote her 50 operations based on the state the document had 30 minutes ago. But the server's current state is completely different. The position numbers, references in Alice's operations — all stale.

The process of merging these two worlds back into one is called reconciliation. In plain terms — "fit your offline changes into the current online document, so everyone has one document and no one's work is lost."

Why is reconciliation hard?

Yes, reconciliation is a hard job — it's not as simple as just "send the offline changes to the server". The difficulty comes from four directions:

1. Position numbers won't match (stale references): Each of Alice's operations has a position number (e.g. "insert at position 11"). But that 11 was relative to the document from 30 minutes ago. Now the server's document has changed — that 11 now points to a completely different place.

2. Someone might delete what Alice was editing: The paragraph Alice was revising offline, Bob deleted it on the server. Where does Alice's edit go now? Delete-restore? Lost?

3. Getting the order right: Alice's 50 ops and the server's 200 ops — what order do these go in the final document so everyone gets the same result? Causal ordering has to be preserved.

4. Performance: 50 × 200 = 10,000 transformation calculations (in OT). For large divergence this gets exponentially expensive.

Example: why position numbers go stale

The first problem (position mismatch) is the most common, so let's understand it through that.

Earlier document (when Alice went offline):
  "Hello world"

Alice wrote offline:
  insert "!" at position 11  →  she thinks the result will be "Hello world!"

But here's what happened on the server in those 30 minutes:
  Bob deleted 'world'.
  Carol inserted 'everyone'.
  Server's current document: "Hello everyone"

Now Alice's "insert '!' at position 11" — what does position 11 mean now?
In "Hello everyone" position 11 is after the 'n' — meaning "Hello every!one".
But Alice wanted to put '!' at the end of the string — meaning it should actually go to position 14.

This position mismatch happens to all 50 of Alice's operations — none will land in the right place unless someone runs a smart transformation. Bigger divergence = bigger mismatch = more transformation = more chance of subtle bugs.

This is exactly why reconciliation is a kernel-level hard problem — multiple concurrent edits, stale state, deterministic ordering, performance — all have to be handled at once.

Why it's easier in CRDT

The core idea in one sentence — in CRDT the position isn't a number, it's a globally unique address. That address never changes, so the address from 30 minutes ago and the address now are the same — there's no mismatch at all.

The CRDT trick — give characters lifelong unique IDs

The insight of CRDT is — if each character's identity doesn't depend on the document's state, then there's nothing to adjust anymore.

So in CRDT each character is given a tuple address: (numeric, replicaId). This address is assigned the moment the character is created, and never changes — not when something else is inserted into the document, not on delete, not when another replica does something.

Think of it like a physical address. Your home's address is "123 Lake Road". The house next door got demolished and turned into a new apartment, a new building went up across the street — your address stays the same, "123 Lake Road". If you want to send someone a letter, just writing this address is enough, you don't have to count the total number of houses in the neighborhood.

OT's position number is "house number 11 on this street" — the count changes when the neighborhood's configuration changes. CRDT's tuple is "123 Lake Road" — absolute.

Concrete trace

Say the initial state was "Hello world". The tuple for each character is like this:

H: (0.1, _)    e: (0.2, _)    l: (0.3, _)    l: (0.4, _)
o: (0.5, _)    ' ': (0.55, _)  w: (0.6, _)   o: (0.7, _)
r: (0.8, _)    l: (0.9, _)    d: (0.95, _)

When Alice wanted to put '!' at the end of the string offline, she computes its tuple locally — some number after the last character ('d' at 0.95), e.g. (0.99, alice). This tuple gets stored in Alice's IndexedDB queue.

In those 30 minutes Bob and Carol did a lot on the server — world deleted, everyone inserted. The server's state is now:

H(0.1) e(0.2) l(0.3) l(0.4) o(0.5) ' '(0.55) e(0.6,bob) v(0.61,bob) e(0.62,bob) r(0.63,bob) y(0.64,bob) o(0.65,carol) n(0.66,carol) e(0.67,carol)

Notice — Bob and Carol put their own replicaId on the characters they created, and Hello's tuples (0.1 to 0.55) are unchanged.

Now Alice comes online. She sends her queued op to the server:

op: insert '!' at (0.99, alice)

What does the server do to apply this op? It just adds it to the character list, then sorts by tuple:

H(0.1) e(0.2) ... e(0.67,carol) !(0.99,alice)
                                  ↑
                  Alice's '!' tuple sorts to the end

Render: "Hello everyone!" — exactly what Alice wanted.

What difficulties auto-solved

• ❌ No need to recalculate position — Alice's tuple is still what it was before
• ❌ No need to run a transformation function on the server — just sort, that's sufficient
• ❌ Order doesn't matter — whether Alice's op comes before or after Bob's, sorting gives the same final result
• ❌ No need to worry about Bob and Carol's existence — Alice knows nothing about them, yet it works

Where OT's reconciliation is a dance of hundreds of carefully-crafted transformation functions, in CRDT it's a single sorted insert. This is why offline-first products (Linear, some features of Notion, Figma) are built on CRDT.

Fun fact: Google Docs uses OT — so it doesn't support long offline sessions well either. Coming back after being offline for a few hours often shows conflict markers, or gets overwritten by the latest server version. With CRDT this problem wouldn't exist.

What happens in Alice's UI?

Disclaimer: The list below is an idealized UX — this is how it should be in some well-designed collaborative editor. Google Docs itself doesn't do all of this exactly. In reality, in offline mode Google Docs shows a "Working offline" badge, syncs silently on reconnect, and for irresolvable conflicts usually does last-writer-wins or both-versions-merge — but Google Docs doesn't show a full "Your version vs Server version" diff (like Git does). So this is a reference for product design, not an exact replica.

While reconcile is in progress, good UX is:

• Show a "Syncing your offline changes…" indicator on reconnect
• In the background the queued ops keep getting pushed to the server
• As the server accepts each one, Alice's local copy updates (Bob/Carol's changes start to appear)
• If there's any irresolvable conflict, show it to the user ("Your version" vs "Server version" diff)

When sync finishes the indicator goes away, and Alice returns to normal collaboration mode.

Concrete timeline example — A offline, B online, insert at the same position

So far it's all theory. Now let's trace a specific timeline — see what happens each second and what shows up on everyone's screen at the end.

Setup

Say User A and User B are editing the same document.

t=1s   A online, B online       Document: "Hello world" (v0)
t=2s   A went offline           Document: "Hello world" (v0)
t=3s   A insert 'X' at 5         A local:  "HelloX world"
                                  Server:   "Hello world" (A's op stuck in queue)
                                  B screen: "Hello world"
t=4s   B insert 'P' at 5         A local:  "HelloX world" (A knows nothing)
                                  Server:   "HelloP world" (v1)
                                  B screen: "HelloP world"
t=5s   A came online            ← reconcile happens here

Until t=5s, A's screen has "HelloX world" and B's screen has "HelloP world" — both have a different document. This is divergence.

What happens at t=5s

The moment A reconnects, two things happen in parallel:

1. A → Server: op_a1 = { insert "X" at 5, parent_version: v0, author: A }
2. Server → A: op_b1 = { insert "P" at 5, parent_version: v0, author: B }
   (A had missed it while offline)

The server sees — both op_a1 and op_b1 have parent_version v0, both at position 5, concurrent same-position insert. A tiebreaker is needed.

Say the convention — the character whose author_id is lexicographically smaller goes first. A < B, so A's X first, B's P after. (This is an arbitrary deterministic rule. As long as every replica follows the same rule, convergence is guaranteed.)

What the server does

Server state now: "HelloP world" (B's op already applied)
A's op_a1 arrives — concurrent, same position, A wins the tiebreak.
Apply op_a1 to "HelloP world":
  insert X at position 5 → P shifts to position 6
  Result: "HelloXP world" (v2)

What happens in A's browser

A's local state was "HelloX world". op_b1 arrives from the server:

A local: "HelloX world"
Apply op_b1 (insert "P" at 5) — but A's local already has X at position 5.
Tiebreak A < B, A's X first — B's P shifts to 6.
Transformed op: op_b1' = insert "P" at 6
Apply: "HelloXP world"

A's screen: "HelloXP world" ✅

What happens in B's browser

B's local state was "HelloP world". op_a1 arrives from the server:

B local: "HelloP world"
Apply op_a1 (insert "X" at 5) — concurrent with B's own op.
Tiebreak A < B, A's X at position 5 — P shifts to 6.
Apply: "HelloXP world"

B's screen: "HelloXP world" ✅

Final convergence

A's screen: "HelloXP world"
B's screen: "HelloXP world"
Server:     "HelloXP world"

Everyone converged on the same document. No one's writing was lost. A's X is there, B's P is there, only the order was fixed by a deterministic rule.

What if the tiebreaker rule were reversed?

Say the convention is "later author_id wins" — then B's P first, A's X after. Final: "HelloPX world". Both are valid. The key thing — every replica must use the same rule. If not, A has "HelloXP" and B has "HelloPX" — divergence permanent.

What if it were CRDT?

No explicit tiebreaker logic needs to be written on the server. The tuple structure does the job:

A's X: (0.55, A)
B's P: (0.55, B)
Lex sort: (0.55, A) < (0.55, B)
Final: ...o(0.5) X(0.55,A) P(0.55,B) ' '(0.6)...
Render: "HelloXP world"

A's browser sorting locally gives this result. B's browser sorting locally gives this result. The same document for both, without any server arbitration — this is the beauty of CRDT.

A hard edge case

User A was offline and wrote a paragraph. User B was online and deleted that paragraph. What happens on reconnect?

• Last-writer-wins (LWW): B's delete wins, A's writing is lost — bad UX.
• Resurrect on edit: A's edit means she wanted that content — bring the paragraph back.
• Show a conflict marker: "<<<< A's version | B's version >>>>" — like Google Docs' "Show changes".

Production systems usually use a combination — guess which is better with a heuristic, show it to the user in edge cases.

Permissions: Zanzibar

At first it sounds like — checking "who can do what" is trivial. An if-statement, a database query, done. But this seeming triviality is the trap. At massive scale this is one of the hardest challenges of a distributed system.

Why is it hard? Think about a few real scenarios:

• A Google Doc shared by link with 5 million users — a permission check is needed on every keystroke of every one of them
• A document shared with a team, with subteams inside the team, and individual users inside the subteams — nested groups
• When someone is removed from a team, access should be lost from all documents immediately — but stale permissions can sit in the cache

If each permission check takes 10ms, then 1 billion checks/sec means countless servers. Fitting it within the latency budget is not trivial.

To solve this problem Google built a system called Zanzibar — paper published in 2019. It handles billions of permission checks per second, with latency usually under 10ms. Not just Docs — Drive, Calendar, YouTube, Cloud — authorization for all Google products runs on it.

Relationship-based model

The traditional approach is role-based — a user has a role (editor, viewer), match the role list against the document. Simple but rigid — nested groups, sharing inheritance are hard to handle.

Zanzibar's approach is different — all permissions are expressed as relationships. A permission means a relation between two entities, inside a tuple.

doc:doc-123#editor@user:alice
doc:doc-123#viewer@group:team-engineering
group:team-engineering#member@user:bob

Format: <object>#<relation>@<subject>. How to read it:

• First line: "the editor of doc-123 is alice" — Alice can edit the document
• Second line: "the viewer of doc-123 is the team-engineering group" — the whole team can view
• Third line: "the member of team-engineering is bob" — Bob is a member of the team

Notice — the third tuple has no reference to the document. Just group membership. This decoupling is the trick.

Permission check = Graph traversal

Question: "Can Bob see doc-123?" Zanzibar treats this question as a graph problem.

Start node: bob
Goal: reach viewer of doc-123 (or any higher role)

Step 1: which groups is bob a member of? → team-engineering
Step 2: is team-engineering the viewer/editor of any document? → viewer of doc-123
Step 3: reached doc-123, target match → yes, Bob can see it

This simple example is 2 steps, but real scenarios can be more complex — groups inside groups, role hierarchy (editor automatically viewer), shared folder permissions that reach a document through inheritance. All resolve in graph traversal.

Benefits

• Easy to express — "all members of the Marketing team can view this doc" can be written in one tuple
• Nested groups natural — the deeper the nesting, the more graph hops, but the same algorithm
• Easy to audit — who got access to which document and why — you can tell by looking at the graph trace
• Multi-product reuse — the same Zanzibar instance can be used across Docs, Drive, Calendar — all

How at scale?

Graph traversal can be expensive — especially with deep nesting. To serve billions of checks at 10ms latency, Zanzibar uses a few tricks:

• Precomputed cache — the answers to common checks are computed in advance.
• Selective denormalization — some relations are kept flattened, less traversal at query time.
• Geo-distributed replica — a data copy in the region near the user, less latency.
• Tunable consistency — for some checks stale is OK (fast), for some strict (slow but accurate).
• Zookies — consistency tokens, "answer using data of this version" guarantee.

Bottom line: Zanzibar isn't just a library, it's a purpose-built distributed system — invoked trillions of times a day just to answer "this user can do this thing". It raises authorization from the rank of a product feature to the rank of infrastructure.

Fault Tolerance — what happens when a machine fails?

When building a large-scale distributed system you have to keep one truth in mind — machines will fail. When thousands of servers are running, every single day some server crashes, a network cable is cut, a datacenter catches fire, electricity goes out, some region goes fully offline. At Google's scale these are daily occurrences.

So when designing the system you have to assume — failure will come, the only question is when and how do we handle it. From the user's side there should be no data loss, the service shouldn't go down, no edit should be lost. Being able to give this guarantee is fault tolerance.

There are four core mechanisms for this job. A quick overview below:

Each is detailed below.

Leader-follower failover

In an earlier section we saw the concept of the "leader collab server" — for each document there's a single authoritative server, all operations go through that server (so there's no conflicting state). But what if that leader suddenly crashes? Someone has to take over quickly, otherwise the system is down for that document.

The solution — each leader has a few follower servers. The followers replicate all of the leader's operations in real time, so that if the leader dies they can take over immediately.

Process:

1. The leader regularly sends heartbeats to the followers — "I'm alive"
2. If no heartbeat comes for a few seconds, the followers understand the leader is dead
3. The followers elect a new leader among themselves with a consensus protocol like Raft/Paxos
4. The new leader takes over, clients reconnect there

The whole failover happens in ~1-5 seconds. To the user this shows up as a small "Reconnecting..." moment.

Split-brain risk

A subtle danger — if a network partition happens, meaning two server groups can't see each other, but clients can connect to both groups. Then each group might think "the other is dead, I'm the new leader" — two leaders, two diverging states. This is called split-brain.

Quorum-based consensus prevents this. Rule: to become leader you must be in contact with the majority (e.g. 3 of 5 servers). On a partition both sides can't have a majority — the servers on one side won't be able to become leader.

Idempotent operations

The network is unreliable. The client sent an operation to the server and is waiting for a response — suddenly the connection drops. The client doesn't know — did the server actually receive the operation or not?

Both are possible:

• The server didn't receive it (request dropped mid-way)
• The server received it, processed it, but the response didn't reach the client

What does the client do? The naive solution — send a retry. But if the server succeeded the first time, retry means a duplicate apply — "Hello" becomes "HelloHello".

The solution — idempotency. Each operation has a unique op_id (a UUID typically). The server keeps track of every processed op_id. When a retry comes, it recognizes the duplicate ID — "this is already processed, just ack again, I won't apply anything".

Client: insert "Hello" at position 5, op_id=abc-123
Server: receive, apply, ack
[network drop, ack lost]
Client: retry — insert "Hello" at position 5, op_id=abc-123  ← same ID
Server: "this op_id's work is already done, just ack again"

The user's data stays correct, no duplicate or loss.

Multi-region replication and snapshots

The other two mechanisms were discussed earlier:

• Multi-region replication — Bigtable, Spanner replicate globally. If one region goes fully down, serving continues from another region.
• Periodic snapshot — a full state checkpoint of the document alongside the operation log. After a crash you can start from the snapshot instead of replaying all ops from scratch.

Scaling — handling millions of users

Running a collaborative editor on a single server is easy. But 100M users, 100 edits per minute each — means billions of operations/sec at peak. This is impossible on one server. So scaling.

The core idea of scaling is one thing — spread the load across many machines. But how you spread it is the tricky part. If a document is split across multiple servers, sync breaks. If a user moves to a different region, latency goes up. These trade-offs have to be balanced carefully.

Horizontal scaling — add servers, not vertical

There are two scaling approaches:

• Vertical — adding more CPU/RAM to the same server. There's a limit — at some point a machine's hardware can't be increased further.
• Horizontal — adding more servers. No limit, but coordination challenges.

Google Docs uses the horizontal approach. Three tricks:

1. Stateless collab server. There's no long-term state in the server's memory — everything is in Bigtable/Spanner. Meaning any server can handle any request. When one server crashes, a new server can read the full state from Bigtable. This is critical for scaling — add as many servers as needed, with no dependency on anyone.

2. Document-based sharding. In an earlier section we saw — each document is assigned to a specific leader server. But how does it get assigned? With consistent hashing. hash(doc_id) → server_X. As servers increase, documents distribute evenly. All operations for a specific document go through that one server, so you get both locality + consistency.

3. Load balancer. The client first comes to the load balancer, and from there routes to the appropriate collab server. For WebSocket connections session affinity is needed — all of a client's messages should go to the same server, so state continuity is preserved.

Geographic distribution — reducing latency

User in America, server in Singapore — round-trip latency 200ms+. For a 100-200ms target this is impossible.

The solution — build edge servers / regional clusters. There will be separate clusters in different regions (US-East, US-West, Europe, Asia, ...). The user connects to their nearest region, latency comes down to 20-50ms.

But a complication — user in London, their collaborator in Tokyo. Both have data for the same document. The two are connected to different regions. How does it sync?

Cross-region replication. In the background data replicates across all regions. There's a small eventual-consistency window (~100-200ms), but same-region latency is fast. The trade-off is accepted because in collaboration a brief delay is manageable.

Where is strong consistency needed? Permission checks yes — if Alice loses access, she has to be blocked immediately, otherwise a security breach. For document content eventual consistency is OK — if Bob's change reaches Alice a few ms later it's not an issue.

Caching — fast data access

Hitting Bigtable on every request is slow. So a caching layer — keeping frequently accessed documents in memory.

Hot document (one someone is editing now):
  └─→ In memory cache + snapshot + Bigtable
        ↑
        all edits instant
Cold document (one no one has opened in a long time):
  └─→ Only Bigtable
        ↑
        loaded into cache on first open

Cache invalidation is tricky for a hot document during active editing — the cache has to be updated on every new operation, avoiding race conditions. Usually the work happens in the leader server's memory, so it's serialized.

Trade-offs — there's no "perfect" answer

In system design every decision has a trade-off. There's no magic combination that's best for everyone. Which is better depends on context.

A few concrete examples:

• Banking system — strong consistency a must. Much slower is OK, but if one dollar goes to the wrong place it's a disaster
• Social media feed — eventual consistency is fine. If you see a friend's post 5 seconds later it doesn't matter
• Google Docs — middle ground. Document content eventual (the user won't notice), permissions strong (security)

Mentioning these trade-offs explicitly in an interview is important. The interviewer wants you to show — "I didn't memorize one optimal solution; I can analyze the situation and make a decision."

Final words

Google Docs looks simple. Underneath — the math of Operational Transformation, Zanzibar's authorization graph, Bigtable's operation log, multi-region Spanner replication, WebSocket fan-out, an idempotency layer, an offline queue, conflict resolution heuristics. All interconnected.

These patterns aren't just in Google Docs — Figma, Notion, Linear, multiplayer games (Among Us, Agar.io), collaborative coding (Replit, VS Code Live Share) — all are variations of the same principles. Real-time multi-user state synchronization is a foundational problem of any distributed system.

In an interview, really — clarify requirements, draw the big picture, deep dive into the hard problems (OT/CRDT, offline sync), state trade-offs explicitly, don't forget production concerns (fault tolerance, idempotency, permissions).

And most important — don't design silently. Explain your reasoning out loud. The interviewer wants to see how you think, not just what answer you give.

Series wrap

Across the two parts we covered:

If you want to read Part 1 again, here.