TL;DR: Shakespeare succinctly describes what will follow, but our history storage mechanism was worse than O(N) — we were writing a whole copy of everything on every change. We moved to using events and an event-driven architecture. This is very much appreciated by our DB. Event-driven systems are an aggregation of the past — what’s past, is prologue! (The Tempest, Act II)

I discuss this topic very briefly here as well (2018 video): https://youtu.be/-VJ6FtTS8zg

Blended Labs is growing. More schools, more terms, more users.

Growth changes the shape of the problem. A design that behaves acceptably at a few thousand snapshots does not behave acceptably at a few hundred thousand, and the failure mode is rarely an outage — it is a cost curve that bends upward with accumulated history. Writes get slower as the history they append to gets larger; reads get slower as the number of historical copies they must diff grows; storage grows with every edit, not with every entity.

This is the story of removing that curve from the marksheet subsystem.

The naive implementation

The write model was — and still is — a flat table of submissions: one row per (period, course, class, column, student) with a value and audit fields. That part is fine.

Alongside every write, the system also stored a snapshot of the entire class marksheet.

POST-WRITE SNAPSHOT
┌──────────────────────────────────────────────────────────────┐
│ snapshot                                                     │
│   periodId, courseId, classId                                │
│   action        marksheetUpdate | periodUpdate | recalc | …  │
│   createdBy, createdAt                                       │
│   cells[]       every cell in the class                      │
│   categoryMeta[] per-category column metadata + links        │
│                                                              │
│   unique (periodId, courseId, classId, createdAt DESC)       │
└──────────────────────────────────────────────────────────────┘

One teacher changing one mark produced one new document containing the whole class. A recalculation of a term produced one snapshot per course, per pass.

The design was not stupid. It bought three things cheaply:

1. Point-in-time reads. “Show the marksheet as it was when this was locked” was a single fetch, not a reconstruction.
2. Decoupling. Snapshot reads did not depend on the write schema evolving.
3. Reproducibility. A generated report could cite the snapshot it was built from.

It worked while the numbers were small. And it was probably fast to implement (though I would contest the “fast” part).

Where it breaks

write amplification (Big-O notation below)
  one cell change  →  O(class size) bytes written
  one recalc       →  O(courses × class size)

storage growth
  proportional to writes × class size, not to the number of marks

read amplification
  history = load N snapshots, diff them cell by cell
  worst bucket: 862 snapshots, ~43 MB, >2,600 queries

dual truth
  a locked period had live rows AND a copy
  the locked read path read the copy, not the source

*period-locking is an internal concept we use for preserving data fidelity — unnecessary for what follows.

The measured cost

We decided to redact the numbers, but it was in the tens of GBs (not the low tens either)

At the current write rate, the design projected roughly 5x additional snapshot storage per year, none of which was required to answer either of the two product questions. And this calculation is just based on user growth. Ideally, from a product perspective, we should encourage more people to use more of our features — but this was gonna kill us and our pockets.

Constraints

The migration had to be invisible at the edges:

Target model

Remove the copy. Keep the write table. Add an append-only log and a projection. Make locked reads read the source.

BEFORE
┌──────────────┐
│  write path  │──► submissions
│              │──► full class copy
└──────────────┘
      │
      ├── grid / reports  ◄── submissions
      ├── history         ◄── load N copies + 3N queries
      └── locked reads    ◄── load copy


AFTER
┌──────────────┐
│  write path  │──► submissions ──► grid / reports / locked reads
│              │──► grade-entry event
└──────────────┘
                        │
                        ▼
                 ┌──────────────┐
                 │  event log   │
                 └──────────────┘
                        │
                        ▼
                 ┌──────────────┐
                 │  projection  │──► history drawer
                 └──────────────┘

What “event-driven” means here

The term is used loosely. In this system it means three concrete things:

1. State changes are recorded as immutable facts. A write appends an event describing what changed, carrying a monotonically increasing sequence. Facts are never edited in place; corrections arrive as new events.
2. Read models are derived, not shared. A consumer reads the log in sequence order and maintains a projection shaped for one question. It keeps its own cursor, so one slow projection does not hold back another, and a new projection is added by replaying from sequence 0.
3. Delivery is at-least-once, so effects must be idempotent. A consumer can see the same event twice after a crash or a leader handover. The projection carries a unique key per event, so the second application is a no-op.

Three things this pattern does not require — and that we deliberately did not build:

The event is appended inside the same transaction as the row it describes, so there is no dual-write window. The dispatcher’s poller owns delivery after commit; an inline wake-up is only a latency hint and is allowed to fail without affecting the write.

The price is eventual consistency: a projection may lag the log. That is acceptable only where the screen has a defined staleness budget. Here, the grid, report generation and locked marksheet paths all read current state and never lag; the history drawer may lag by one consumer batch.

Invariants first

Before any code, we wrote down what “correct” meant. Everything after that was a design search constrained by five sentences.

1  every change is observed exactly once                (no gaps)
2  observing it twice has no additional effect          (no duplicates)
3  changes are observed in the order they happened      (no reordering)
4  the derived view can be thrown away and rebuilt      (no lock-in)
5  no screen changes what it shows                      (no visible delta)

Seams

In Working Effectively with Legacy Code, Michael Feathers describes seams; that is what we picked.

The tempting move was to event-source the domain: make events the write model, derive everything from them, and let the submissions table become a projection too. That would have been more elegant and much more expensive — every write path, every migration, every report input, and the entire blast radius of the domain. We chose a narrower seam:

keep:      the write table, the domain model, the API surface
add:       event emission beside the write
add:       one projection, for one screen
remove:    the snapshot write
change:    locked reads from copy → source

Staleness

Event-driven systems trade consistency for availability, but “eventually consistent” is not a design — it is a question. The question is: which screens are allowed to be behind, and by how much?

grid / marksheet        current-state read        never lags
report generation       current-state read        never lags
locked marksheet        current-state read        never lags
history view            projection                may lag one batch
analytics               projection                may lag one batch

Putting the history view on the projection and everything else on current state was the decision that made the migration safe. If the projection is wrong, one screen is wrong. If we had moved the grid onto it, a consumer bug would have been a grading incident.

We treated the projection as infrastructure

A projection is not a cache. A cache can be dropped and recomputed on demand; a projection is a read path with its own cursor, its own failure modes, and its own recovery procedure. We designed it that way from the start:

- idempotent writes, keyed per source event
- a cursor that advances only after the write succeeds
- a documented rebuild path: truncate, reset cursor, replay

That last one matters more than it sounds. The ability to rebuild from the log is what makes the projection a derived artifact rather than a second source of truth.

Execution

The migration was sequenced so that no phase required rolling back the previous one.

PHASE 1   emit events beside the write
          snapshot still written, projection still empty
          gate: events exist, payloads are complete

PHASE 2   backfill history before the cutoff
          gate: parity against the old copies

PHASE 3   replay events after the cutoff
          gate: cursor reaches the head, no failures

PHASE 4   switch the history read to the projection
          gate: view output matches the diff-based history

PHASE 5   switch locked reads to current state
          gate: locked marksheet identical to the last copy

PHASE 6   stop writing snapshots
          gate: no new documents; old ones frozen, not dropped

The snapshot was never deleted. Freezing it is cheap; deleting it removes the rebuild source and the ability to answer historical questions if the projection is ever suspect.

The backfill had to assume interruption

Two eras meet at a cutoff: before it, copies are the only record; after it, the event log is.

  2024 ────────────────────────── cutoff ───────────────────── today
  │◄────── copies only ─────────►│◄────── event log ─────────►│
                │                            │
                └──────── backfill ──────────┘
                     one pass, restartable

We assumed the job would be interrupted — by a deploy, a crash, an operator, or a deadline. So every emitted row carried a deterministic identity derived from the copy, the student and the column. Re-running the job produces the same identities; the storage rejects the ones that already exist. The job is safe to run twice, and safe to stop halfway.

We also knew the traversal order had to be cheap. Walking the copies in the order the history needs them let the database serve the scan from an index instead of sorting twenty gigabytes in memory. That was the difference between a backfill and an outage.

Cutover without a flag day

There was no moment where we turned the old system off and the new one on. The phases overlapped.

Events were emitted for weeks before anything read the projection. The backfill ran while the old reads were still live. The replay followed the backfill, and the cursor caught up while the old code served the same screens. Reads switched one surface at a time, each behind a parity check.

The only irreversible step was the last one. Once snapshot writes stopped, rolling back the image would have meant old code reading a copy that no longer advanced. We treated that as a one-way door and wrote it down before deploying.

The token

Locked terms had one more problem: an old report had to be recognisable as stale.

The old system identified a locked term by the copy it was generated against. That identity had to survive the switch, or every existing report would look outdated on day one and the first regeneration would produce a wave of spurious warnings. So we gave each locked term a generation counter, seeded it at one for every term already locked, and made the new identity a pair of the term and its generation. Old reports kept a token that no longer matched the new format, but nothing compared it for correctness — only for equality when deciding whether to warn. The wave never happened.

The night it ran

18:05   backfill starts in the first pod of the new rollout
19:14   the last snapshot is written
19:50   the backfill and the generation stamp both commit
20:51   the replay reaches the head

It was a nerve-racking time, at least for me. I sat there, watching migrations apply and the cursor move.

Outcome

history view       1 indexed range read (was: up to 862 copy loads)
recalculation      no copy write per course per pass
locked reads       one query, same data as the copy
copy growth        stopped (~41 GB/year avoided)
API surface        unchanged
report checksums   unchanged

The generalisable point is not “event sourcing”. It is narrower:

A projection is a read model with a correctness story of its own: a monotonic cursor, an idempotent write key, a rebuild path, and a defined staleness budget. Treat it as production infrastructure, not as a derived table.

The snapshot was never wrong. It was a write-path materialisation of a read-path question, and we paid for it on both sides. (Thanks to AI for this convoluted sentence)

References

• Martin Fowler, Event Sourcing, 2005.
• Martin Fowler, What do you mean by “Event-Driven”?, 2017.
• Chris Richardson, Event-driven architecture pattern.
• Chris Richardson, Transactional outbox pattern.
• Microsoft Azure Architecture Center, Event Sourcing pattern.
• Martin Kleppmann, Designing Data-Intensive Applications, Ch. 11 (“Stream Processing”), O’Reilly, 2017. dataintensive.net
• Pat Helland, Immutability Changes Everything, ACM Queue 13(9), 2015.
• Vaughn Vernon, Implementing Domain-Driven Design, Ch. 8 (“Domain Events”), Addison-Wesley, 2013.
• Greg Young, Versioning in an Event Sourced System, Leanpub, 2017.

ME: https://youtu.be/-VJ6FtTS8zg

The exam ends: in the last two minutes of the window, most of the class hits submit at the same time. It’s not only the class — these are central exams, everyone in the country is submitting at the same time.

When the system shows “Submitted”, this is a promise to Sara that her exam is now safely stored with it and she doesn’t need to worry.

Acknowledgement

The acknowledgement is a promise. How is it implemented? What is the formal and informal literature available for this? These are questions a systems designer or architect has to answer. Jim Gray’s “Queues are Databases” is a foundational paper. He points out that queues and databases are not different species — a queue is a database albeit with specialized semantics (enqueue, transactional dequeue, visibility, ACK).

What does an ACK mean in your system? What guarantees does your system provide around storage of the user’s data?

The dual-write problem is the big question any distributed system has to answer, and its canonical solution is commonly called a transaction. As Chris Richardson explains it: a database update and a broker publish must be atomic; 2PC is not viable, so you persist the message with the domain write and let a relay publish it.

• Transactional outbox (for the atomicity)
• Polling Publisher + Transaction Log Tailing (relay implementations)

An ACK must not outrun durability: persist before you externalize. This is write-ahead logging, formalized in Mohan et al., “ARIES” (ACM TODS 1992). The older, more general formulation is in Hohpe & Woolf, Enterprise Integration Patterns (2003), which covers Guaranteed Delivery, Message Store, Transactional Client, and Idempotent Receiver.

Two Pat Helland papers frame the distributed-systems theory:

• “Life Beyond Distributed Transactions: An Apostate’s Opinion” (CIDR 2007): don’t span resources in distributed transactions; use messages plus idempotence.
• “Idempotence Is Not a Medical Condition” (ACM Queue 2012): exactly-once is not achievable as delivery but is achieved as effect.

For us, this means that the submission document and its result event commit in one database transaction, and the ACK is that commit. No window of uncertainty exists in this case. In the modern path below, this promise splits in two: the gate’s “Submitted” is the durable acceptance write (the queue document), and the worker’s finalize transaction — submission document plus result event, atomically — is the ACK for the result itself.

AWS have documented this same pattern for their Lambda service: Lambda makes sure that your function is run, even in the face of server failures, so it needs a durable queue in which to store your request. With a durable queue, your request can be re-driven if your function fails the first time. (AWS Builders’ Library)

Legacy Architecture

This is what the legacy architecture looks like (high level):

Everything is inline. At scale, the budget is very tight: we want to save the document coming in as quickly as possible so we can save the next one. Any slowdown downstream ripples up, as can be seen in the picture.

Side effects had no single delivery story and were spread all over the place: sometimes they are direct calls, sometimes enqueued with a poller implemented to drain the outbox. Each context has its own retry semantics, as well as its own failure handling. And its own idea of an event — there is no shared ledger, no canonical envelope, and no single place to look to answer whether a side effect happened and how many times.

Reads were coupled similarly: the de-facto shared read model is a snapshot collection whose documents fully embedded their submissions — one of the largest collections in the database, written by several contexts and read through one another’s services. No single context owned it and it made it expensive to both change and read.

At scale, this fails: the deadline burst arrives at a request path wired everywhere. The transaction fails if any downstream fails. We have also had cases of partial successes because there’s no overarching manager (e.g. a durable execution engine).

Modern Architecture

Write Path

Although fairly complex, I have tried to summarize it in the following table. The idea is to get the submission asap into a ledger — one that we can later build upon.

All the expensive work is done by workers. A worker would claim a pending queue document and process it. Details omitted for brevity.

Gate total: ~6 indexed reads + 1 majority write + 1 best-effort Redis op — no transaction, no external calls, no scans, no Person read.

Downstream of the finalize transaction, the result flows through the event layer:

Infrastructure

The queue is built on Redis. The configuration of Redis is another blog post, but the instance of Redis we’re using doesn’t support Append Only File (AOF), which can lead to data loss. It is tolerated here because we’re persisting first and foremost on the primary data store. Is a queue a good source of truth? We try to answer that next…

Is a queue a good source of truth?

As established earlier, a queue is a specialized database. So the question is: what is the durability guarantee that this queue makes?

BullMQ’s production guide — Going to production:

“Since BullMQ is based on Redis, persistence needs to be configured manually… We recommend enabling AOF… maxmemory-policy must be noeviction. This is the only setting that guarantees the correct behavior of the queues.”

While we wait for AOF to become available, I found literature that raised alarm bells: Sidekiq has documented job loss on hard failover (unless you are on Pro with super_fetch), and Celery’s Redis visibility-timeout semantics bite in exactly the same way. The synthesis rule is short: your ACK’s durability is the minimum of everything you wrote before acking.

Queueing theory: bursts, knees, and backlogs

This section is more about literature, with the design-specific things towards the end. Much of it was done by AI.

Exam windows are a queueing-theory problem wearing a school uniform.

Little’s Law, L = λW (John D. C. Little, Operations Research, 1961): concurrency equals arrival rate times latency. If service latency spikes 100x, in-flight work spikes 100x — threads, heap, and database sessions become a queue you never designed. It is why we bound consumers rather than “drain as fast as possible”, and why a slow downstream doesn’t get to inflate the submission request.

The utilization curve (M/M/1, Kingman’s formula; Gross et al., Fundamentals of Queueing Theory): latency behaves like ≈ 1/(1−ρ) — flat until about 70% utilization, then a knee, then a wall. Every exam window walks toward that knee. This is also why an autoscaler that only reacts to CPU is too blunt for bursts: the queue signals load before the CPU does.

AWS Builders’ Library, “Avoiding insurmountable queue backlogs” (David Yanacek) — aws.amazon.com/builders-library/avoiding-insurmountable-queue-backlogs — is the essential operational text: queues are bimodal (a fast mode and a backlog mode); backlog mode is dramatically slower to drain than to fill, so an hour of outage can need ~2x capacity for an hour to recover; backpressure must kick in before the backlog; alarm on message age, not just depth; DLQ volume is an availability metric.

Google SRE Book, “Handling Overload” (sre.google/sre-book/handling-overload) and “Addressing Cascading Failures” (ch. 22) supply the controls: load shedding, criticality tiers, adaptive client-side throttling, and retry budgets (≤3 attempts, ≤10% retry ratio) — with the design goal that a backend keeps serving its provisioned rate while receiving 2–10x excess.

Also worth knowing: Nichols & Jacobson, “Controlling Queue Delay” (ACM Queue 2012) — bufferbloat: unbounded buffering trades latency for throughput until the buffer itself collapses. Queues are never free; every unbounded queue is a latency debt. And Dean & Barroso, “The Tail at Scale” (CACM 2013) — fan-out amplifies tail latency, which matters when one submit triggers grading, recalc, projections, and notifications.

The exam-specific corollaries:

• The deadline is a self-inflicted load spike. You cannot flatten it (students submit when they submit), so you absorb it: an accept-only gate plus bounded, idempotent consumers.
• Retries are the second self-inflicted spike. A transient failure right at the deadline would otherwise multiply traffic — which is what retry budgets, backoff, and jitter are for.
• Consumer lag is the metric to watch, not queue depth. Depth tells you how much work is queued; age tells you whether students are waiting.
• Per-student concurrency is naturally bounded (one active attempt), so the burst is wide, not deep — thousands of small, independent transactions rather than one hot document.

Idempotency: how “exactly once” is actually achieved

You cannot deliver a message exactly once; you can only make delivering it twice harmless. At-least-once delivery + idempotent effects = effectively-once.

The database arbitrates “first writer wins” with a unique key — the per-attempt idempotency key. The gate checks it against both finalized submissions and pending queue documents, and the key stays unique when the worker finalizes. Retried submits and duplicate deliveries land on the same document; the second writer gets a duplicate-key error and is resolved to the stored attempt, not to a failure. The key is derived from natural identity, not generated randomly, so network retries, double-clicks, and tab reloads all converge.

Consumers are idempotent the same way: upserts keyed by the result id, monotonic guards (only a newer change moves the latest state; the earliest creation timestamp is kept with a minimum operation), and changes recorded as a set. Re-running any consumer over any stretch of the log produces the same state.

Queues, notably, cannot do this for you. A queue can deduplicate within its retention window (with the right configuration), but after that, only the system of record can arbitrate a duplicate — which is why queue products document idempotent-consumer patterns (BullMQ ships an “idempotent jobs” page) instead of promising effective exactly-once. The best industry reference is Stripe’s idempotency keys (“Designing robust and predictable APIs with idempotency”, Stripe blog, 2017) — the same contract we implement for exams. Not exactly Stripe’s scale (yet).

Streaming-theory equivalents: MillWheel (Akidau et al., VLDB 2013) and the Dataflow model (VLDB 2015) — dedup via stable IDs.

One practical footnote: make the acceptance check read the same store the ACK promises. An attempt-key lookup served from a cache is not an idempotency guard; it is a cache hit with a database problem behind it.

Reading path

One hour: the microservices.io transactional outbox page; the BullMQ production guide; AWS, “Avoiding insurmountable queue backlogs”; SRE ch. 21 “Handling Overload”.

Half a day: Richardson, Microservices Patterns, ch. 3–4 (IPC and transactional messaging); Hohpe & Woolf, Enterprise Integration Patterns (the four patterns above); Gray’s “Queues are Databases”.

Deep/formal: Helland’s two papers; Kreps, “The Log”; Gross et al., Fundamentals of Queueing Theory; ARIES; Dean & Barroso, “The Tail at Scale”; Kafka exactly-once semantics; MillWheel.

The short version, if you take one line: the database is the record, the queue is a relay, and the ACK is only as durable as the least durable thing you wrote before saying it.

Conclusion

Before computers, an exam ended with a handover: the student gave the script to the proctor, and from that moment the paper was the institution’s responsibility — one script, one record, one result weeks later. Everything this post described is that same handover, rebuilt in machines: the gate is the hand, the queue document is the receipt, the finalized transaction is the script reaching the examiner, and the log is the sealed archive. The medium changed, the failure modes changed, but the product did not: exactly one durable, attributable result per attempt, identical wherever it is read.

The deadline burst may change when the result is produced, never what it is. The exam window is that contract’s first stress test; the report card months later is its acceptance test.