The Attention Timeline ⧗
Attention mechanisms · 2014 → 2026 · in launch order

Vanilla attention was not wrong. It was expensive.

Every mechanism since is somebody looking at that bill and trying to pay less of it. In launch order, you can watch the field change its mind: first it wants exactness, then its memory back, then length, then its memory back again.

— mechanisms every one as a decision, not a fact every date checked against the paper pros and cons for all of them 1 decision graph + 8 live demos
00 · Start here

The plain one, because nothing after it makes sense first

Scaled dot-product attention, June 2017. Every token asks a question, every other token answers, and the best matches contribute most. Four steps, then a mask.

Each token is projected three ways. The query is what I am looking for. The key is what kind of information I contain. The value is what I hand over if you pick me. Then:

  1. Score. Dot every query against every key. One pair, one number.
  2. Scale. Divide by √dk. Dot products of d-dimensional vectors grow like d, and large logits push the softmax into saturation where the gradient is nearly zero.
  3. Mask, then softmax. Add −∞ wherever a token would be reading the future, then turn each row into positive weights summing to one.
  4. Weighted sum. Combine the values with those weights.

Two facts from this picture drive everything below. Every token is compared with every other token — 1K tokens is a million comparisons, 100K tokens is ten billion. And attention has no idea what order the tokens are in: two identical tokens at positions 2 and 20 produce identical queries, keys and values. Something else has to supply position.

01 · The two bills

Attention sends two invoices, and they arrive at different times

Compute during training, growing like T². And memory during generation — the KV cache, which is private to each conversation and can't be shared. Almost everything below is aimed at one bill or the other.

Bill one · compute
T²
Double the context, quadruple the work. Paid once per forward pass, by whoever is training or prefilling.
Bill two · KV cache
2·L·Hkv·d·T·B
Grows linearly with context and linearly with concurrent users. Paid continuously, by whoever is serving.
2019–2021 fights the compute bill. 2023 fights the memory bill. The switch wasn't gradual — ChatGPT went into production, and a 2022 kernel paper (FlashAttention) showed the compute bill had been misdiagnosed all along.
02 · The decision graph

Every mechanism is a forced move. Here's what forced it

Each node is a mechanism, placed by date and by which bill it fights. Each arrow means "answers a limit of". Walk one of the four decisions below, or click any node to light up what it came from and what it led to.

03 · The timeline

The timeline, one entry at a time

Every mechanism in launch order, each with its problem, its trade-offs, and when you'd pick it. Filter below, or jump here from any node in the graph.

04 · Which would you actually pick

Right for a 2K chatbot, wrong for a 1M agent — and that's fine

Every mechanism is a trade, and the right side depends on the workload. Pick one and see what wins, what to rule out, and why.

05 · What comes next

What comes next

Six guesses — predictions, not facts. Each one just extends a trend already visible on the timeline by a single step.

06 · Dates and sources

Every date, and where it came from

Unless noted, the date is the arXiv v1 submission timestamp — not the conference date, not the journal date. Two paperless entries are flagged and dated indirectly.

The two paperless entries (▲): NTK-aware scaled RoPE was a Reddit post, dated from the YaRN citation plus the first implementation request; "Attention Is Off By One" was a blog post, dated from its Hacker News submission.
DateMechanismPrimary sourceHow the date was established