How a language model works — as a city you can watch, one token at a time
TokenTown is an isometric city where every district is one stage of a transformer language model. A convoy carries a hidden state along the roads: it is cut into tokens at the docks, cast into a vector at the foundry, stamped with its position, then driven around the layer ring — attention, residual, feed-forward, residual — once per layer, before the stadium turns it into a probability distribution and the sampler picks one token. That token drives back up the feedback highway and the whole city runs again.
Genuinely computed, live, in your browser: the tokenizer split; the embedding lookup; sinusoidal positional encoding; LayerNorm; multi-head scaled dot-product attention with causal masking over a real growing KV cache; the residual adds; a GELU feed-forward; and temperature / top-p sampling. The bars on the truck are the actual vector. The beams over the warehouse are the actual softmax weights. Prefill really does process every prompt token at once while decode really does process only one.
Scaled down: 12 dimensions instead of thousands, 2 attention heads instead of dozens, 2–12 layers instead of 80, and a vocabulary of a few hundred words instead of 100k+.
Deliberately faked: the weights are random — nothing here was trained — so a random-weight model would emit noise. To keep the output readable, the final logits blend the real hidden-state projection with a bigram prior built from a small fixed corpus. The attention scores are also sharpened, and given a small first-token ("sink") and recency bias, so the map looks like the patterns trained models actually produce. Treat the text this city writes as scenery; treat the mechanism as the lesson.
The first time the convoy reaches a district it stops long enough to read that district's explanation — between 9 and 26 seconds depending on how much there is to say. A progress bar under the panel text shows how long the stop has left. Once every district has been explained the city runs at a watchable pace instead of a readable one, and the repeated layers fast-forward because they are the same road with different weights. Space holds any stop indefinitely, S steps one stage at a time, and the Speed slider scales everything — including the reading stops — from 0.4× to 8×. Reset (⟲) replays the slow tour from the beginning; Run keeps what you have already read.
Inspired by the idea behind PGSimCity (a city-shaped model of PostgreSQL); all code, art and copy here are original.