Mark Levy/Writing/The Word Machine
[ Floor 2 of 3 ] Behind the glass
The Word Machine
From your question to its answer: inference, thinking, effort, and pictures on the same loop.
New here? Clear Floor 1, the monster itself, first.
Floor 1 established the machine: given some text, predict a plausible next word. Trillions of frozen knobs, attention look-backs, run in a loop, with the context as its only memory. If any of that sounds new, start there.
Floor 2 answers the questions crawlers ask next. How does a question ever start an autocomplete? What is "inference"? What are the machines doing when they "think," and what does an "effort" setting actually buy? And how does a text-completion machine look at a screenshot?
Five rooms. As before, every demo is a hand-built toy with illustrative numbers; the point is the mechanism, not the measurements.
Room 1 of 5The Masquerade
How a question starts a completion
The machine doesn't answer questions. It continues text. So the product never hands it your question alone. It assembles a transcript, a little screenplay in which a helpful assistant is about to speak, and asks the machine to continue that. Your message is one block in the stack; the last line is the trigger.
Press the button and watch the costume go on: this is what the machine actually receives.
One thing is still missing: why would a raw autocompleter continue that script helpfully? Out of big training (Floor 1, Room 3), it wouldn't. It would just continue the pattern. So models get a second, smaller round of training on example conversations, written and rated by people, until "what a helpful assistant says next" becomes the likeliest continuation. Same question, before and after:
Chat is a costume. The machine never stopped being an autocompleter. The product dresses your question in a transcript where an assistant speaks next, and tuning made the helpful continuation the likely one. Even the reply's ending is a prediction: the end-of-turn token is an ordinary candidate on the list, learned like every other word.
Room 2 of 5The Forge and the Field
Inference: running the machine
The machine has two lives, and the vocabulary keeps them straight. Training is the rare one: knobs moving, trillions of examples, months in a datacenter. It happens before you ever meet the model. That's the forge, where the blade was hammered into shape, once. Inference is the other life, and it's the only one you ever touch: the frozen machine, run forward. Text in, one word out. That's the field, where the blade gets swung. Every message you send triggers it.
The word is borrowed from statistics: the machine infers the next word from the input. Flip the switch and compare the two lives.
If you're chatting, it's inference. Nothing you type moves a knob. When a conversation seems to "learn," that's the growing context doing the work (Floor 1, Room 5), not the machine changing. Learning-the-machine-way happened once, in the forge, before release.
Room 3 of 5The Wizard's Study
Thinking: muttering to itself first
Here's a hard limit from Floor 1: the circuit is one-way, and every token gets the same fixed slice of computation, one pass. A hard problem doesn't get a bigger slice; it gets a plausible-shaped guess.
The workaround is delightfully low-tech: before answering, let the machine write to itself. Scratch text, generated by the same next-word loop, goes into the context, where the final answer's attention look-backs can anchor to worked steps instead of leaping to a guess. Like a wizard who writes the working out on a scroll before casting: no new magic, just more ink.
Thinking is more turns of the same crank. The machine buys computation by spending tokens, and the scratch scroll lives in the context where attention can use it. This is also why "think step by step" helps in a plain prompt: you're inviting the scratch work into the transcript, the only workspace the machine has.
Room 4 of 5Spell Slots
Effort: how many slots to spend
If thinking is spending tokens, someone has to set the budget. That's all an effort setting is: a cap on how much scratch text the machine may write before answering. Spell slots, in other words. It doesn't make the caster smarter (same knobs at every position); it buys more passes.
Choose a budget and watch what it costs, and what it's worth, on an easy question versus a genuinely hard one.
TASK A — "What's the capital of France?"
TASK B — "Three staff, six scheduling rules — build Friday's shift plan."
Effort is a budget, not a brain transplant. Match it to the problem: an easy question at high effort buys the same answer at thirty times the price, and a hard one at low effort gets Room 3's first button, a confident guess.
Room 5 of 5The Gallery
Pictures on the same loop
The loop carries tokens: numbers standing for word-pieces. Nothing in the machinery cares that they started as words. So to let the machine see, a second, smaller machine (a vision encoder) chops the image into a grid of patches and turns each patch into the same kind of number-list a word gets, a point in the meaning-space from Floor 1, Room 1. Those patch-tokens are spliced into the context right next to your words, and from there it's business as usual: attention reaches into the picture exactly the way it reaches back at words.
The machine doesn't see pictures. It reads them. Once an image is patches in the context, it's text-like all the way down. That's why it can describe a chart beautifully and still misread the fine print: a patch, like a token, is a coarse unit, and detail smaller than the unit is a bet, not an observation.
InventoryFloor 2 vocabulary
What you're carrying now
Floor 2's product words, each with its exact address in the rooms above:
- inferenceRunning the frozen machine: one pass, one token (Room 2). Every message you send triggers it. Training is the other, rarer life.
- streamingWords appearing one by one is not a display effect. It's the loop itself, each token shipped as soon as its pass finishes.
- roles · system/user/assistantThe transcript blocks from Room 1's masquerade. An API request is that stack, written out as a list.
- stop tokenA token meaning "the assistant's turn is over": an ordinary vocabulary entry that wins the candidate list like any word, because every training conversation ended with it. Predicting it is how a reply ends.
- fine-tuning · RLHFFinishing school (Room 1): a second training round on example conversations. RLHF, Reinforcement Learning from Human Feedback, is the part where people rate answers and the ratings nudge the knobs toward answering instead of pattern-continuing.
- thinking · extended thinkingScratch tokens generated before the visible answer (Room 3), usually folded away in the interface. Same machine, pointed at a scratch scroll first.
- reasoning modelA model finishing-schooled on worked solutions, so its scratch text is worth the tokens.
- effort · thinking budgetRoom 4's spell slots: how many scratch tokens are allowed. Buys passes, not intelligence.
- multimodal · visionThe gallery (Room 5): images chopped into patches, embedded into the same space as word-tokens, spliced into the context.
- embeddingThe coordinate-list a token, or an image patch, becomes: a point on Floor 1, Room 1's map. "Similar meaning" literally means "nearby numbers."
- vector database · RAGA filing cabinet of embeddings, searched by nearness. RAG embeds your question, fetches the nearest-sitting documents, and pastes them into the context: tools plumbing feeding the same machine.
- latency · tokens/secHow fast inference cranks for you today. "The model feels slow" is a statement about the datacenter's load, not about the knobs.
Floor 1's punchline still holds. None of this replaced the machine. Inference runs it, thinking runs it longer, effort chooses how much longer, and vision feeds it different freight. It's next-word prediction all the way down.
Floor clearedWhat this buys you
Four more skills unlocked
Four more rules fall straight out of these five rooms:
- Give it the script, not just the question. The reply continues a transcript, so the rules, facts, and examples you put in the context are the script. A vague scene gets a vague next line.
- Spend thinking where it pays. Hard, multi-step, checkable problems earn a high effort setting. For lookups and rewrites, extra effort mostly buys latency and cost.
- "It learned!" means the context, not the knobs. Correcting it improves this chat only. The model that greets your next chat is the same frozen arrangement; carry anything worth keeping forward yourself.
- Images are read, not seen. Coarse patches are great for "what is this chart saying," risky for fine print, tiny numbers, and counting. Treat what it "saw" like what it wrote: check it.
That's the whole floor: a completion machine in a costume, run frozen, sometimes allowed to mutter to itself first, fed pictures chopped into words. Climb back up to Floor 1 whenever the fundamentals need a refresher.
Stairs down. Floor 3 is where the monster gets a party: the loop that lets it act on the world, the character sheet that tells it who to be, the spellbook of tools it can cast, familiars it can summon, wards at every door, and the map that decides what it gets to read. Same monster. Now with a harness.
Descend to Floor 3 Back to the mapThe mental model continues John Mount's "A Simplified Mental Model of LLMs" (Win-Vector): the machine's mechanics are Floor 1's territory; this floor extends the same non-magical framing to the plumbing around it.
The teaching style, poking the simulation instead of reading the bullet list, follows Laurentiu Gabriel's "How I Use LLMs to Learn". All demos on this page are illustrative toys, not real model outputs.