Inside a language model

The previous chapter showed training with two weights and a straight line. A language model has billions of weights, but the question worth asking it is the same one you’d ask the line: what goes in, what comes out, and what did it adjust to get there?

This chapter doesn’t ask for math. It asks you to follow a sentence from the moment it’s typed until the answer shows up — and to stop at each step and ask what can go wrong there.

What does the model receive when I write?

Not your sentence. The text is cut into pieces called tokens — sometimes a whole word, sometimes a piece of one — and each piece is swapped for a number.

It’s worth pausing here. If the model counts tokens and not words, what does that say about the limits on how long a conversation can be? And about the bill at the end of the month, when you’re charged by usage?

How does a number carry meaning?

On its own, it doesn’t. Each token’s number is just an address. What carries something is the next step: each token is swapped for a long list of numbers — a vector, like the ones in chapter 2. These vectors have a name: embeddings.

Nobody wrote those numbers. They were adjusted during training, until tokens used in similar contexts ended up close to each other. “School” ends up near “college” and far from “screw” without anyone having taught it what a school is.

A question to take with you: if closeness comes from the training texts, what happens to the unfair associations that are in those texts? (Chapter 4 is about that.)

How does it know who “she” refers to?

Read: “The student told the principal that she needed to work.” Who needed to work?

You worked it out by weighing the other words in the sentence and what you know about the world. The model does something similar, but only the first part: for each token, it calculates how much each of the others matters at that point. This mechanism is called attention, and the architecture built around it, introduced in a paper that became a landmark, is the Transformer. To watch the machinery turn, at your own pace, I recommend two visits: the 3Blue1Brown video series and the Transformer Explainer, which shows each step on a real sentence.

Two practical consequences follow. The first: everything in the conversation can influence everything — including what you pasted in by accident. The second: the more text there is, the more that attention gets divided. That’s the origin of the problem chapter 8 will deal with.

So then it answers?

Not exactly. After all that, what the model produces is a list: for each possible token, the chance that it’s the next one. One of them is drawn at random, goes into the sentence, and the process starts over — one token at a time, until the end.

The control you just moved is called temperature. Low, the draw almost always lands on the most likely option; high, the unlikely ones get a turn. There is no “right” value: there is the question does this task call for variety or repetition?

Now go back to Lia’s scene. The assistant didn’t look up anything about that student. It continued the text with what usually comes after similar questions. Most of the time, what usually comes next and what is true coincide. When they don’t, the name people give it is hallucination — but notice that the machine did exactly the same thing in both cases.

If it’s just the next word, why does it seem to understand?

That’s the question that divides people far more qualified than I am, and I’m not going to settle it here. I can offer three angles for you to think about:

• To predict the next word well in a text about physics, it helps a lot to have captured something of physics. How much, nobody really knows how to measure.
• Fluency isn’t evidence. We humans have always confused speaking well with knowing.
• The question “does it understand?” may be less useful, for the person using it, than “how do I check?”

What do I do with this?

Not answers, but four questions that from now on go with you into every use:

1. Was this in what I provided, or is it filling in? If the answer depends on a fact about your organization and you didn’t give it that fact, the answer was made up, and nicely written.
2. How old is what it knows? Training ended on some date. What came after, and what was never in public text, isn’t in there.
3. Do I need the same result every time? If so, the draw is a problem — and the way out is in chapter 10.
4. Does it fit? Everything is counted in tokens: what you write, what you attach, what it answers.

It’s what I repeat in every training session: before any sophisticated technique, the basics. Understanding how the model works, knowing how to choose one, and knowing what to put in the input solves most of the problems.