Context is everything

If I had to pick a single idea for someone to take from this guide, it would be this: the model doesn’t know you, doesn’t remember yesterday, and knows nothing about your project. It only knows what is written in the window, right now.

Everything that looks like memory, personality, or “it already gets me” is someone — you or the product — putting text back into that window.

What does the model see when you ask?

The best image I know for this comes from Andrej Karpathy, who describes language models as a new kind of computer: the model plays the role of the CPU, and the context window is the memory — the RAM. (The whole talk is worth watching.)

A computer’s CPU only processes what has been loaded into RAM. The model only reasons about what has been loaded into the context: your instructions, the attached documents, the conversation history, the descriptions of the tools it can use, the results from those tools. All of it is measured in tokens (pieces of words) and competes for the same space.

Context engineering is the name that stuck for the work of deciding what goes into that RAM. It’s the natural evolution of prompt engineering: we used to polish a single instruction; now we manage everything the model sees over the course of a long task. When the term showed up, I wasn’t sure we needed yet another concept, but I’ve always argued for the idea behind it: the most important human skill right now is knowing how to communicate and hand over the right information and the right tools.

Is more context better context?

Intuition says: “when in doubt, throw everything in.” It’s the most common mistake I see.

Language models work by relating each token to all the others. The more stuff in the window, the thinner that attention gets spread. Some people call this the attention budget: it’s finite, and every new token spends a little of it. When the budget runs out, you get what has come to be called context rot, the drop in performance that has already been measured as the input grows. The model forgets an instruction from the beginning, mixes up documents, repeats what it already did. It doesn’t “freeze” — it just gets gradually worse.

That’s where the practical principle comes from: look for the smallest set of high-signal tokens that gets the task done.

What is the right altitude for an instruction?

A piece on context engineering for agents has an idea that changed the way I write instructions: there is a right altitude, a “Goldilocks” zone between two mistakes.

IF the question contains "taxi" AND amount < 80 THEN answer "Approved".
IF the question contains "taxi" AND amount >= 80 THEN answer "Needs approval".
IF the question contains "car service" THEN treat it as "taxi".
IF the question contains "lunch" AND it is a business day THEN ...
(36 more rules)

You are a helpful assistant. Help employees with reimbursements.
Be polite and accurate.

<context>
You help company employees with questions about expense reimbursement.
The current policy is in <policy>. The people asking are usually in a
hurry and want to know: can I or can't I, and what is the next step.
</context>

<instructions>
1. Answer based only on the policy. Cite the item you used.
2. If the policy doesn't cover the case, say so clearly and point to
   who approves exceptions. Do not estimate amounts or deadlines.
3. Start with the answer; explain afterward, in up to three sentences.
</instructions>

<policy>
(only the relevant excerpts from the policy)
</policy>

What about day to day?

• I explain the why, not just the what. “Keep it short because the person is reading on their phone between two meetings” works better than “keep it short.”
• I separate the parts. Markdown headings or tags like <context> help the model not mix up instructions and documents.
• I bring only the excerpt. Instead of the 300-page manual, the three pages that matter. If I don’t know which ones those are, that’s my first request: “find the passages about X.”
• I start a new conversation when the subject changes. Old history is noise. If I need to carry something forward, I ask for a summary and paste it at the start of the new one.
• I turn on few tools. Every connector takes up space in the window just by existing. And I keep an eye on the meter the tool shows: when a session goes past two-thirds of the window, I’m already thinking about closing it and starting another.
• I keep the context outside the conversation. Everything I don’t want to explain again becomes a file in the project’s repository, which is the subject of chapter 11.