Your whole history now fits. That doesn't make it useful

Until recently, setting up an AI agent always started with the same awkward question: out of everything your business knows, what do we show it? You had to choose, because it wouldn't fit. Twenty FAQs, the price list, and that was about it. Now it fits — frontier models can hold the equivalent of fifteen hundred pages in view at once — and a comfortable idea has taken hold along with it: just give it everything and let it work the rest out.

This one is for small business owners who already run an agent or are about to. By the end you'll know what large context windows actually changed, why handing over your entire history usually makes the answers worse rather than better, and what to hand over instead.

A million tokens, in plain terms

The context window is the model's working memory: everything it can see while it writes a reply. It isn't what the model learned during training — that part comes built in. It's what you pass it on that specific call: instructions, documents, the conversation so far.

That window has grown enormously. Anthropic's documentation puts its current models at a one-million-token window, with no special flag and no premium rate: a million is simply the default. The other major providers are in the same territory.

In terms you can picture: a token is roughly three quarters of a word, so a million tokens is somewhere around 750,000 words. Fifteen hundred pages. Three years of one salesperson's email. Your entire product manual, the price list, the warranty terms and the last two thousand WhatsApp threads, all in view simultaneously.

It looks as though the "what do I show the agent" problem solved itself. It didn't, and it's worth understanding why before you decide it the wrong way round.

Fitting is not the same as reading well

Here's the part the demos leave out. Anthropic's own documentation says it plainly: as the token count grows, accuracy and recall degrade. They call it context rot, and the conclusion they draw is that choosing what goes into the context matters as much as how much room you have.

This isn't a hunch, it's been measured. A landmark Stanford study, Lost in the Middle, found that models do far better when the relevant fact sits near the start or the end of what you give them, and noticeably worse when it's buried in the middle — including in models built specifically for long contexts.

Think of it the way you'd think about a new hire. You can hand them a twenty-page folder you chose carefully, or the whole company archive. With the folder they're working by Tuesday. With the archive they need a month at best, and at worst they quote a customer 2023 prices, because those were in there too.

That second scenario does the most damage and is the hardest to spot. An agent looking at three versions of your returns policy doesn't tell you there are three. It picks one and answers, perfectly confidently.

What genuinely did change

None of this makes big windows a gimmick. Three things that used to be expensive or impossible are now within reach of any small business.

1. Getting started is no longer a data project. Making an agent answer from your own documents used to mean chunking them, indexing them and running a search layer behind the scenes. That still exists and still makes sense at thousands of documents — but if your own material runs to forty pages, none of it is necessary. It fits as-is. That's weeks of plumbing and an invoice you don't pay.
2. One-shot jobs come out brilliantly. "Read these four tender documents and tell me where they contradict each other." "Go through everything we've discussed with this client since January and give me a summary before Thursday's meeting." Bounded material, a clear question, and a result you're going to read yourself. That's exactly what the big window was missing before.
3. Long threads stop resetting. A WhatsApp conversation that stretches over weeks used to run out of memory halfway through, and the agent would start re-asking things the customer had already answered. In practice, that's over.

Notice the pattern: big windows shine on the bounded and the one-off, with somebody watching. Not on the permanent and the vague.

The context pack that actually works

For the agent that works every day — answering customers, triaging email, booking appointments — the right answer is never the archive. It's a small, maintained pack that somebody owns. Five pages will do it:

• The current version, not all of them. One price list, one returns policy, one set of opening hours. If there are two, delete one. This is 80% of the errors you'll ever see.
• The twenty questions that really come in. Pulled from your inbox over the last fortnight, not invented in a meeting.
• Your business rules as short sentences. "We don't book a service without an approved quote." "Deliveries outside the mainland are always confirmed by a person."
• The escalation cases. What it never answers alone, who it pings, and how fast.
• Three or four of your own real replies, copied verbatim. They teach tone infinitely better than any description along the lines of "friendly but professional".

Then, separately, targeted access: let it look up that specific customer's record when that specific customer writes in. Having the whole archive in front of it at all times is not the same as being able to fetch one fact on demand. The second is cheaper, more accurate, and far easier to explain the day someone asks what data your system handles.

A big window removes the obligation to choose. It doesn't remove the benefit of choosing.

What shouldn't go in, however well it fits

That a million tokens will swallow your customer folder doesn't mean you should upload it. And there's a technical detail here with very practical consequences: context isn't stored anywhere. It's re-sent in full on every single call. Load in three years of conversations and three years of conversations travel down the wire every time someone types "hi".

You pay for it every time. An agent with twenty pages of context and three hundred conversations a month is cheap. The same agent carrying the full dump is not. Caching techniques make those repeated tokens cheaper, but they don't make them go away.

It's personal data. Three years of conversations contain names, phone numbers, addresses and, if you run a clinic, health information. Sending all of it on every query is data processing, and the GDPR's data minimisation principle asks for the opposite: data that is adequate, relevant and limited to what's necessary. "It fit" is not a lawful basis.

What you don't curate, nobody reviews. Five pages get read by someone on your team each quarter, and they'll notice the shipping policy is out of date. A three-year dump never gets read by anyone, and the mistakes in it move in permanently.

That said, there's one clear case for handing over a lot: the supervised one-off. Dump the four contracts in, ask the question, check the answer, close the tab. Volume works in your favour precisely because you're standing there. The trouble starts when that dump gets wired permanently into a process nobody looks at any more.

Where to start this week

If you already have an agent running, give it a ten-minute audit: take the five questions it answers worst and, for each one, count how many different versions of that answer are sitting in its context. Nine times out of ten the problem isn't missing information. It's surplus.

If you don't have one yet, don't start by gathering material. Start by writing the pack above into a five-page document. It's the part that decides whether the agent answers well or badly, and it's the one part no supplier can do for you.

Much of this is what we sort out when we help a team put some judgement around the AI they're already using: what goes in, who keeps it current, and what never leaves the building. A big context window is a convenience, not a criterion. The criterion is still yours to set.