Lost in the Middle of an LLM Context Window
In one experiment, researchers gave GPT-3.5-Turbo twenty documents and asked a question that exactly one document could answer. They kept the question, documents, and expected answer fixed. Then they moved the answer-bearing document through the input.
Accuracy was 75.8 percent when the answer appeared in the first document. It fell to 53.8 percent when the answer appeared in the middle. It recovered to 63.2 percent when the answer appeared last.
The evidence did not change. Its position did.
Every prompt fit inside the model’s context window. The model accepted the evidence each time, but it did not use that evidence with equal reliability. This is the distinction behind the observation known as lost in the middle.
Accuracy on the 20-document question answering condition reported in Table 6 of Lost in the Middle: How Language Models Use Long Contexts by Liu and colleagues. The chart recreates the published values rather than reproducing the paper’s figure.
A context window measures capacity#
A context window is the token budget available to one model request. The budget includes the input and generated output. A larger window lets the surrounding application send more text in one request.
This is useful. A coding assistant can receive more source files, a longer conversation, additional tool results, and more detailed instructions. A document system can send more retrieved passages. A support assistant can retain more history.
The size of the window answers one question. Can this information fit inside the request? It does not answer the second question. How reliably will the model use information from each position in that request?
That second property is context utilization. If a model receives the same evidence at different positions and its accuracy changes, capacity stayed fixed while utilization changed. Capacity is easy to advertise. Utilization depends on the model, task, prompt, and input length.
Fit is a limit. Use is behavior.
What the researchers observed#
Liu and colleagues (TACL 2024) studied long context use through two controlled tasks. The first was multi-document question answering. Each prompt contained 10, 20, or 30 Wikipedia passages. Exactly one passage contained the answer. The researchers changed the answer passage’s ordinal position while preserving the question and document set.
Performance was strongest when the answer appeared near the beginning (primacy bias) and recovered when it appeared near the end (recency bias). Plotting accuracy by position produced a U-shaped curve. In the worst case, GPT-3.5-Turbo with the answer in the middle performed worse than GPT-3.5-Turbo with no documents at all.
The paper also compared standard models with their extended context counterparts when the same prompt fit inside both windows. Their performance was nearly identical. A larger maximum context did not produce more reliable use of the shared input length.
The pattern persists in newer models#
The original study tested 2023-era systems. Since then, follow-up research has confirmed the same pattern in more capable models.
Li and colleagues (LongICLBench, 2024) evaluated 15 long-context models including GPT-4 and Gemini on classification tasks with input lengths from 2K to 50K tokens. They found a consistent bias toward labels presented later in the sequence and degrading performance as context grew. Levy and colleagues (ACL 2024) tested GPT-4 and Claude on reasoning tasks padded to different lengths. Performance degraded well before the technical maximum context length.
The most direct confirmation comes from Anthropic’s own prompting documentation. Their current best practices page states: “Put longform data at the top. Place your long documents and inputs near the top of your prompt, above your query, instructions, and examples. This improves performance across all models.” It adds that “queries at the end can improve response quality by up to 30 percent in tests, especially with complex, multidocument inputs.”
Position still matters. The models got better. The effect got smaller. It did not disappear.
What to do about it#
When you are deep into a long coding session and the assistant starts missing constraints you stated earlier, the context window has not shrunk. But the material in the middle of that session, the early decisions, the constraints you set in message four of a thirty-message thread, may be contributing less to the model’s output than what came first or what you just said.
The practical response is straightforward. Pay attention to where important information sits in your context. When results degrade, a fresh context window with the relevant constraints restated at the top often produces better output than continuing to append to a long thread.
This connects to why code locality matters for LLM-assisted development. Code that concentrates its relevant context locally, rather than spreading constraints across distant files, places less demand on the model’s ability to use information from every position equally.
“The model can accept the repository” is a capacity statement. “The model reliably follows constraints throughout the repository” is a performance statement. The first is a product spec. The second is something you observe through results.
Capacity tells us what can fit. Only observed performance tells us what was used reliably.