You paste a 40-page policy doc into the prompt, ask one question, and the model confidently answers as if page 19 doesn't exist. The fact you needed was right there. It just happened to be in the middle.

That's not bad luck. It's a well-measured pattern called lost in the middle, and the one lesson worth taking away today is simple: put the highest-signal stuff at the start and the end of a long prompt, never only in the middle.

What the research actually found

In Lost in the Middle: How Language Models Use Long Contexts (Liu et al., TACL 2024), researchers from Stanford, UC Berkeley and Samaya AI ran a clean experiment. They gave models a question plus 10, 20 or 30 retrieved Wikipedia passages, where exactly one passage held the answer. Then they moved that one passage around: first, somewhere in the middle, last. Nothing else changed.

If a model used its context evenly, accuracy would be flat no matter where the answer sat. It wasn't flat. It was a U-shape:

  • Start of the context: highest accuracy. The paper calls this primacy bias.
  • End of the context: second best. That's recency bias.
  • Middle: a clear dip.

The numbers are what make it stick. With 20 documents, GPT-3.5-Turbo answered correctly about 75.8% of the time when the right passage came first, 63.2% when it came last, and 53.8% when it sat in the middle. Its closed-book score, answering with no documents at all, was 56.1%. So in the worst position, giving the model the answer did slightly worse than giving it nothing.

Two other findings matter for builders:

  • Bigger context windows didn't fix it. Extended-context versions (16K, 100K) performed almost identically to their shorter siblings when the input fit both. A larger window means you can send more, not that the model uses it well.
  • More retrieved docs hit a ceiling fast. In their open-domain QA case study, going from 20 to 50 retrieved documents improved GPT-3.5-Turbo by only about 1.5% (Claude-1.3 by about 1%), while costing a lot more tokens and latency.

Why it happens

A follow-up paper, Found in the Middle (Hsieh et al., ACL Findings 2024), links the effect to the model's attention itself: tokens at the beginning and end of the input get more attention regardless of how relevant they are. Their calibration method reduces that bias and improved RAG results by up to 10 percentage points over existing methods.

Think of it like a long meeting. People remember how it opened and how it ended. The important point someone made at minute 34 is the one nobody writes down.

Newer models do much better on simple "find the needle" tests than the 2023 models in the paper. But position effects haven't gone away on real multi-document tasks, so treat placement as something you design, not something you hope works out.

Before and after

Here's a typical long prompt, built in the order the data arrived:

You are a support assistant. <documents> [doc 1: general company FAQ] [doc 2: shipping policy] [doc 3: refund policy <- the answer is here] [doc 4: warranty terms] [doc 5: holiday hours] </documents> Answer the customer's question. Only use the documents. Keep it under 80 words. Cite the doc number. Question: Can I get a refund after 45 days?

The key rules ("only use the documents", the word limit) and the one doc that matters are all wedged into the middle of the stack.

Same content, placed on purpose:

You are a support assistant. Answer only from the documents below. If the answer isn't in them, say so. Keep it under 80 words. Cite the doc number. Task: answer a customer question about refunds. <documents> [doc 3: refund policy] <- most relevant first [doc 4: warranty terms] [doc 1: general company FAQ] [doc 5: holiday hours] [doc 2: shipping policy] <- second most relevant last </documents> Question: Can I get a refund after 45 days? Reminder: answer only from the documents, under 80 words, cite the doc number.

The rules are at the top, the question and a one-line restatement of the rules are at the bottom, and the strongest chunks sit at the two edges where the model looks hardest.

This lines up with Anthropic's long-context tips, which recommend putting long documents above the query and say that queries at the end can improve response quality by up to 30% in their tests, especially with complex, multi-document inputs.

The RAG trap: sorted is not the same as well-placed

Most RAG pipelines retrieve top-k chunks, sort them by score, and drop them in. That puts chunk #1 at the top, which is fine, but chunks #2 to #4, often also important, end up in the middle once you have 10 or more.

Better options:

  • Rerank, then trim. Use a reranker and keep fewer chunks. Five good chunks beat twenty mediocre ones.
  • Place the best chunks at both edges. Put rank 1 first, rank 2 last, rank 3 second, rank 4 second-to-last, and let the weakest fall into the middle. LangChain ships this idea as LongContextReorder, and it's about ten lines to write yourself.
  • Don't pad the context. Every extra chunk pushes something else toward the middle. 
def edge_order(chunks_by_score): """Best chunks at the start and end, weakest in the middle.""" front, back = [], [] for i, chunk in enumerate(chunks_by_score): (front if i % 2 == 0 else back).append(chunk) return front + back[::-1] # [r1, r2, r3, r4, r5] -> [r1, r3, r5, r4, r2]

Quick checklist

  • Instructions and hard constraints at the top
  • The question or task restated at the end
  • For RAG, the most relevant chunks near the start or end, not only sorted by score into the middle
  • Rerank and keep fewer chunks
  • Keep the context as short as the task allows
  • Test it: move the key fact to the start, middle and end of a real prompt and see if your model's answer changes

That last one takes ten minutes and tells you more about your own model than any benchmark.

References

Have you ever tested what your model does when the key fact sits in the middle?