Summarising documents of up to a million tokens by reframing the task as question answering. A 3B model answering 24 learned questions by retrieval produces better summaries than a 70B model reading the whole text, because the standard framing of summarisation is underconstrained.
Role Project lead. Period 2025. Scope Problem formulation, agentic pipeline, contrastive question selection architecture, evaluation.
We were able to make a 3-billion-parameter model produce better summaries of very long documents than a 70-billion-parameter model, while the small model never read the document.
It works by asking 24 questions and retrieving the answers, then assembling the summary from those answers. The 70B model reads up to a million tokens and writes a summary directly. The small model wins anyway.
That is a surprising result and the interesting part is why it happens, which has less to do with modelling than with the fact that the summarisation task as normally defined does not describe what anybody actually wants.
Summarisation research almost universally treats one reference summary as the correct answer and scores a system by its similarity to that reference.
The conjecture this project is built on is that this is wrong in a specific and consequential way. A summary is not a property of a document. It is a property of a document and a reader together. What belongs in a summary depends on what the reader already knows and what they are trying to find out, and two competent summaries of the same text written for different readers should disagree substantially.
Treating a single reference as the gold label makes the task underconstrained. There are many defensible outputs and the metric rewards only one of them, which is a plausible explanation for the long-standing observation that automatic summarisation metrics correlate poorly with human judgement.
If that is right, the fix is not a better model. It is a task definition that includes the reader.
If a summary is defined relative to what a reader wants to know, then the natural representation of “what a reader wants to know” is a set of questions.
So the pipeline conditions on questions. Given a long document, the system answers a set of questions about it by retrieval, and the summary is assembled from those answers. The document is never read end to end by the generating model.
Two things fall out of this that are worth stating separately from the quality result.
The first is that it makes the summary auditable. Every sentence traces to a question and a retrieved passage, so a reader who disagrees with the summary can see which question produced the disagreement.
The second is that it makes the summary steerable. Change the questions and you change the summary, which is exactly the user conditioning the framing argued for. A reader who already knows the background can drop the background questions.
The obvious objection is that the whole result now depends on the questions, and choosing 24 good questions by hand for arbitrary documents does not scale.
So the questions are learned. Questions are generated in bulk from the summaries in the training data, which produces a large and very noisy pool: most generated questions are trivial, redundant, or unanswerable from the text. The task is to find the small subset that actually carries a summary.
I treated this as weak supervision and designed a custom architecture trained with a contrastive objective, so that questions whose answers reconstruct a good summary are pulled apart from questions whose answers do not. The supervision signal is weak because the training summaries are themselves only one defensible summary each, which is the same limitation the project is arguing about, applied one level down.
The efficiency framing is the one that gets attention: a model roughly twenty times smaller, not reading most of its input, producing better output. That matters practically, because 70B inference over million-token contexts is expensive in a way that 3B retrieval is not.
But the reason I think the result is worth attention is what it implies about long-context work generally. The 70B model had all the information and used it worse. That points at the framing rather than the capacity, and it suggests a general move: when a task is defined loosely enough that many outputs are defensible, adding capacity is a weak lever and constraining the task is a strong one.
Research project, led by me, 2025. The pipeline, the contrastive question selection and the comparison are built and run.
The evaluation is a comparison against the larger baseline on summary quality; I have deliberately not published a single headline metric on this page, because the argument the project makes is that single-number summarisation metrics are the thing not to trust.