Four weeks of remote research with mentors from EleutherAI and Apart Research, re-implementing a published prompt compression method on benchmarks it was never tested against. The contribution is the reproduction itself: moving a 2023 technique onto LongBench v2 and MS MARCO, and recasting it as a token level classification problem.
Role Research contributor, Summer of Open AI Research. Period 2025. Scope Re-implementation of the compression method, the port to two new benchmarks, the token classification reframing, and the inference infrastructure to run it all.
Prompt compression means shortening the text you send to a language model before it arrives, so you pay for fewer tokens and wait less time, while the answer stays as good as it was. It matters because the input is usually the expensive part: a retrieved document set or a long report can be tens of thousands of tokens, and most of that text does not change the answer.
Over four weeks of remote research organised as EleutherAI’s Summer of Open AI Research, with mentors from EleutherAI and Apart Research, I took a published state of the art compression method, Selective Context, and rebuilt it against benchmarks it had never been run on, then rewrote its decision procedure in a different form. The interesting work is the reproduction rather than a novelty claim, and I want to be direct about that.
A model charges you, in money and in latency, roughly in proportion to how many tokens it reads. Long context work makes that bill grow fast. Compression attacks it from the cheapest possible angle: do not change the model, do not retrain anything, just delete the parts of the prompt that were not carrying information, and hand the model a shorter version of the same question.
The catch is that “not carrying information” has to be decided without knowing the answer, by something small and fast running ahead of the expensive model. That constraint is why compression methods tend to look like cheap statistical filters rather than like reasoning systems.
Selective Context, from Li and colleagues at EMNLP 2023, makes that decision with a small language model and a quantity called self information: for each unit of text, how surprising was it given everything before it. Predictable text carries little information, so it is dropped first. Surprising text is kept. The method scores lexical units, ranks them, and prunes the bottom of the ranking until the prompt fits the budget.
The appeal is that it needs no training and no labels. The risk is that surprisal is a proxy for relevance, not relevance itself. A rare token can be surprising and irrelevant, and a completely predictable phrase can be the one the question turns on.
A method that works on the evaluation it was published with tells you less than most people assume. So the target was two settings the original work did not use.
LongBench v2, from Bai and colleagues at ACL 2025, is a long context benchmark of multiple choice questions over contexts running from roughly eight thousand words up into the hundreds of thousands, across document question answering, dialogue, code repositories and structured data. It is deliberately hard: the benchmark’s own authors report that human experts under time pressure land near 53.7 percent accuracy. MS MARCO is the other end of the shape, a large retrieval and question answering set built from real search queries, where the prompt is a bundle of retrieved passages rather than one long document.
Those two settings stress a compressor differently. On a long document, redundancy is spread thin across the whole text. On a retrieved passage set, redundancy is concentrated: several passages often say the same thing, and one of them says the thing that matters. A surprisal ranking has no notion of which of the near duplicates to keep.
The second piece of work was reframing the method. Instead of scoring lexical units and pruning a ranked list, the compression decision is expressed as a classification made per token: for each token, keep it or drop it.
This is a small change in phrasing and a large change in what becomes possible. A ranking rule is fixed, so the only lever is the threshold. A classifier is a learnable object, which means the keep or drop decision can be supervised, evaluated with the usual classification tooling, and compared against alternative decision rules on equal terms. It also makes the evaluation cleaner, because per token predictions can be measured directly rather than only through the downstream answer.
The pipeline is four stages with one decision point in the middle. A small causal language model runs once over the prompt and emits, for every token, how surprising that token was given the tokens before it. That number is the informativeness score, and producing it is the only place a model reads the prompt in full before the expensive model does. Scores are then aggregated to whatever unit the run is configured for, individual tokens or the lexical spans the original method used, so one scorer serves both framings.
The decision point is the cut. A target compression ratio, set per run rather than learned, fixes how many units survive: the scores supply the ordering, the ratio supplies the boundary. Units above the boundary are kept in their original positions, units below are deleted, and what remains is concatenated back into a prompt the answering model reads as ordinary text. Nothing is rewritten or paraphrased, so the shortened prompt is always a subsequence of the original.
Downstream, that prompt goes to the answering model and its output is handed to the benchmark harness: multiple choice grading on LongBench v2, retrieval style question answering on MS MARCO. Because the ratio is a run parameter, sweeping it traces answer quality against prompt length rather than reporting a single operating point.
The reframing swaps exactly one component. The ranking rule becomes a per token keep or drop classifier, which gives the decision a label, a loss, and its own error analysis, so it can be measured without routing everything through the downstream answer.
The figure traces one prompt through the four stages; the part to watch is the pair of boxes feeding the cut, which is the only interchangeable piece.
Most of the four weeks was inference at a scale that does not fit on one machine’s patience. Long context evaluation means feeding very large prompts repeatedly through models on rented cloud hardware, and the experiment code has to survive that: configuration separated from run logic so a variant is a config change rather than an edit, runs resumable so a crashed job does not restart from zero, and outputs stored so results can be recomputed without paying for inference twice. That work is invisible in any writeup and it is the reason a four week window produced a complete run rather than a partial one.
Four week research collaboration, 2025. The re-implementation, the benchmark port and the token classification framing were built and run.
I am not reporting a compression ratio or an accuracy figure here, because I do not have a result I am willing to stand behind as a clean comparison against the original paper’s published numbers. The benchmark fork used for the work is public at LongBench-Compression; the compression code on top of it is not yet pushed to that repository, so treat the link as the evaluation harness rather than as the method implementation.