A domain-specific retrieval encoder for Indonesian regulation, trained on one consumer GPU. It significantly beats the commercial embedding API it was built to replace on the query distribution the product actually serves, and ties it on the other, at roughly an order of magnitude fewer parameters. The result rests on two benchmarks built before any model was trained.
Role Researcher, SOAR-26 Domain Encoder track, working on the Klaussa AI corpus. Period June to July 2026. Scope Benchmark construction, training data generation, fine-tuning, evaluation, paper draft.
Klaussa’s legal search runs over 305,628 Indonesian regulations, and the component deciding which of them a question reaches is an embedding model: a neural network that turns text into a vector so that a query and the right document land near each other. Production used a commercial embedding API. That is a per-call cost, a network round trip, and a vendor dependency on the single most product-critical piece of the system.
The result is a 568-million-parameter encoder, trained on one consumer GPU, that significantly beats that API on the queries the production system actually issues and ties it on the other query style. It also beats an 8-billion-parameter general-purpose retrieval model, fourteen times its size, on both: domain specificity outweighed scale by a clear margin. What made that possible was not a training trick, but noticing that quality had been measured on the wrong queries all along.
Klaussa’s search does not receive human questions. It sits behind an agent that decomposes a question into short keyword searches, so the retriever is benchmarked on one distribution and deployed on another. Call these two shapes registers.
The split is sharp and measurable. Human questions name the governing law 73 percent of the time. Real production agent queries name it 12.9 percent of the time. The synthetic training data generated for fine-tuning named it 0.6 percent, because the generator had been told not to cite regulations.
Naming a law makes retrieval partly a lexical matching problem; not naming it makes it purely semantic. So you can tune on one register, silently lose the other, and see no warning, because your offline numbers are computed on the register you tuned for.
No public benchmark reflected either register on this corpus, so both were built first. AQUAK is 855 real production agent queries with graded relevance labels, stratified by intent. KlaussaBench is 648 natural-language legal question and answer items in the human register, plus 488 graded at the level of the pasal, the numbered provision an answer actually cites.
Two pieces of measurement discipline mattered more than the benchmarks themselves. Evaluation runs against a restricted candidate pool proven to reproduce full-corpus scores within 0.002, making each evaluation roughly a hundred times cheaper. And an audit found between 43 and 70 percent of the original labels questionable, fixing a noise floor: differences below about 0.01 are not called wins, and headline claims are checked with a significance test rather than by eye.
The first serious fine-tune improved the agent register and regressed the human one, and the regression sat entirely in queries that named a law. The model had traded citation matching away for semantics, exactly as the 0.6 percent training statistic predicted.
The fix was data, not architecture. The mix was rebuilt with a citation slice: queries naming a regulation by number and year, paired with wrong answers chosen to be nearly right that differ only in that number or year. Those force the model to read the citation instead of discarding it. After that, one model beat the base encoder on every metric of both registers.
Every run was a full-parameter fine-tune on a single RTX 5060 Ti with 16 GB, using cached contrastive training to reach an effective batch of 1,024 within that memory, at roughly two hours per run. Each checkpoint is then interpolated back toward the original weights, a merge that recovers what fine-tuning forgets.
Every run starts from BGE-M3, a 568M-parameter multilingual encoder producing 1024-dimensional CLS-pooled vectors with symmetric prompts: query and document are embedded with no instruction prefix. That symmetry has to be replicated exactly in training and in evaluation, or scores drift for reasons that have nothing to do with the model.
Training is a full-parameter fine-tune, no adapters, over roughly 100,000 query and document pairs. Fifty-five percent of them are the citation slice: templated queries naming a regulation by number and year, a cross-reference slice derived from the corpus citation graph, and longer application-style questions. Negatives are mined by the previous champion and filtered so that near-duplicates of the true answer never get used as negatives; the citation negatives are deliberate near-misses, same number and different year. Every gold document from both benchmarks is excluded from the training targets.
The objective is CachedMultipleNegativesRankingLoss, which processes the batch in slices of 16 so a contrastive batch of 1,024 still fits in 16 GB. With bf16 and gradient checkpointing the whole thing is one RTX 5060 Ti: sequences capped at 288 tokens, learning rate 1e-5, 10 percent warmup. A one-epoch run takes about 100 minutes; the two-epoch run that produced the champion took about four and a half hours.
The fine-tuned weights are not what ships. WiSE-FT interpolates them back toward the original base, theta equals alpha times the fine-tune plus one minus alpha times the base, swept from 0.7 to 0.85, with 0.75 winning. The merge costs nothing at inference.
Evaluation then embeds one shared 39,153-document pool per model and scores both benchmarks with the same kit, calling win, tie or loss per metric from a 10,000-sample paired bootstrap.
The diagram traces one model from base weights to that verdict. The step to watch is the merge in the middle, because what gets scored is never the fine-tune on its own.
Scored on a shared candidate pool with identical labels, using nDCG at 10: a score between 0 and 1 rewarding correct documents near the top of the first ten results.
| Model | Size | Agent queries | Human questions |
|---|---|---|---|
| Commercial embedding API | Large, undisclosed | 0.413 | 0.856 |
| General 8B retrieval model | 8B | 0.340 | 0.704 |
| Base open encoder | 568M | 0.437 | 0.735 |
| This work, dense | 568M | 0.454 | 0.864 |
| This work, plus keyword signal | 568M | 0.496 | 0.880 |
On agent queries the win over the API is significant on every metric. On human questions the dense model is a statistical tie, with all ten confidence intervals spanning zero. Adding a light classical keyword signal, which costs no parameters and runs locally, pushes past the API on both.
Several substantial efforts produced nothing and are logged as such. A twelve-experiment campaign distilling a stronger reranking model into the encoder worked as an objective and still degraded performance on the real query distribution. Fine-tuning on purely synthetic queries backfired twice. Cross-corpus hard negative mining was a clean negative result. An architecture comparison concluded architecture was not the lever at all.
The most useful negative result concerns the agent design itself. At equal retrieval budget, decomposing a question into sub-queries found the right regulations more often on multi-step questions but retrieved fewer of the correct provisions and answered no better than one search with the whole question. The stated scope limit matters: this tests parallel one-shot decomposition, not iterative agents that read and re-query.
The research is complete and unshipped, which is uncomfortable but accurate. The model, both benchmarks, the evaluation kit and the experiment record exist and are reproducible. Production still runs the commercial API. A head-to-head comparison dashboard is designed but not deployed. A six-page resource paper draft exists with a pre-submission checklist that is not cleared: labels remain machine-generated pending human validation, the released queries need a privacy scrub, and the author field is a placeholder.
The last commit is from mid June, with about three weeks of further results uncommitted since. Nothing is blocked technically. What remains is the distance between a result and a released artifact, and none of it is research.