A retrieval augmented question answering system for Indonesian consumer health questions, built so that it goes back for more evidence mid sentence whenever the model produces a low confidence token. First author work, three person team, second place at Gemastik 2024 Data Mining.
Role First author and project lead. Period 2024. Scope Research framing, data scraping, retrieval and generator inference across four models, and the active retrieval mechanism itself.
We built a question answering system for Indonesian consumer health questions, the kind an ordinary person types rather than the kind a clinician writes, and made it check its sources in the middle of answering instead of only at the start. The paper, “Exploring Indonesian Consumer Health Question Answering using Active Multi Retrieval”, took second place at Gemastik 2024 in the Data Mining category, the largest technology competition in Indonesia, run by the Ministry of Education, Research and Technology.
The reason to care is the mechanism rather than the placing. Almost every retrieval augmented system works in one shot: read the question, fetch documents, then generate an answer and hope the fetched documents covered it. Ours watches the model while it writes. When the model emits a token it is not confident about, that is treated as a signal that the evidence in front of it has run out, and the system stops and retrieves again before continuing.
Consumer health questions are written by people who do not know the vocabulary of the thing they are worried about. They describe symptoms in everyday language, bundle three questions into one, and leave out the details a clinician would ask for first. Doing this in Indonesian narrows the ground further, because the medical text available to retrieve from is far thinner than it is in English.
That combination is exactly where a language model hallucinates, meaning it produces fluent text that is not supported by any source. The failure is worse here than in most domains, because a confident wrong answer about a drug interaction is not an inconvenience. So the target was not only accuracy but a reduction in the number of answers produced under low confidence with no evidence behind them.
Retrieval augmented generation, usually shortened to RAG, means fetching relevant documents and putting them into the prompt so the model answers from sources instead of memory. The standard version retrieves once, using the question as the query.
The structural problem is that the question is a bad description of what the answer will need. A user asks about a symptom; the answer turns out to require a dosage, a contraindication and a referral threshold, none of which appear in the question, so none of which the initial query retrieved. The model reaches that point, finds nothing in the context supporting it, and fills the gap from its parameters. No amount of improving the first retrieval fixes this, because the need only becomes visible after generation has started.
The system exposes a signal it already produces. A generating model assigns a probability to every token it emits. When it writes a well supported clause it commits confidently; when it is guessing at a fact it does not have, that confidence drops.
Active Multi Retrieval treats a low probability token as an interrupt. Generation pauses, the partially written answer forms a new query, fresh passages are retrieved into the context, and generation resumes. The retrieval budget is spent where it is needed rather than uniformly, and later queries are informed by what the model has already committed to saying, which is strictly more information than the original question carried.
The general idea of triggering retrieval on low confidence has a precedent in the literature, notably FLARE from Jiang and colleagues at EMNLP 2023. Our contribution is applying it to Indonesian consumer health, where the retrieval corpus is sparse and the cost of an unsupported claim is high, and running it across several generator models rather than one.
The system is a decoding loop with a monitor attached to it. The question is sent to a ColBERT index built over the scraped corpus, the passages that come back are placed in the prompt ahead of the question, and generation starts. After each decoding step the monitor reads the probability the generator assigned to the token it just emitted. That value is the only signal the loop runs on: no separate classifier, no second model judging the output.
The probability is compared against a fixed threshold. Above it, decoding continues untouched and the monitor costs nothing beyond a comparison. Below it, the loop breaks. The text generated since the last break is turned into a query, the index is hit again, the returned passages are added to the context, and decoding resumes from the point where it paused, now conditioned on the larger evidence set. The question is never re-sent as the query; the partial answer is, which is why the second retrieval can find things the first one had no way to ask for.
Three things are configuration rather than learned behaviour, and each is a place the loop can be tuned: the threshold that defines hesitation, the span of generated text that becomes the new query, and how much of the earlier context is carried forward when new passages arrive. Set the threshold at zero and the system collapses back into ordinary retrieve once generation, which is what makes the two directly comparable on the same generator and the same index.
The figure puts the two shapes side by side; the arrow to watch is the dashed one running backwards from the confidence check into retrieval.
Retrieval uses ColBERT, from Khattab and Zaharia at SIGIR 2020, which scores a document by matching every query token against every document token rather than compressing each side into a single vector. That fine grained matching is worth its extra cost here, because the useful overlap between a colloquial symptom description and a clinical passage is often a couple of terms rather than a whole sentence.
On the generation side I ran four systems: SAILOR-7B, chosen for its Southeast Asian language coverage; Llama 3 8B Instruct; OpenOrcaPlatypus 13B; and GPT-3.5 through its API as the commercial reference point. Running open weight models of three sizes alongside a hosted one keeps the finding about the mechanism rather than about one model’s quirks.
I was responsible for the data scraping that produced the corpus, the encoder and generator inference for all four systems, and the active retrieval loop itself.
I ran the brainstorming that produced the research question, which meant finding the point where an unmet research gap and an unmet application gap overlapped, then holding a three person team to that scope.
Competition and research project, 2024. First author, roughly 3,000 lines of code, second place at Gemastik 2024 Data Mining.
The comparative numbers behind the system, hallucination rates, retrieval counts and per model accuracy, are in the paper and are deliberately not restated on this page. The honest limitation is that token probability is a proxy for knowing, not knowing itself: a model can be fluently confident and wrong, in which case no retrieval is ever triggered, and that failure mode is not addressed by this design.