A self-updating index of the AI research literature, from conference rosters through metadata enrichment to extracted full text, exposed to language model agents as a small set of query tools. Built so an agent can check whether a problem has already been solved without depending on a hosted search API.
Role Sole designer and engineer. Period 2026. Scope Data pipeline, deduplication, retrieval layer, agent tool surface, scheduled operations.
Paperbase is a local copy of the AI research literature, indexed and queryable, built so a language model agent can answer one question reliably: has somebody already done this?
That question is the bottleneck in research tooling. Hosted search APIs answer it shallowly, over abstracts, with ranking you cannot inspect and coverage you cannot bound. The alternative is to own the corpus: 163,849 papers from AAAI, NeurIPS, CVPR, ACL, EMNLP, ICML, ICLR, IJCAI and ICCV, plus workshops and several journals, of which 106,501 have their full body text extracted and searchable. It is the substrate the gapfinder project runs on.
The payoff of ownership is that search becomes exhaustive within a scope you can state. An agent can say “not found in AI venues from 2015 onward” and mean it, rather than “the first page of results did not have it”.
Every stage is idempotent, meaning running it twice produces the same result as running it once. The roster stage asks DBLP, a computer science bibliography, for everything published at a venue in a given year. That choice was empirical: the first design used OpenAlex and found its venue filter unreliable for recent proceedings, returning zero papers for NeurIPS from 2022 onward, so DBLP became the spine and OpenAlex was demoted to metadata. Semantic Scholar supplies citation counts and abstracts, and each provider carries its own rate limit, backoff and on-disk cache.
Deduplication is deliberately conservative: identifiers from different providers are collapsed into canonical clusters, with hard merges only on shared identifiers or an exact normalised key of title, year and first author. Fuzzy matches are flagged, never auto-merged. That crosswalk holds 373,013 rows, which measures how much of this problem is identity rather than retrieval. Full text then arrives as PDFs converted to markdown, routed per venue because AAAI, IJCAI and MIT Press each expose their files differently. Every stage writes to a durable ledger, so a job killed halfway resumes without duplicating anything.
Retrieval is a pluggable strategy rather than a hardcoded engine. The default shells out to a Rust trigram and regular expression search tool over a corpus of one text file per paper. The fallback is SQLite’s built-in full text search, which needs no external binary. Both implement the same interface, so switching is a config change.
Neither is semantic. There are no vector embeddings in the search path, and that is a decision rather than an omission. Semantic search is good at “papers about roughly this” and bad at the query that matters here, which is “does the exact string BookSum appear in the body of any paper”. Exhaustive lexical matching gives an agent a genuine brute force pass, at the cost of queries phrased as terms rather than prose.
The corpus is exposed over the Model Context Protocol, a standard interface for giving agents access to external systems, as nine tools: search, fetch metadata, read a window of a paper’s text, find related work, list a venue, report coverage, walk citations in either direction, and ingest a paper on demand.
The reading model took the most thought. An early instinct is a tool returning a whole paper, which burns the context window on one document and makes long investigations impossible. Instead the pattern mirrors how an engineer reads code: search returns ranked snippets, which answer most questions outright, and the reader pages a window only when a snippet is not enough. Forward citation lookup is fetched live and cached, returning papers inside the index plus stubs for those outside it that can be ingested on request, so the index grows in response to use rather than only on a schedule.
A scheduled job runs roughly every two weeks with no supervision. It re-rosters the recent year window across all core venues, enriches missing abstracts, recovers PDF URLs that failed the first time, then drains and indexes new full text. The last run added 4,029 papers and recovered 678 AAAI PDF links.
It also exposed the real operational constraint. Full text at this scale is heavy: the archived PDFs alone occupy 261 GB, the search index 13 GB, the extracted text 8 GB. That refresh processed two chunks and stopped itself cleanly, logging that free disk had fallen below its 15 GB floor and that a watchdog would resume when space was freed. Halting on a stated threshold rather than filling the disk is correct, but the underlying problem is unaddressed: the PDFs are an archive nothing reads once text is extracted, and pruning them has not been automated.
Paperbase is one Python package with a command line entry point and a thin agent wrapper over the same calls. Ingest runs as five idempotent stages. DBLP supplies the roster for a venue and year, which is why it is the spine: OpenAlex was tried first and its venue filter returned nothing for NeurIPS from 2022 onward. Every identifier that arrives is collapsed by union find into a canonical cluster and written to an alias crosswalk, so re-ingesting a known DOI or arXiv id resolves to the record that already exists. OpenAlex and Semantic Scholar then fill abstracts, citation counts and one line summaries, each sitting behind a shared HTTP layer that owns the token bucket, the retry backoff and an on-disk response cache. PDF fetching is routed by venue: NeurIPS to papers.nips.cc, ACL to the Anthology, ICML to PMLR, CVPR and ICCV to the CVF open access host, IJCAI and AAAI to their own proceedings, everything else to arXiv, and all of it through a rotating proxy pool because venue hosts throttle per source IP. PyMuPDF extracts the body, and that text is projected into one file per paper for the search index.
Storage is four artifacts side by side: a SQLite database in write ahead log mode holding papers, aliases and a per paper stage ledger; the extracted text; the PDF archive; and the trigram index. Retrieval sits behind a strategy interface, so the Rust trigrep binary and SQLite FTS5 are one config flip apart. FastMCP publishes the nine tools over stdio. A systemd user timer fires on the first and fifteenth of the month, re-rosters the current year and the two before it across every venue stream, and pauses the full text drain when free disk falls under the floor.
Two decision points are worth finding in the diagram below: the rejected roster source on the left, which is why DBLP sits where it does, and the dashed fallback under the search box, which is the swap point the whole retrieval layer was designed around.
Functional and running, with the scheduled refresh live and firing on time. The first milestone is complete end to end: ingest, deduplication, resumable jobs, full text extraction, pluggable search, citation tools, agent tool server and the scheduler.
Development has stopped. The last commit was 14 July 2026 and the remaining roadmap is untouched. Full text is fetched almost entirely through preprint servers; routing to publisher-hosted proceedings for ACL, ICML, NeurIPS and the vision conferences is still stubbed, which is a large part of why 57,348 papers have no body text, and abstract coverage stands at 101,927. The most valuable planned feature, ingesting peer review text so an agent can report how a paper was received and not only what it claims, is blocked: the free tier of that platform’s API stops at 2023, and the years worth having need an authenticated client that was never set up. Semantic retrieval, roster polling and citation context extraction are specified and unbuilt.