A pipeline that scrapes metadata for every paper from fourteen top computer science conferences, extracts the text from each PDF, and stores it as markdown in object storage. The metadata index is complete at 169,864 papers. The text extraction reached 1.5 percent and has not moved since March 2026.
Role Sole author, personal infrastructure project. Period March 2026. Scope Scrapers, extraction pipeline, object storage.
The goal was a local, searchable copy of the machine learning literature: every paper from the fourteen conferences that CSRankings treats as top venues, pulled down as PDF, converted to plain markdown text, and parked in object storage where a language model could read it later.
Half of that goal is met. There is a metadata index of 169,864 papers across AAAI, ACL, CVPR, ECCV, EMNLP, ICCV, ICLR, ICML, IJCAI, KDD, NAACL, NeurIPS, SIGIR and WWW, spanning 2010 to 2026, and a working pipeline that turns any record in it into a markdown file in a Cloudflare R2 bucket. The other half is 1.5 percent finished. 2,545 papers have been through the pipeline, 2,481 successfully and 64 with errors. The other 167,000 are still sitting in the index as rows.
One thing to say plainly, because the repository name implies otherwise: there is no MCP server here. MCP, the Model Context Protocol, is the standard by which a language model calls external tools, and serving this corpus through one was the intended endpoint. The extraction code was deliberately factored into a reusable module so an MCP tool could call it later. That is as far as the idea got.
Conference proceedings do not live in one place, so the scraping layer is six modules, one per publishing platform, each exporting the same function shape and returning the same record: ACL Anthology covers ACL, EMNLP and NAACL; the CVF open access site covers CVPR, ICCV and ECCV; DBLP covers AAAI, IJCAI, KDD, SIGIR and WWW; and NeurIPS, OpenReview and PMLR each cover one venue. Records deduplicate on title, conference and year, which is what keeps the count honest across sources that overlap.
The extraction pipeline is deliberately small. It downloads a PDF to a temporary file, converts it to markdown, prepends YAML front matter with the paper’s metadata, uploads the result to object storage under a conference and year key, and deletes the temporary file. Nothing accumulates on disk except a SQLite file tracking which papers are done, which means a re-run skips finished work and the whole thing can be interrupted and restarted without losing anything. That resumability is the reason a stalled run is recoverable rather than wasted.
The most useful decision on this project was undoing an earlier one.
The first design ran OCR through GLM-OCR, a vision model served behind vLLM, which meant 16 GB or more of VRAM, a model download, a second process to babysit, and a batching structure in the pipeline that existed purely to keep an expensive GPU busy. A design note two days before the change made the observation that killed it: conference papers are born digital, not scanned. They already carry a text layer. Reading a text layer is not an OCR problem, and a vision model was solving a problem this corpus does not have.
Swapping to PyMuPDF text extraction removed the GPU requirement, the model download, the second process and the batching complexity in one commit. It is worth contrasting with the OCR work on Indonesian government documents, where the input really is scanned paper and a vision model is unavoidable. The lesson is not that vision models are heavy, it is that matching the tool to the input is worth checking before buying hardware for it.
Scraping and processing are two separate programs over two SQLite files. The scraper entry point imports six modules by name, each exporting the same scrape function of a start and end year and returning the same record shape, and inserts what comes back into papers.db under a uniqueness constraint on title, conference and year. That constraint is doing real work: DBLP and the ACL Anthology overlap, and an ignored insert is cheaper than a deduplication pass afterwards.
The pipeline reads that table back, newest years first, in batches of twenty. Each batch first drops any paper already marked done in the tracking database, then downloads the survivors with eight threads into one temporary directory. Every thread keeps its own scraper session, and a per host delay table throttles requests before they leave. A 429 backs off exponentially and retries up to four times. If the bytes that come back are not a PDF, the HTML is parsed for a citation PDF meta tag and the real file fetched once.
Extraction is deliberately not threaded. PyMuPDF converts the batch’s downloads one paper at a time, an output under fifty characters counts as a failure, and the markdown gets a YAML header carrying title, conference, year and source URL before a boto3 client puts it into R2 under a conference and year key. The temporary directory is a context manager, so the PDFs are gone the moment the batch ends and nothing but the tracking database grows. Credentials come from a local environment file that is never committed.
Every outcome goes back into that database as ok, skip or error with the object key and character count. The ledger is the whole idempotency story, and it is also the honest measure of progress: it holds 2,545 rows against an index of 169,864.
The figure shows that path plus the one that was deleted. Look at the dashed box under the extraction step, and at the bar along the bottom.
Two obstacles were about being an unwelcome client rather than about code.
OpenReview sits behind bot protection and returned 403 to a plain HTTP client. The fix was a scraper library that negotiates the challenge, one session per thread rather than one shared session, and leaving the default user agent alone instead of spoofing one. Getting past that produced the next problem, 429 rate limiting, which was settled by raising the per request delay for that host to two seconds and retrying on rejection. Each publisher domain now has its own delay in a small table, because the polite rate for a university proceedings server and for a large commercial platform are not the same number.
There was also a performance rewrite, threads for downloading and separate processes for extraction, that was committed and reverted to the simpler batch pipeline within the same day. The simpler version was fast enough given that the real constraint is how quickly the publishers will let you ask.
Stalled, and has been since 24 March 2026. Eighteen commits over thirteen days, then nothing.
Nothing technical is blocking it. The pipeline is idempotent, the index is complete, and finishing the remaining 98.5 percent is a matter of wall clock time and bandwidth rather than unsolved problems. There is no test suite, no linter, no retrieval layer over the extracted text, and no MCP server, so even a completed extraction run would leave the original purpose unbuilt.
It stopped for the ordinary reason: nothing downstream depended on it yet. That is an honest description of a side project, not a failure of the design, but it does mean the useful artifact today is the metadata index and the scraper set, not the corpus.