Samhita @sushrutalgs.ai

See the full portfolio →

Built with

  • Python
  • Pydantic
  • BioLORD
  • Cloudflare R2

Links

  • live product↗
  • Request repo access→

Source is private; sushrutalgs.ai is a live product. Happy to walk through the code or grant read access on request.

A Python pipeline that turns full-length surgical-textbook PDFs into clean, structured, machine-readable data: 220 chapters into a searchable knowledge base of sections, figures and tables, exported as a versioned, hash-verified package the search platform loads directly into its graph and vector databases.

The problem

sushrutalgs.ai answers surgical-exam questions and traces every claim back to a line in a standard textbook, so a citation is only worth anything if the text behind it is intact and addressable down to the section, figure or table. What actually existed was three full surgical textbooks as print PDFs: 220 chapters and 5,941 pages laid out for a typesetter, where a table is drawn rather than encoded and a figure's caption is a floating run of text that happens to sit near it. None of the retrieval work could start until that was structured data, and nobody was going to re-key three textbooks by hand.

Why it is hard

PDF extraction fails silently. A parser that loses content does not raise an error, it returns a slightly smaller document, and every downstream count still looks plausible, so the loss only surfaces months later when someone asks about a topic the book plainly covers and the answer has nothing to cite. One bug in the extraction path quietly dropped 968 table elements while the run reported success. The second half of the problem is that layout rules tuned to one publisher break on the next, so a single static rule set cannot be trusted across three books, and "it ran without errors" is not evidence of anything.

The design

Two rules shaped the pipeline: structure is decided only by code that behaves the same way twice, and every stage has to be able to say what it lost.

  • Recovery as its own phase, not as error handling. Parsing starts from Adobe's PDF extraction JSON and runs parse, recover and structure, then cleaning, enhancement and annotation in parallel; the six-phase recovery stage exists to detect and repair the specific ways each book breaks rather than to skip the pages it cannot read.
  • Deterministic structure, models only at the edges. The tempting alternative was to hand whole pages to an LLM and ask for structured output, which is far less work to build, but it reruns differently and gives you no way to tell a hallucinated section boundary from a real one, which is fatal when the citation is the product. Claude is used only to describe figures and tables that a deterministic pass has already located.
  • Structure before embeddings. Content is assembled into a knowledge graph of 71,621 nodes and 130,057 edges with 5,987 cross-references resolved and a 17-domain taxonomy over the top, and only then embedded, into 52,871 vectors from BioLORD, a medical-domain model, rather than a general-purpose one that flattens surgical vocabulary.
  • Immutable, content-hashed exports. Each run publishes versioned chapter packages under SHA-256 hashes to Cloudflare R2 with manifest drift detection, so the retrieval backend pins a known version instead of reading a bucket that can change underneath it, and a CI workflow with mocked services runs the pipeline so a parser change cannot quietly alter an export.
System architecture. Tap to enlarge.

Where it stands

All 220 chapters and 5,941 pages went through end to end, and all 220 exported chapter packages pass structural validation. The check that mattered most was a clean rebuild from the source PDFs reproducing every count, because that is the only way to know the output does not depend on incremental state left over from earlier runs. The honest limit is that the validation is structural: it shows nothing went missing or landed in the wrong chapter, but a model-written figure description is still a model's description, and those are spot-checked by hand rather than measured. A fourth textbook would still need parser work; the pipeline adapts, it is not publisher-agnostic.

In numbers

220
chapters ingested
71,621
knowledge-graph nodes
52,871
dense vectors
SHA-256
content-hashed exports

The pipeline's job was never speed, it was trust: 5,941 pages a citation product has to be able to stand behind. A parser that fails loudly instead of returning a slightly smaller document, and an export that is hash-verified end to end, is what makes that trust something other than a hope.