Every conclusion deserves evidence.
PrüfPilot is a compact Document AI concept for auditors: grounded RAG, explicit evidence gates, human approval and measurable reliability — designed around the kind of production constraints that matter in GovTech.
three synthetic funding cases
The expense maps to eligible equipment costs and the supporting invoice and procurement record agree. Final action still requires human approval.
Claims must be supported before release
core controlInteractive synthetic edge cases
demoUnsupported material claims allowed
policyApproval remains consequential action
guardrailNo benchmark performance is claimed here. The page demonstrates the evaluation contract; measured results belong in the repo once the suite is executed.
Reliability before fluency.
The core idea is simple: the assistant may only make an important assessment when it can show where the claim came from, how strong the support is, and what evidence is still missing.
Claim → source → passage
Every material statement is linked to a concrete document span. Unsupported claims are blocked before they reach the auditor.
Abstain intelligently
When the evidence is insufficient, the system says so clearly and identifies the exact missing documents or facts.
Human approval by design
The model can prepare, compare and recommend — but consequential decisions remain reviewable and attributable.
Designed like a production system.
The proof-of-work is intentionally small, but the interfaces mirror a production architecture: FastAPI services, MCP-compatible tools, retrieval, verification and explicit handoff points.
GovTech systems should fail visibly.
In public-sector workflows, a confident but unsupported answer is often worse than no answer. PrüfPilot makes uncertainty inspectable and turns “I don’t know” into a useful next step.
Never optimise only for answer rate. Optimise for trustworthy outcomes.
Evaluate the behaviour that matters.
The benchmark is designed before the score: answerable and intentionally unanswerable cases test whether the system grounds, cites, abstains and retrieves — rather than merely sounding convincing.
Measured values will only be published after running the benchmark. No decorative percentages.
50 cases. 15 impossible on purpose.
Most RAG demos never test whether the model knows when not to answer. This one does: missing evidence, conflicting documents and near-match citations are first-class test cases.
Why I built this for aconium.
The Document AI role sits at an intersection I care about: agentic AI, evidence, MCP and public-sector systems where reliability is part of the product — not a layer added afterwards.
What I’d build next with the team
Built to start a technical conversation.
If this direction is useful, I’d love to show the implementation choices, failure cases and what I would test next with real aconium workflows.