Grounded AI · PDF Processing · Multi-Agent Analysis
Research-Paper Deconstruction Agent
A research-paper analysis system built around a strict separation between document extraction and LLM reasoning. The first pipeline reconstructs a PDF into structured Markdown and JSON; the second runs a LangGraph multi-agent workflow over that normalized representation to produce grounded technical analysis and reproduction guidance.
Architecture
Deterministic preprocessing + LangGraph MAS
Implemented analysis phases
5 of 8 planned agents
Primary design goal
Grounded analysis with explicit hallucination controls
01 · Engineering problem
You cannot ground analysis if the source representation is unreliable
Research PDFs are not clean text documents. Multi-column layouts, detached captions, tables, equations, images, heading ambiguity, and page-order issues can corrupt the context before an LLM ever sees it. The system therefore does not begin with “send the PDF to an LLM.” It first reconstructs a structured, inspectable representation and only then runs research agents over that output.
02 · System architecture
Two pipelines with different responsibilities
Pipeline A
Paper Preprocessor
Converts PDF layout into structured Markdown + JSON. The stages are modular so extraction, metadata, tables, reading-order logic, or LLM repair can be swapped independently.
Pipeline B
Research MAS
Loads the preprocessor output into shared LangGraph state, then runs specialized agents sequentially so later analysis can depend explicitly on earlier structured reports.
03 · PDF preprocessing pipeline
Reconstruct the paper before reasoning about it
Layout extraction
PyMuPDF extracts text blocks, coordinates, and embedded images while preserving page-level layout information.
Block classification
Heuristics classify headings, paragraphs, captions, lists, headers/footers, while table extraction is merged into the document model.
Reading-order reconstruction
The pipeline reconstructs multi-column reading order before section boundaries are derived.
Structured Markdown
The normalized document is rendered as readable Markdown with headings, paragraphs, images, captions, and tables.
Structured JSON
A section-wise JSON tree is produced together with indexes for images, tables, and equations.
Optional structural refinement
An LLM can repair structure section-by-section without overwriting deterministic outputs, allowing before/after comparison.
Metadata can optionally come from a reachable GROBID server, with heuristic extraction as fallback. Equations are extracted separately. Deterministic Markdown/JSON outputs are preserved even when optional LLM structural refinement is enabled.
04 · Semantic paper schema
Paper headings are not the analysis schema
An “Introduction” can simultaneously contain motivation, background, and a problem statement. The Paper Structure Agent therefore uses semantic classification rather than assuming section titles directly represent analytical categories.
Required framing categories—problem statement, motivation, and background—receive a targeted recovery pass when the initial classifier cannot map any chunk to them. Recovery is restricted to title, abstract, and early sections; unsupported content remains explicitly missing rather than being invented.
05 · Multi-agent research workflow
Specialized analysis stages over one shared state
Paper Structure Agent
ImplementedMaps chunks into a fixed research schema: problem statement, motivation, background, proposed method, technical details, experimental setup, results, limitations, future work, and other.
General Section Analysis Agent
ImplementedSummarizes problem, motivation, results, and future work into structured claims, evidence, assumptions, importance, confidence, and hallucination metrics.
Background Knowledge Agent
ImplementedSeparates paper-grounded background answers from supplementary model knowledge instead of blending them into one untraceable response.
Proposed Method Deep-Dive Agent
ImplementedAnalyzes the proposed method in detail using only the extracted paper structure and document content.
Experiment Reproduction Agent
ImplementedBuilds an actionable reproduction checklist, environment/setup details, dataset/training/evaluation steps, missing information, and reproducibility risk.
Ablation & Significance Agent
PlannedPlanned analysis of component contribution and experimental significance.
Critic / Consistency Agent
PlannedPlanned cross-checking of claims, contradictions, and consistency across analysis outputs.
Final Synthesis Agent
PlannedPlanned final synthesis across all previous reports.
06 · Grounding & hallucination control
Verification changes the output, not just the score
The most important design choice is that faithfulness checking is not merely a confidence label attached after generation. Unsupported statements are actively removed from grounded outputs.
Source-scoped prompts
Each paper-grounded analysis receives only the relevant extracted categories rather than unrestricted paper + model context.
Separate verification pass
Claims and evidence generated by analysis are checked in a second LLM call against the original source text.
Drop unsupported claims
Unsupported claims are removed from final grounded output and preserved separately as dropped claims/evidence for auditability.
Deterministic hallucination score
The hallucination score is computed in Python from unsupported / checked claims rather than trusting the model to grade itself.
Fail closed on verification
If verification fails, the system marks the output maximally uncertain instead of pretending it was validated.
Paper vs external knowledge split
Background analysis keeps paper_answer and external_answer separate, with external model knowledge explicitly labeled as unverified internal knowledge.
07 · Background knowledge separation
Paper-grounded knowledge and model knowledge are intentionally different fields
The Background Knowledge Agent does something unusual: it does not silently enrich the paper with general LLM knowledge. It stores a verified paper_answer separately from external_answer. External items are explicitly labeled as model-internal knowledge rather than pretending to be retrieved citations. The code identifies retrieval-backed RAG as the future replacement for this placeholder.
08 · Reproduction analysis
Convert method sections into an implementation-oriented checklist
The Experiment Reproduction Agent consumes experimental setup, results, appendix content, and method implementation notes to produce environment requirements, dataset preparation, training steps, evaluation steps, missing details, a reproducibility-risk assessment, and checkable source claims. Those claims are then verified against the supplied source material.
09 · Quality evaluation
Extraction quality can be evaluated independently of research analysis
The CLI supports an extraction-quality evaluator and emits both human-readable and machine-readable reports. When a ground-truth transcript is supplied it can measure text fidelity using CER/WER; without one, it can measure structural-refinement drift between deterministic and LLM-refined output.
10 · Architecture decisions
Keep PDF extraction deterministic by default and make LLM structural repair optional.
Never overwrite the pre-LLM representation; always preserve before/after artifacts for comparison.
Separate semantic structure classification from deeper analysis so downstream agents consume a stable schema.
Use shared LangGraph state with one typed report per analysis phase rather than passing uncontrolled prose between agents.
Use a second faithfulness pass and deterministically drop unsupported grounded claims.
Separate paper-grounded statements from external model knowledge instead of mixing both into a single answer.
Treat missing information as a first-class output, especially in experiment reproduction, rather than filling gaps with plausible defaults.
Engineering takeaway
The project's core idea is not “summarize a PDF with an LLM.” It is to build a controlled information pipeline where source extraction, semantic normalization, analysis, verification, uncertainty, missing information, and external knowledge have explicit boundaries. That makes the final analysis easier to inspect, test, and defend.