Work / Research / Research-Paper Deconstruction Agent
Grounded AI · PDF Processing · Multi-Agent Analysis
Research-Paper Deconstruction Agent
A research-paper analysis system built around a strict separation between document extraction and LLM reasoning. The first pipeline reconstructs a PDF into structured Markdown and JSON; the second runs a LangGraph multi-agent workflow over that normalized representation to produce grounded technical analysis and reproduction guidance.
Architecture
Deterministic preprocessing + LangGraph MAS
Phases
5 of 8 planned agents
Design goal
Grounded analysis, explicit hallucination controls
Status
Working · 5 of 8 phases implemented
01 — Problem
You cannot ground analysis on an unreliable source
Research PDFs are not clean text documents. Multi-column layouts, detached captions, tables, equations, images, heading ambiguity, and page-order issues can corrupt the context before an LLM ever sees it. The system therefore does not begin with “send the PDF to an LLM.” It first reconstructs a structured, inspectable representation and only then runs research agents over that output.
02 — Architecture
Two pipelines with different responsibilities
Pipeline A
Paper Preprocessor
Converts PDF layout into structured Markdown + JSON. Stages are modular so extraction, metadata, tables, reading-order logic, or LLM repair can be swapped independently.
Pipeline B
Research MAS
Loads the preprocessor output into shared LangGraph state, then runs specialized agents sequentially so later analysis can depend explicitly on earlier structured reports.
03 — Preprocessing
Reconstruct the paper before reasoning about it
Layout extraction
PyMuPDF extracts text blocks, coordinates, and embedded images while preserving page-level layout information.
Block classification
Heuristics classify headings, paragraphs, captions, lists, headers/footers, while table extraction is merged into the document model.
Reading-order reconstruction
The pipeline reconstructs multi-column reading order before section boundaries are derived.
Structured Markdown
The normalized document is rendered as readable Markdown with headings, paragraphs, images, captions, and tables.
Structured JSON
A section-wise JSON tree is produced together with indexes for images, tables, and equations.
Optional structural refinement
An LLM can repair structure section-by-section without overwriting deterministic outputs, allowing before/after comparison.
Metadata can optionally come from a reachable GROBID server, with heuristic extraction as fallback. Equations are extracted separately. Deterministic Markdown/JSON outputs are preserved even when optional LLM structural refinement is enabled.
04 — Schema
Paper headings are not the analysis schema
An “Introduction” can simultaneously contain motivation, background, and a problem statement. The Paper Structure Agent therefore uses semantic classification rather than assuming section titles directly represent analytical categories.
Required framing categories — problem statement, motivation, and background — receive a targeted recovery pass when the initial classifier cannot map any chunk to them. Recovery is restricted to title, abstract, and early sections; unsupported content remains explicitly missing rather than being invented.
05 — Agents
Specialized analysis stages over one shared state
Implemented
Paper Structure Agent
Maps chunks into a fixed research schema: problem statement, motivation, background, proposed method, technical details, experimental setup, results, limitations, future work, and other.
Implemented
General Section Analysis Agent
Summarizes problem, motivation, results, and future work into structured claims, evidence, assumptions, importance, confidence, and hallucination metrics.
Implemented
Background Knowledge Agent
Separates paper-grounded background answers from supplementary model knowledge instead of blending them into one untraceable response.
Implemented
Proposed Method Deep-Dive Agent
Analyzes the proposed method in detail using only the extracted paper structure and document content.
Implemented
Experiment Reproduction Agent
Builds an actionable reproduction checklist, environment/setup details, dataset/training/evaluation steps, missing information, and reproducibility risk.
Planned
Ablation & Significance Agent
Planned analysis of component contribution and experimental significance.
Planned
Critic / Consistency Agent
Planned cross-checking of claims, contradictions, and consistency across analysis outputs.
Planned
Final Synthesis Agent
Planned final synthesis across all previous reports.
06 — Grounding
Verification changes the output, not just the score
The most important design choice is that faithfulness checking is not merely a confidence label attached after generation. Unsupported statements are actively removed from grounded outputs.
Source-scoped prompts
Each paper-grounded analysis receives only the relevant extracted categories rather than unrestricted paper + model context.
Separate verification pass
Claims and evidence generated by analysis are checked in a second LLM call against the original source text.
Drop unsupported claims
Unsupported claims are removed from final grounded output and preserved separately as dropped claims/evidence for auditability.
Deterministic hallucination score
The hallucination score is computed in Python from unsupported / checked claims rather than trusting the model to grade itself.
Fail closed on verification
If verification fails, the system marks the output maximally uncertain instead of pretending it was validated.
Paper vs external knowledge split
Background analysis keeps paper_answer and external_answer separate, with external model knowledge explicitly labeled as unverified internal knowledge.
07 — Background knowledge
Paper knowledge and model knowledge are different fields
The Background Knowledge Agent does something unusual: it does not silently enrich the paper with general LLM knowledge. It stores a verified paper_answer separately from external_answer. External items are explicitly labeled as model-internal knowledge rather than pretending to be retrieved citations. Retrieval-backed RAG is identified as the future replacement for this placeholder.
08 — Reproduction
Convert method sections into an implementation checklist
The Experiment Reproduction Agent consumes experimental setup, results, appendix content, and method implementation notes to produce environment requirements, dataset preparation, training steps, evaluation steps, missing details, a reproducibility-risk assessment, and checkable source claims. Those claims are then verified against the supplied source material.
09 — Evaluation
Extraction quality can be evaluated independently
The CLI supports an extraction-quality evaluator and emits both human-readable and machine-readable reports. When a ground-truth transcript is supplied it can measure text fidelity using CER/WER; without one, it can measure structural-refinement drift between deterministic and LLM-refined output.
10 — Decisions
Engineering decisions and what they cost
Keep PDF extraction deterministic by default and make LLM structural repair optional.
Never overwrite the pre-LLM representation; always preserve before/after artifacts for comparison.
Separate semantic structure classification from deeper analysis so downstream agents consume a stable schema.
Use shared LangGraph state with one typed report per analysis phase rather than passing uncontrolled prose between agents.
Use a second faithfulness pass and deterministically drop unsupported grounded claims.
Separate paper-grounded statements from external model knowledge instead of mixing both into a single answer.
Treat missing information as a first-class output, especially in experiment reproduction, rather than filling gaps with plausible defaults.
11 — Stack