Back to Portal

Error Analysis Synthesis Pipeline

A post-Stage-5 workflow for turning selected papers into auditable failure evidence, canonical challenge clusters, temporal/community dynamics, and research-gap synthesis.

Cached data LLM extraction/synthesis Analysis layer Audit interface Human validation
Position in the larger workflow: this pipeline starts after the literature-review pipeline has already produced a frozen Stage 5 seed set. It does not expand the corpus or decide which papers belong in scope. Its job is to mine the selected papers for explicit evidence about failures, limitations, diagnostic breakdowns, and unresolved gaps.
Current snapshot: 81 Stage 5 papers 639 merged evidence rows 20 challenge clusters 639 embedding points 4029 KNN edges
Input

Frozen Stage 5 Set

Seed papersdata/stage5/stage5_paper_index.json
Cached full textdata/stage5/extracted_text/
Research boundaryManual labels and paper cards define the scope.
Human checkpointStage 5 membership is already frozen before this analysis begins.
Extraction

Dual Evidence Mining

Candidate chunksRecall-heavy extraction from sections likely to contain limitations, errors, ablations, and discussion gaps.
Full text passPaper-level extraction for high-level framing and less obvious failure evidence.
Structured fieldsQuote, paraphrase, mechanism, implication, mitigation, page, section hint, relevance score.
Smoke first10-paper smoke set checks extraction quality before the full run.
Merge

Evidence Deduplication

Raw overlapMerge same-context evidence across extraction strategies.
Semantic mergeLLM checks ambiguous candidate pairs with conservative merge rules.
Stable IDsEach merged row receives all_merge_xxxx for cross-view traceability.
Audit browseraudit_browser_all_merged.html exposes every merged evidence row and source item.
Taxonomy

Challenge Clustering

Provisional clustersBatch-level LLM clustering over evidence mechanisms.
Canonical clustersMerge provisional clusters into stable Cxx challenge categories.
Coverage repairAssign omitted evidence and normalize legacy repair IDs back to the Cxx namespace.
Cluster browserclusters_v1/cluster_browser.html supports challenge-level inspection.
Dynamics

Temporal and Community Analysis

Venue timelineUse event dates where available, year-only fallback otherwise.
Community flowTrack HCI/CHI, ML/data-science benchmarks, human-agent collaboration, and graph benchmarks.
Year dynamicsFind persistent, newly visible, increasing, and fading challenges.
CautionCounts are curated evidence counts, not field-wide bibliometrics.
Visual Space

Interactive Evidence Map

SankeyYear/Event to community to challenge flow, with presentation and detail modes.
Semantic mapEvidence embeddings projected with UMAP; KNN links show high-dimensional semantic neighbors.
HeatmapChallenge-by-year evidence dynamics and ranking panels.
Evidence chipsClick IDs to inspect quote, paraphrase, paper, and challenge in context.
Synthesis

Research Gap Report

Challenge evolutionGenerate bilingual synthesis over clusters, dynamics, and representative evidence.
Candidate gapsForm claims about persistent, emerging, migrating, and reframed failure mechanisms.
TraceabilityEvery high-level claim keeps challenge IDs and evidence IDs.
Human validationUser reads evidence, cross-checks overclaims, and builds an independent landscape understanding.

Human-Audit Thread

Before extraction
Confirm that the Stage 5 set is the right reading boundary. This analysis assumes the paper set is already worth deep reading.
After smoke
Inspect whether extracted evidence captures real failure analysis rather than generic limitations or unsupported summaries.
During review
Read all merged evidence rows from audit_browser_all_merged.html to build an independent human landscape.
After synthesis
Cross-validate report claims against evidence IDs, cluster definitions, and original paper context before using them in a proposal.

Traceability Chain

Evidence ID
all_merge_xxxx is the stable bridge between audit browsers, visual-space evidence cards, cluster assignments, and synthesis citations.
Challenge ID
Cxx names canonical challenge clusters, such as evaluation misestimation, state drift, verification breakdown, or strategy failure.
Claim support
High-level research-gap claims cite both Cxx clusters and all_merge_xxxx evidence rows so they can be audited.
Raw source
Each evidence row preserves quotes, paraphrases, page/section hints, raw context, and source extraction items.

Evidence Audit

data/analysis/error_patterns_20260723/audit_browser_all_merged.html is the best starting point for reading all merged evidence.

Cluster Audit

data/analysis/error_patterns_20260723/clusters_v1/cluster_browser.html groups evidence by canonical challenge clusters.

Visual Report

data/analysis/error_patterns_20260723/clusters_v1/visual_space/cluster_visual_space.html combines Sankey, semantic map, yearly dynamics, and bilingual synthesis.

Control Notes

docs/stage5/error_pattern_post_analysis.md records reproduction commands, audit rules, and interpretation cautions.