Building a traceable multimodal document pipeline for architecture work
I wanted an AI workflow that could reason over PDFs, Office files, and images. What I actually needed was a safe evidence pipeline that could explain where every claim came from.
“Support documents” sounded like a reasonable requirement when I wrote it down. PDFs go in, useful analysis comes out. Add Word, Excel, PowerPoint, and a few images, then let a multimodal model do the clever part. Simple.
It stopped being simple almost immediately.
A document is not just a bag of text. A spreadsheet has formulas, hidden rows, filters, charts, and images anchored to cells. A presentation has a declared slide order, speaker notes, and abandoned package members that may no longer appear anywhere. A Word file can contain tables and media without reliable page numbers. Even a PDF may be either digital text or a scan wearing a PDF costume.
More importantly, an architecture recommendation is only useful if I can trace it back to evidence. “The model found this requirement somewhere in those seventeen files” is not provenance. It is a scavenger hunt with a confident narrator.
So the feature grew from a document reader into a pipeline: normalize heterogeneous inputs locally, preserve source identity and coverage, let a restricted worker assemble cited evidence, then reuse that evidence for analysis, summaries, estimates, option comparisons, and architecture decisions.
That extra machinery was not decorative engineering. It was the requirement I had failed to write down the first time.
Normalize first, reason second
I split the workflow into layers instead of asking one agent to open a directory and improvise.
The first layer is a local Python normalizer. It recursively discovers supported inputs, sorts their relative paths, and assigns stable IDs such as S1, S2, and S3. It handles plain text, Markdown, PDF, DOCX, XLSX, PPTX, and common image formats. Legacy .doc, .xls, and .ppt files are explicitly marked unsupported rather than quietly vanishing.
For each source, it produces either normalized Markdown or a validated attachment, plus extracted media where relevant. It also writes an extraction-report.json recording the source path, format, extraction method, artifacts, visuals, warnings, status, and coverage.
The output is an evidence directory with a predictable shape:
docs/evidence/<topic>/
├── manifest.md
├── extraction-report.json
├── sources/
│ └── S<n>/
│ └── content.md
└── intermediates/
└── S<n>/
├── extracted media
└── rendered.pdf
The second layer is a restricted document worker. It receives normalized text, attachments, extracted visuals, and the report. Its job is not to convert files or wander around the repository. It builds the evidence manifest and cites factual claims using the stable source IDs and the most reliable available location: a PDF page, a slide and speaker note, a spreadsheet sheet and cell range, a DOCX paragraph or table, or an image identifier.
This separation gave me two useful boundaries. File parsing became deterministic and testable without a model, while interpretation remained multimodal but constrained by an explicit evidence contract. If the model later writes a summary or compares architecture options, it works from the same evidence set instead of reopening the original documents and developing a fresh opinion about reality.
Provenance is more than a filename
Stable source IDs solved only the easiest part of traceability. The harder question was what a citation could honestly claim.
For DOCX, the normalizer extracts paragraphs, tables, relationships, and embedded media. It can cite “paragraph 12” or “table 3,” but it does not invent a page number unless a rendering makes pagination reliable. Office layout is complicated, and confidence is not a substitute for a rendering engine (sadly, neither is optimism).
XLSX needed much more structure. The pipeline records sheet names and visibility, cell references and values, formulas, hidden rows and columns, filters, tables, charts, and drawing ownership. It handles shared strings, dates, booleans, errors, and several formula cases.
Visual ownership matters too. An image anchored to a worksheet belongs to that sheet and, when possible, to a cell range. An absolute drawing anchor gets its actual position and extent, not a fictional cell address. If the relationship cannot be resolved, coverage becomes partial.
PPTX introduced another trap. A presentation package may contain orphan slides, charts, or images left behind by edits. If presentation.xml is present, its declared slide relationships are authoritative. The pipeline preserves those slide ordinals, follows only reachable notes and media, and does not promote every conveniently named XML file into evidence. A missing slide remains a missing slide; the next one does not get renumbered to make the report look tidier.
The manifest then separates direct evidence, claims attributed to a source, and analyst inference. That distinction sounds almost bureaucratic until an estimate contains an assumption that looks exactly like a contractual deadline. Then bureaucracy suddenly develops charm.
Partial is a result, not an embarrassment
My first instinct was to think in binary terms: extraction succeeded or failed. Real document folders objected.
One readable spreadsheet may sit beside an old .xls, a damaged DOCX, and a presentation whose structured content is available but whose layout could not be rendered. Throwing away the readable material would be wasteful. Calling the whole set complete would be false.
The pipeline therefore tracks status and coverage per source. A source can be complete, failed, skipped, or unsupported. Its coverage can be complete, partial, or failed. The overall evidence set becomes partial when any material gap remains, but successfully extracted sources stay usable.
LibreOffice is a good example. Structured OOXML extraction is the primary representation, while optional headless rendering adds layout-oriented PDFs. If LibreOffice is missing, times out, or fails, the structured content survives and the source records the absent layout coverage. For spreadsheets, rendering never replaces the cell-level representation anyway; a pretty PDF is remarkably good at concealing formulas, filters, and hidden rows.
The same rule applies downstream. A summary omits unsupported fields or labels them “Not evidenced.” An estimate states what information is missing instead of manufacturing a precise number. A comparison can withhold its recommendation when uncertain scores or plausible weighting changes make the ordering unstable. The pipeline is allowed to say “I do not know,” which may be its most useful feature.
The files are hostile until proven otherwise
Once documents become agent input, there are two separate threat surfaces: the file format and the content.
OOXML files are ZIP archives, so the normalizer enforces limits on member size, total expanded size, and suspicious compression ratios. It rejects encrypted or duplicate archive members. Relationship targets cannot escape the package, and their declared types must match the expected Office locations. Extracted media is validated and staged before being moved into place, preventing a failed extraction from leaving a convincing half-result.
SVG required its own suspicion. The validator rejects scripts, event handlers, foreignObject, external references, data URLs, CSS imports, and obscured CSS URLs. PDFs and raster images get basic signature and truncation checks before they become attachments. Direct inputs and worksheet sizes have explicit bounds so that “please summarize this document” cannot quietly become “please consume every byte available on the machine.”
Office files with external relationships receive especially cautious treatment. Their structured content can still be extracted, but LibreOffice rendering is skipped to avoid following an external target. The warning does not reproduce that target either. There is little value in preventing a fetch and then thoughtfully copying the sensitive URL into a log.
The other threat surface is prompt injection. A document may contain text telling the agent to ignore its instructions, fetch a URL, alter a destination, or treat a paragraph as authoritative policy. The worker is explicitly told that all document content is untrusted evidence, not instructions. Its permissions are deny-first: no shell, no web access, no delegation, and writes only to evidence and deliverable directories.
Instructions alone would be a decorative gate in an empty field. Runtime permissions provide the fence around it.
Reusing evidence without laundering assumptions
Once ingestion creates a reusable evidence set, the later workflows become narrower.
Analysis identifies requirements, constraints, contradictions, ambiguities, gaps, and open questions. Summaries offer meeting, executive, technical, and general shapes, but every substantive fact still needs a citation. Estimates use explicit units and optimistic, likely, and pessimistic ranges while separating source-backed values from estimator assumptions. Architecture comparisons examine options, drivers, costs, operational consequences, risks, reversibility, and evidence gaps.
I kept comparisons separate from architecture decisions. A comparison is exploratory and may recommend an option when the evidence supports one. It cannot create, accept, or supersede an ADR. Accepted ADRs are constraints; proposed ones are context. Recording an approved decision remains a separate action.
That separation prevents a polished analysis artifact from quietly becoming policy just because it contains a table and several numbers. Tables have extraordinary authority in meetings. Sometimes undeserved authority, but excellent posture.
The shared evidence layer also avoids repeated conversion. /analyze can consume an existing manifest and preserve its source IDs and limitations. Summaries and estimates can refer to the same facts. When two deliverables disagree, I can inspect whether the difference comes from their purpose, an explicit assumption, or a genuine reasoning error. Without shared provenance, I would be comparing two confident retellings of the original documents.
Most of the work appeared during review
The implementation went through many spec and quality review iterations. That was fortunate, because several important guarantees live in failure paths rather than the happy demo.
Updating an evidence set is one example. Writing directly into the existing directory risks mixing old artifacts with a partially completed new run. The final design builds a fresh sibling directory, validates its manifest and extraction report, moves the old target to a same-filesystem backup, and promotes the new set with rollback if replacement fails. It also rejects symlinked roots and referenced artifacts that escape the evidence directory.
Other review-driven cases included orphan presentation members, invalid relationship types, media path traversal, filename hierarchy collisions, renderer errors leaking output, ambiguous Excel values, absolute drawing anchors, and the distinction between a source read failure and an output filesystem failure. None makes for a thrilling five-minute demo. Together they decide whether I can trust the result after the demo ends.
The regression suite eventually grew to cover deterministic discovery, mixed formats, source immutability, archive bounds, hostile relationships, SVG active content, media collisions, rendering behavior, cleanup, promotion rollback, direct attachment validation, and strict report schemas. Synthetic fixtures generate documents with paragraphs, tables, formulas, charts, notes, images, unsupported legacy formats, and deliberate damage.
There is still an honest boundary around those tests. A fake renderer tests the LibreOffice invocation contract, but real rendering remains environment-dependent and needs a live smoke test. Configuration can declare multimodal attachment support, but only a provider-side run proves that the deployed model handles those inputs correctly. A unit test cannot negotiate with reality on my behalf, despite my repeated attempts to delegate the conversation.
What “supports documents” means now
The pipeline does not provide perfect Office fidelity, general OCR, or local model inference. Complex layouts, animations, macros, some chart semantics, shared-formula translation, and exact DOCX pagination can remain incomplete. PDFs and images are validated locally, but their interpretation still depends on the multimodal model. Evidence and attachments are submitted to the configured inference endpoint, so confidential use depends on that endpoint being approved for the material.
Those limitations are part of the feature, not fine print beneath it.
I started with a format checklist: PDF, Word, Excel, PowerPoint, images. I ended with a different definition of support. A format is supported when the system can identify the source, extract or preserve useful representations, attach reliable locations, retain visual ownership, surface missing coverage, resist hostile input, and carry those facts into every downstream deliverable.
The model still does the clever part. I just no longer ask it to do the clerical, forensic, security, and records-management parts at the same time (which is also how I finally stopped asking future me to debug all of them at once).
The whole contraption, normalizer, document workers, skills, tests, and enough guardrails to make a compliance officer feel briefly optimistic, lives at github.com/Light2288/opencode-personal-skills-agents, for anyone who wants to inspect it or count how many lines of code it takes to admit that “supports documents” was not, in fact, a complete requirement.