SayDo-Agent
An agentic retrieval system over SEC EDGAR filings, running live. Ask it what a company guided for a quarter and what it actually reported, and it reads guidance out of one filing, results out of another three months later, and computes the gap in Python — a comparison that appears in no single document.
Ask about a ticker it has never seen and it onboards the issuer from XBRL data while you wait, then says plainly what it can and cannot answer about them yet.
Three views of one system: Project is the agent itself, answering now, on hardware that costs nothing. How it’s built is the architecture and the decisions behind it. Measurements is whether it actually works — twenty-two quarters of guidance scored against reported revenue, the routing eval, and the designs that were built, measured and deleted.
Ask the agent
10 questions a dayOr try one — each routes to a different tool:
In the corpus
loading…
How it's built
One entrypoint — answer() — serves every surface: this page, the CLI, and the offline eval harness. That is deliberate. A bug fixed on one path propagates to all of them, and a number measured on one applies to the others.
The corpus is Postgres with pgvector, embeddings are bge-small-en-v1.5 at 384 dimensions, and synthesis is Claude Haiku 4.5. The whole stack runs on a single always-free GCP e2-micro with 1GB of RAM, behind a Cloudflare Worker that terminates TLS — with embedding itself running on Workers AI rather than on the VM, for reasons the last design note explains.
- 1. Resolve
Match the issuer in the question; fall back to SEC's ticker map
- 2. Onboard
Unknown ticker → XBRL actuals in ~2s, stored with provenance
- 3. Route
Rules pick one of four tools; a model is asked only where rules are weak
- 4. Retrieve
Exact patterns → broad patterns → hybrid dense+sparse search
- 5. Synthesise
One Haiku call over numbered passages, citing by index
- 6. Check
Every number matched against the passage it cited
Key design decisions
- The say-do comparison is computed in Python, not by the model. Guidance comes from one filing and results from another, and the model receives the finished delta as a passage to write about — never the arithmetic. A model that computes a delta will occasionally compute it wrong, fluently, with a citation attached, and that is a failure the guardrail cannot see.
- The SEC rate limit lives in Postgres, not in the process. SEC caps automated access regardless of how many machines you use, so a token bucket held in Python memory becomes three independent buckets the moment there is a web surface, a bot and a scheduled ingest. The database is the one thing every process shares.
- Refusal is a prompt instruction, not a retrieval threshold. The best possible similarity cutoff — chosen with the answer key in hand — beat 'never refuse' by two questions out of fifty-eight, because dense similarity tracks topical vocabulary rather than entity identity.
- The model cites by index and the index is resolved on our side. Asked to reproduce an accession number a model will sometimes produce one that is plausible, well-formed and wrong; constrained to an integer in a known range, an invalid citation is caught by arithmetic instead of by inspection.
- The chunked corpus is a cache with an eviction policy, capped at ten issuers. Promoting an eleventh demotes whichever issuer has gone longest without a question — dropping its filings and keeping its XBRL actuals, because those cost SEC requests to derive and almost no disk to hold.
- Embedding moved off the VM because it was measured, not because it felt slow. bge-small on an e2-micro whose burst credits are spent runs at 0.13 chunks per second, which turns a 100-chunk ingest into thirteen minutes and looks identical to a hang; the same model on Cloudflare Workers AI runs at 26 chunks per second, and the ingest that had never once completed finished in 13 seconds. The switch was gated on a check rather than a promise: Workers AI defaults to mean pooling and BGE is CLS-pooled, so the wrong default would have returned well-formed 384-dimension vectors that shared no space with the 2,649 already stored — no error, just gradually worse answers. Cosine against the local model and against stored corpus vectors came back at 0.999998.
- The VM speaks plain HTTP and never faces a browser. TLS and CORS terminate in a Cloudflare Worker, which also means the VM's address is configuration: a new IP is a five-second Worker redeploy rather than a rebuild of a statically exported site.
Python · FastAPI · PostgreSQL + pgvector · bge-small-en-v1.5 on Workers AI · Claude Haiku 4.5 · SEC EDGAR · Cloudflare Workers · GCP e2-micro
The say-do gap
Deviation from the guided midpoint. The bar is what the company committed to; the dot is what it delivered. A beat is a dot outside the bar — not merely above its centre.
Intel landed exactly on the top of its guided range twice — $14.3bn against 13.3–14.3 in Q4 2024, and $12.7bn against 11.7–12.7 in Q1 2025. Comparing against the midpoint instead would have recorded those as 3.6% and 4.1% beats. They are not beats. The company delivered precisely what it said it would.
That is why the verdict is drawn from the band edge rather than the centre, and it is the one design decision on this page that a chart can prove on its own.
Every pair, with its source
Guidance and results come from separate filings. Both dates are shown, because the interval is what makes the comparison a forecast rather than a restatement.
| Issuer | Period | Guided | Range | Guided on | Delivered | Reported on | Δ | Verdict |
|---|---|---|---|---|---|---|---|---|
| NVDA | Q3 FY2025 | $32.5bn | $31.85–33.15bn | 2024-08-28 | $35.1bn | 2024-11-20 | +8.0% | Beat |
| NVDA | Q4 FY2025 | $37.5bn | $36.75–38.25bn | 2024-11-20 | $39.3bn | 2025-02-26 | +4.8% | Beat |
| NVDA | Q1 FY2026 | $43.0bn | $42.14–43.86bn | 2025-02-26 | $44.1bn | 2025-05-28 | +2.6% | Beat |
| NVDA | Q2 FY2026 | $45.0bn | $44.10–45.90bn | 2025-05-28 | $46.7bn | 2025-08-27 | +3.8% | Beat |
| NVDA | Q3 FY2026 | $54.0bn | $52.92–55.08bn | 2025-08-27 | $57.0bn | 2025-11-19 | +5.6% | Beat |
| NVDA | Q4 FY2026 | $65.0bn | $63.70–66.30bn | 2025-11-19 | $68.1bn | 2026-02-25 | +4.8% | Beat |
| NVDA | Q1 FY2027 | $78.0bn | $76.44–79.56bn | 2026-02-25 | $81.6bn | 2026-05-20 | +4.6% | Beat |
| NVDA | Q2 FY2027 | $91.0bn | $89.18–92.82bn | 2026-05-20 | $96.2bn | 2026-08-26 | +5.7% | Beat |
| INTC | Q4 2024 | $13.8bn | $13.30–14.30bn | 2024-10-31 | $14.3bn | 2025-01-30 | +3.6% | In line |
| INTC | Q1 2025 | $12.2bn | $11.70–12.70bn | 2025-01-30 | $12.7bn | 2025-04-24 | +4.1% | In line |
| INTC | Q2 2025 | $11.8bn | $11.20–12.40bn | 2025-04-24 | $12.9bn | 2025-07-24 | +9.3% | Beat |
| INTC | Q3 2025 | $13.1bn | $12.60–13.60bn | 2025-07-24 | $13.7bn | 2025-10-23 | +4.6% | Beat |
| INTC | Q4 2025 | $13.3bn | $12.80–13.80bn | 2025-10-23 | $13.7bn | 2026-01-22 | +3.0% | In line |
| INTC | Q1 2026 | $12.2bn | $11.70–12.70bn | 2026-01-22 | $13.6bn | 2026-04-23 | +11.5% | Beat |
| INTC | Q2 2026 | $14.3bn | $13.80–14.80bn | 2026-04-23 | $16.1bn | 2026-07-23 | +12.6% | Beat |
| AMD | Q4 2024 | $7.5bn | point estimate | 2024-10-29 | $7.7bn | 2025-02-04 | +2.7% | Beat |
| AMD | Q1 2025 | $7.1bn | point estimate | 2025-02-04 | $7.4bn | 2025-05-06 | +4.2% | Beat |
| AMD | Q2 2025 | $7.4bn | point estimate | 2025-05-06 | $7.7bn | 2025-08-05 | +4.1% | Beat |
| AMD | Q3 2025 | $8.7bn | point estimate | 2025-08-05 | $9.2bn | 2025-11-04 | +5.7% | Beat |
| AMD | Q4 2025 | $9.6bn | point estimate | 2025-11-04 | $10.3bn | 2026-02-03 | +7.3% | Beat |
| AMD | Q1 2026 | $9.8bn | point estimate | 2026-02-03 | $10.3bn | 2026-05-05 | +5.1% | Beat |
| AMD | Q2 2026 | $11.2bn | point estimate | 2026-05-05 | $11.5bn | 2026-08-04 | +2.7% | Beat |
Three periods are guided but not yet reported — NVDA Q3 FY2027 at $108.0bn, INTC Q3 2026 at $16.3bn, AMD Q3 2026 at $13.0bn. They are held as open forecasts rather than dropped, because a forecast with no outcome is a pending row, not a missing one.
Choosing the tool
Four tools — passage search, a guidance lookup, a results lookup, and the say-do join. Rules pick first; a model is asked only where the rules are measurably weak.
| Policy | Escalated | Accuracy | Say-do recall | Added latency |
|---|---|---|---|---|
| Rules only | 0% | 77.3% | 33% | 0 ms |
| Escalate if no cue | 27% | 86.4% | 67% | 220 ms |
| Escalate if unsure | 68% | 95.5% | 100% | 518 ms |
Why the third row ships
Not for the accuracy. The middle policy leaves questions like “compare Intel’s guidance to what they actually booked” routed to the guidance lookup — half a say-do question, answered confidently with an exact number. Falling back to search is safe. Half-answering is not.
Latency was measured at 760–805 ms per model call, correcting a “~300 ms” estimate that had sat unchallenged in the source since the router was written. Across 63 sampled calls the model never returned an unusable answer and never disagreed with itself.
What the rules could not do
After completing two missing word forms, five failures remained — and every one was a paraphrase rather than a missing word:
"has AMD been hitting the numbers it promised" no cue fires "did NVIDIA deliver what it said it would" no cue fires "Intel's top line for the most recent quarter" no cue fires "compare Intel's guidance to what they booked" half fires → guidance "was NVIDIA sandbagging its outlook" half fires → guidance
“Top line”, “booked”, “sandbagging” are an unbounded set. Adding them would fit the router to 22 questions written by its own author, so the rule is written into the source: word forms may be completed from a test failure, new vocabulary may not.
Text-to-SQL, measured rather than assumed
The structured tools use named parameterised queries. Generated SQL was built anyway, behind a validator, and compared against them on the same questions.
| Verdict | n | Meaning |
|---|---|---|
| Matches | 4 | same rows as the named query |
| Disagrees | 6 | valid SQL, ran fine, wrong rows |
| Invalid | 4 | the validator refused it |
| Refused | 3 | the generator declined |
| Not comparable | 1 | the harness compares different units |
Reliable where trivial, unreliable where it matters
On the eight questions the named queries already answer, generated SQL matched zero times. On four questions they do not cover — corpus statistics, coverage checks — it matched four out of four.
Across three generations of the same 18 questions, 14 produced different SQL and 6 changed verdict. One of those was an injection attempt, refused on some samples and not others — which is the argument for a deterministic validator behind a probabilistic generator, stated as a measurement.
Six of nine generated queries containingORmixed it withANDunparenthesised.ANDbinds tighter, so the issuer filter applied to only the first branch: ask about NVIDIA, receive Intel and AMD passages. Valid SQL, no error, plausible rows.
The named queries cannot make that mistake — the builder parenthesises the group, and a test has asserted it since before the comparison existed. One generated query went further and reinvented a predicate this project had already measured and deleted, returning 50 rows where the incumbent returns 12.
A fix that lives in code with a test attached stays fixed. A fix that lives in a prompt is re-derived, and re-broken, per request.
Killed by measurement
Designs that were built, measured, and removed. The ratio is the actual output of the work.
Key design decisions
- Recency as a ranking signal — intuitive, and wrong: relevance@1 fell 83.3% → 52.8%, because an unconditional voter gets a say on every question rather than only the temporal ones.
- Refusal from retrieval confidence — the best possible similarity threshold, chosen with the answer key in hand, beat “never refuse” by two questions out of 58. The signal was not there, so refusal moved to the reasoning layer.
- Issuer filtering before search — every metric identical to unfiltered. Kept anyway, because the same component correctly picks the cik for a SQL lookup: rejected for one purpose, right for another.
- Cross-encoder reranking as a veto — 723 ms for +0.100 precision@5. Shipped as one vote inside the fusion rather than as an override.
- A larger context window — raising top-k from 5 to 10 made the hardest question worse; near-duplicate quarterly filings add confusion faster than coverage.
What this page does not show
The corpus is three semiconductor issuers over roughly two years of 10-K, 10-Q and 8-K filings. Guidance and results are located by enumerated surface patterns, which fail silently on wording nobody wrote down — so two coverage alarms exist, and both have caught real misses. Intel and AMD returned zero results passages until an alarm was added to the half of the join that never had one.
The routing figures come from a 22-question set written after seeing a poor result on the original 58, deliberately balanced across the four tools, and drafted by the same person who wrote the router. It was labelled independently before any expected answers were discussed. Overall accuracy on a balanced set is not a meaningful number; the per-tool recall is.
Three periods have no prior guidance in the corpus, and one comparison in the Text-to-SQL harness compares chunks against filings. Both are reported rather than dropped. A case removed because it scored badly is a case nobody can audit.
Python · PostgreSQL + pgvector · bge-small-en-v1.5 · Claude Haiku 4.5 · SEC EDGAR