SIX TRACES
Six agents said 400. Which one can you check?
Every vendor you've heard from this year has an agent that'll answer questions about your data. I asked six of them the same one: how many home runs have I seen in person?
All six said 400. All six were right.
Then I opened every trace and scored it on five things: does it show its plan, its query, its source, its tokens and time, and can you even get the trace out? No agent went five for five. Each one hides something different, and whatever it hides is what you'll be QA'ing blind.
The game. Saturday, July 30, 2022. Old-Timers' Day, the first since 2019. Royals at Yankees, 8–2. The crew back together: Al, Bergan, Matty, on the Yankee train in. I kept score, and helped the son of one of Matty's friends keep his. Mo was on the field before first pitch. The Claude agent's one action in this piece was a note to Al about this day. It drafted it and stopped. No recipient. I'm the send button.
#Six traces, one rubric
Measured September 3 to 5, 2026, one question, one run each, from the trace each platform actually produces. Not a benchmark. A vendor may have closed its gap since.
| Agent | Shows its plan? | Shows the query? | Names its grounding? | Tokens & time? | How you get the trace |
|---|---|---|---|---|---|
| Agentforce | A 21-step sequence, no stated reason | The action and its input, not the SQL underneath | Absent. Provable only from the Apex source | Latency per step. Zero token fields in 85 KB | The CLI writes a local file. Copy it or lose it |
| Claude SDK | Tool sequence, thinking billed but not surfaced | The metric name; there is no SQL by design | In the tool's own text: the governed catalog | Both, fully. $0.18, 5.7 s of model in 38.7 s | Straight from the SDK's own message stream |
| BigQuery | Plain sentences, the plainest of the six | SQL verbatim, twice | The table and the BigQuery job id | 9.5 s; 0 bytes billed, a cache hit | The official Python client, streamed |
| Databricks Genie | Four typed thoughts; states its own assumption | MEASURE(home_runs_witnessed) by name | Named and self-qualified | 17.9 s; model called twice per turn | The Python SDK, polled. No stream |
| Snowflake Cortex | A status ladder; thinks aloud when it fails | SQL, and the first attempt failed in the open | The semantic view, then re-derived by hand | Richest platform log: 12 rows, per-step tokens | A stream, plus an event table you can query with SQL. No setup |
| Tableau Agent | No reasoning text | The structured semantic query, verbatim | Named: the semantic model, on every answer | 9.1 s; zero token fields | The hosted MCP server, over HTTP. One browser login |
Tableau Agent's confident zero was diagnosable only because it shows the query. Baseball Scout's 9 was diagnosable only because Agentforce writes tool_invocations: null to a file. Genie was the one agent that volunteered a caveat. Cortex was the one that walked past the definition it was handed and rebuilt it. BigQuery narrated in English and gave a job id you can look up.
#Four calls, nine calls, and where the time went
The Claude agent has no SQL tool. Its whole world is list the metrics, query a metric by name, draft an email. Asked about home runs and the Yankees' win rate, it listed the metrics, asked for two by name, done: 4 tool calls. A twin with a raw read-only SQL tool instead spelunked the schema, rediscovered that a home run is event code 23, found the team-grain table, and got the same right answers in 9. Good modeling narrows the gap, because the marts are clean. The governed path still answered in under half the moves with zero rediscovery, and the metric name is right there in the trace. A smaller model got 400 through the same path too, after fumbling seven tool-loading calls first. Correctness survived the small model.
Then the clock. That one turn cost $0.18 and took 38.7 seconds. The model's share was 5.7 seconds. 21.7 seconds was the local metric engine starting up. The bottleneck was not the model, and if you'd only had the chat window you'd have blamed it.
One more from the same family, from Baseball Scout: an action returned 22 venue groups and the prose said “21 different stadiums.” Grounded, and miscounted. The tool did its job; the sentence didn't. That's a column the rubric doesn't have yet.
#From the shoebox
#Try this
Ask your agent one question you already know the answer to. Then find its trace and fill in the five columns. Any column you can't fill is what you'd be QA'ing blind.
#What it taught me
- Right answers are the floor. Legible answers are the job. Six agents said 400. The one you can defend in a meeting is the one whose trace shows the query and names the model it ran against. Ask that of any agent you're handed before you ask whether it's accurate.
- The tool set is the policy. An agent with no SQL tool cannot go ungoverned. This is Baseball Scout's “gate structurally” applied to reads.
- Measure the whole turn, not the model. 33 of 39 seconds wasn't inference. In the age of the agent, you're going to be asked why it's slow, and the answer is going to be in the trace, not the prompt.
Next, batting 6th: Where does the fan-out hide? →