Every number on this page comes from a real client (Codex, Claude Code, or Agy) answering
the same question twice against a pinned open-source repo: once with ordinary grep/read
exploration, once with one GitCortex MCP call. Token counts are the client's own reported
usage. Nothing here is simulated or hand-picked after the fact — the harness and raw logs
ship in tools/agent-bench/.
15 harder tasks — deep blast-radius, cross-file architecture, refactor-safety — across 5 pinned repos (ripgrep, requests, hono, cobra, gson), each client run twice per task (baseline vs. GitCortex), one round.
Beyond simple "where is X" search, the release suite adds three question types that mirror real pre-edit workflows.
| Task type | Question shape | GitCortex path |
|---|---|---|
| Impact | "Before I change this function, what's the full blast radius?" | find-callers --depth 2 / pre_edit_impact |
| Architecture | "How does this fit into the codebase — what's it connected to, both directions?" | get-subgraph --depth 2 --direction both |
| Refactor | "Is it safe to refactor this? What's the full surface area?" | symbol-context |
Before any paid client run, the same 35-task suite runs against the gcx CLI directly — no LLM in the loop — checking evidence contracts, ranking, and response budgets.
Real bugs, found and fixed before this suite's final numbers above:
symbol_context had no ambiguity handling and could silently resolve a common short name (e.g. beginArray, which matches 4 methods in gson) to the wrong one. It now goes through the same candidate-surfacing resolver as find_callers.ToolSearch) wasn't accounted for in the harness prompt, so the agent reported GitCortex as "unavailable" on one task. Fixed and re-verified.Remaining misses are scoring strictness, not product bugs: a few samples cite the correct symbol (e.g. ArrayTypeAdapter.read) without repeating the full file path our automated grader string-matches on. The underlying evidence was correct in every case we checked.