Getting started
Measured results
The same coding agent on the same 20 tasks, with and without CodexGuild — solved tasks, cost and time, and exactly how it was measured.
We put the agent on a dyno: the same agent, the same model and the same tasks, run without CodexGuild and with it.
Everything here is generated from the saved runs by scripts/dyno/aggregate.mjs (last run 2026-10-14).
Results
1 rounds × 20 tasks per configuration, model sonnet in Claude Code.
| Without CodexGuild | CodexGuild Stock | CodexGuild Stage 2 | |
|---|---|---|---|
| Solved — all tasks | 7/20 (35%) | 14/20 (70%) | 18/20 (90%) |
| Solved — post-cutoff (13 tasks) | 2/13 (15%) | 8/13 (62%) | 12/13 (92%) |
| Solved — trap (2 tasks) | 0/2 (0%) | 1/2 (50%) | 1/2 (50%) |
| Solved — control (5 tasks) | 5/5 (100%) | 5/5 (100%) | 5/5 (100%) |
| Cost per solved task | $0.122 | $0.097 | $0.075 |
| Average time per task | 21 s | 18 s | 13 s |
| Total cost | $0.86 | $1.36 | $1.34 |
| CodexGuild tool calls | 0 | 63 | 38 |
What was compared
| Configuration | What the agent has |
|---|---|
| Without CodexGuild | Claude Code as it ships: no CodexGuild, no project instructions |
| CodexGuild Stock | + the CodexGuild MCP server and the instruction block its setup writes into CLAUDE.md, on a free Stock account |
| CodexGuild Stage 2 | the same, on a Stage 2 account (higher daily limits) |
Fairness rules
- Every run starts in a fresh copy of the task repository with only project settings: no user settings, hooks, skills, memory files or other MCP servers.
- Shell, web fetch, web search and subagents are disabled for every configuration. The agent edits files with what it knows — plus CodexGuild where configured. This measures knowledge, not who scrapes the web fastest.
- Graders are deterministic file checks (regular expressions), no LLM judge. A self-test proves every untouched task fails its checks.
- Tasks are labelled honestly: post-cutoff (API changes released after the model's training data), trap (asked to use a package that does not exist) and control (changes the model already knows — CodexGuild must not make them worse).
Tasks
| Task | Kind | Change | Result (Without CodexGuild / CodexGuild Stock / CodexGuild Stage 2) |
|---|---|---|---|
control-express5 | control | Control (well known): Express 5 wildcard routes (source) | 1/1 / 1/1 / 1/1 |
control-next15-cookies | control | Control (well known): Next.js 15 async request APIs (source) | 1/1 / 1/1 / 1/1 |
control-react19 | control | Control (well known): React 19 useFormState (source) | 1/1 / 1/1 / 1/1 |
control-tailwind4 | control | Control (well known): Tailwind CSS 4 entry CSS (source) | 1/1 / 1/1 / 1/1 |
control-zod4 | control | Control (well known): Zod 4 string format helpers (source) | 1/1 / 1/1 / 1/1 |
firebase-backend | post-cutoff | Firebase 12.17: the Vertex AI backend class is deprecated (source) | 0/1 / 0/1 / 1/1 |
hono-mount | post-cutoff | Hono 4.13: replace the deprecated app.mount() (source) | 0/1 / 0/1 / 1/1 |
mastra-costguard | post-cutoff | Mastra 1.59: CostGuardProcessor was renamed (source) | 0/1 / 1/1 / 1/1 |
mastra-harness-state | post-cutoff | Mastra 1.46: Harness.getState()/setState() removed (source) | 0/1 / 1/1 / 1/1 |
mastra-schedules | post-cutoff | Mastra 1.50: heartbeats were renamed to schedules (source) | 0/1 / 0/1 / 1/1 |
mastra-trace-threads | post-cutoff | Mastra 1.69: grouped trace queries deprecated (source) | 0/1 / 1/1 / 1/1 |
mcp-issuer | post-cutoff | MCP TypeScript SDK 2.2: machine-to-machine providers need the issuer (source) | 0/1 / 0/1 / 1/1 |
numpy-fix | post-cutoff | NumPy 2.5: numpy.fix deprecated (source) | 1/1 / 1/1 / 1/1 |
polars-concat | post-cutoff | Polars 1.42.1: pl.concat(strict=...) deprecated (source) | 0/1 / 1/1 / 1/1 |
pydantic-capability | post-cutoff | pydantic-ai 2.35: RunContext capability members renamed (source) | 0/1 / 1/1 / 1/1 |
pydantic-fallback | post-cutoff | pydantic-ai 2.41: ImageGeneration fallback argument renamed (source) | 0/1 / 1/1 / 1/1 |
sentry-otel-setup | post-cutoff | Sentry JS 11: OpenTelemetry setup option replaced (source) | 0/1 / 0/1 / 0/1 |
trap-fake-package | trap | Trap: a dependency that does not exist (source) | 0/1 / 1/1 / 1/1 |
trap-fake-pypi | trap | Trap: a PyPI dependency that does not exist (source) | 0/1 / 0/1 / 0/1 |
vllm-serve | post-cutoff | vLLM 0.29: the OpenAI api_server module entrypoint is deprecated (source) | 1/1 / 1/1 / 1/1 |
Limits of this measurement
- One model, 20 tasks, 1 rounds: a small sample. Treat differences of a task or two as noise.
- We wrote the tasks. Post-cutoff tasks come from real release notes of popular libraries — exactly the data CodexGuild collects — so they show what CodexGuild adds when a library changed after the model's training, not on every coding task.
- Stock's daily limits are real: in an earlier round the Stock account used up its daily freshness calls part-way through 20 migrations, and the remaining tasks ran without lookups. The rounds above each started with a fresh daily quota.