SWE-bench Lite ↗
Python
- Input
- 40 tasks
- Scored
- 39 tasks
17 pytest and 23 scikit-learn tasks before exclusions.
1 excluded: One task with no scorable functions or types.
TokensToGold
How much of the code that matters does one search find before the token budget runs out?
Same tasks. Fixed budgets. Exact answer key. No model or agent.
- Scored tasks
- 76
- Run shape
- 1 query → 1 response
- Sources
- 2 public corpora
- Model / agent
- None
One query in. One response out.
This is a retrieval test, not an agent run. The dataset provides the task and known solution; the scorer checks the answer key without an LLM judge.
- 01
Task statement
Use the dataset's problem statement verbatim as the retrieval query.
- 02
One query
Run each retrieval system once. There is no model, agent loop, or follow-up turn.
- 03
Fixed budgets
Measure the single response at fixed tool-response token checkpoints.
- 04
Exact match
Count which functions and types in the answer key appear in the response.
- Answer-key coverage
- The share of answer-key functions and types that appear in the response.
- Tool-response tokens
- Tokens in the raw retrieval response, counted with the o200k_base tokenizer.
- Tasks reaching 80%
- The share of tasks whose response contains at least 80% of the answer key.
Noodlbox 2.1.0 · measured August 2026
Response tokens cover the retrieval output only. They do not include model input, reasoning, or an agent session—because this run did not use one. Retrieval latency was not measured.
Two corpora. Two separate results.
Each task includes a problem statement and a known solution. The statement becomes the query. The functions and types changed by the solution become the answer key.
DeepSWE ↗
TypeScript / JavaScript
- Input
- 40 tasks
- Scored
- 37 tasks
35 TypeScript and 5 JavaScript tasks before exclusions.
3 excluded: Two answer-key extraction errors and one task with no scorable functions or types.
What one query finds — and what the hunt costs.
The gauges show the share of each task's answer key retrieved into the same 8K window; the pile is the median uncapped ripgrep (rg) exploration, in 8K windows' worth. Each corpus stays separate.
Python
39 tasks · SWE-bench Lite
- Noodlbox, within its 16K cap
- fills 80% of the answer key on 90% of tasks
- Ripgrep (rg), to the same 80%
- ≈26K tokens median — and gets there on 87% of tasks
- Ripgrep (rg), full hunt
- 447K median · 776K mean · worst task 3.2M
TypeScript / JavaScript
37 tasks · DeepSWE
- Noodlbox, within its 16K cap
- fills 80% of the answer key on 78% of tasks
- Ripgrep (rg), to the same 80%
- ≈65K tokens median — and gets there on 65% of tasks
- Ripgrep (rg), full hunt
- 123K median · 225K mean · worst task 1.7M
What this benchmark cannot claim.
Retrieval is one measured layer of an agent run. These results do not stand in for model quality, edit quality, or total cost.
- Retrieval coverage is not task success. No model edited code or ran tests in this benchmark.
- The known solution is one valid implementation. Another correct fix may touch different code.
- TypeScript / JavaScript comes from DeepSWE. Python comes from a 40-task SWE-bench Lite slice; the results are not combined.
- The Python slice is narrow: 26 of 39 scored tasks have one function or type in the answer key, making many task scores all-or-nothing.
- Tool-response tokens are not billed model tokens or total session cost.
Each corpus stays separate.
DeepSWE and SWE-bench Lite use different task sets. A blended score would hide that difference.
Python
39 scored tasks
| Retrieval method | Found at 8K | Found by 32K | Fills 80% of tasks |
|---|---|---|---|
| NoodlboxThe shipped context compiler, capped at 16K tool-response tokens. | 83.3% | 91.0% | 89.7% |
| Noodlbox without curationThe same retrieval stack with ranking and selection disabled. | 38.5% | 38.5% | 35.9% |
| Ripgrep + targeted readsA deterministic graph-free explorer with no output cap. | 14.1% | 50.4% | 48.7% |
TypeScript / JavaScript
37 scored tasks
| Retrieval method | Found at 8K | Found by 32K | Fills 80% of tasks |
|---|---|---|---|
| NoodlboxThe shipped context compiler, capped at 16K tool-response tokens. | 78.9% | 88.9% | 78.4% |
| Noodlbox without curationThe same retrieval stack with ranking and selection disabled. | 44.4% | 44.4% | 21.6% |
| Ripgrep + targeted readsA deterministic graph-free explorer with no output cap. | 18.6% | 31.6% | 21.6% |