TokensToGold

How much relevant code can a search tool find within a token budget?

Search results are text an agent has to read. We measure how much of a reference fix each tool surfaces before that budget runs out.

Scored tasks
146
Run shape
Single query and response
Sources
3 public corpora, 4 language groups
Model / agent
None

Coverage by token budget.

0%50%100%8K32K100K1MMedian response · 447K · 56×14%50% by 32K83%91% at cap2.3.18 harness-scored0%50%100%8K32K100K1M447K median14%50% by 32K83%91% at cap2.3.18 harness-scored
noodlboxripgrep with targeted reads
Share of the answer key found · response tokens, log scale. Dashed lines connect measured checkpoints.
Coverage in the first 8K response tokensSelecting a corpus draws it in the chart above.
Corpusnoodlboxripgrep with targeted reads
83%14%
79%19%
69%8%
53%18%
2.3.18 harness-scored

One task, up close.

Unroll the iterable for all/any calls to get better reports

pytest-dev/pytest · SWE-bench Lite · Reference fix

Instruction

Sometime I need to assert some predicate on all of an iterable, and for that the builtin functions all/any are great - but the failure messages aren't useful at all! For example - the same test written in three ways:

  • A generator expression
    def test_all_even():
        even_stevens = list(range(1,100,2))
>       assert all(is_even(number) for number in even_stevens)
E       assert False
E        +  where False = all(<generator object test_all_even.<locals>.<genexpr> at 0x101f82ed0>)
  • A list comprehension
    def test_all_even():
        even_stevens = list(range(1,100,2))
>       assert all([is_even(number) for number in even_stevens])
E       assert False
E        +  where False = all([False, False, False, False, False, False, ...])
  • A for loop
    def test_all_even():
        even_stevens = list(range(1,100,2))
        for number in even_stevens:
>           assert is_even(number)
E           assert False
E            +  where False = is_even(1)

test_all_any.py:7: AssertionError

The only one that gives a meaningful report is the for loop - but it's way more wordy, and all asserts don't translate to a for loop nicely (I'll have to write a break or a helper function - yuck) I propose the assertion re-writer "unrolls" the iterator to the third form, and then uses the already existing reports.

  • Include a detailed description of the bug or suggestion
  • pip list of the virtual environment you are using
Package        Version
-------------- -------
atomicwrites   1.3.0  
attrs          19.1.0 
more-itertools 7.0.0  
pip            19.0.3 
pluggy         0.9.0  
py             1.8.0  
pytest         4.4.0  
setuptools     40.8.0 
six            1.12.0 
  • pytest and operating system versions platform darwin -- Python 3.7.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
  • Minimal example if possible

pytest-dev__pytest-5103 · SWE-bench Lite problem statement, verbatim

Reference-fix functions found within 8,000 returned tokens
Search toolFunctions found
noodlbox2 of 2
ripgrep with targeted reads0 of 2
2.3.18 harness-scored

What we measure.

Tasks
Merged fixes from public repositories. The problem statement is the query.
Answer key
The functions and types the fix touched. A reproducible proxy, not the only valid fix.
Budget
Tool-response tokens on the wire, measured at 8K and 32K. noodlbox is capped at 16K. Not a provider bill.
Scope
One query, not a session. Retrieval, not task success. No model, no judge.

The full results.

Python39 tasks · SWE-bench Lite
Measurementnoodlboxripgrep with targeted reads
Coverage at 8K83.3%14.1%
Coverage by 32K91.0%50.4%
Output cap16KNo cap
Median tokens returned447K
Mean tokens returned776K
Most tokens returned3.2M
2.3.18 harness-scored · — not reported

Coverage is the share of each task's answer key delivered within the budget. Tokens are tool-response tokens, wire-priced.

On 2026-09-07 the ripgrep with targeted reads basis moved from the in-engine rollup to the 2.3.18 harness rollup of the same engine numbers (harness commit 7bd405ff9ffa269202ce01fa69086d97c9ff465c, in release v1-2.3.18). The figures did not change — the basis did.

Reports, frozen gold and checksums verify offline from the release assets: github.com/noodlbox/tokens-to-gold

benchmark V1 · harness release v1-2.3.18 · engine noodlbox 2.3.18