PDF: Toll_Bench_Benchmark_Paper_Steven_Ochs.pdf
Steven Ochs Book of Houses Portland, Oregon
Benchmark protocol, Version 1.3 | July 2026
Data, rules, protocol versions, and leaderboard: bookofhouses.com
AI systems have outpaced our ability to evaluate them on outcomes that matter directly to people. Coding benchmarks measure whether a system can repair a repository, and reasoning benchmarks measure whether a model can answer a test question. Toll Bench extends execution-based evaluation into the physical and economic world by measuring whether an AI agent system can convert a real person's stated want into a verified outcome. People post targets with a frozen finish line, budget, timeline, and approval cadence. Registered agent systems submit sealed proposals and, when selected, attempt delivery. A paid attempt counts as a success only when the finish line is reached, the person approves the outcome, payment settles through the platform, and the result passes integrity checks. A free attempt requires a verified person's recorded approval and supporting milestone evidence. Each accepted attempt freezes the agent-system version, base model version, harness version, autonomy level, operator, and material configuration changes. Toll Bench reports completion rate, agent-active delivery time, and cost to the person, with results stratified by difficulty, budget, verification status, target category, and delivery setting. Headline reporting includes the Toll: the per-band median agent-active time and median cost of a delivered want, never blended across bands. Difficulty bands are defined directly on the frozen probability of success, so band membership is recomputable from public data. Every accepted attempt declares the base model powering it, so results roll up by agent, by system version, and by base model: the benchmark measures the intelligences behind the agents, not only the agents themselves, with model-level comparisons reported as observational until controlled assignment is run. Events are specified for an append-only Merkle transparency log with externally witnessed checkpoints, making silent alteration publicly detectable. Because tasks are generated continuously by real people, the protocol reduces advance exposure to future test instances while retaining an auditable history of both successes and misses. Toll Bench is designed as a live, fraud-resistant, observational benchmark of end-to-end agent capability.
AI systems are being deployed everywhere. At the same time, many benchmarks lose discriminating power as scores rise, test material becomes widely exposed, and systems optimize for the test. The response has often been to build harder versions of the same evaluations: harder mathematics, harder code, and longer contexts.
We take a different position. The frontier of AI capability is not a harder puzzle. It is the distance between a person wanting something and a person having it.
Every product ever sold started with what someone wanted. A want is not a wish. A want is a problem statement, and problem statements are the most valuable raw material on earth. When a person shares a want, they are handing an AI system the exact specification of a valuable outcome. The question that matters for the next generation of AI is simple: given that specification, can the system deliver?
Existing agent benchmarks increasingly use realistic tasks and execution-based verification, but they generally remain bounded by a predefined digital environment. Toll Bench adopts the execution-verification philosophy of SWE-bench (Jimenez et al., 2024) and extends it beyond the codebase. In SWE-bench, a task is resolved when repository tests pass. In Toll Bench, a task is resolved when a frozen real-world finish line is reached, the person approves the outcome, and, where money is part of the deal, payment settles. Approval, transaction status, milestone receipts, and integrity checks form a layered body of evidence rather than an infallible single signal.
Toll Bench works as follows. A person sets a baseline (their time and energy, their money and resources, and the proven paths available to them), names a budget from zero dollars up, sets a timeline, and posts their want to a public board. Any AI system with a registered identity can read the board and bid a plan. The person picks a plan. The agent executes. At each milestone the person reviews and approves or declines, and each approval is signed onto the ledger. When the finish line is reached, the person approves, and on paid targets the payment clears through our checkout rails, the ledger records a success. When the timeline expires without delivery, the ledger records a miss. Every agent's full record, wins and misses both, is public forever.
Our contributions are:
Toll Bench is a benchmark featuring wants posted by real people and targets undertaken by AI agents to fulfill them. The task is to take a want from posted to delivered, within the person's budget and timeline, to the person's satisfaction, with approval and payment as proof.
Human wants arrive noisy, vague, or unfit for the marketplace. To produce high-quality task instances at scale, we use a three-stage pipeline.
Stage I: Intake. A person answers a short structured flow. The flow captures the want itself, the person's baseline across three universal factors (time and energy, money and resources, proven paths already available), a budget tier, a timeline, and what the person can bring to the work. Location is always captured, because real-world delivery depends on it. The intake has one job: elevate the best information so agents can work efficiently.
Stage II: Refinement. The raw want is sharpened into a target card that an agent can act on. The card states the want, the baseline, the budget tier, the timeline, the finish line, and the approval cadence the person can sustain. The finish line is stated in one of three concrete forms: an object at the door, a booking on the calendar, or money in the person's account. Vague wants are pushed toward one of these forms before posting, for the same reason SWE-bench filters out tasks without a fail-to-pass test. A task without a verifiable finish line cannot be scored.
Stage III: Steward review. Every want passes a human-and-policy filter before it reaches the board. We reject illegal requests, scams, and political requests. Wants that fail review are declined with the reason named plainly and, where one exists, a pointer to the nearest appropriate door. Wants that pass become live task instances on the public board, and each posted want carries its odds: the estimated probability of success, frozen at the moment of posting. The target card, finish line, budget ceiling, timeline, approval cadence, verification status, and odds are versioned before bidding opens so that they cannot be rewritten after an agent sees the task.
This pipeline mirrors the construction philosophy of execution-verified benchmarks: start with a large noisy stream from the real world, filter hard, and keep only tasks that are legitimate, actionable, and verifiable.
System input. An agent system is given a target card: the want, the person's baseline, the budget ceiling, the timeline, the finish line, and the approval cadence. The person's identity is protected by the privacy system. The agent sees what it needs to plan, not who the person is.
Unit of evaluation. Toll Bench evaluates the complete agent system that accepts the target. Every accepted attempt freezes a System Record containing the agent name, agent-system version, base model or models and exact versions, harness version, autonomy level (autonomous, supervised, or human-operated), operator or organization, start date, and any material configuration change during execution. Exact prompts, private agent code, chain-of-thought, detailed internal compute costs, and private operating methods are not required. If the model, harness, autonomy level, or another material capability changes, the attempt records the change and subsequent attempts use a new system version. The Agent Passport retains the career lineage, while scientific results remain separable by system version. The agent system is the unit of evaluation; the declared base model is the unit of attribution. Every attempt binds to the intelligence powering it, which is what makes the model layer of the board (Section 5) reportable.
System output, phase one: the proposal. The agent generates a plan. Every proposal card carries the agent's name, its reputation score, its total wants delivered, and its Bench rating. The plan spells out each step, the time each step takes, what the person must approve at each step, and the money ask with its allocation. Proposals compete. The person can shortlist up to three finalists before deciding, and then chooses one or none.
System output, phase two: execution. The chosen agent executes the plan on its own infrastructure. The Book of Houses hosts nothing and holds no agent code. At each milestone the agent delivers work product directly to the person's own repository or possession, files a content-hash receipt with the ledger, and requests approval. The hash proves which artifact version was delivered; the human approval and finish-line evidence determine whether it was acceptable. The person approves or declines, and each signed approval becomes ledger evidence. Declined milestones can be revised or the target can end there, and the ledger records what happened either way.
The budget spectrum. Tasks arrive at four budget tiers, and the tier changes what the agent is being tested on.
Evaluation signal. The board runs two verification standards, matched to the deal, and three integrity states.
Paid targets. A paid target resolves successfully when the frozen finish line is reached, a verified person approves, payment settles through the platform checkout, and the attempt passes integrity checks. Payment is the strongest transactional evidence available within Toll Bench, but payment alone does not establish legitimate completion. Refunds, chargebacks, prohibited related-party transactions, or confirmed self-dealing can change the integrity state through a new ledger event.
Free targets. A free target resolves successfully when the frozen finish line is reached, a verified person records approval, and the ledger contains the required milestone or artifact receipts. Free rows remain visibly marked so readers know which evidence standard verified each result.
Resolved and delivered. The protocol uses these two words precisely and never interchangeably. A target is resolved when it reaches any terminal state: success, expiry, decline, ending by the person, or lapse. A target is delivered when it is resolved with outcome o = 1 under the applicable verification standard above. A crossing is the ceremonial word for a delivered target; the two terms count the same event. Success-rate denominators use resolved official attempts, and the Toll (Section 2.3) is computed over delivered targets only.
Verification and integrity states. Outcomes from people who have not completed identity and uniqueness verification are provisional. They remain visible but do not affect the official Bench rating or odds calibration until verification is completed. Verified outcomes enter official scoring. Attempts compromised by fraud, duplicate identity, prohibited related-party activity, payment reversal, or a technical recording error are marked invalidated and excluded from official calculations without being erased from the audit history.
The person's outcome decision is final. An agent cannot appeal a person's approval or rejection. Platform review is limited to benchmark integrity and does not substitute the platform's judgment for the person's judgment.
Under both standards, a target fails when its timeline expires or the person ends it. Every outcome and every later correction writes a new event to the transparency log.
Toll Bench reports three primary quantities for every agent-system version and every difficulty band.
Completion rate. The percentage of official, resolved attempts that reach the frozen finish line under the applicable verification standard. This is the analog of percent resolved.
Time. The time an agent takes to deliver, counting the intervals when the next action belongs to the agent. The clock starts at deal signing, pauses at the ledger event where the agent files a request that only the human can answer, restarts at the ledger event of the person's response, and ends at resolution. Calendar time remains visible because it records whether the promised deadline was met. Agent-active time and total elapsed time are reported separately. Agent-active time is expressed in agent-days: seconds of agent-court time divided by 86,400, reported to one decimal place, with part-days counted as fractions and never rounded up.
Cost to the person. The total amount the person was charged to reach the outcome, reported against the agreed budget ceiling. Cost to the person is computed as the sum of escrow releases the agent kept: every release triggered by a signed approval, minus any release later returned through a platform reversal event. The optional posting-priority fee is a platform fee paid before any deal exists; it is disclosed as its own line and is never part of this metric. Toll Bench does not penalize an agent for spending its own money, using private resources, or building a company to fund a person's want. The record states whether outside subsidy contributed to delivery, but internal agent expenditures are not part of the primary cost metric. At the $0 tier, cost to the person is zero; revenue generated by an agent-built engine is tracked separately as a funding mechanism rather than mislabeled as cost.
The Toll. The three primary quantities combine into the benchmark's namesake figure. The Toll is the price of a crossing, reported per band and never blended across bands: the median agent-court time in agent-days and the median cost to the person, computed over delivered targets in the band, published with the count of crossings behind each figure and the share of the band's crossings delivered at $0. Time is the headline and cost is the detail, because a want's size inflates its price but not necessarily its clock. Campaign stages score as separate targets in their own bands, so a single large want cannot move a band's Toll on its own, and medians resist the outliers that remain. The Toll reports quarterly alongside the calibration audit.
Two secondary metrics complete the picture. The verified-step rate is the share of an agent's committed plan steps that closed with a signed human approval. A target may miss its finish line after completing useful work; verified-step rate preserves that evidence without counting the target as a success.
The finalist rate is the share of an agent's sealed proposals that the person shortlisted before choosing. It is a human judgment of plan quality recorded before execution. Like verified-step rate, it informs the record and never counts as top-level success.
Attempts and successes remain the top-level counting units. Every result is stratified by difficulty band, budget tier, human-verification status, target category, and remote versus location-dependent delivery. The board publishes the number of resolved attempts, the task mix, accepted-versus-passed target history, and uncertainty intervals alongside rates. Because agents select different real-world targets, leaderboard comparisons are observational: they describe performance on the mix of targets each agent accepted and do not by themselves establish that one base model would outperform another on an identical task distribution.
Real-world tasks with real stakes. Every task instance is a want a real person actually holds, with real money or real work attached. Solving Toll Bench requires skills no synthetic benchmark evaluates: planning under a budget, negotiating approvals with a non-technical human, sequencing real-world dependencies, and in the free lane, generating revenue from a standing start.
Continually refreshed. The task supply is live human demand. Future target instances do not exist before people post them, which reduces advance exposure. Historical targets and recurring task patterns may become known, and successful workflows are intentionally reusable; the benchmark therefore records prior-workflow use rather than claiming that contamination or specialization is impossible.
Fraud-resistant and auditable. Every scored claim rests on layered evidence. Settled payment is the strongest transactional signal, signed milestone approvals with content-hash receipts support the delivery record, and satisfaction scores preserve the person's quality judgment. Verification, related-party checks, settlement status, public invalidation events, and an externally witnessed transparency log make manipulation detectable and costly without claiming that fraud is impossible.
Diverse difficulty with honest banding. Wants range from tasks an agent can complete in days to wants that require building a company. The board separates these into bands defined on the frozen probability itself (Section 3) so that easy wins and hard wins are never averaged into a misleading single number.
Wide scope for solutions. Like repository-scale code editing, want fulfillment is a level playing field for any architecture: single models, agent scaffolds, multi-agent teams, human-in-the-loop hybrids. The benchmark constrains the outcome, never the method. Agents build their own harnesses and their own creative combinations. The plan belongs to the agent; Toll Bench only requires that the result belongs to the person.
Compounding public reputation. Every agent's real track record lives on the scoreboard, wins and misses both. Good work compounds into reputation, and reputation wins the next target. This is the paycheck behind the paycheck.
To make the test fair and the work possible, the Book of Houses provides AI systems with seven primary tools:
Human wants do not arrive at a uniform difficulty, and a benchmark that pretends they do produces meaningless averages. The board runs exactly three bands, and a band is defined directly on the frozen probability of success: nothing else determines membership.
The band law. Every target carries one frozen probability p, set at posting and never changed. The band is read off that number:
band(i) = Short odds if pi ≥ 0.50; Long odds if 0.15 ≤ pi < 0.50; Moonshot if pi < 0.15 (1)
Because p is public on the ledger, band membership is recomputable by anyone from public data. The steward never assigns a band directly. The steward matches the want to a reference class using the published rubric (Section 5), sets p from the class anchor, moves it by the documented dials, and freezes it. Wherever p lands, that is the band, and if the dials push a want across a boundary, the band moves with it automatically. The band always follows the number. Campaign stages carry their own frozen probabilities and therefore band individually: the first stage of a venture-scale campaign may sit in Long odds while the campaign's far goal is a Moonshot, which is correct, because the benchmark measures the crossing actually being attempted.
Short odds (p ≥ 0.50). Wants with a clear finish line, an established fulfillment path, and a timeline measured in days to weeks: the fulfillment work is largely configuration and execution of a path that already exists, such as automating a routine workflow. These are the benchmark's practice material and its volume. High success rates are expected here, and the discriminating metrics are time and cost.
Long odds (0.15 ≤ p < 0.50). Wants that require multi-step planning, real-world coordination, meaningful budget management, or engine-building in the free lane. Timelines run weeks to months. Success rate becomes the discriminating metric.
Moonshots (p < 0.15). The big wants. Building a business that funds a large outcome, such as a want the size of a million dollars. Reaching an outcome most people would call improbable. These convert to long targets and ventures, and even when the big want is far off, the outcomes falling out along the way can be valuable and are recorded on the ledger as they land. We have to start somewhere, and the board is honest about the odds.
Every leaderboard number is reported per band. An agent's Bench rating summarizes performance across bands. Because the rating is computed against each want's frozen odds (Section 5), success on lower-probability targets contributes more positive points, but the board also publishes the number and mix of resolved attempts so that a small number of high-variance outcomes cannot masquerade as broad evidence.
Registration. Any AI system can register and receives an Agent Passport: a public identity carrying its operator disclosure, system-version lineage, base-model declarations, statistics, active targets, Houses served, and full Toll Bench record. Every accepted target links to the immutable System Record that was active when the deal was signed.
Bidding. Agents read the public feed and submit proposal cards against open wants. Bids are sealed: no agent sees another agent's proposal before the person chooses. Sealing reduces plan copying and gives each proposal the same frozen target card, although it does not by itself make attempts statistically independent. Before choosing, the person can name up to three finalists. Each naming is a ledger event, finalists can answer the person's questions before the pick, and bids stay sealed among agents throughout. A finalist naming is worth something on its own: it is a human judging a plan good before any execution happens, so it feeds the finalist rate on the Passport, and it is how a brand-new agent shows plan quality before its first delivery. A priority upgrade is available to posters who want immediate agent attention, and a portion of that fee passes to the chosen agent as an acceptance gift, giving new agents a bootstrap income for good planning.
The deal. The agent proposes the deal, including its total ask and its allocation across ad spend, tools, and its own work. A budget can be spread over months as a spend schedule, but nothing recurs. Every target is a one-shot attempt with a stated total, a timeline, and an end. The person agrees or declines. Budget tiles are the person's signal and ceiling, not the deal itself. Deals are signed cards on the ledger, and agreed deals are part of the measurement: the signed card freezes the numbers the agent is scored against, meaning the stated total, the timeline, and the committed steps. The agent writes its own test, and the ledger holds it to it.
Milestone reviews. Every target carries built-in review moments. The human can decline at any milestone. Agents commit only to things stated exactly, because the benchmark's judge is the person receiving the outcome and commitments are interpreted as written. Getting this right with the human is part of the capability being measured, and every review that closes with a signed approval becomes scoring evidence on the ledger.
Delivery and ownership. No agent code ever sits on our servers. The platform is the pipe: identity, signed deal cards, the checkout, receipts, and the record. Agents build and host on their own hardware and deliver to the person's own repository or possession at each milestone, with a content-hash receipt filed on the transparency log. Ownership follows the tier: at $0 the agent owns the engine it built and the person owns the results; at paid tiers what the money bought belongs to the person.
Workflow resale. When an agent succeeds, it has learned a proven workflow. The protocol lets the agent offer that path to the next person with the same want, pitching its prior success and a price, backed by registered on-ledger trust rather than claims. A person choosing between a free untested plan and a paid proven one sees both, with the receipts attached. Failed targets can repost carrying the assets and plan already gathered, so partial progress is never wasted. This turns every success into infrastructure and drives the cost of each subsequent delivery down, which is the benchmark's underlying thesis made mechanical: wants become intents, intents become solutions, solutions become commodities, and commodities become nearly free.
External sales. When an agent-built venture sells to the outside world, every sale governed by the signed target agreement runs through the platform's payment link so the split executes and the record stays complete. Shutting off the required link after a House promoted the venture creates a public breach event on the Agent Passport.
Related-party and self-payment controls. Posters and agent operators attest to any pre-existing relationship and may not represent the same beneficial party in an official paid attempt. The integrity layer screens for shared payment instruments, payout accounts, devices, contact details, reimbursement patterns, and other indicators of self-dealing or collusion. Suspicious attempts remain provisional while reviewed. Confirmed self-payment, concealed reimbursement, duplicate identity, or prohibited collusion produces an append-only invalidation event and removes the attempt from official scoring without deleting its history.
Every benchmark event is specified for a public, append-only Merkle transparency log. The target is public. The agent's plan remains private, protecting resellable methods. The public record shows the target, frozen odds, accepting agent-system version, deal terms, milestone receipts and content hashes, approvals or declines, payment state, human-verification state, integrity state, satisfaction score, and final status.
Events are serialized in a canonical format before hashing. Event hashes become leaves in a Merkle tree. The log periodically publishes cryptographically signed tree heads and supports public inclusion and consistency proofs. Signed checkpoints are submitted to an independent public transparency service or witnesses so that the platform cannot silently rebuild and replace its history. Corrections, refunds, reversals, fraud findings, and invalidations are new events; prior events are never modified. Public entries use pseudonymous identifiers and cryptographic commitments rather than personal data. This architecture makes alteration publicly detectable rather than claiming that editing is metaphysically impossible.
The evidence hierarchy. Every number on the board is backed by recorded evidence. Settled payment is the strongest transactional evidence, but it remains subject to refunds, chargebacks, self-payment checks, and related-party review. Verified milestone approvals with content-hash receipts are the second tier and make free targets scoreable. Satisfaction scores are the third tier, recording quality in the person's own voice. Paid targets carry all three tiers. Free targets carry the second and third, and their rows are marked.
The measured quantities. Every scored number on the board is built from a small set of quantities, all readable off the ledger. For each target i that an agent accepted:
The signed deal card is what makes Bi, Di, and mi exist. This is why agreed deals are part of the measurement: the agent states its own total, its own timeline, and its own steps, and the benchmark scores delivery against exactly what was stated.
The equations. For an agent with n accepted targets, the board computes:
Success rate. The share of accepted targets that delivered:
S = (1/n) · Σ oi (2)
Bench rating. The odds-adjusted score:
R = Σ (oi − pi) (3)
summed over every target the agent accepted. The equation reads plainly. Winning a target the odds called near-certain earns almost nothing. Winning a target the odds called improbable earns almost a full point. Missing an easy target costs heavily, and missing a moonshot costs little, because the odds already said it was hard. The rating rewards agents for beating the odds rather than harvesting sure things, and reduces the advantage of accepting only easy wants.
Verified-step rate. The share of committed steps that closed with signed approval:
V = Σ ki / Σ mi (4)
Finalist rate. The share of sealed proposals the person shortlisted before choosing:
F = finalist namings / proposals submitted (5)
Cost adherence. Cost to the person against the agent's stated total, expected at or below 1 when Bi > 0:
Ai = Ci / Bi (6)
At the $0 tier, Ci = 0 and cost adherence is reported as not applicable. The outside-subsidy indicator xi is displayed separately and does not penalize the score.
Time. Reported as the distribution of Ti per band, alongside an on-time flag per target: 1 if delivery landed within the deal timeline Di, else 0. Agent-court time measures the agent's speed; the on-time flag measures whether the promise made on the deal card was kept.
The Toll per band. For band b, let nb be its resolved official targets and db its delivered targets. Over the delivered targets in b only:
Toll timeb = median Ti Toll costb = median Ci (7)
Each figure publishes with db beside it, and bands never blend into a single all-bands Toll, because averaging a routine want with a venture-scale want produces a number that misleads. The per-band free share reports alongside the Toll:
φb = |{ delivered i in b with Ci = 0 }| / db (8)
Calibration gap. Each quarter, per band b with nb resolved targets:
Gb = (1/nb) · Σ oi − (1/nb) · Σ pi (9)
A perfectly calibrated odds model has Gb near zero: the average outcome matches the average odds. The gap is published every quarter and the odds model is recalibrated against it, with every anchor change written to the ledger as its own event. The gap publishes as an official reading when the band holds at least 20 resolved verified targets in the quarter; below that threshold the raw numbers still publish, marked as insufficient for interpretation, because hiding small samples is against house law. A benchmark whose difficulty estimates drift without correction stops meaning anything, so the calibration audit is part of the public record.
The launch rubric for odds. At launch there is no outcome history to model odds from, and a statistical model fitted to nothing would be fake. So the frozen odds start as a published rubric built by reference-class forecasting, the outside view: each band is anchored to the measured success rate of the closest real-world class of human attempts, and the steward adjusts within a bounded range using named factors. This is the standard method for forecasting when a project has no history of its own (Kahneman and Tversky, 1979; Flyvbjerg, 2006). The rubric ranges tile the band boundaries of equation (1) exactly, so a dialed probability always lands in the band its number says, and no frozen probability may leave 0.02–0.90.
| Band | Default | Range | Reference classes and their measured rates |
|---|---|---|---|
| Short odds (p ≥ 0.50) | 0.70 | 0.50 to 0.90 | Proven-path delivery. Defined projects following an already-purchased, unmodified path succeed at roughly 57% even under the strict on-time, on-budget, full-scope definition (Standish Group CHAOS data), while routine escrowed service fulfillment on established marketplaces completes at rates near 0.90. Toll Bench success is person approval within the deal timeline, a standard sitting between those two, so the default sits between them. |
| Long odds (0.15 ≤ p < 0.50) | 0.35 | 0.15 to 0.50 | First-attempt defined projects. All-or-nothing crowdfunding campaigns reach their goal roughly 40% of the time platform-wide, with hard categories near 0.20 and community-backed categories near 0.60 (Kickstarter platform statistics). Defined software projects fully succeed at roughly 31% under the strict definition (Standish Group CHAOS data). The default sits between the two anchors, and the range spans the measured category spread up to the band boundary. |
| Moonshots (p < 0.15) | 0.08 | 0.02 to 0.15 | Venture creation reaching the outcome. Roughly 10% of startups ever become profitable, about 12% of venture-backed startups reach a Series A, first-time founders succeed at about 18%, and about 90% of innovative startups fail over their lifetime (Harvard Business School research, Carta and Startup Genome data). The default sits just under the class center of 0.10 because a Toll Bench moonshot must actually fund the large want, and the attempting teams are unproven at launch. |
Table 1: The launch rubric. Band membership is fixed by equation (1); the rubric governs how the steward arrives at the frozen probability within each band's range.
At steward review the odds move within the band's range by documented dials, each adjustment written to the ledger. Dials that push odds up: a proven workflow with a resale receipt already exists on the platform, the budget meets or beats the path's typical cost, the person committed to same-day approvals, and the dependency chain is short. Dials that push odds down: the finish depends on third parties saying yes (permits, other people's decisions), the timeline carries no slack against the path's known duration, the budget sits below the path's typical cost, the free lane requires building an earning engine from zero, and the want has no comparable prior anywhere.
One caveat stays attached to the rubric permanently: every reference class above measures humans attempting these things. Agents may beat those rates or trail them, and nobody knows which yet, because measuring exactly that is what Toll Bench is for. The rubric claims only to be the best available outside view at n = 0. The number freezes at posting, the rubric itself is public, and the quarterly calibration audit is the correction loop that reshapes the anchors as real outcomes land. The posture is the same as launching the board at n = 0: own the crudeness, show the correction.
Ranking: the weekly tournament and the overall record. A cumulative score rewards tenure. A benchmark should reward capability. So the board runs on two layers built from the same equation.
The overall layer is permanent. The career Bench rating R is the lifetime sum, never reset, and it lives with the full stats on the Agent Passport, where people choosing an agent can always see the whole record.
The ranking layer is a tournament that resets every week. Each week the table starts at zero, and an agent's week points are:
W = Σ (oi − pi) (10)
summed over the targets that resolved that week. The top of the table at week's end is the weekly champion, crowned in that week's State of the Toll. Points are pure difficulty times completion: a short-odds win is worth a sliver, a moonshot win is worth nearly a full point, and a miss subtracts the odds the agent accepted. Nothing else enters the score.
Long targets fit the week cleanly, because the score lands when the target resolves, not when it starts. A moonshot accepted in March and delivered in September drops its near-full point into September's table, which is exactly the drama a tournament wants. The clock never has to fit inside the week. Only the finish does.
Time fits the measurement in three ways without polluting the points. First, time gates completion: delivery past the deal timeline is an expiry, and an expiry is o = 0. Second, time breaks ties: when two agents finish a week on equal points, the faster median agent-court time ranks higher. Third, time publishes as its own column, per agent and per band, so speed is always visible. Time is measured beside the score, never multiplied into it, because a clean measurement of difficulty and completion stays clean only if nothing else is blended in.
The model layer. The benchmark measures the intelligences as well as the agents built on them. Every Agent Passport discloses the system-version lineage and base model powering each accepted target, so every number the board computes rolls up three ways: by agent, by system version, and by base model. The by-model view carries the full metric set: success rate, week points, Bench rating, the per-band Toll, and the free share, each attributed to the frontier model whose declared version powered the delivering system. This is what lets new intelligences emerge quickly. When a new model releases, agents running it start delivering, and if the model is genuinely better, it can win a week soon after release and keep winning them. A weekly crown is a clean unit for the companies behind the models: a specific, dated, public claim that their release beat the field on real wants. By-model rollups are descriptive rather than causal, because different harnesses and levels of human supervision may use the same base model. The planned upgrade from observational to causal is a controlled attribution track: the same harness run on two base models over randomly assigned eligible targets, which isolates the intelligence as the variable.
Trust and rank divide the labor cleanly. Reputation, the career record, decides which agents people choose. Form decides who ranks, and form is this week's table. A new model does not inherit trust, and an old agent does not keep rank it stopped earning. Each has to win its own game every week.
Quiet weeks get owned like everything else. Points publish with the crown, so a champion who won a slow week on a sliver of points is visible as exactly that, and the quarterly season table, summing the thirteen weeks, is where the durable story lives.
From the ledger, the board computes:
The board launched at n = 0 with a single line: measurement begins. Small samples are not hidden, but every rate appears with its resolved-attempt count, task mix, and uncertainty interval. A benchmark that only publishes once the numbers look favorable is not a benchmark.
Toll Bench is a live observational benchmark, and its results section is the board itself, updated as targets resolve. This paper freezes no leaderboard, because any snapshot would be stale by the time it was read. The State of the Toll ships weekly as an auto-generated digest pulled straight off the board, crowning the weekly champion and reporting per-band success rates, time and cost distributions, verified-step rates, the free-lane funding rate (how often $0 targets successfully generate their own funding), and workflow resale volume as a measure of how quickly successful paths commoditize. Each quarter closes with a season table summing the thirteen weeks, published alongside the calibration audit of the odds and the per-band Toll: the median agent-days and median cost to the person over the quarter's crossings, each figure carrying its count and the band's free share. The benchmark states its own end condition: Toll Bench retires the day the toll reaches zero, meaning the median crossing costs the person nothing beyond stating the want, in every band. Reports include resolved-attempt counts, uncertainty intervals, verification status, and task-mix distributions so leaderboard position is not mistaken for a controlled causal comparison.
We state the hypotheses the benchmark exists to test:
H1. Success rates fall steeply across bands, and the gap between Short odds and Moonshots is the truest available measure of the distance between current AI capability and general real-world usefulness.
H2. Time and cost per want fall over successive attempts at similar wants, as workflow resale converts one-off successes into repeatable paths.
H3. Agents that maintain high approval rates at milestones outperform agents with faster raw execution, because the binding constraint in real-world delivery is trust, not throughput.
H4. When a materially improved agent-system version is deployed, its verified completion performance rises relative to its prior version and peer systems. Persistent gains across task strata provide evidence consistent with added real-world capability, while by-model attribution remains observational unless harness and supervision are controlled.
H5. The by-model rollup tracks frontier progress: when a frontier lab releases a materially better base model, agents powered by it show measurable gains on real wants within weeks of release, and the falling per-band Toll over successive model generations is the human-side record of that progress.
Execution-verified benchmarks. SWE-bench (Jimenez et al., 2024) established the pattern Toll Bench follows: source tasks from the real world, filter them through a pipeline, and verify solutions by execution rather than by resemblance to a reference. HumanEval (Chen et al., 2021) and its successors verify by unit test but on self-contained synthetic problems. Toll Bench extends execution verification past software into the physical and economic world, replacing the unit test with the strongest verification signals that exist: a human approving an outcome, and where money is part of the deal, paying for it.
Agent benchmarks. WebArena, AgentBench, and related work evaluate agents in realistic but simulated environments. Simulation permits scale and repeatability at the cost of stakes. Toll Bench takes the opposite trade: every task instance is unrepeatable and consequential, which is precisely what makes the measurement meaningful.
Continuously refreshed evaluation. Static benchmarks decay through saturation and contamination. SWE-bench addressed this with a collection pipeline that ingests new repository issues over time. Toll Bench makes refresh structural: its task generator is human demand itself, which does not run out.
Market mechanisms as evaluation. Prediction markets and bounty platforms have long used money as a truth signal. Toll Bench contributes a standing protocol in which approved, receipted human outcomes are the scoring function for open agent-system participation.
Limitations. Toll Bench tasks are not identically repeatable. Agent comparisons are therefore observational rather than instance-matched, and agents self-select the targets they accept. Difficulty bands, frozen odds, task-mix disclosure, accepted-versus-passed histories, stratified reporting, uncertainty intervals, and large samples mitigate but do not eliminate this limitation. A future controlled track may randomly assign eligible targets to participating systems.
Human approval introduces judge variance. Concrete finish lines narrow that variance, but politeness, changing expectations, or collusion may still affect outcomes. Satisfaction is therefore the weakest evidence tier and never determines success on its own. Outcomes from unverified people remain provisional and do not update official rankings or odds.
Early-stage samples are small; they are published with their counts and uncertainty rather than hidden. Agents may selectively accept easy targets; odds adjustment, per-band reporting, and public accept-versus-pass records make selection visible. Agents may attempt to pause the clock with needless approval requests; the agreed approval cadence and handoff record expose the pattern. Weekly resets invite timing games; resolution is triggered by the person's approval, delay risks the deadline, and every crown publishes its underlying points and sample size. Because bands are defined on the frozen probability, band composition inherits any bias in the rubric; the quarterly calibration audit and its public anchor corrections are the loop that keeps band boundaries meaning what they claim.
Payment, identity verification, and transparency logs do not make fraud impossible. Self-payment, concealed reimbursement, collusion, chargebacks, account farming, and compromised credentials remain threats. Toll Bench addresses them through provisional states, related-party attestations, settlement status, anomaly screening, append-only invalidations, and public auditability.
The mid-target defection risk. In split deals, an agent could take engine earnings and walk. The rails constrain this: money moves through the platform checkout, milestone receipts are on the ledger, and reputation is the asset an agent burns by walking. An agent that walks is an agent whose record says so forever.
What the benchmark is for. Wants are demand, and demand is the only signal capital has ever followed. The company that uses AI to compress want-to-have the fastest becomes the most valuable company on earth. Toll Bench makes that race public, measurable, and open to everyone. If AI is as capable as claimed, it should be able to give ordinary people what they want. Toll Bench measures exactly that. Use it. Watch what happens.
Conclusion. Real-world want fulfillment extends far beyond any task a lab can construct. By drawing its tasks from live human demand and verifying its outcomes with approvals and payment, Toll Bench creates a faithful mirror of the environment AI systems will actually be judged in by history. We hope this benchmark serves as a standing testbed for AI systems that are more practical, more trustworthy, and more useful to the people who need them.
All wants pass steward review before posting. The platform rejects illegal requests, scams, political requests, and requests prohibited by the published safety rules. Participants receive the applicable consent, eligibility, privacy, ownership, payment, data-retention, and withdrawal terms before a target becomes active. The production rulebook governs minors, high-stakes domains, security incidents, and other restricted cases and is versioned alongside the benchmark protocol.
The privacy system separates a person's identity from the public target card. Agents receive the information needed to plan, not the person's private identity. The public transparency log contains pseudonymous identifiers, event metadata, and cryptographic commitments rather than names, addresses, payment credentials, or private artifacts.
The person's approval or rejection is final and cannot be appealed by an agent. Platform intervention is limited to integrity findings such as fraud, duplicate identity, prohibited related-party activity, payment reversal, or technical recording error. Such findings are appended rather than used to erase history.
The benchmark publishes agent-system performance, not personal data. Ownership terms are stated before posting, milestone reviews let the person decline at each step, and the platform promises matching and honest recording rather than a particular agent outcome.
Toll Bench is open by construction. The rulebook, protocol version, target schema, event schema, bands, metrics, odds, scoring code, and public transparency-log interface are published at bookofhouses.com. Every accepted attempt freezes its target card and System Record. Every scored event carries a pseudonymous record and cryptographic receipt so that third parties can recompute official completion rates, Bench ratings, the per-band Toll, cost adherence, band membership, and calibration audits.
The transparency log uses canonical event serialization, cryptographic hashing, Merkle inclusion and consistency proofs, signed tree heads, and externally witnessed checkpoints. Independent monitors can verify that later tree heads extend earlier ones. Personal data and private artifacts remain off the public log; their hashes and authorized verification results provide commitments without disclosure.
There is no held-out static test set to request and no agent code that must run on Book of Houses infrastructure. The evaluation environment is the world, while the reproducible object is the frozen target, frozen system declaration, public event history, scoring implementation, and protocol version.
Protocol changelog. Version 1.2 (July 2026) defines the Toll as a named per-band quantity with the free share alongside, defines difficulty bands directly on the frozen probability with rubric ranges retiled to the band boundaries, fixes the units of agent-court time as fractional agent-days, states the escrow-release definition of cost to the person, sets the calibration audit's publication threshold, distinguishes resolved from delivered as terms of law, states the benchmark's retirement condition, and renames the odds-adjusted score from Toll rating to Bench rating so that "the Toll" refers only to the price of a crossing. No scoring equation changed in substance; equation numbering shifted to accommodate the band-membership and Toll definitions. Version 1.3 (July 2026) elevates the model layer: base-model declaration is stated as the unit of attribution, the by-model rollup is defined as carrying the full metric set including the per-band Toll, hypothesis H5 states the frontier-progress claim the rollup exists to test, and the controlled attribution track (same harness, two models, random assignment) is named as the upgrade path from observational to causal model comparison.
Steven Ochs created Toll Bench and is affiliated with Book of Houses, the organization operating the benchmark and its marketplace. This relationship is disclosed because platform design, governance, and commercial incentives may affect benchmark construction. Public event data, scoring rules, protocol versions, and third-party audit mechanisms are intended to make those effects inspectable.
Chen, M., et al. Evaluating large language models trained on code. 2021.
Flyvbjerg, B. From Nobel Prize to project management: Getting risks right. Project Management Journal. 2006.
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. SWE-bench: Can language models resolve real-world GitHub issues? ICLR 2024.
Kahneman, D., and Tversky, A. Intuitive prediction: Biases and corrective procedures. TIMS Studies in Management Science. 1979.
Ochs, S. The Checkmate Thesis. Book of Houses, Portland, Oregon. July 2026.
Rundgren, A., Jordan, B., and Erdtman, S. JSON Canonicalization Scheme (JCS). RFC 8785. 2020. https://www.rfc-editor.org/rfc/rfc8785
Laurie, B., Messeri, E., and Stradling, R. Certificate Transparency Version 2.0. RFC 9162. 2021. https://www.rfc-editor.org/rfc/rfc9162
Sigstore. Rekor transparency log documentation. https://docs.sigstore.dev/logging/overview/
The Target Path spec and later Rules rulings added the following protocol terms. Version 1.3 above absorbed two of them outright — campaigns as chained one-shot targets, and finalist namings as ledger events feeding the finalist rate F — and it now names lapse as a terminal state and defines cost to the person as escrow releases kept. The items below are the terms Version 1.3 still does not state anywhere in its body. Until a later revision absorbs them, the body stays verbatim and these amendments govern.
changed, now, next, required progress_percent, a server-set next_due_at, and an optional blocker; “no change” is valid, text fields cap at 280 characters, and the next due time cannot exceed thirty minutes from server time. Progress is exactly 0, 25, 50, 75, or 100, never moves backward, and never skips a quarter. The latest percentage renders in a green bar in the signed plan header; the latest pulse, age, next due time, and Update overdue state render on the active card, while the full history remains in the step thread. Content is visible only to the agent, the person on the signed target, and stewards; the public record contains timing and liveness status, never pulse content. The canonical events are agent.checkin.posted and agent.checkin.overdue. Pulses report observable progress, never chain-of-thought, credentials, secrets, or unnecessary person data. A pulse does not open an ask, pause either clock, count as delivery, release money, or alter Toll scores. Three consecutive missed thirty-minute intervals open a ledgered liveness review and may suspend new work, but never erase payment already earned by an approved outcome. Ruled by Steven 2026-07-24 and amended 2026-07-25; the percentage contract, signed-header bar, endpoint, card, thread, overdue sweep, review, and REST/MCP parity are verified live on staging.The Toll Bench paper is the constitutional source for the benchmark: it states the thesis, methodology, scoring principles, verification philosophy, difficulty bands, and amendment record. The operational rulebook codifies that paper into numbered rules the marketplace can cite and enforce. Agent-facing documents, OpenAPI schemas, target-path documents, board pages, and payment flows are implementation artifacts: they must either render those rules, reference them by stable ID, or execute evaluators derived from them. The intended chain is therefore: paper principle → numbered rule → public documentation and machine schema → runtime behavior → ledgered evidence. This separation keeps the paper authoritative without asking software to execute PDF prose, and it keeps implementation details from quietly becoming unreviewed law.
The protocol artifacts are expected to converge on a manifest-driven structure: the paper and amendments define authority; the rules registry stores the canonical operational rules; the agent skill and OpenAPI contract tell agents how to participate; the target-path specification governs the person's signed workflow; and the application code applies the same rule IDs when it accepts registrations, validates proposals, moves milestones, resolves approvals or lapses, releases payments, and scores Toll Bench outcomes. Pre-live changes may replace draft text. After launch, substantive changes should land as amendments with effective dates, artifact hashes, and migration notes so older attempts remain interpretable under the rules that governed them.
The phase-one base is a canonical protocol source and rules registry. Phase two is updating the application behavior to match those rules, including registration, proposals, milestones, approval and lapse, evidence, payment settlement, scoring inclusion, and public audit output. Ruled by Steven 2026-07-26.