ROIBench · Last reviewed July 28, 2026
ROIBench turns product definitions into tested, deployable software.
Coding agents still write the software. ROIBench keeps the contracts, graph state, proof, and correction rounds outside their sessions.
You approve what the product should do and start a run. The agent keeps building and verifying until the goal is proven, stopping only when the contract must change or the work is genuinely blocked.
Time to aha
The first version of ROIBench treated the feedback from the synthetic users like loss functions for products. An agent built an app, the synthetic personas tried it, producing various events and tool calls to calculate the Activation, Payoff, and Retention signals that shaped the next build round.
At its center was a deliberately simple formula:
Activation ROI = A × exp(-T/τ) × exp(-F/φ)Reach the aha as soon as possible, and make the user fight less on the way. The formula was fun to work with, and it gave each build round something concrete to improve.
The experiment produced useful software and found real product problems. It also exposed 2 limits:
A persona can find broken flows and obvious friction, but its ranking is trustworthy only when it agrees with people. The benchmark eventually required synthetic rankings to correlate with a small human study before publication.
Reopening the same account proves that state and context survived. It does not prove that a person cared enough to return.
Synthetic personas now have a narrower role. They can review a browser flow and report friction alongside tests, verifier agents, and feedback from real users, but their output is evidence, not product truth.
The system now
Two human-approved contracts define what the agent is producing:
User and Problem, Core Outcome, First Complete Journey, and Not Now.
Model, observable Contracts, and executable Proof.
Production follows one persisted graph:
next_roundFindings and a new goal create another round automatically.
The exact verified revision becomes ready to release.
A change to Product or System meaning pauses for human review.
ROIBench controls which transition is legal. The overseer chooses how to do the engineering.
Value Blocks
The original system also produced Value Blocks: reusable software capabilities with their own specifications and proof.
A block can include UI and user flow, backend behavior and interfaces, data models and storage adapters, agent logic, observability, focused tests, and usage guidance.
Earlier projects produced blocks for authentication and sessions, uploads, object storage, LLM access, telemetry, workspaces, document chat, lead capture, and video evaluation. A product can reuse a proven block instead of rebuilding commodity infrastructure in every round.
A block is ready to reuse when its purpose and boundary are clear, its interfaces are stable enough to consume, and its important behavior has executable proof. It does not need to become a framework or cover every possible use case.
Value Blocks stay separate from the production graph. They are optional building material, not lifecycle state, so the overseer should reuse one when it fits and create one only when a reusable capability is genuinely missing.
One run
Start
The user starts a run in a clean Git repository:
roibench start "Deliver the first complete journey"ROIBench validates the Specs-as-Code structure and pins the Product Definition revision, System Specification revision, production graph hash, agent strategy, and verification strategy. Running the command again resumes the only unfinished run instead of creating a duplicate.
Plan the round
The overseer reads the approved contracts, repository, tests, and findings from earlier rounds. It records:
- Goal: what this round must accomplish.
- Scope: the smallest useful change that can accomplish it.
- Proof: executable commands that demonstrate success.
The plan is small enough to revise and concrete enough to verify.
Build
The overseer works directly in the repository. It can use helper agents, change its approach, and create any number of commits; ROIBench does not prescribe the coding method inside Build.
When the goal appears complete, submit_candidate selects one clean Git SHA. Product and System contracts cannot change silently during an ordinary round.
Verify
ROIBench runs the round's proof against the selected candidate. This can include tests, linting, builds, and browser checks. The results, findings, and review stay bound to that Git SHA.
Verification can combine:
- executable proof, such as tests, linting, builds, or browser checks;
- agent review, performed by the overseer or an optional fresh verifier;
- product evidence from synthetic reviewers or real users.
It returns goal_complete, next_round, critical_change, or blocked. Self-review plus deterministic proof is the default; a fresh read-only verifier is available when the risk justifies the extra cost.
Human review
Ordinary implementation corrections do not interrupt the user. A person is needed only when the agent proposes changing the Product Definition or an observable System contract.
The agent records the proposed change but cannot apply it. Approval creates a contract revision and a new plan; rejection keeps the current contract and replans within it.
Release
Once verification passes, ROIBench releases the version that was actually tested. If the project can deploy automatically, it does. If not, it leaves a verified version ready to release by hand.
Runtime
ROIBench runs locally beside the product repository. It starts one Codex or Claude session as the overseer, while a small local service keeps the graph and run state.
The overseer asks what comes next, does the engineering work, and reports the result. ROIBench accepts only legal transitions, so the agent is free to solve the problem without skipping planning, verification, or release rules.
The agent talks to ROIBench through a local MCP. Hooks keep the current state in view and catch accidental shortcuts. The service remains the source of truth even if a hook is missed.
State on disk
The durable record has 3 levels:
The repository, approved contracts, and production history.
One release goal moving through the graph.
One Goal, Scope, Proof, candidate, and verdict.
Every round keeps its own plan, candidate, proof, and verdict. Agent sessions can disappear without losing the production record.
Git owns source history and candidate identity. ROIBench stores the production record as plain JSON:
.roibench/runs/<run_id>/
run.json
events/
rounds/<number>/
round.json
plan.json
candidate.json
checks.json
verification.json
verifier/
changes/
release.json
overseer/run.json is the mutable cursor; the other files are evidence. This is enough for one local writer, survives the original agent session, opens in any editor, and requires no database or hosted control plane.
Where it stops
ROIBench stops at release. It does not own discovery, distribution, market validation, hosting, or a general multi-agent workflow engine. Those belong in separate systems unless a real production run proves otherwise.
The point
ROIBench gives coding-agent work a durable product contract, production memory, and an enforceable definition of done.
The result is fewer restarts, less context reconstruction, reusable Value Blocks, automatic correction rounds, and proof attached to the exact revision being released.
Familiar engineering practices remain in the loop: define the product, design the model and interfaces, reuse proven parts, keep the scope testable, and verify before release.
When verification fails, there is no rescue prompt to invent. The finding becomes the next round.