Open source template

Compare AI coding models on your codebase.

10x-bench-kit creates a private, reproducible benchmark for your repositories, tasks and rubrics. Run the same work across models and compare quality, cost and runtime.

View on GitHub See how a trial runs
acme-bench / main

$ 10x bench-kit init acme-bench

✓ benchmark instance created

✓ base repository registered

✓ agent skills installed

✓ runner dependencies installed

Next: open acme-bench in your coding agent

Ask it to use bench-wiring

Pinned baselineSame repository commit
Hidden checksMounted after execution
Versioned outputComparable within an era

Why test locally

Public benchmarks don't show which model fits your team

A public leaderboard ranks models on its own tasks and environment. Your decision depends on how those models perform in your repositories, under your constraints.

01

Training-data overlap

Public benchmark tasks can appear in training data. A higher score may reflect a better model, familiarity with the test, or both.

02

Generic tasks

Leaderboards use their own tasks and environments. Your team works in an existing product repo with internal conventions, dependencies and CI checks.

03

Limited reproducibility

Published scores do not always include the prompts, environment, rubric and model settings required to reproduce the evaluation.

Shortlist models with public benchmarks, then test the finalists on your own tasks.

What's included

Start with the benchmark infrastructure in place

We built 10x-bench-kit from the infrastructure behind 10xBench. The template includes the runner, evaluation flow, versioning and CI setup. Add tasks and criteria from your own work.

A working repository from the start

Run 10x bench-kit init to create a repository with tasks, an evaluation pool, configuration and CI workflows. The generated manifest records the template version.

Isolated trials

Each trial runs in a disposable container with a fresh copy of your base repo at a pinned commit. Evaluation assets are mounted only after the agent finishes.

Quality, cost and runtime

Score quality with static checks, hidden tests, e2e tests and an LLM judge, weighted per task. Each trial records its score, cost and wall-clock time in result.json.

Versioned comparisons

Each result records the template version, task hash, judge model and rubric version. Results are compared only within the same evaluation era.

Guided setup

Included skills guide setup and task authoring: bench-wiring, bench-new-task, bench-build and bench-rubric.

Defined update boundaries

The runtime updates atomically, changes to shared skills arrive as a diff for review, and updates do not modify your tasks or evaluation pool.

How it works

What happens in each trial

Each model × task × trial combination runs as a separate matrix job. Evaluation assets are added only after the agent has finished its work.

  1. 01

    Workspace

    The runner creates a fresh copy of your base repo at a pinned commit, applies the task overlay and starts with empty agent state.

  2. 02

    Execution

    The agent runs prompt.md non-interactively under a fixed timeout.

  3. 03

    Metrics

    An adapter writes OpenCode data to metrics.json, while the workspace changes are saved as patch.diff.

  4. 04

    Evaluation

    The runner mounts the evaluation pool and runs static checks, tests, e2e checks and the LLM judge. Weights come from task.yaml.

  5. 05

    Artifact

    result.json records the score, metrics and evaluation versions used for the trial.

Quickstart

Five steps to your first run

  1. 01

    Create the instance

    Run the installer inside your product repository so it can register that repository as the first benchmark target.

    10x bench-kit init my-benchmark
  2. 02

    Open it in your coding agent

    The installer places the benchmark skills where your selected agent can use them.

  3. 03

    Use bench-wiring

    Ask the agent to configure model keys, the judge, repository access and the container image. The skill runs the validation checks.

  4. 04

    Add real tasks

    Use bench-new-task and bench-build to turn representative work into tasks with hidden evaluation assets.

  5. 05

    Run the matrix

    Start the bench-run workflow in GitHub Actions and select the models, tasks and number of trials.

Updates

Engine separated from execution

The instance separates kit-owned files, shared skills and company-owned evaluation assets. bench-kit update handles each group differently:

Zone Owner On update
.bench-kit/ the kit replaced atomically with the updated runner, container and workflows
.agents/skills/ shared proposed as a diff for your team to review
tasks/ · evaluation-pool/ · bench.config.yaml your company left unchanged by the updater

What you need

Before you run the kit

Prepare the repository, credentials and evaluation inputs once. The validation command checks them before the first run.

GitHub Actions

Trials run as a model × task × trial matrix through workflow_dispatch. You select the models, tasks and number of trials for each run.

10xCLI

The 10x CLI creates and updates the kit instance with bench-kit init and bench-kit update.

Model API keys

Keys for the models under evaluation, plus one for the judge model that scores rubric-based criteria.

A base repository

The product repository your tasks run against, pinned at a commit. Private repositories require a fine-grained token with read-only access to contents.

OpenCode as the runner

The kit currently runs agents through OpenCode. The template pins the OpenCode version used by each evaluation era.

Your evaluation inputs

Before the first run, add secrets, prepare evaluation assets and calibrate the rubrics. The bench-wiring skill runs the validation checks.

Run your first comparison

Create an instance, add one representative task and compare the models your team is considering.

Get the kit 10xDevs ↗