Docs › Bench
Projects, experiments and parameters
A project is a folder of code under git. Its steps are commands; Bench runs them as experiments and keeps what each run did, so results can be compared and reproduced.
A project
~/Bench/projects/<name>/ a git repository
project.json {"description", "params": {...}, "steps": [{"name", "run", "doc", "needs"}], "libs": {...}}
*.py, notebooks/*.ipynb your code
out/ not versioned: what runs write (out/exp/<id>/ per experiment)
Edits — yours in the page, the agent's, or made in VS Code — go to the working tree and show as uncommitted changes until you commit them (Review & commit). Every commit can be reviewed and reverted from History.
Steps and the bench module
A step is any shell command run in the project; {python} is the Python you picked. "needs" lists modules the step imports, so Bench can offer to install them. Scripts use the bench module to find their files and report results:
| call | what it does |
|---|---|
bench.out("model.pkl") | a path in this step's output folder (created as needed) |
bench.inp("prepare", "splits.json") | a path in an earlier step's output — of this experiment, or of the one it reuses |
bench.params(lr=0.01) | the parameters: these defaults, then project.json, then the experiment's |
bench.record(auc=0.97) | numbers: columns of the experiment table, compared between experiments |
bench.series("loss", points) | a curve, overlaid when two experiments are compared |
bench.chart(title, "line"|"bar"|"scatter", series) · bench.table(title, columns, rows) | what the experiment page shows |
bench.data_dir("source") | a folder under ~/Bench/data/, shared by every project |
bench.compare(reference, result) | do two DataFrames agree, column by column, within a tolerance (for reproductions and refactors) |
Outside an experiment (a terminal, a notebook) the same calls fall back to the project's out/, so the code runs anywhere.
Parameters
project.json "params" are the defaults. The run form shows them as JSON to change for one experiment; the agent can pass them too. The experiment keeps the values it ran with: they appear in its details, in the experiment table, and in comparisons. Use them for seeds, hyperparameters, sample sizes — anything you want to vary without editing code.
Experiments
Every run keeps: the code snapshot (a commit on a hidden ref, uncommitted edits included, not on any branch), the parameters, the commits of the libraries used, the Python version, each step's command, exit code, log, outputs, numbers and charts, and the hashes of data files a step names (bench.note_data). Give each a title and a hypothesis; after reading the results, record a verdict (winner, loser, inconclusive) and a note. Compare puts two side by side: parameters, metrics with differences, code diff, overlaid charts.
Rerun
Rerun runs an experiment's snapshot again: same code, parameters, libraries and computer. From a later step, the earlier steps' outputs are reused — after installing a missing package or fixing data, without touching the code. In a new experiment, untick a step to reuse its outputs from an earlier experiment.
Servers and clusters
Install Bench on a server and run psbench start (it listens on 127.0.0.1 only). On your laptop, add the server as a workspace: Bench connects through ssh with your own keys and keeps the connection up. Experiments can also run on a server, a Slurm cluster or Kubernetes / OpenShift (Run on · servers…).