Docs › Bench
Bench quick start
Install PriorStates Bench, create a project from the example, run your first experiment, change a parameter and compare — about ten minutes.
1 · Install
Bench is Python and a browser window, so the same download runs on any Linux (and macOS) with Python 3.9 or newer and git. It installs for your user only; no root.
curl -LO https://priorstates.com/download/bench/psbench-0.1-50-linux.tar.gz tar -xzf psbench-0.1-50-linux.tar.gz && cd psbench-0.1-50-linux ./install.sh psbench
The installer copies the app to ~/.local/share/psbench, adds the psbench command to ~/.local/bin and a PriorStates Bench entry to your applications menu, and tells you what is missing. Claude Code, signed in, is needed only for the AI agent. psbench opens Bench in its own window; your work goes in ~/Bench.
Upgrading is the same lines with the new file: a running Bench is restarted on the new version. ~/.local/share/psbench/current/standalone/uninstall.sh removes the app; ~/Bench stays.
Or in Docker
For Windows, or to keep everything in a container. From the unpacked folder:
./psbench-docker # builds the image once, starts Bench, opens http://127.0.0.1:8940 ./psbench-docker login # once: sign Claude Code in, inside the container ./psbench-docker stop | logs | shell | rebuild
The image has Python with numpy, pandas, scikit-learn and JupyterLab, git and Claude Code. Your work stays in ~/Bench on your computer.
2 · Create a project
On the Projects tab, type a name, pick the mlstudy template and press Create. The project opens: its files on the left, the editor in the middle, Steps, History and Agent on the right.
The example asks a real question: does a random forest beat a logistic-regression baseline on scikit-learn's breast-cancer data, and does the gap hold across random splits? It runs offline. Its four steps are listed in project.json:
| step | does | records |
|---|---|---|
| prepare | loads the data, makes one stratified train/test split per seed | rows, features, positive rate |
| train | fits the model and the baseline on each split, keeps held-out probabilities | a score table per seed |
| evaluate | AUC of each, the gap and a bootstrap interval | auc_model, auc_baseline, auc_gap, its interval, a chart |
| notebook | runs notebooks/explore.ipynb top to bottom | the best single-feature AUC |
If the Python you picked lacks a package a step needs (here scikit-learn), the step shows needs … · Install: one click installs it.
3 · Run an experiment
In Steps, give the run a title (what it tries) and, if you like, a hypothesis (what you expect, and what would show you are wrong). Press Run experiment. The code is snapshotted, including uncommitted edits, and the steps run from the snapshot. When it finishes, open the experiment: each step's numbers, tables and charts, its outputs and its log.
4 · Change a parameter, compare
The Parameters box holds project.json's "params". Change "model" to "logistic", untick prepare (its outputs are reused from the first run) and run again. Then open Experiments, tick both and press Compare: the parameters, metrics and code diff side by side, and the AUC charts overlaid. With the model equal to the baseline, the gap is zero — a quick check that the evaluation is honest.
Next
- Projects, experiments and parameters: how a project is laid out, what an experiment keeps, reruns, servers.
- Libraries, notebooks and the agent: sharing code between projects, notebooks in JupyterLab or VS Code, working with the agent.