Docs › Bench

Libraries, notebooks and the agent

Code several projects need lives in a library. Free-form work lives in notebooks beside the code. The agent works in the project with the project's tools.

• Libraries
• Notebooks
• VS Code
• The agent

Libraries

Create one on the Libraries tab from the statlib template. It is a git repository like a project:

~/Bench/libs/<name>/
    python/<name>/       the package: import <name> in any project that uses it
    tests/test_*.py      checks that pin its behaviour (the test step runs them)
    lib.json             {"name": "<name>"}

A project uses it by version in project.json:

"libs": {"statlib": "main"}      follow the branch
"libs": {"statlib": "v0.1"}      a release (Release tags the library's last commit)
"libs": {"statlib": "work"}      the uncommitted working tree, to try a change before committing it

Steps and notebooks can then import statlib. Every experiment records the library commit it ran with, so a result can be reproduced after the library has moved on. A library's page shows which projects use it.

Notebooks

Notebooks live in notebooks/ and open in JupyterLab (JupyterLab ↗) or VS Code, with the Bench (Python) kernel: your chosen Python, with bench, the project's own modules and its libraries importable. Git stores notebooks without outputs, so diffs show code. Add a step {python} -m bench nb run notebooks/x.ipynb to run a notebook as part of an experiment: its numbers (bench.record in a cell) and the executed copy are kept. When exploration settles, move the code into the project's .py files and make it a step.

VS Code

The whole workspace is one VS Code workspace (~/Bench/psbench.code-workspace): VS Code opens it, locally or through Remote-SSH when the workspace is a server's. Two tasks come with it: open this file in Bench, and run this project's steps as an experiment.

The agent

The Agent tab is a continuous chat with your own Claude Code, restricted to the project's tools: read files and results, edit files, list and compare experiments, run steps, commit — plus your PriorStates Memory and journal when installed. It has no shell and no other files. It works incrementally: it discusses first, edits when you agree (edits show as uncommitted changes), runs only when you ask, and commits only when you say the change is settled. Pick a faster or a more capable model per chat.

Good first requests: "Add a gradient-boosting option and run it against the baseline with 10 seeds." · "Why is seed 3 worse? Look at the predictions." · "Move the bootstrap into the statlib library with a test."