Skip to main content

Getting started

What you need​

  • Git
  • A running Docker daemon
  • A GitHub or GitLab repository with merged pull requests
  • Credentials for the git host and the AI tool you want to test

Install​

curl -fsSL https://get.berbench.ber.run/install | bash

This downloads the release build for your platform (Linux and macOS, amd64 and arm64), verifies its checksum, and installs it to ~/.local/bin. Two optional environment variables:

  • BERBENCH_BIN_DIR — install somewhere else
  • BERBENCH_VERSION — pin a release instead of taking the latest

Check the result:

berbench doctor

Or build from source, with Go 1.26:

go build -o berbench ./cmd/berbench
install -m 755 berbench ~/.local/bin/berbench

Optional: let your coding agent handle setup​

Install the BERBench agent skill if you want a coding agent to prepare the repository for you:

berbench skill install

Then ask the agent to set up BERBench in your repository. The skill tells it how to inspect the project's CI and dependencies, write Dockerfile.berbench, run berbench init and berbench doctor, and work through any failed setup checks.

1. Prepare your repository​

BERBench builds a Docker image for each challenge. Add a Dockerfile that copies the repository to /workspace and installs the dependencies needed to run its tests.

Example for a Python project:

FROM python:3.12-slim

RUN apt-get update \
 && apt-get install -y --no-install-recommends git ca-certificates \
 && rm -rf /var/lib/apt/lists/*

WORKDIR /workspace
COPY . /workspace
RUN pip install --no-cache-dir -e .

Save it as Dockerfile.berbench in the repository root. BERBench can also use an existing Dockerfile; set its path in .ber/bench/config.yaml.

Do not install Claude Code or Codex in this image. BERBench adds the selected tool itself.

2. Configure credentials​

For the git host, use an environment variable or an authenticated CLI:

# GitHub: GITHUB_TOKEN or GH_TOKEN, or `gh auth login`
# GitLab: GITLAB_TOKEN, or `glab auth login`

For AI tools, BERBench recognizes:

# Claude Code
export ANTHROPIC_API_KEY=...
# or CLAUDE_CODE_OAUTH_TOKEN / ~/.claude/.credentials.json

# Codex
export OPENAI_API_KEY=...
# or ~/.codex/auth.json

Credentials are passed to the agent container at run time. They are not baked into images or copied to run results.

Using a cloud provider instead of the first-party API? See Run Claude Code on Amazon Bedrock.

3. Initialize BERBench​

Run this from the repository you want to benchmark:

cd /path/to/your/repository
berbench init
berbench doctor

init creates .ber/bench/, detects the git remote, and records the Dockerfile. doctor checks Git, Docker, host access, credentials, configs, and existing challenges.

Commit .ber/bench/. It contains benchmark definitions, not run output.

Run results live outside the repository under $XDG_DATA_HOME/berbench/<repo-id>/, or ~/.local/share/berbench/<repo-id>/ when XDG_DATA_HOME is unset.

Next​

Read Challenges, then follow the end-to-end run guide.

Configuring a specific setup? The how-to guides cover the cases that need extra steps.