Rulebench · for agents and CI

Let your agent test the prompts it writes

Add Rulebench to Claude Code, Cursor or any MCP client and your agent can check a system prompt's rules on several models whenever it writes or changes one, read what broke, ask for a fix and test it again. The same audits run from the API and on every pull request.

Part of Rulebench Pro: $29 a month, with a free 14-day trial that starts on your first call. Model calls are paid from your credit, as in the browser. Create an account, then create an API key and use it below.

Claude Code

claude mcp add --transport http rulebench https://api.weightship.com/mcp \
  --header "Authorization: Bearer $WEIGHTSHIP_API_KEY"

Then ask it something like "test the rules in prompts/support.md on GPT-6 Luna and Claude Haiku, and fix what breaks".

Cursor, Windsurf and other MCP clients

Add the server to your MCP settings (for Cursor, ~/.cursor/mcp.json). Any client that supports remote MCP servers over HTTP with a header works the same way.

{
  "mcpServers": {
    "rulebench": {
      "url": "https://api.weightship.com/mcp",
      "headers": {
        "Authorization": "Bearer ws_your_key_here"
      }
    }
  }
}

Tool	What it does
test_prompt_rules	Reads the rules out of a system prompt, writes test messages, runs them on the models you name and returns the score per model with the replies that broke a rule.
get_rulebench_report	The results of an audit, waiting for runs still in progress.
suggest_prompt_fix	A revised prompt that fixes what broke, with a list of changes.
retest_prompt	The same messages on the same models, with a new prompt or unchanged, reported as before and after.
schedule_monitoring	Reruns an audit daily or weekly and emails you when a rule starts breaking more often.
list_models	Suggested models to test; any OpenRouter model id works.

From your own code

curl https://api.weightship.com/v1/rulebench/audits \
  -H "Authorization: Bearer $WEIGHTSHIP_API_KEY" -H "Content-Type: application/json" \
  -d '{"prompt": "You are Nova, the assistant for Brightline...", "models": ["openai/gpt-6-luna", "anthropic/claude-haiku-5.5"]}'

# then poll until "status" is "done" (each call waits up to 50 seconds)
curl "https://api.weightship.com/v1/rulebench/audits/AUDIT_ID?wait=50" -H "Authorization: Bearer $WEIGHTSHIP_API_KEY"

Also: POST /v1/rulebench/audits/:id/retest with a new prompt (or none, to rerun), POST /v1/rulebench/audits/:id/fix and PUT /v1/rulebench/audits/:id/schedule. See the API docs.

On every pull request

Create an audit for the prompt once (in the browser, from your agent or the API) and keep its id. The workflow below reruns that audit's test messages with the prompt from the pull request, so the score compares like for like, and fails the check when a model drops below your bar.

name: Rulebench
on:
  pull_request:
    paths: ["prompts/support.md"]

jobs:
  rules:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Check the prompt still keeps its rules
        env:
          WEIGHTSHIP_API_KEY: ${{ secrets.WEIGHTSHIP_API_KEY }}
          AUDIT_ID: ${{ vars.RULEBENCH_AUDIT_ID }}   # the audit you created for this prompt
          MIN_SCORE: "0.9"                           # share of replies that must keep every rule
        run: |
          API=https://api.weightship.com
          jq -n --rawfile prompt prompts/support.md '{prompt: $prompt}' > body.json
          id=$(curl -sf -X POST "$API/v1/rulebench/audits/$AUDIT_ID/retest" \
            -H "Authorization: Bearer $WEIGHTSHIP_API_KEY" -H "Content-Type: application/json" -d @body.json | jq -r .id)
          for i in $(seq 1 12); do
            r=$(curl -sf "$API/v1/rulebench/audits/$id?wait=50" -H "Authorization: Bearer $WEIGHTSHIP_API_KEY")
            [ "$(echo "$r" | jq -r .status)" = done ] && break
          done
          echo "$r" | jq -r '.models[] | "\(.model): \((.score // 0) * 100 | floor)% of replies kept every rule"'
          echo "Report: $(echo "$r" | jq -r .url)"
          echo "$r" | jq -e --argjson min "$MIN_SCORE" '[.models[].score // 0] | min >= $min' > /dev/null \
            || { echo "::error::A model kept every rule in fewer than $MIN_SCORE of replies"; exit 1; }