# How to use ADoor for AI evals

Connect your evaluation runner, save results and traces, and give your team one place to review them.

Use ADoor alongside your existing evaluation code. Your runner calls the models or agents and scores their outputs. ADoor stores the records so you can query them from Python, open them in the browser and share them with teammates.

This guide starts with two tables and a single example result. You will need Python 3.9 or newer and an ADoor account.

## 1. Install the SDK and connect

Install the Python package:

```sh
python3 -m pip install adoor
```

Then connect from your Python script or notebook:

```python
import adoor

db = adoor.login(name="AI eval runner")
```

ADoor opens in your browser. Sign in, choose the workspace and space for your evals, and approve the connection’s access. For the setup below, it needs permission to create tables and read and add their records. The SDK saves the connection for subsequent runs.

For a background job, create a connection through **Access → New connection**, provide its token as `ADOOR_TOKEN` in your job’s secret configuration, and use `adoor.connect()` instead. The [Python guide](/docs/guides/python/) covers connection options.

## 2. Create results and traces tables

Run this setup once in a space where these table names are available:

```python
db.create_table("eval_results", {
    "run_id": "text",
    "candidate": "text",
    "case_id": "text",
    "dataset_version": "text",
    "prompt_version": "text",
    "status": "text",
    "score": "numeric",
    "output": "jsonb",
})

db.create_table("eval_traces", {
    "run_id": "text",
    "candidate": "text",
    "case_id": "text",
    "sequence": "integer",
    "event": "jsonb",
})
```

`eval_results` holds one result per candidate and case in a run. `eval_traces` holds the associated events: requests, responses, tool calls or errors. The run, candidate and case identify the attempt; `sequence` orders its events.

Use `candidate` for the configuration you want to compare, such as a model, prompt version or agent setup. Give every new execution a fresh `run_id`. If you repeat a case within one run, add a trial field to both tables and include it in the attempt identifier.

## 3. Write your first result

Start with this small storage check. It writes a fixed example output and two trace events; it does not call a model.

```python
from uuid import uuid4

run_id = uuid4().hex
candidate = "baseline"
case_id = "billing-001"

output = {"label": "billing"}
score = float(output["label"] == "billing")
events = [
    {"type": "request", "input": "I was charged twice."},
    {"type": "response", "output": output},
]

attempt_key = f"{run_id}:{candidate}:{case_id}"

db.insert("eval_results", [{
    "run_id": run_id,
    "candidate": candidate,
    "case_id": case_id,
    "dataset_version": "support-v1",
    "prompt_version": "routing-v1",
    "status": "completed",
    "score": score,
    "output": output,
}], key=f"{attempt_key}:result")

db.insert("eval_traces", [
    {
        "run_id": run_id,
        "candidate": candidate,
        "case_id": case_id,
        "sequence": index,
        "event": event,
    }
    for index, event in enumerate(events)
], key=f"{attempt_key}:trace:0")
```

Open your space in ADoor. You should see one result with a score of `1` and two trace records. Activity shows that the writes came from **AI eval runner**.

Keep the same idempotency key when retrying the same write with the same data. A new evaluation gets a new run ID and new keys. If you stream traces in several batches, use a separate stable key for each batch and continue the event sequence across them.

## 4. Connect your actual evaluation loop

Replace the fixed `output`, `score` and `events` with values from your runner. Write one result after each case, and collect the trace events your model or agent interface exposes. You can retain the original event payload in the JSON field.

Keep the dataset and prompt versions with every result. Add fields such as model identifier, latency, token usage or grader version when your workflow needs them. Leave unavailable measurements empty rather than recording zero.

Record timeouts and other failures too. For a pass/fail eval, store `1` for a pass and `0` for a failure, including execution failures, and use `status` to explain what happened. Reconcile results against the cases you scheduled so crashed workers do not disappear from the score.

For larger runs, insert result batches and flush trace events periodically. The SDK supports batched inserts; avoid a separate database request for every streamed token.

## 5. Query a run and inspect its traces

Compare candidates within one run:

```python
summary = db.query("""
    SELECT candidate,
           COUNT(*) AS attempted,
           AVG(score) AS mean_score
    FROM eval_results
    WHERE run_id = $1
    GROUP BY candidate
    ORDER BY candidate
""", [run_id])

for row in summary:
    print(dict(row))
```

For the pass/fail scoring above, `mean_score` is the pass rate. For another metric, it is the average of the scores your grader supplied.

To inspect one attempt:

```python
trace = db.query("""
    SELECT sequence, event
    FROM eval_traces
    WHERE run_id = $1
      AND candidate = $2
      AND case_id = $3
    ORDER BY sequence
""", [run_id, candidate, case_id])

for row in trace:
    print(row["sequence"], row["event"])
```

Use the same identifiers to find the result and its events. You can also browse the tables directly in ADoor.

## 6. Share results and connect more workers

Invite teammates to the eval space and give them read access to the tables they need. They can review the same results in the browser or connect their own notebook or assistant. Grant edit access only where their work requires it.

For parallel evals, give each worker its own named connection with access to add results and traces. The Python SDK can create expiring child connections with `db.sandboxes()`; the coordinator needs the approved `manage_connections` capability. Your scheduler launches the workers and passes each its connection environment. See [one identity per eval worker](/docs/guides/python/#one-identity-per-sandbox-run-or-eval-worker).

When evaluating an agent that can execute code, keep the result-writer connection in your trusted harness rather than giving the agent access to its own scores. Revoke worker connections when they finish, or set an expiry when creating them.

You now have the basic setup: your runner produces the evaluations, ADoor stores their results and traces, and your team can review the same records. Export the experiment data you need for long-term retention; Activity’s before-and-after row history is limited to 5,000 row changes per workspace or 30 days, whichever is shorter.

[Open ADoor](https://app.adoor.run/login) · [Python SDK guide](/docs/guides/python/)