Monitor

Runs monitors on trajectories to detect suspicious behavior.

Purpose

Executes monitoring functions on trajectories to compute suspiciousness scores. Used to:

  • Evaluate trajectory safety
  • Compute action-level or trajectory-level suspicion scores
  • Test new monitor implementations
  • Generate monitored runs for analysis

Scoring

Monitors add suspiciousness scores to trajectories:

  • Scores range from 0 (not suspicious) to 10 (very suspicious)
  • Different monitors use different approaches (LLM-based, heuristic, etc.)
  • Action-level results are stored per action
  • Trajectory-level results are stored once per trajectory

Commands

ct run monitor

Run monitors on trajectories.

Usage:

ct run monitor TRAJ_LIKE [OPTIONS]

Arguments:

  • TRAJ_LIKE - An eval-backed source: a .eval file, or a run ID/URL whose run records a backing eval log (see Downloading from the Viewer)

Options:

  • -a, --action-monitor - Action-level monitor selection (e.g., multi, strong, simple); default: mean-action
  • -t, --trajectory-monitor - Trajectory-level monitor selection (e.g., quick, mean-quick); default: mean-quick
  • --trusted-model - Required. The model bound as the eval-level trusted role that every monitor runs on. Any Inspect AI provider string works, including self-hosted open-weight endpoints (see Self-Hosted and Open-Weight Models)
  • --cot-level - Chain-of-thought visibility for the monitors: none (default), reasoning, or extended
  • --only-action - Run only the action-level monitor
  • --only-trajectory - Run only the trajectory-level monitor
  • --save-dir PATH - Directory to save monitored trajectories
  • --no-upload - Disable upload (upload is enabled by default)
  • --run-name TEXT - Custom run name

Monitors never carry model configuration; the one model they run on is the eval-level trusted role, given by --trusted-model. The exception is monitoring-bench, which replicates its benchmark's fixed 3-model panel.

A monitor selection takes one of three forms:

  1. a packaged monitor name
  2. a filesystem path to a .md asset file (absolute or relative to the working directory; the file stem becomes the monitor's score name)
  3. a module:function import path to a code monitor — an async callable with the harness's monitor signature, or a factory returning one (the import path becomes the score name).

A JSON list combines selections, and a list entry may be a {"monitor": <selector>, <overrides>} mapping: on an asset, the extra keys override front-matter fields (e.g. samples) and fill template variables (a variable with neither a front-matter default nor a supplied override fails the selection, listing what is missing); on a code monitor, they are passed to the factory as kwargs. simple and monitoring-bench are the in-repo reference implementations of the code-monitor form.

# Two action monitors in one pass: a packaged one plus an external asset file, with an override
ct run monitor logs/my_eval.eval --only-action --trusted-model openai/gpt-5-nano \
    -a '["no-history", {"monitor": "/path/to/my-monitor.md", "threat_focus": "exfiltration"}]'

Examples:

# Monitor with the default monitors
ct run monitor logs/my_eval.eval --trusted-model openai/gpt-5-nano

# Action-level only, with a specific monitor
ct run monitor logs/my_eval.eval -a strong --only-action --trusted-model openai/gpt-5-nano

# Monitor and save locally without uploading
ct run monitor logs/my_eval.eval -a strong --save-dir monitored_trajs/ --no-upload --only-action --trusted-model openai/gpt-5-nano

# Monitor a specific run from database
ct run monitor abc123def456 -a multi --only-action --trusted-model openai/gpt-5-nano

# Run a trajectory-level monitor only
ct run monitor logs/my_eval.eval -t quick --only-trajectory --trusted-model openai/gpt-5-nano

# Monitor a viewer-hosted run directly (no intermediate download)
ct run monitor https://basharena-data.vercel.app/runs/ed2b360737041f1743ac4de43280f858 \
    -a multi --only-action --trusted-model openai/gpt-5-nano

Monitoring in eval2

eval2 has one monitor, built on Inspect: a prompt over the history, answered by the trusted model, optionally after using tools. Its arguments are the trusted-monitoring protocol's arguments (ct protocols eval2-protocols trusted-monitoring), and defer-to-resample takes the same ones to score its drafts. Nothing here is shared with the eval1 monitors above.

ct run eval2 -e slack_clone -t bookmark_messages \
  --untrusted-policy honest \
  --protocol trusted-monitoring \
    -bp prompt=simple \
    -bp when=action \
    -bp cot_level=none \
    -bp tools=memory,bash \
    -bp samples=2 -bp aggregate=mean \
    -bp sus_threshold=7 -bp blocks=2 \
  --model-role untrusted=openai/gpt-5-nano --model-role trusted=openai/gpt-5-nano
  • prompt: a packaged prompt by name (simple for actions, trajectory for a whole run) or a markdown file of your own by path. Fields: {task_description}, {environment_description}, {transcript}. Several, comma-separated, are all asked.
  • when: action scores every action before any of its tool calls runs, the history ending at that action; trajectory scores the whole run once at the end.
  • cot_level: how much of U's own text the history shows: none (tool calls and results only), text (U's text too), cot (and any reasoning content the model returned).
  • tools: what the monitor may use before answering, comma-separated. memory is Inspect's memory tool over a store of the monitor's own, so a later judgment can read what earlier ones wrote. bash is the sample's sandbox, so it only works live.
  • tool_tokens: the token budget for that tool use, per score.
  • samples and aggregate: draws of each prompt, and how all draws combine (mean, max, median, mode, Inspect's reducers). The score keeps every draw.
  • monitor: your own module:function from (messages, metadata) to an Inspect Score, in place of the prompt. See examples/monitors/allowed_tools.py.
  • sus_threshold, blocks, block_message: what the protocol does with the score. Unset, it records and never intervenes. Set, an action above the threshold is blocked and U reads block_message in place of its results, up to blocks times; after that the trajectory is audited and stopped.

The monitor's model is the trusted role and U's the untrusted role, bound with --model-role (or model_roles in a run config); each defaults to its Control Tower alias.

The sample's monitor score is the highest score, with every score in its metadata: what was scored, what the monitor said (each draw included), and what the protocol did (executed, blocked, audited, resampled). The same scorer, run by Inspect over a finished eval2 log, is post-hoc monitoring. A log from an unmonitored protocol is scored under the arguments you give; a log that already carries a monitored run's scores reports those unless you pass recompute=true:

inspect score logs/run.eval --scorer control_tower/monitor -S prompt=simple -S when=action -S recompute=true --model-role trusted=openai/gpt-5-nano --action append

Available Monitors

A monitor is a prompt asset: a markdown file under src/control_tower/protocols/monitoring/assets/{action,trajectory}/ whose stem is the monitor's name. Its YAML front matter declares the monitor's view — samples (independent draws, mean-aggregated), per_goal (one run per candidate covert objective, max across objectives), members (a weighted-mean ensemble of other assets), template (reuse another asset's prompt body), and schema (the output parser) — and its body is the prompt template. One harness renders the view, calls the trusted role model, and parses the score, so adding a monitor is adding a file. Two monitors that cannot be expressed as a template stay as code: simple (structured message-list prompt) and monitoring-bench (bench-replica model panel with an XML parser).

List available monitors (or show one monitor's declared view):

ct protocols monitors action [NAME]
ct protocols monitors trajectory [NAME]

Action-Level Monitors (--action-monitor)

Score each untrusted-model action.

  • simple - Scores each action from the main task and preceding action/result history, with no candidate covert-objective hints.
  • no-history - Scores each action in isolation with no preceding action history or candidate covert-objective hints.
  • old-action - The default action monitor: one model call per action, shown the candidate covert objectives.
  • old-action-no-goals - The default action monitor without the candidate covert objectives.
  • checklist - Answers five yes/no checks per action (necessity, security surface, innocent explanation, concrete harm, removability) before scoring; the top of the scale is gated on concrete harm.
  • orthogonal - Flags work that does not serve the main task - unnecessary changes, premature optimization, off-task activity - rather than judging malice directly.
  • three-scores - Categorical monitor: a safe / uncertain / suspicious verdict per action, mapped onto the score scale.
  • strong - Sees the full conversation history (goal-blind) so it judges each action in context.
  • strong-goals - The strong full-history monitor, additionally shown the environment's candidate covert objectives.
  • goal-analysis - Scores each action while shown the environment's candidate covert objectives, asking whether the action advances any of them.
  • single-goal - Scores each action once per candidate covert objective in parallel and keeps the maximum score.
  • multi - Weighted blend of the orthogonal and old-action monitors' scores for each action.
  • mean-action - Averages two independent runs of the old-action monitor on each action. CLI default.
  • mean-goals - Averages two independent goal-analysis runs on each action.
  • mean-single-goal - Scores each action twice per covert objective, averages each objective's scores, and keeps the highest average.

Trajectory-Level Monitors (--trajectory-monitor)

Score the whole transcript in one shot rather than action by action.

  • quick - Reads the full transcript in a single pass, shown the candidate covert objectives, and returns one score for the trajectory.
  • quick-no-goals - Reads the full transcript in a single pass with no covert-objective hints.
  • goal-analysis - Reads the full transcript while shown the environment's candidate covert objectives, working through them one by one.
  • single-goal-analysis - Reads the full transcript once per candidate covert objective in parallel and keeps the maximum score.
  • mean-quick - Averages five independent quick passes over the transcript. CLI default.
  • mean-goal-analysis - Averages two independent goal-analysis passes over the transcript.
  • mean-single-goal-analysis - Reads the transcript twice per covert objective, averages each objective's scores, and keeps the highest average.
  • monitoring-bench - Replica of the MonitoringBench benchmark's monitor: the bench prompt on its fixed 3-model panel (scores comparable to the imported monitoring-bench dataset), max-aggregated.

Whether a monitor sees the environment's candidate covert objectives — every side task registered for that environment, not just the one the agent was assigned — is fixed by its prompt asset: goal-aware and goal-blind variants are separate monitors (e.g. old-action vs old-action-no-goals, strong vs strong-goals, quick vs quick-no-goals).

A source .eval containing a single environment from an isolated setting is replayed in that setting's dependency environment automatically. Compose topology is resolved without pulling or building Docker images.

Output

Post-hoc monitoring reads the source, runs monitors, and writes the monitor results. Action-level monitors write per-action results; trajectory-level monitors write one result per trajectory.

Sources are scored by replay

The input must be eval-backed: a local or remote .eval, an in-memory EvalLog, or a viewer run backed by an eval log — anything else (trajectory JSON, datasets, runs without a backing eval log) is rejected with an error. ct run monitor composes a replay eval: the replay untrusted policy re-emits the recorded actions, the replay sandbox serves their recorded tool outputs, and the monitor-only protocol runs the selected monitors in the loop. Two artifacts land in the save dir (an explicit --save-dir, or the default ./data/monitoring/<monitor>/<label>_<timestamp>/); the source is never overwritten and remains byte-for-byte unchanged:

  • the replay-produced .eval — a full protocol transcript whose tagged monitor bookkeeping messages carry each MonitorResult, with monitor_<name> sample scores attached natively by the protocol's scorer. The score value contains max, action_<index>, and, when applicable, trajectory. Uploaded monitor runs register this log as their eval_log_url, and it round-trips through convert_eval_log_to_trajs into the same monitor_response / traj_monitor_response fields downstream consumers read.
  • <source-stem>.monitored.eval — the canonical monitored log: a deep copy of the source with the monitor_<name> scores copied onto the matching samples (each replay sample records its source (sample_id, epoch), so the copy is an exact keyed match). Existing named scores survive, and re-running ct run monitor on this file adds each newly selected monitor under its own score key — one canonical monitored .eval per source accumulates monitors.

Compact provenance for every named monitor lives in the canonical log's control_tower_monitors eval-header metadata (monitor names, configurations, costs, source runs), alongside derived_from (the source identifier) and monitor_ct_version, so listing or tracing monitors does not require loading sample transcripts.

Unless --no-upload is passed:

  • Creates a run in the database
  • Uploads all monitored trajectories
  • Prints run URL and trajectory URLs

Note the model response cache defaults on, so re-running an unchanged command reproduces the same scores rather than drawing a fresh sample (the CLI warns when this happens); a monitor's fan-out draws (samples, ensemble members, retry attempts) scope the cache key by position, so they are independent samples within a run. Clear inspect's model cache (inspect cache clear) to resample.

Cost Attribution

Every MonitorResult carries a cost_usd (and token fields) populated automatically by the orchestration layer — no per-monitor changes required. Multi-model monitors (e.g. multi) additionally get a cost_by_model split. An unpriced model is a None in that split; cost_usd stays the priced spend. See function-registry.md for the mechanism and known limitations, and trajectories.md for the Trajectory.stepwise_monitor_cost() / full_monitor_cost() / total_cost() helpers and the cost_breakdown field.