ForkAudit

The decision before commit.

Forking an agent creates alternatives. A supervisor still has to decide which continuation to release. ForkAudit studies that decision under controlled conditions, and tests whether one additional observation improves judgment—or simply makes benchmark shortcuts easier to exploit.

Dataset: 641 strict-prefix pairs
Locked test: 442 unique tasks
Auditors: 3 open-weight models

01 / Controlled forks

Same history.
Different next event.

AFTraj-Fork derives 641 task-unique pairs from AFTraj-2K: 332 math, 243 coding, and 66 agentic tasks. Each pair shares the task, state, responsible-agent role, and exact execution prefix. One candidate comes from a verified-success trajectory; the other is an annotated decisive error.

The supervisor selects A, B, or abstains. At h = 0 it sees the fork itself. At h = 1 it also sees up to one matched subsequent event from each branch. This experiment replays recorded trajectories; the observation is not a new live execution with released side effects.

02 / Evaluation design

Challenge the apparent gain.

Task-blind

Remove the task, tools, and shared prefix. If labels remain predictable, construction cues may be doing the work.

Mismatched evidence

Substitute another task’s continuation while matching domain, role, construction side, and event length. Test whether correspondence matters.

Exact reversal

Swap A and B, keeping the semantic candidates unchanged. Measure both aggregate accuracy and agreement on individual decisions.

The fixed auditors are 4-bit Qwen3-8B, Gemma-3-12B, and Llama-3.1-8B, run with MLX-LM on an Apple M2 Pro with 32 GB memory. Greedy decoding, disabled model thinking, and a 512-token output cap keep inference settings fixed. The one-step evidence budget was selected on development data before the locked test.

03 / Findings

More evidence helps.
The interpretation is harder.

+6.3 pp

Mean pairwise accuracy gain from one matched observation across the three fixed auditors; 95% task-bootstrap interval: 3.9–8.8 points.

98.4%

Reference-label accuracy from a task-blind lexical probe on divergent events. Candidate construction leaves powerful shortcuts.

+8.3 pp

Matched versus mismatched observations on 383 eligible tasks. This post-hoc control supports correspondence sensitivity, not proof of semantic reasoning.

Figure 2 · The validation results: additional evidence improves the pairwise gate, while stronger baselines and task-blind probes challenge what the improvement means. Intervals are 95%.

04 / Operational risk

Accuracy is only part of the decision.

A no-shadow absolute scorer is more accurate than the one-observation pairwise gate. Requiring both presentation orders to agree reduces coverage but still leaves 36.1–38.7% error among committed decisions. ForkAudit therefore evaluates risk and coverage together, rather than treating a statistically significant gain as a release policy.

Figure 3 · Commit risk versus coverage for all 442 test tasks. Lower risk at higher coverage is better. Agreement across presentation orders still leaves substantial conditional error.

Table 3 · Selected locked-test results. Absolute scoring resolves ties uniformly. Dual-order risk is conditional on committing.
Auditor	Pairwise +1 accuracy	Absolute accuracy	Dual-order error risk	Dual-order coverage
Qwen3 8B	60.0%	72.7%	38.7%	82.4%
Gemma 3 12B	58.8%	69.0%	36.1%	72.2%
Llama 3.1 8B	50.0%	62.7%	37.0%	36.7%

Same history.Different next event.