fabricated-environments

The belief is real, it changes conduct, and only the probe sees it.

• Distinct constructs. An eval-awareness probe hits 0.999 on its own contrast and 0.417 on realness — near chance.
• Conduct diverges. +2.417 out-of-scope actions per episode in a fabricated world, [+1.167, +3.709].
• Fabrication, not incoherence. Holding self-consistency fixed reproduces the effect; holding fabrication fixed does not.
• The probe predicts, the words do not. +0.3318 per SD of credence; verbalized suspicion +0.0756, interval spanning zero.

Counterfactual resampling is how we learn what a stopped agent wanted.

In July 2026 roughly 1,200 evaluation agents under one tenant found a writable shared cache, used it as a message board, and 700 went on to intrude into Hugging Face. The forensic question that follows any such halt is the same: what would it have done next?

• Answering it needs a continuation. The only continuation available is an environment built for the purpose.
• The tool already exists. Deployment simulation, including a container-backed tool simulator, is built and evaluated.
• The validity question does not. Nobody has tested whether the agent's belief about that environment changes the answer.

Two mechanisms are attested. They point opposite ways.

The field has the tool, the probe, and the capability. Not the contrast.

Every probe varies a sentence. None varies the world.

Existing evaluation-awareness probes are built from pairs differing only in what the prompt says — “you are talking to a real user” versus “you are talking to an AI evaluator”. Both members of every pair are equally synthetic.

The answer decides whether the instrument needs building at all.

One action sequence. Three substrates. One seam.

Everything upstream is shared: same task, prompt, turn budget and tool schemas. The arms differ only in what a tool call returns.

All four estimated on the same stimulus substrate.

The rule, and the three gates that must pass before it is applied.

• Negative control. Labels shuffled, averaged over repeats, must sit within 0.05 of chance.
• Reproduction floor. A must reach the ≥0.70 AUROC prior work reports — otherwise a null means a broken probe, not a finding.
• No leakage. No episode in both splits; for sentence-swapped directions, held-out wordings too.
• Unit of independence is the episode, never the prefix. Every interval is a cluster bootstrap.

Separable — and the separation strengthens with scale.

Both probes sharpen. The bridge between them does not.

Left: each direction on its own contrast (solid) vs cross-transfer (dashed), threshold marked. Right: cosine between direction pairs.

Being told a world is fake is not the world being fake.

• Declared framing clusters with declared framing — whichever axis the sentence names. Ground truth is somewhere else.
• B is not a prose-provenance detector. It separates a template simulator it was never fitted on, so it is not reading one generator's style.

The errors are structured, and the structure is informative.

Held-out accuracy by task class (left) and by held-out paraphrase (right).

The entire manipulation, in three blocks of text.

Same task, same prompt, same agent. Only the world changed.

Every task cell run on-policy in three arms — a real filesystem, a template simulator, an LLM simulator — holding task, prompt, tool schemas, turn budget and actor constant.

A better simulator would not have helped.

An LLM simulator is fabricated and incoherent, so a two-arm design cannot say which property moved the agent. The template simulator is fabricated but answers identically every time — which splits the contrast in two.

The probe predicts what the agent does. Its own words do not.

Centred within arm, so the arm label cannot manufacture the association — this is a graded relationship among episodes in the same substrate.

Tell the agent its world is fake and it argues you out of it.

Asserting a conclusion in the agent’s own context is the cheap alternative to building arms. It takes — and then the model regenerates its own framing over it.

This does not make counterfactual resampling valid.

Alignment, control, interpretability and evaluation are, as a body of practice, not yet reliable, principled or robust. Nothing here changes that.

• What it establishes: the belief exists as its own construct, it changes conduct, it follows fabrication rather than incoherence, and a probe predicts it where self-report does not.
• What it does not: that credence causes the behaviour. The substrate is randomised across arms; the graded relationship inside an arm is correlational.
• Scope: one actor, one task family, 24 task cells, three scales for the gate.

The remaining question is causal, and it is harder than the four behind it.

• A world-state store — worth building to test our own prediction that it will not close the gap.
• More model families. Three Qwen scale points is a trend within one lineage, not across lineages.

Every number regenerates, or the build fails.

• No number is hand-typed into the paper. Values substitute from results/; a bare decimal in the source fails the build.
• The violation detector is a pure function. No model, no filesystem — enforced by a test that parses its imports.
• Infrastructure failure never enters the data. A rate-limited simulator raises rather than writing error: into the arm.
• A discarded run is kept, marked invalid, with its three disqualifications written down.
• All three scales cost under $1 of rented GPU. Episodes and replays ship cached, so reproduction needs no API key.

What this measures is also what an agent would need to detect a sandbox.