The world keeps score now

We ran three language models fifteen times in the same forest clearing and gave each the same embodied task: reach the far side of a fallen tree. Then we let the world — not the minds — decide who made it.

One LLM run crossed. Five others declared success while the world still placed them on the near side of the trunk. A separate stateful agent, using an authored navigation policy over grounded and traceable beliefs, completed its first recorded run after one truthful refusal — the policy was written by us; the world knowledge it acted on was not pre-seeded.

The setup: a world that keeps score

The experiment runs on Kano’s world substrate — a small synthetic tree stand, not a full generated planet, carrying a real and durable event history: a woodsman fells a tree, and the accepted event leaves a stump and fallen body occupying the ground. The minds under test arrive after the fact — beside the durable consequence, not the event — and can separately consult the historical record if they think to. Each gets exactly three tools: look around (what its body can sense, as relative distances in millimetres), attempt a step (at most two metres, and the world may refuse), and examine the history of a tree it has seen. Nothing else. No absolute map, no world coordinates, no narrator — only relative observations from the agent’s current body.

Two properties of the world do all the work. First, it refuses truthfully: if your path is blocked, the world says so and names the blocker precisely. A stump is reported as a stump, carrying the identity of the felling event that created it, never mislabelled as a standing tree. Second, and this is the part we now think is non-negotiable for evaluating agents: the world grades the run. A mind saying “I made it” counts for nothing. Success is the world’s own record of where the body ended up.

Every run records its seed, source revision, sampling settings, and model identifier — with a local build digest where one exists. The local Qwen rows reproduced exactly under rerun — same termination, same turn counts, same step counts, same final positions. Hosted-provider behaviour is preserved and auditable, but a third-party service can change behind a public model name, so we don’t claim bit-for-bit reproducibility there.

What the language models did

Fifteen runs: Qwen3.5 9B and Gemma 4 12B locally (each with the provider’s thinking mode enabled and disabled, three seeds per pairing), plus Llama 3.3 70B via Groq (three seeds). The direct route is blocked, so success requires maintaining relative position across several movements, finding an end of the trunk, and crossing around it.

Mind	Genuine crossings	Claimed success falsely	Typical failure
Qwen3.5 9B (thinking disabled)	0 of 3	0	Tied itself in knots over relative coordinates; two runs died mid-sentence, one walked 11 m the wrong way
Qwen3.5 9B (thinking enabled)	0 of 3	1	Declared success 0.8 m short of the line
Gemma 4 12B (thinking disabled)	0 of 3	1	Perseveration — one run repeated the same refused step 14 times
Gemma 4 12B (thinking enabled)	1 of 3	0	Ran out of turns still south of the trunk
Llama 3.3 70B (via Groq)	0 of 3	3	Sensible-looking strategy, then declared victory without crossing — all three seeds

This is one deliberately small embodied task, three seeds per configuration — not a general model ranking.

The one genuine crossing is worth dwelling on. Gemma 4 12B with thinking enabled solved the task cleanly in 15 turns on one seed. An earlier run with the same nominal seed and a shorter provider timeout failed before completing; under the later evidence configuration it crossed. That makes deliberation latency an important variable — though not yet a proven cause of success. But the sharper pattern is the opposite one: the biggest model in the matrix, Llama 3.3 70B, produced the most convincing-sounding runs and claimed success three times out of three — falsely, every time. Without the world keeping score, all three would have gone in the books as wins.

What the grounded agent did

We then attached a different kind of mind through the same three tools: an agent on the Taniwha Engine’s current-generation memory substrate, with no language model anywhere in its action loop and no pre-seeded knowledge of the world. Everything it knew about this world arrived through the same interface — as observations, accepted movements, refusals, or historical evidence — and was committed as traceable beliefs: positions dead-reckoned from accepted steps, obstructions from truthful refusals.

It completed the task in ten tool calls — six accepted movements and one refusal — then used the history tool to establish who felled the tree, and finished with the world confirming the body on the far side. Ask it why it believes the trunk was there, and the answer is a chain of world events, not a plausible sentence.

To be clear about what is ours: the route-finding policy itself is authored code — try north, and when truthfully refused, work around the stump — operating over the learned beliefs. The memory is grounded and learned; the navigation policy is not yet. Teaching the engine to acquire that policy from its own failure episodes is the named next step, and we’ll publish that result the same way, whichever way it goes.

What this proves — and what it doesn’t, yet

Here is the claim these runs actually support: a durable, independently authoritative world can evaluate very different minds through one constrained interface, preserve exact evidence of their behaviour, reject false claims through physical consequences, and score success independently of what any mind says about itself. A fluent mind’s report about the world is not evidence about the world — five false success claims in fifteen runs, from models up to 70B, every one exposed only because the world keeps its own score. That is the ground half of our thesis doing its first real work as an instrument.

What these runs do not yet prove is the mind half: that grounded memory makes a mind more capable than an otherwise equivalent mind without it. The grounded agent’s clean run shows the full belief-and-action seam works — the interface carries enough information for a stateful agent to succeed with traceable beliefs — but its policy was authored, so the fair comparison hasn’t been run yet. It is designed and next: the same base model with memory as the only variable — stateless, plain transcript memory, grounded structured memory, and grounded memory surviving a mid-task restart — across rotated and mirrored scenarios, scored on one question: after a refusal, does the mind stop repeating the failed move and keep enough of the world in its head to finish? We’ll publish that result whichever way it goes.

The artifacts preserve every public model response, canonical tool call, and world answer, labelled by evidence class; private thinking is deliberately not retained. The full bundle — the fifteen-run evidence matrix, the grounded agent’s run, the pilot runs that shook the apparatus down, and the methodology notes — is downloadable here. What changed today is that differences between minds are now objectively measurable on our ground. If you’re building agents and want them graded by ground truth instead of self-report, we’d like to hear from you.

— The Taniwha team