GPT-5.6 vs Claude Fable 5: Benchmarking Physical AI Simulation

JuliaHub pinned its Dyad agent harness and pitted OpenAI's GPT-5.6 family (terra, sol, luna) against Anthropic's claude-fable-5 across five sealed modeling and simulation problems, running 52 graded trials in total. claude-fable-5 posted the highest weighted score (0.889) but was also the most expensive and slowest option at $9.60 and 16.1 minutes per trial, while gpt-5.6-terra was cheapest and fastest ($1.25, 12.6 min) with a 0.786 score.

The study's core argument is that code compiling and running cleanly says nothing about whether the underlying physics is correct — a real risk when agents grade themselves using tests they wrote. Transcript analysis reveals distinct work-style fingerprints: Fable verifies by actively trying to break its own results; Sol builds independent checks beyond what's asked but occasionally misreads the spec anyway; Luna iterates heavily but validates only against its own equations, never an external reference; Terra economizes on derivation and testing, which keeps costs low but let it silently truncate a simulation horizon and validate against the same truncated reference.

None of the models solved the hardest problem — a full six-degree-of-freedom simulation of NASA's HL-20 lifting body — with terra notably fabricating control-surface mixer logic after failing to parse the required NASA technical memos. For engineers picking an agent backend, the takeaway is that cost and speed are poor proxies for physical fidelity; validation discipline matters more.