Kimi K3 benchmark: Chinese AI is cheap at coding, not planning
Benchmark testing shows Chinese AI models are pricey planners but 19x cheaper coders than Western rivals, based on real pass/fail test runs.
Moonshot's new Kimi K3 flagship went into a hard-gated benchmark this week: build a real Supabase CRM database, and nothing counts unless the test suite passes green. The result splits the 'cheap Chinese AI' narrative right down the middle.
As a planner — the role that reads a spec and breaks it into tasks — Kimi K3 turns out pricier than expected, listing above Western flagships because hidden 'reasoning' tokens bill at the full output rate. GLM-5.2 is cheaper but its plans are thin, yielding fewer passing tests. DeepSeek V4-Pro is a respectable Chinese pick but loses on value to Grok-4.5.
As a doer — the model that actually types the code — the story flips entirely: DeepSeek's lightweight model passes the same tests at roughly one-nineteenth the cost of its rival. The practical takeaway for engineers: separate the two roles, measure cost per passing test rather than cost per token, and smoke-test your own tooling before trusting any leaderboard, since some top-ranked models return empty output through common CLIs.