When AI Reviewers Disagree: A Multi-Agent Code Review Tribunal

ShiftLeft Society, built for the Qwen Cloud Global AI Hackathon 2026, tackles a core flaw in existing AI code review tools: a single AI voice rendering judgment with no mechanism to challenge it. The system pits a security-focused and a performance-focused AI reviewer against each other in a real negotiation, where each agent picks a categorical stance—DEFEND, PARTIAL, or CONCEDE—while deterministic Python code, not the LLM, computes the budget cost and resulting severity. This keeps the model out of arithmetic entirely, making every verdict auditable and reproducible regardless of temperature settings.

The tribunal also tracks each agent's credibility over time: past negotiation outcomes are Bayesian-smoothed, capped at ±15 budget points, and persisted across every PR review, so agents start future negotiations with an adjusted budget based on their track record. On the infrastructure side, Qwen-Max runs through DashScope's OpenAI-compatible endpoint orchestrated by LangGraph, with FastMCP providing real security tooling, and FastAPI plus SQLite deployed via Docker and Caddy on an Alibaba Cloud ECS instance in Singapore.

Across a 40-case benchmark, the single-agent baseline scored 82.5% accuracy while the multi-agent tribunal reached 95%, with the biggest gain coming from eliminating false positives on six safe-code samples. The developer also reported that Qwen-Max's OpenAI-compatible API integrated cleanly, structured output succeeded roughly 99% of the time, and a more powerful reasoning variant (qwen3.7-max) was ultimately impractical due to latency exceeding webhook timeout limits.

For engineers, the project offers a concrete pattern for taming non-determinism in multi-agent LLM systems: restrict the model to bounded categorical decisions and let deterministic code handle consequences—a design choice that proves critical for auditability and reliability in production settings.