RK3576 NPU on Mainline Linux: Single-Task Works, Multi-Task Stalls

Rockchip's RK3576 packs a 6 TOPS NPU whose only fully open software path runs through the mainline drivers/accel/rocket driver and Mesa's Teflon delegate — byte-exact on the sibling RK3588 chip but producing all-zero output on RK3576. Root-causing traced the fault to an int8 requantization bug: the rescale multiplier was emitted at Q14 when the hardware expects Q4, a 2^10 over-scale that saturated every output. With the fix in place, a single int8 convolution is now byte-exact against a CPU reference.

The deeper finding is that any job with more than one task only computes correctly for the first task issued after the NPU power domain resumes. Every subsequent task engages its compute units, DMAs real input, and advances the sequencer — yet its multiply-accumulate result never commits, confirmed for int8 (fp16 behavior is inferred from mechanism, not yet measured). A falsification-first methodology systematically ruled out dispatch grammar, the full register write set, configuration geometry, buffer/cache/IOMMU state, clock/power sequencing, and firmware as causes, even though the vendor's closed-source stack chains dozens of tasks per session on the same kernel without issue.

The best-supported explanation points to an on-chip CBUF -> CSC -> CMAC consume-arm resource tied to the power session, one with no register footprint and invisible to black-box probing. An independent RK3568 bring-up hits the same engaged-but-empty wall, while a separate open-stack RK3588 effort runs multi-op fp16 matmul and LLM prefill through the same driver, cross-confirming five of the findings. The team has published the full investigation ledger, two upstreamable bugs, and the conditions that would reopen the case.