AI Agents Take on Embedded RTOS Vulnerability Research

A security researcher used OpenAI's Codex coding harness (gpt-5.6-sol/gpt-5.5, xhigh thinking mode) to run largely autonomous vulnerability research against a Netgear CG3700B cable modem's eCos/BFC-based MIPS firmware. Unlike browser or OS targets, real-time operating systems lack modern exploit mitigations, but present a different challenge: agents have far less stable tooling for navigating firmware and debugging live devices.

The setup combines Trail of Bits' skills marketplace (code review, architecture context-building, false-positive verification), a ghidra-rpc integration for scripted reverse engineering, and a custom 'ecos-offensive-research' skill set distilled from the researcher's own prior eCos work. The agent renames functions and variables via ghidra-rpc, cross-references a BSIM database of standard eCos functions, and performs live debugging on the physical device using a custom RAM-resident GDB stub built by the agent itself in an earlier session.

For engineers, the notable part is the operational architecture: Codex runs sandboxed inside a NixOS VM with serial console and LAN access to the target, launched with approval/sandbox bypasses to work independently for extended periods. The case demonstrates that agentic workflows can meaningfully accelerate embedded systems reverse engineering — traditionally a slow, manual-tooling-constrained domain — including building novel debugging infrastructure from scratch.