gui-tool: A Dependency-Free Rust CLI for Desktop GUI Automation

gui-tool is a cross-platform, dependency-free Rust CLI that lets AI agents (Claude Code, Codex, Gemini CLI, and similar) interact with desktop applications without guessing pixel coordinates. Its core idea is to overlay a labeled grid with crosshairs at each cell's center on a screenshot; an agent simply names a cell (optionally zooming into a sub-grid, e.g. C7.F3) and the click lands precisely on that point.

Crucially, the tool never reads the OS accessibility tree — it works purely from what's rendered on screen. That means it can drive games, canvas/WebGL apps, custom-drawn UIs like Flutter, remote desktop/VNC sessions, and any application where accessibility data is missing or wrong. Everything, including the PNG encoder, D-Bus client, JSON output and DEFLATE implementation, is hand-written against raw OS APIs (CoreGraphics, user32.dll, D-Bus) with no external crates, compiling to roughly 9,100 lines of Rust in a single small binary.

It also runs natively on GNOME/Wayland via XDG Desktop Portals and the window-calls extension, working where tools like xdotool and pyautogui typically fail. Every command — screenshots, window management, mouse and keyboard control — returns structured JSON, and a bundled Agent Skills-compliant definition makes integration into agent workflows straightforward.

For engineers, the tool matters because it automates GUI-only software with no CLI or API, gives agents a way to visually debug applications by reproducing and screenshotting issues, and reaches UI elements that accessibility trees can't represent. Its zero-dependency, auditable codebase and direct OS API usage also make it notable from a security and portability standpoint.