Hazshield AI: I Killed the Database, the Alarms Survived
A Rust-based industrial gateway processed 10,000 sensor readings per second while its database was deliberately killed mid-test, ending with zero lost alarms and zero request errors.
This second devrazzi entry follows the build of hazshield-ingest, a single Rust binary running on a tightly constrained 2-vCPU, 512MB VM that ingests readings from 3,000 industrial sensors. The gateway validates and classifies each reading in microseconds, guaranteeing delivery for critical alarms through a dedicated 'hot lane' while routine telemetry is shed under load through a separate 'warm lane.' The author first stress-tests this design with a malfunctioning sensor flooding the system, then runs a full simulated methane leak across thousands of virtual sensors to generate a genuine cascading emergency.
The centerpiece of the post is a deliberate database kill mid-load-test, performed while alarms are actively firing. The result: zero request errors, uninterrupted critical alarm delivery, and an automatic degradation into sampled storage mode with data explicitly flagged as incomplete. Once the database returns, buffered data is replayed and the system silently resumes normal operation. Rather than relying on dashboards, the author validates correctness through an exact numerical audit, matching every generated alarm against the stream that received it.
The piece argues that graceful degradation must be engineered in advance rather than improvised during an incident, and that testing safety-critical systems requires making data loss provably countable. It closes by previewing the next phase, where these alarm streams will feed a locally hosted LLM tasked with planning physical isolation responses.