Solo Dev Built a Sharded Go Library with WAL Instead of Kafka

A solo developer building a social platform needed a way to group high-throughput events by key and batch them into database writes without pulling in heavyweight infrastructure like Kafka or Redis. The result is Flux, an in-process Go library that shards keys across N buffers to minimize lock contention, with each shard holding its own lock and batches flushing on either a size or time threshold.

Flux offers an optional write-ahead log with three durability tiers—SyncAlways, SyncPeriodically, and SyncOS—letting developers trade off throughput against durability guarantees. Deletes are handled via O(1) tombstones with lazy compaction rather than rewriting the log. Benchmarks show throughput scaling from about 5.9 million to 11.5 million operations per second as shard count increases, while WAL write speed ranges from roughly 4,200 to 1.25 million writes per second depending on the sync mode chosen.

The project reportedly passes race-detector tests, concurrent stress tests, and crash-recovery scenarios verifying write-then-replay behavior. It's already running in production for use cases like vote counters, user reputation scores, and keeping multiple databases eventually consistent. The author notes the implementation was heavily AI-assisted, though the architectural decisions and trade-offs were their own, and the code is open source on GitHub for community review.