How We Self-Host DeepSeek V4 Flash on AWS Spot Instances

A team building an AI product wanted a genuinely free tier without eating negative margins, so they chose to self-host DeepSeek V4 Flash rather than pay per-token API costs. The mixture-of-experts model has 284B total parameters but only 13B active per token, making it feasible to run on hardware a startup can afford. Its public checkpoint ships already quantized to MXFP4, shrinking disk footprint from roughly 570GB in BF16 down to 150-160GB, which pointed the team toward NVIDIA's Blackwell generation and specifically the RTX PRO 6000, available on AWS as g7e.24xlarge instances with four 96GB GPUs.

Key engineering decisions include storing model weights in same-region S3 rather than baking them into machine images, since spot instances get reclaimed frequently and each restart requires re-fetching ~150GB of weights onto fresh NVMe storage. Speculative decoding via DeepSeek's own DSpark module lets a single stream hit roughly 300 tokens/sec, with aggregate throughput climbing toward 3,000 tokens/sec under load, though each box tops out around 64 concurrent sequences.

The most consequential design choice, per the team, was V4's attention mechanism: it compresses older conversation history aggressively while keeping recent tokens at full resolution, preventing long agent sessions from exhausting GPU memory despite growing token counts. Production caps context at 256K tokens even though the model supports up to 1M. The entire setup, including inference server configuration and image tags, has been open-sourced as a reference for teams considering similar self-hosting tradeoffs.