Self-Host Your Own LLMs with Docker Compose

This piece walks through a minimal Docker Compose setup for running a private LLM assistant locally, aimed at engineers who want to keep personal data, private notes, and half-formed ideas off cloud services. The stack is just two containers: Ollama, which pulls and serves models over an HTTP API, and Open WebUI, which provides a polished ChatGPT-style front end. The services discover each other via Compose's internal DNS using service names, and critically, Ollama's unauthenticated API is never published to the host—only Open WebUI's port is exposed.

The article highlights GPU passthrough as the most common sticking point, explaining the deploy block's capabilities and device_ids settings and their dependency on the NVIDIA Container Toolkit; getting this right is what determines whether models run smoothly in VRAM or become unusably slow. A CPU-only variant is also covered by simply dropping the deploy block, useful for testing but impractical for daily use with larger models. Named volumes persist models and chat history across restarts, and a healthcheck gives Docker genuine readiness signals for dependent tooling.

For engineers, the takeaway is that self-hosting an LLM is no longer a weekend project but a five-minute setup with real trade-offs: near-zero marginal cost if hardware is already owned, complete data privacy since nothing leaves the local network, and competitive latency on decent GPUs. Open models are deemed more than sufficient for summarizing, drafting, and coding assistance, though frontier cloud models still retain an edge for the most demanding tasks.