Trendyol's agent tunes LLM serving configs for 4x speedup

Trendyol Tech engineers built autooptimizer, an autonomous agent loop that tunes LLM serving configurations for frameworks like vLLM and SGLang without human intervention. Inspired by Karpathy's autoresearch project, it follows a propose-hypothesis, edit-config, launch-server, benchmark, score, keep-or-revert cycle, with the agent strictly barred from touching the measurement harness itself — only the serving config is fair game, preventing it from gaming the benchmark instead of improving real performance.

In a real run against Gemma 4 26B-A4B-it served via vLLM with default flags, the baseline scored 167. The agent's first move—fp8 quantization, tuned batching, and memory utilization—jumped the score to 524, and after 18 total experiments it settled on a config scoring roughly 640, nearly 4x the baseline, entirely unattended. The winning setup stacked six validated changes: fp8 quantization, tuned batching/memory settings, performance mode, prefix caching disabled, block size 32, and chunked prefill disabled.

Key design principles include deterministic benchmarks (identical workload every run), a single composite score balancing throughput against latency, and a pluggable framework architecture supporting vLLM, SGLang, and planned TensorRT-LLM adapters. For engineers, this turns what's typically a day of manual flag-flipping and spreadsheet comparisons into an unattended loop that can run overnight, making production serving optimization systematic and reproducible rather than guesswork.