Stop Fine-Tuning Everything: A Framework for Model Adaptation

Teams adapting foundation models often default straight to fine-tuning without questioning whether it's the right move. Fine-tuning is just one of six adaptation strategies — spanning prompt engineering, in-context learning, RAG, fine-tuning, domain-specific pre-training, and training from scratch — and picking the wrong one can cost weeks of engineering time and significant compute for little accuracy gain.

The framework works through a sequence of questions: Can prompting alone hit your accuracy target? Do you have 5-50 high-quality examples for few-shot/ICL? Is the model failing because it lacks information (RAG) or because it misuses information it already has (fine-tuning)? Is your domain close to the model's pre-training distribution? If you have large volumes of unlabeled domain data, self-supervised domain pre-training followed by fine-tuning on a small labeled set often outperforms fine-tuning alone; training from scratch remains a last resort.

The steps teams most often skip are distinguishing knowledge gaps from behavior gaps, and checking domain distance before committing to fine-tuning. While parameter-efficient methods like LoRA and QLoRA make fine-tuning far cheaper, RAG better serves frequently changing knowledge, and domain-specific pre-training pays off when labeled data is scarce and the domain diverges sharply from the base model's training distribution.