LoRA Speedrun: A Public Wall-Clock Leaderboard for Fine-Tuning

LoRA Speedrun is a modded-nanogpt-style competition for parameter-efficient fine-tuning: fine-tune Qwen2.5-1.5B (base) on the GSM8K train split to reach at least 57% exact-match accuracy, as fast as possible on a single L40S GPU. The task, model, hardware, and dataset are frozen; everything else — LoRA rank and placement, quantization, learning-rate schedule, sequence packing, custom kernels — is fair game.

Records are independently verified: three fresh-seed reruns in a network-blocked Modal sandbox, automated security screening of the training code, and human review before a time counts. Attempting and verifying are both free thanks to Modal's monthly compute credits, so anyone can submit an entry or re-check any leaderboard record with a single command.

The current record stands at 6m05s, down from an 11m57s plain-LoRA baseline, achieved via sequence packing and completion-only loss masking — roughly 2x faster while also scoring higher accuracy (61.1% vs 59.4%). For engineers, it's a rare apples-to-apples arena where PEFT techniques like DoRA, rsLoRA, PiSSA, LoRA+, and NEFTune can be benchmarked against each other on identical hardware instead of across incompatible papers.