Fine-Tuning Qwen2-VL on AMD MI300X: The ROCm Pitfalls Docs Skip
A development log on fine-tuning Qwen2-VL for blockchain graph classification on AMD MI300X, detailing real ROCm setup and debugging friction.
Blockchain transaction graphs carry topology signals — like a hub node surrounded by dozens of short-lived wallets — that read as generic tables in text but appear instantly as recognizable shapes once rendered as images. That observation drives the choice of a vision-language model over a graph neural network: Qwen2-VL's native dynamic-resolution tokenization handles high-resolution transaction graphs without downsampling, and it ships with open weights plus working fine-tuning paths through LLaMA-Factory and ms-swift.
The real friction isn't the hardware, it's the software ecosystem. Nearly every fine-tuning guide assumes CUDA, so running the same workflow on AMD MI300X with ROCm 6.x means building flash-attn from source against ROCm (the CUDA-built pip wheel installs cleanly but silently corrupts gradients), and getting environment variables like HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, and HSA_OVERRIDE_GFX_VERSION to agree — otherwise ops fall back to CPU without any error, loss curves still look plausible, and you only notice when throughput lands 20x below expectations.
MI300X's 192GB HBM3 pool reframes what's feasible on a single node: full-parameter fine-tuning of a 7B model like Qwen2-VL fits comfortably under 80GB, shifting the real bottleneck from memory capacity to keeping compute units saturated. But that headroom doesn't remove the need for careful environment verification — subtle ROCm mismatches can corrupt a run in ways that never surface as a crash.