Six Hidden Bugs Found in a Vertex AI and WIF Setup on EC2
A detailed account of six distinct bugs hit while deploying Gemini via Vertex AI with Workload Identity Federation on AWS EC2, and how each was diagnosed.
This post walks through six distinct bugs encountered while deploying a keyless Gemini/Vertex AI setup on AWS EC2 using Workload Identity Federation, in the order they appeared. Most initially looked like variations of the same 401 Unauthorized error, and realizing they were six unrelated problems rather than one stubborn bug turned out to be the real challenge. The bugs ranged from a PM2 filename typo that silently caused environment variables never to load, to a bare, unbranded HTML 401 page that turned out to come not from Google but from AWS's EC2 Instance Metadata Service (IMDS) a collision caused by AWS and Google Cloud sharing the same link-local metadata address.
Further issues included a missing IMDSv2 session-token field in the WIF credential_source config, a service account name that had drifted from what the docs assumed, and a WIF setup that never actually used the impersonation pattern common in tutorials. At each step the author isolates the root cause with concrete commands like curl tests, pm2 env, and gcloud iam service-accounts list.
The key engineering takeaway is that when an error's shape doesn't match what a given service should return, that mismatch is itself the clue pointing to a different layer of infrastructure. Shared conventions between cloud providers, like the metadata IP address, can make debugging deceptively confusing, and security features such as IMDSv2 or service account impersonation fail silently if not configured explicitly.