Don't Let the LLM Speak, Just Probe Its Hidden State

When an LLM is asked whether a piece of text satisfies some criterion, the answer typically already exists inside its hidden state during the forward pass, well before any token is generated. Building on this, the technique skips generation entirely: it grabs the hidden state at the final prompt token — usually ending in a judgment cue like 'Assessment:' — from a mid-to-upper layer, feeds it into a small MLP or linear probe, and calibrates the output with isotonic regression to produce a genuine probability.

The key trick is in how the training data is built: thousands of (criterion, content, label) triples are generated with the criterion varied each time, so the probe learns the general pattern of 'does this content satisfy this criterion' rather than memorizing one specific task. That means any criterion phrased in English at inference time can be scored. An optional LoRA adapter is trained not to classify but to write a short justification for the known label; this text is never actually generated at inference — its sole purpose is to reshape the geometry at the seed token so the decision is more legible to the probe.

A KV-caching optimization lets one piece of content be scored against many criteria cheaply by prefilling once and running each criterion as a short continuation — but this means the content is encoded before seeing the criterion, which can hurt accuracy on complex or counterfactual phrasing, mirroring the classic cross-encoder versus late-interaction trade-off seen in systems like ColBERT.

The approach now powers a system called Predicate inside NOPE's safety stack, where running structural checks on every message of every conversation makes LLM-judge economics infeasible. For engineers, it offers a practical middle ground: richer structural judgments than embedding classifiers can provide, at a fraction of the cost and latency of full LLM-as-judge pipelines, using calibrated probabilities instead of vague scores.