Probelock: A Lockfile for LLM Tool-Calling Capabilities

Probelock introduces a 'capability lockfile' for local models: it automatically derives a battery of tool-calling and output tests from an agent's existing OpenAI-style tool schemas, scores them deterministically (JSON-schema validation, exact match, tool-name checks), and commits the result as a lockfile. In a worked example, swapping the same model from Q8_0 to Q4_K_M quantization drags capabilities like tool_selection and structured_output from a perfect 1.00 down to 0.33-0.67, and probelock's gate catches the drop and fails CI.

Unlike frameworks such as promptfoo, which require authoring test cases, probelock generates a fixed, reproducible set of probes straight from the tool definitions you already ship — no test writing involved. It also avoids LLM-graded judging entirely; because every probe is scored by code, running it twice on the same model yields identical numbers. Rather than producing an absolute leaderboard, it compares a model against its own prior baseline across a model/quant/runtime swap, sidestepping typical 'benchmarks are gameable' objections.

The tool works out of the box with OpenAI-compatible endpoints like Ollama, vLLM, llama.cpp and LM Studio, and reaches providers such as Anthropic, Gemini or Bedrock through any-llm or litellm adapters. The entire pipeline is deterministic except for the model call itself, so the same inputs always yield the same lockfile and diff. For engineers, the practical payoff is catching silent capability regressions from quantization or runtime changes before they reach production.