You Don't Need an LLM to Cluster LLM Traces: Seldon's Trace Audit
Seldon's Trace Audit clusters LLM gateway traces into reusable program groups using deterministic features and DBSCAN—no LLM needed, near-perfect accuracy.
Seldon, a project aimed at compiling repetitive LLM calls into cheaper deterministic pipelines, published results from its Trace Audit v0 system, which groups LLM-gateway traces by the underlying program they represent rather than by topic. Tested on more than 12,500 traces spanning classification, span extraction, structured extraction, entity resolution, and similarity tasks, the system achieved near-perfect pair precision, recall, and purity (1.000/0.9999/1.000) without using any LLM calls — relying instead on hashed contract features (response format, schema, output shape) and DBSCAN clustering over a short deterministic descriptor string.
Notably, inserting LLM-based summarization before clustering — tested with GPT-4.1, a model called GPT-5.6-luna, and Phi-4 — did not improve results and actually hurt clustering purity, reinforcing the core claim: semantic similarity is the wrong objective for identifying 'the same program.' Two invoice traces with different output shapes should not merge; two traces with different entities but identical JSON schemas should.
The work matters for engineers running LLM gateways at scale, where a large share of calls are effectively ETL disguised as prompts — same schema, same instructions, different payloads. By partitioning traffic into contract-compatible clusters cheaply, Seldon lays groundwork for the next step: treating each cluster as a specification for program synthesis against a typed operator library, falling back to the LLM only when the compiled pipeline underperforms.