Community LLM Fine-Tunes Often Perform Worse Than Base Models, Contamination-Controlled Audit Finds
A systematic audit of 150 matched base-model and community fine-tune pairs across 164 models found that fine-tunes published on HuggingFace often underperform their base models once benchmark contamination is controlled. The study tested grade-school math (GSM8K), general knowledge (MMLU), and instruction following (IFEval) using both standard published questions and freshly generated, never-published equivalents, exposing “benchmaxxing” where higher scores came from leaked training data rather than real improvement.
High-performance tooling also stood out. The new Rust crate Csveee parses CSV files in parallel across cores and uses speculative parsing to avoid pre-scanning chunk boundaries, reaching a claimed geometric-mean 7.7 GB/s across 1,000 Kaggle files compared with 0.8 GB/s for rust-csv, and up to 192 GB/s in benchmarks. Separately, WebLLM brings in-browser LLM inference to browsers via WebGPU with OpenAI API compatibility.
Several stories highlighted reliability gaps in AI systems. An audit of ambient AI clinical-note scribes found LLM judges verify added or altered facts but miss omissions — the dominant error class — across 500 single-error note pairs. A local coding-agent experiment using Ollama, Qwen3.8 27B, and the pi CLI failed 13 times over 70 minutes because the agent assumed a 32,768-token context window while the server used a smaller limit. A token-level audit of an autonomous SDLC pipeline that generated an 861,601-line application found 77 user stories consumed 595.7 million tokens and cost $837.53, or about $10.88 per story.
Elsewhere, Perplexity’s “best software” answers were found to cite many low-authority SEO domains — 59.8% of 7,534 citations pointed to domains ranked worse than 100,000 on Tranco, with wifitalents.com, worldmetrics.org, gitnux.org, and zipdo.co appearing as a coordinated operation. PostgreSQL 19 adds system views for lock, recovery, autovacuum, and DSM registry statistics, and a community port now brings Compose Multiplatform to Apple tvOS. Finally, William Kent’s 1982 guide to the five normal forms remains a clear, plain-language reference for relational database design.
» Statistics
- Posts
- 22
- Reads
- 0
- Avg. score
- 7.7
» Most read
- Digital Sovereignty Is Written in PHP
- Audit Finds Community LLM Fine-Tunes Often Perform Worse, Not Better
- Latest BGP Hijack Targets Hosting Software Vendor Softaculous
- I Gave an On-Device LLM a Search Tool, It Ignored It and Fabricated Data
- Co-Designing AI Models Using Speculative Decoding
- What an AI-Built User Story Really Costs: A Token-Level Audit
- LLM Judges Verify Presence, Not Absence: Blind Spot in AI Clinical Notes
- Study: Perplexity Grounds 'Best Software' Answers in 215,128 SEO Pages
- WebLLM: High-Performance In-Browser LLM Inference Engine
- Csveee: Rust CSV Parser Hits 192 GB/s With Speculative Parsing
» Top scored
- Audit Finds Community LLM Fine-Tunes Often Perform Worse, Not Better
- Csveee: Rust CSV Parser Hits 192 GB/s With Speculative Parsing
- LLM Judges Verify Presence, Not Absence: Blind Spot in AI Clinical Notes
- Compose Multiplatform Now Available for Apple tvOS
- When a Local Coding Agent Hits a Context Wall: 13 Failures in 70 Minutes
- Study: Perplexity Grounds 'Best Software' Answers in 215,128 SEO Pages
- What an AI-Built User Story Really Costs: A Token-Level Audit
- William Kent's Classic Guide to the Five Normal Forms (1982)
- WebLLM: High-Performance In-Browser LLM Inference Engine
- New System Views in PostgreSQL 19