Flawed LLM Benchmark, New Spectre Bypass, and Sparsity Attacks Expose Critical Evaluation and Security Gaps
A domain-expert audit of the widely used SciCode coding benchmark uncovered 263 defects, with 192 spread across 91% of main problems causing correct, instruction-following solutions to be marked as failures. The findings explain why state‑of‑the‑art language models had plateaued around 60% subproblem accuracy—the stagnation lay in the test itself, not model capability. The audit, which examined all 65 SciCode problems, underscores how flawed benchmarks can mask true progress in AI coding.
Two new security exploits surfaced. The TONTOU CPU attack bypasses recently deployed Spectre v2 mitigations, successfully leaking sensitive data—including Linux passwords—by exploiting gaps in neutralization-based defenses on both AMD and Intel processors. Meanwhile, SparSEEty demonstrates a side-channel token extraction attack against sparse‑LLM serving systems. By constructing a neuron‑activation oracle from input‑dependent weight accesses, attackers can reconstruct prompt and response tokens with BLEU scores exceeding 0.95 across multiple models and datasets, raising serious privacy concerns for inference deployments.
On the performance front, a pull request to llama.cpp introduced an x86 VNNI implementation for the Q2_0 dot product, yielding a 3.0–3.6× throughput improvement on an AMD EPYC 9645 across Bonsai models ranging from 1.7B to 27B parameters. However, the VNNI path may be unavailable on Intel’s 12th to 14th generation CPUs. In the datacenter, Google analysis revealed that up to 97% of bytes in active memory pages are cold—a problem created by size‑based allocation interleaving hot and cold objects. OBASE tackles this with a compiler‑runtime system that separates objects by access pattern, dramatically reducing DRAM overprovisioning. And recent WebAssembly runtime benchmarks show Wasmer leading overall performance, with WAVM excelling in optimization, while new wide arithmetic instructions promise to accelerate cryptographic workloads.
Several developer tools addressed long‑standing friction points. A deep dive into iOS standalone PWAs dissected why the WebKit‑ignored interactive‑widget directive and 100dvh fail to keep a message composer above the keyboard; the investigation provides accurate viewport measurement techniques that bypass naive detection logic. Elsewhere, the posix‑regex library delivers POSIX‑compliant leftmost‑longest matching to JavaScript, Doubletake enables AirPlay 2 screen mirroring from Linux to Apple TV with hardware‑accelerated encoding, and the TutorMoments benchmark—built from real math tutoring sessions—found that large language models consistently over‑scaffold students, highlighting a gap in pedagogical judgment that current prompting alone cannot close.
» Statistics
- Posts
- 20
- Reads
- 0
- Avg. score
- 7.6
» Most read
- Concurrency vs. Throughput: Why More Parallelism Can Slow Down Databases
- TutorMoments: Testing If AI Tutors Know When to Help or Hold Back
- LLM-Generated Tests Influence Implementation Selection Outcomes
- How Netflix Developed a Real-Time Distributed Graph for Efficient Querying
- Doubletake – AirPlay/TV Screen Mirroring Sender for Linux
- Llama.cpp PR Boosts Q2_0 Performance 3.0–3.6x on x86 CPUs
- When AI Automation Measures the Wrong Thing
- Benchmarking Cheap LLMs for Production Agent Traces
- Informing HTTP Extension Design with Data
- Show HN: posix-regex – POSIX Standard Regex Engine for JavaScript
» Top scored
- SciCode-Verified: Flawed Benchmark Masked True LLM Coding Skill
- iOS standalone PWA: keeping a chat composer above the keyboard
- TutorMoments: Testing If AI Tutors Know When to Help or Hold Back
- OBASE: Object-Aware Address-Space Design to Fix Memory Tiering
- Show HN: posix-regex – POSIX Standard Regex Engine for JavaScript
- Doubletake – AirPlay/TV Screen Mirroring Sender for Linux
- Llama.cpp PR Boosts Q2_0 Performance 3.0–3.6x on x86 CPUs
- Performance of WebAssembly Runtimes in 2026
- New TONTOU CPU Attack Bypasses Spectre v2 Fixes, Leaks Linux Passwords
- SparSEEty: Extracting Tokens from Sparsity-Exploiting LLM Serving Systems