» Tag
nlp
23 postsGigatoken: ~1000x faster tokenization than HuggingFace tokenizers
Gigatoken delivers ~1000x faster tokenization than HuggingFace tokenizers, offering GB/s throughput as a Rust-based drop-in replacement.
Cross-tokenizer distillation can silently erase 85% of teacher signal
Naive logit projection across mismatched tokenizers can erase 85% of teacher information; chain-rule conditioning restores retention to 86%.
Mechanistic View Reveals How Bias Lives Inside LLM Judges
Study shows LLM-as-judge bias is encoded in activation geometry, enabling causal steering and better failure prediction than text-based methods.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comGigatoken: Rust BPE Tokenizer Hits 24.53 GB/s, Up to 989x Faster
Rust-based Gigatoken hits 24.53 GB/s BPE tokenization using SWAR and dual-cursor techniques, vastly outpacing HuggingFace tokenizers and tiktoken.
pdfmuse: A Deterministic PDF Parser That Failed on Real Resumes
pdfmuse's deterministic PDF parser looked solid until real resumes exposed a silent 10% failure rate from unhandled form XObjects.
Why AI Models Perform Worse Outside English
Why do LLMs underperform outside English? Training data, tokenizer inefficiency, and instruction-tuning gaps explain the cost and accuracy divide.
StoryScope Reveals the Narrative Fingerprints of AI Fiction
StoryScope focuses on narrative structure, not style, detecting AI fiction with 93% accuracy and revealing model-specific fingerprints.
Open, structurally chunked EU AI Act corpus released for legal RAG
An open, 933-chunk SQLite dataset splits the EU AI Act by its real legal structure, with embeddings and EUR-Lex links for RAG and legal engineering.
Bioinformatics meets prompt injection defense: the Smith-Waterman trick
An open-source technique adapts the 1981 Smith-Waterman DNA alignment algorithm to catch paraphrased prompt injections that regex and classifiers miss, boosting F1 by 34 points.
RAG Evaluation: Faithfulness, Context Recall, and Answer Relevance
RAGAs evaluation enhances AI systems' reliability by utilizing three key metrics: faithfulness, context recall, and answer relevance.