» Tag
data-engineering
29 postsDuckDB Parquet Paging: file_row_number vs OFFSET, and a Correctness Trap
DuckDB Parquet paging compared: file_row_number vs OFFSET performance, plus a hidden data-correctness risk in parallel scans.
78,000-Tweet LLM Study Debunks the Trump-Tweet Market-Mover Myth
A 78,000-tweet LLM study tests whether Trump's posts move markets — and uncovers seven statistical bugs before landing on a null result.
Building a Deal Finder Bot from Catawiki's Expert Estimate Data
Catawiki quietly embeds expert price estimates in its NEXT_DATA JSON on every lot page. Here's how that discovery became a live deal-finding Apify Actor for auction bidders.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comThe New Physics of Columnar Storage: Parquet, Lance, Vortex, Nimble
How AI workloads and modern hardware are reshaping columnar file formats, from Parquet's renovation to new entrants Lance, Vortex, and Nimble.
How Airbnb rebuilt its data architecture for a multi-product world
Airbnb explains how its data teams built a consistent yet flexible data architecture using core principles and modeling guidelines for Homes, Experiences and Services.
Netflix's Data Canary: Validating Catalog Metadata at Scale
After an outage caused by corrupted data rather than code, Netflix built a production-traffic data canary system that detects catalog issues within 10 minutes.
Lakehouse Table Formats 2026: Iceberg, Delta, Hudi, Paimon, DuckLake
A 2026 technical comparison of lakehouse table formats: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, their architecture, state, and roadmap.
data-eng-bench: Snowflake's dbt benchmark for coding agents
Snowflake's data-eng-bench tests coding agents on 103 realistic dbt data-engineering tasks across DuckDB and Snowflake, via the open-source Harbor framework.
Meta Rebuilds BLOB Storage to Cut GPU Stalls in AI Training
Meta redesigned its BLOB-storage architecture with unified metadata and direct-to-client streaming to eliminate GPU stalls in AI training.
data2prompt: CLI Tool Fits Data-Heavy Projects into LLM Context Windows
data2prompt CLI profiles and samples CSVs, Excel files, and SQLite databases to fit data-heavy projects into any LLM context window.