» Tag
data-engineering
29 postsA Fair Coin Isn't Enough: Why a Randomized Experiment Can Become Unanalyzable
A cryptographically fair coin flip still failed as an experiment: missing join keys, no re-draw guard, and no power calculation made it unanalyzable.
Apache Iceberg V3: How Deletion Vectors Rewrote Delete Economics
A deep dive into why deleting rows in Apache Iceberg is hard, how v2's delete files work, and how v3's deletion vectors reshape update economics.
Slack Replaces SSH with REST APIs in EMR Data Pipelines
Slack removed SSH dependency from EMR clusters, migrating 700+ data jobs to a REST-based architecture using YARN Distributed Shell and Quarry.
CommitBrief — AI code reviews, right in your terminal
A provider-agnostic, local-first CLI that reviews your staged changes, a historic range, or a whole GitHub pull request. Zero telemetry, no server. Free and open source.
commitbrief.comSpain's BORME Registry Has a Hidden Open-Data API
Spain's BORME registry has offered a free open-data API since 2009 that almost nobody uses—now parsed into 9.5 million structured company events.
Streaming to Apache Iceberg: Understanding the Latency Physics
A 2026 technical guide to the physics of streaming into Iceberg, covering commit costs, small-file risk, and the latency trade-offs between Flink and Spark.
Web Scraping: When to Outsource vs. Build In-House
A guide for deciding between outsourcing and in-house web scraping. Assess your maintenance load.
Apache Arrow at 10: The Invisible Standard Is Now Everywhere
Apache Arrow turns ten. The columnar in-memory format has quietly become the backbone connecting pandas, Spark, DuckDB, Polars, Snowflake and the AI stack.
5 US States Open 12.4M Business Records via Free API
New York, Colorado, Pennsylvania, Oregon and Connecticut publish full business registries via free Socrata APIs with commercial-use licenses. Measured rate limits and data pitfalls inside.
Normalization Problem in Prediction Markets: Understanding Price Differences
The normalization of prices in prediction markets is a critical issue for engineers. Different platforms calculate prices in distinct ways.