How Netflix Outgrew Casspactor for Cassandra-to-Iceberg Data Movement

Netflix discovered that Casspactor, its in-house engine moving roughly 1,200 jobs and 3 PB of data daily from Cassandra to Apache Iceberg, had hit fundamental limits. The system relied on multiple independent services to determine backup completeness, leading to stale metadata, fragile synchronization, and cluster-wide movement failures triggered by a single node replacement. Higher-level abstractions like Key Value and Time Series inherited Casspactor's constraints, including out-of-memory failures on skewed partitions, lack of data-model awareness, bloated intermediate tables, and no ability to time-travel across backups.

To address this, Netflix built a new layered architecture that reads metadata directly from backup files in S3, eliminating the fragile multi-service dependency chain. Its foundation, the Cassandra Analytics Wrapper, converts raw backup data into standard Spark DataFrames, while a new 'Connector Factory' model lets each data abstraction build its own model-aware, optimized connector on top.

The result is an engine that handles skewed partitions efficiently at the Spark executor level, removes the need for intermediate Iceberg tables, auto-sizes jobs based on table characteristics, and relies on a single source of truth for backup metadata. It's a practical case study in replacing a monolithic connector with a shared, reusable foundation for building purpose-built data movement pipelines at scale.