Netflix's Data Projects: Managing Data Assets at Scale

Netflix's data platform spans millions of tables and tens of thousands of scheduled workloads, historically managed through per-table ACLs and workflows tied to the identity of the engineer who authored them. This fine-grained model broke down as teams reorganized and people changed roles or left the company, flooding support teams with permission requests and causing workflows to fail when their owning identity became stale.

To address this, Netflix built Data Projects: an abstraction that acts both as a container grouping related tables, workflows, and assets under one logical umbrella, and as a durable synthetic identity that scheduled workloads can run under independent of any individual's employment lifecycle. Management granularity shifts from the individual asset to the project, so hundreds of tables can be governed through a single set of grants instead of hundreds of separate ACLs.

Each project supports role-based grants (Contributor, Viewer, etc.) for users, groups, applications, and CI jobs, and is provisioned with a Netflix application identity plus an optional AWS IAM role that workflow orchestrators like Maestro use to execute jobs securely. A feature called 'gravity' automatically attaches newly created assets to the project that produced them, giving teams organization as a natural side effect of execution rather than manual bookkeeping.

The approach offers a broader lesson for engineering teams operating at scale: moving identity and access management from individual assets to a meaningful project-level container reduces operational overhead and makes workflows resilient to the organizational churn that is inevitable at large companies.