Slack Replaces SSH with REST APIs in EMR Data Pipelines
Slack removed SSH dependency from EMR clusters, migrating 700+ data jobs to a REST-based architecture using YARN Distributed Shell and Quarry.
By 2024, Slack's data platform had grown around 700+ SSH-based operators that let Airflow trigger jobs directly on EMR clusters, a pattern dating back to 2017. While it powered critical pipelines like daily search indexing and analytics, it created a large attack surface, complex key and permission management, and operational headaches such as broken connections after Kubernetes pod restarts, orphaned 'zombie' jobs, and no reliable way to check job status. This SSH dependency also blocked further infrastructure work, including Spark on Kubernetes and EMR on EKS.
While Spark and Hive jobs could move to existing REST interfaces like Livy and HiveServer2, MapReduce jobs and hundreds of CLI-based scripts running arbitrary shell commands had no equivalent. The breakthrough came from YARN Distributed Shell, a lesser-known YARN feature that runs any shell script inside a managed YARN container using the same standard REST APIs, with proper resource allocation, retries, and logging, and without requiring a custom security layer.
Slack combined this with Quarry, its internal REST-based job submission gateway for EMR/YARN, Trino, and Snowflake, to migrate all 700+ jobs across eight data regions with zero downtime. For engineers, the takeaway is how replacing a fragile, connection-based execution model with a server-managed, observable REST architecture removed a major blocker for broader infrastructure modernization.