At Aledade, we perform ETL on the healthcare data of millions of patients from thousands of different sources, and the primary tool we leverage is the workflow management tool Airflow. Because the amount of data we process is growing exponentially, we have quickly outgrown the ability to scale our dockerized Airflow deploy horizontally. We decided to move Airflow into Kubernetes to take advantage of their native support for scaling pods up and down, as needed, to handle tasks. With zero experience running a Kubernetes cluster, EKS allowed us to get up and running rapidly. Here is how we did it.

Read more

LEAVE A REPLY

Please enter your comment!
Please enter your name here