Speaking of Airflow…
Today, Google Cloud is announcing the beta launch of Cloud Composer, a managed Apache Airflow service, to make workflow creation and management easy, powerful, and consistent.
Over the past 2-3 years, Airflow has gone from being one option for orchestration to being the option for orchestration. It’s now an Apache project, has hundreds of contributors, and I almost never find myself having conversations about what were previously other solutions to this problem (Luigi, Azkaban, Oozie…).
There are certainly other managed Airflow solutions, but this is the first one released by a major cloud provider to my knowledge, which means its going to be a lot cheaper than the other options out there today. Likely more reliable, too.
If you’re running Airflow and managing the infrastructure yourself, this deserves a look.