I love this post—this is my favorite thing I’ve read for a little while. It talks about the Quickbooks data engineering team’s tools and process for monitoring their data pipelines, from ingestion to transformation to serving. Data quality is often the single biggest time suck for a data team, and too few teams have automated tools to monitor it.
I think their concept of a circuit breaker is interesting, but I’m not sold on it. Instead, the thing that is so fascinating is all the work they’ve done prior to that point. Their thousands of jobs all collect both operational profiling and data profiling metrics, and yours probably should as well.
We’re still early in the modern data engineering game. Most data engineers are focused on making pipelines work at all, and haven’t yet had the luxury of building robust tooling for monitoring. The industry as a whole will get there, but it will take time. This is a topic that I plan on following closely.