I’ve been thinking a lot about this topic recently. The modern data stack is fantastic
, and for those of us who have literally decades of experience in data, it turns us into superhumans. But it actually can create a significant barrier to entry to getting many questions answered for many people. From my just-written post, Analytics Engineering for Everyone
, on the topic:
In a prior generation of data, a single person could acquire all of the relevant skills to become an expert: both a domain area expert as well as an expert in the relevant data technology. Data wasn’t that big or diverse, and typically people acquired the necessary data skills (Excel, simple SQL, SAS) as their problems required. The data was small in size and scope, but the process worked fine. Just grab a CSV and get to work.
This broke down with the advent of the modern data stack. Today, the possibilities for analysis have grown dramatically, but the breadth and depth of skills required has also grown. This means that it’s no longer enough for a single domain expert to sit down at a computer and geek out for a while to get an answer. Getting answers to questions in the modern data ecosystem requires teams working together in lockstep, using a well-thought-out workflow, with tooling built to enable this collaboration.
Modern data tech, especially the hyperscale cloud data warehouse, has unlocked so much potential
. And we as an industry haven’t come even close to figuring out the tooling, workflows, and skillsets required to actually digest this change. This makes me think a lot about a recent Ben Thompson
post where he makes the point that car companies
stopped seeing a lot of innovation after 1920, but that the car
continued to drive massive shifts in society throughout the entire 20th century.
It’s completely possible that the race to build a lot of the core technology in the modern data stack is already over! But we are at the very beginning of seeing the changes that will drive in our organizations. That topic—how organizations will adapt in response to modern data tech—is on my brain these days.