Data Science Roundup #70: The Rise of the Data Engineer, GPUs, and Some Amazing Data Viz

This is a must-read post. Here’s the entire intro:

There is a massive amount of knowledge in this post. Read it.

If you’re a data scientist at a large company, or your datasets primarily come to you in CSVs that you store locally on your hard drive, you may not have much of a relationship with anyone calling themselves a data engineer. Give you a Jupyter or RStudio notebook and you’re good to go. So why is there such a fuss about this new role?

Data engineers are particularly critical because of the data environments that have become commonplace at technology companies. It is the complexity involved in the environment of, say, Uber or Facebook that makes data engineering so critical. 

This article goes into depth on exactly what such a complex data environment looks like and does an excellent job discussing how data engineers create solutions in environments like these.

This article is a real gem. Written by an expert in GPU technology (MapD builds a SQL database that is powered by GPUs), it goes back to basics in explaining the fundamentals of GPU-based computing. I actually hadn’t realized that the primary benefit of GPUs is their massively parallel architecture, which is something of a historical accident:

Apparently each GPU chip has thousands of cores, and as such is far superior at handling “embarrassingly parallel” operations. Like analytics.

You’re likely familiar with the core concepts behind predictive analytics—choosing your variable, choosing an algorithm, creating test and training sets, etc.—but unless you’ve actually gone through the process yourself, you may feel a bit shaky on the implementation details. If this describes where you are, this article, and its followup, were written for you. With this post and some basic R or Python skills, you have everything you need to build a simple predictive model.

I doubt you are an assembler hacker (and I’m not linking to this article because I think you should become one). But the author makes a really fascinating observation: there are elements of deep learning algorithms that are not well-optimized by modern compilers. Because of the massive datasets required by these algorithms, the performance implications of even minor optimizations become very significant.

This feels very much to me like when it was still challenging to accept credit card payments online, circa 1998—a great reminder that we are still at the very beginning of this particular era.

Fishtown Analytics works with venture-funded startups to implement Redshift, BigQuery, Mode Analytics, and Looker. Want advanced analytics without needing to hire an entire data team? Let’s chat.

Developers shouldn’t have to write ETL scripts. Consolidate your data in minutes. No API maintenance, scripting, cron jobs, or JSON wrangling required.