If you’re a data scientist at a large company, or your datasets primarily come to you in CSVs that you store locally on your hard drive, you may not have much of a relationship with anyone calling themselves a data engineer. Give you a Jupyter or RStudio notebook and you’re good to go. So why is there such a fuss about this new role?
Data engineers are particularly critical because of the data environments that have become commonplace at technology companies. It is the complexity involved in the environment of, say, Uber or Facebook that makes data engineering so critical.
This article goes into depth on exactly what such a complex data environment looks like and does an excellent job discussing how data engineers create solutions in environments like these.