It’s not often Jeff Dean puts out new work. This paper is brand new, from a talk at a recent conference, and is jam-packed with interesting stuff if you care about the intersection of chip design and ML. I was particularly interested in the section Machine-Learning-Specialized Hardware, which was the best overview of the differences between a classic microprocessor and a TPU that I’ve read.
The size of some of the recently-released language models are intense. This is problematic for two reasons:
First, it hinders democratization. If we believe in a world where millions of engineers are going to use deep learning to make every application and device better, we won’t get there with massive models that take large amounts of time and money to train.
Second, it restricts scale. There are probably less than 100 million processors in every public and private cloud in the world. But there are already 3 billion mobile phones, 12 billion IoT devices, and 150 billion micro-controllers out there. In the long term, it’s these small, low power devices that will consume the most deep learning, and massive models simply won’t be an option.
This is the best post I’ve read on model efficiency. It goes deep in certain tactical areas but remains extremely accessible at all points.
My feelings on this report by the Brookings Institute: ¯\_(ツ)_/¯
The main takeaway is that white-collar jobs are likely to be more impacted than blue-collar jobs from the widespread deployment of AI, and they got there via a bunch of NLP work using a couple of different datasets. Here’s the big problem with the analysis though:
…the exposure measure employed here only suggests that in particular occupations some kind of impact can be expected, whether positive or negative.
The report is just saying that certain fields are more “AI-exposed” than others. For instance, software engineers are listed as being very highly AI-exposed. That seems quite obvious, given that software engineers literally…build AI systems. Other top areas listed also fall under “I could have just told you that without needing to do a bunch of language NLP”.
I include this link here because it is going certainly made the rounds in the last couple of weeks, worth a scan just to have the water cooler conversation.