Yeah, I can see why that comment would be confusing - the datacenters are indeed huge.
The ‘advancement’ Sutskever is referring to is, in the long term, the performance improvement that comes with new computer hardware. Future hardware will be faster than hardware is today, so if the primary obstacle to AI improvement is hardware limitations then it’s just a matter of time.
But Sutskever is probably also referring to something particular to neural network processing right now, which is that existing chip architectures - mainly discrete CPUs and GPUs designed to work well on a wide range of historically significant work loads - are a poor fit for NN processing and the near future introduction of NN specific accelerators is going to provide dramatic increases in the capacity for training NNs over the next several years.
The first generation of NN specific silicon is just arriving on the scene and it’s giving 10x improvements just for the most low hanging of the potential improvements. So while von neumann style computation capability has been improving about 30% per year for the last couple of decades, we could see 100% or 150% annual improvement in NN training specific hardware over the next decade or so. These anticipated capacity improvements are likely to have a big impact on what is doable with NNs even if there weren’t further progress at the software end.
Sutskever is trying to make the case to a skeptical audience that big advances in AI are likely in the next several years, and that the advances could be big enough that we get qualitatively different systems in the near future.