Very thoughtful piece, thanks. "...a self-reinforcing loop that prioritizes elite branding over actual pedagogical work" this has been somewhat partially true for academia for some time. I can feel under-currents (or should I say temptation) pulling in the direction of "Labs" in math too, though it is still quite far from the norm.
Elite branding indeed. I agree with all you say ... but haven't the last five years seen the absolute unexpected triumph of incrementalism? Incremental scaling and tweaking and tinkering with small variations of architectures, optimisers, training schedules, hyperparameters, choice of training data, types of post-training, have provided work for at least 10,000 clever people, for there are exponentially many variations to try. A 'bitter lesson' is that we cannot yet argue with the results! There are times when boring incrementality has spectacular payoff, no?
Isn't it a comedy that deep learning has developed in the opposite way from most high technology? Instead of engineers having a clear idea of what they want to achieve, and working to solve a series of well defined problems to get there (as in building a moon rocket or a nuclear power station where the underlying principles were long understood), we have the opposite process: astonishing and completely unexpected success produced by scaling up rather simple ideas, and then a mad scramble of hackery to understand why these LLMs work so well.
I remember when people complained that the field of speech recognition was an intellectual desert and that any radical new ideas were blocked, because the best models were highly trained HMMs and it required large teams to tweak and train these models, while the same teams also influenced the test competitions. Progress was measured by performance in open competitions, and even a good new idea could not beat highly trained HMMs, because of the amount of work and size of team required to train a good model. So how could any new ideas emerge?
I’m not an AI guy but is not the AI related breakthroughs already happening in industry recently? I think industry-based labs (e.g., google) is leading AI now. If that’s true, how is the research culture there? Do they suffer the same prestige flywheel?
Very thoughtful piece, thanks. "...a self-reinforcing loop that prioritizes elite branding over actual pedagogical work" this has been somewhat partially true for academia for some time. I can feel under-currents (or should I say temptation) pulling in the direction of "Labs" in math too, though it is still quite far from the norm.
Elite branding indeed. I agree with all you say ... but haven't the last five years seen the absolute unexpected triumph of incrementalism? Incremental scaling and tweaking and tinkering with small variations of architectures, optimisers, training schedules, hyperparameters, choice of training data, types of post-training, have provided work for at least 10,000 clever people, for there are exponentially many variations to try. A 'bitter lesson' is that we cannot yet argue with the results! There are times when boring incrementality has spectacular payoff, no?
Isn't it a comedy that deep learning has developed in the opposite way from most high technology? Instead of engineers having a clear idea of what they want to achieve, and working to solve a series of well defined problems to get there (as in building a moon rocket or a nuclear power station where the underlying principles were long understood), we have the opposite process: astonishing and completely unexpected success produced by scaling up rather simple ideas, and then a mad scramble of hackery to understand why these LLMs work so well.
I remember when people complained that the field of speech recognition was an intellectual desert and that any radical new ideas were blocked, because the best models were highly trained HMMs and it required large teams to tweak and train these models, while the same teams also influenced the test competitions. Progress was measured by performance in open competitions, and even a good new idea could not beat highly trained HMMs, because of the amount of work and size of team required to train a good model. So how could any new ideas emerge?
Well where are the HMM speech recognisers now?
Change will happen.
I’m not an AI guy but is not the AI related breakthroughs already happening in industry recently? I think industry-based labs (e.g., google) is leading AI now. If that’s true, how is the research culture there? Do they suffer the same prestige flywheel?
thank you