I'm really annoyed by this. Not because I think Ilya is wrong, but because I think he is right. Because for years I think his current statement is right but for the same years I think his previous statement was wrong.
My stance hasn't changed, his has.
There's a big problem in that we reward those who hype, not merit. When the "era of scaling" happened there was a split. Those that claimed "Scale is all you need" and those that claimed "Scale is not enough". The former won, and I even seem to remember a bunch of people with T-shirts at NeurIPS with "scale is all you need" around that time.
So then, why are we again rewarding those same people when they change tunes? Their bet lost, sorry. I'm happy we tried scale and I'm glad we made progress, but at the same time many of us have been working outside the SIAYN paradigm and we struggled to get papers through review[0]. Scaling efforts led to lots of publications and citations, but you got far less by working outside that domain. And FFS, the reason most of you know Gary Marcus is because he was a vocal opposition to SIAYN and had enough initial clout. So as this tune is changing does the money shift towards us? Of course not.
I don't care about being vindicated, I care about trying to do research[1]. I don't care about the money, I care about trying to make AGI. Even Sutton has said that the Bitter Lesson was not about SIAYN!
So why I'm annoyed is that it seems we're going to let those who made big claims and fell short rather than those who correctly predicted the result. Why do we reward those who chase hype more than we reward those who got it right?
[0] a common criticism being "but does it work at scale?" or "needs more experiments". While these critiques/questions are legitimate they are out of place. Let us publish the small scale results first so that we can evidence our requests for more scale. Do you expect us to start at large scale first?
[1] I'm anonymous here, I don't care about the internet points. For the sake of this comment I might as well be any one of those Debbie Downers who pushed back against SIAYN and talked about the limits and how we shouldn't put all our eggs in one basket. There's thousands of us
I think you are getting caught in marketing semantics. The scale is all you need movement was mostly a way to funnel money to LLMs. Did Ilya ever actually believe transformers with no other improvements would lead to AGI, or did he believe that it would lead to a much more useful AI and wanted to raise money for that but found it hard without claiming it would lead to AGI? At the end of the day it is probably a good thing that so much money went into scaling recently, because it did work, as long as your measure of success is more nuanced than “did it lead to AGI”. And even then it may lead to AGI as the amount of money spent on ai research is much higher now and that new money may be what is actually needed.
My issue isn't that they are wrong, my issue is with rewarding those who are wrong. But your argument is it's fine to reward those who lie? I'm not sure how this isn't worse.
I'm sure that you're right that many people used it as a vehicle rather than being just true believers (I know some people that do), but there were also a lot of true believers.
The movement also stopped a lot of research. It has also resulted in a lot of money being dumped into companies betting on it being true. If we are in fact in a bubble (and it looks this way) then all that damage is on the hands of the SIAYN crowd.
Not being a true believer makes it better, it makes it worse. A lie is far worse than being wrong. Being wrong isn't a big issue, especially in the world of research. But lying is a major issue. It ruins it for everyone. We don't have to do this cycle of boom and bust to get things done. That's literally destructive
The issue as I see it is that google invented transformers way before they were released publicly. Clearly there were not enough resources being spent on them which is why the scale movement came about. Would google still be hoarding transformer based LLMs today without Ilya’s hype? Seems like a real possibility to me.
No one was releasing a product based on the paper though. Ilya had to go and raise a bunch of money for that to happen. Maybe I’m just more cynical and accepting of lying as the way things are done compared to you.
My stance hasn't changed, his has.
There's a big problem in that we reward those who hype, not merit. When the "era of scaling" happened there was a split. Those that claimed "Scale is all you need" and those that claimed "Scale is not enough". The former won, and I even seem to remember a bunch of people with T-shirts at NeurIPS with "scale is all you need" around that time.
So then, why are we again rewarding those same people when they change tunes? Their bet lost, sorry. I'm happy we tried scale and I'm glad we made progress, but at the same time many of us have been working outside the SIAYN paradigm and we struggled to get papers through review[0]. Scaling efforts led to lots of publications and citations, but you got far less by working outside that domain. And FFS, the reason most of you know Gary Marcus is because he was a vocal opposition to SIAYN and had enough initial clout. So as this tune is changing does the money shift towards us? Of course not.
I don't care about being vindicated, I care about trying to do research[1]. I don't care about the money, I care about trying to make AGI. Even Sutton has said that the Bitter Lesson was not about SIAYN!
So why I'm annoyed is that it seems we're going to let those who made big claims and fell short rather than those who correctly predicted the result. Why do we reward those who chase hype more than we reward those who got it right?
[0] a common criticism being "but does it work at scale?" or "needs more experiments". While these critiques/questions are legitimate they are out of place. Let us publish the small scale results first so that we can evidence our requests for more scale. Do you expect us to start at large scale first?
[1] I'm anonymous here, I don't care about the internet points. For the sake of this comment I might as well be any one of those Debbie Downers who pushed back against SIAYN and talked about the limits and how we shouldn't put all our eggs in one basket. There's thousands of us