For what it’s worth, while I do think this will matter 5-10 years down the line, or more years, I am currently relatively skeptical that this will be achieved soon, and my current view is that a lot of these papers will have gotchas that make them less useful than they think.
The HRM paper has a gotcha pretty much immediately:
For what it’s worth, while I do think this will matter 5-10 years down the line, or more years, I am currently relatively skeptical that this will be achieved soon, and my current view is that a lot of these papers will have gotchas that make them less useful than they think.
The HRM paper has a gotcha pretty much immediately:
https://www.lesswrong.com/posts/tEZa7PouYatK78bbb/?commentId=ELTcESCdWjikCq3HT
That said, worth keeping an eye out here.
I’d put my probability on a new architecture surpassing transformers within 2 years to be much closer to 1%.