I think he was saying:
By the time the new chip is ready, that will be 1.5 years which implies 5x growth if we assume 3x per year. So; by the time OpenBrain is ready to build the next datacenter, we’re in middle/late 2026 instead of beginning of 26.
Aside from that, the idea that investment will scale proportionally seems like a huge leap of faith. If the next training run does not deliver the goods there is no way softbank et al. pour in 100B.
I have a question that I didn’t see anyone ask, but I don’t frequent this site enough to know if it was mentioned somewhere.
Are we sure there will be a 2-OOMs-bigger training run at all?
After the disappointment that was GPT 4.5, will investors give them the 100B (according to TFA) they need for that? In general I’d like to see more discussion about the financial side of the AGI Race. How will OpenBrain get the funding to train Agent-4?
I’ve been looking for markets on manifold to bet this and I couldn’t find a good one. I would bet we don’t get a 2-OOMs-Bigger model until at least 2027 by which chip costs come down enough. My prediction would be OpenBrain etc would focus on fine tuning / wrappers / products / ux /ui etc. for the next 2-3 years.
If I understand correctly (I very well might not), A “one bit LLM” has to be trained as a “one bit LLM” in order to then run inference on it as a “one bit LLM”. I.e this isn’t a new Quantization scheme.
So I think training and inference are tied together here, meaning; if this replicates, works, etc. we will probably have new hardware for both stages
Which no one does anymore because it doesn’t work...