I posted some of my thoughts on their website, might aswell share it on here:
What I don’t understand is why they would need as much inference compute as a human? Maybe future architecture will make it way more inference cheaply compared to the human brain. And I don’t know how you can compare the human amount of inference and the amount of inference ai needs in order to automate remote work. Also, sample efficiency doesn’t apply to inference since weights are not updated at this stage (yet, some research suggest we should do that) and maybe it’ll end up more sparse (like transformers), in the sense that we can reduce the amount of compute we need for that. I also think you exaggerate how compute bound we are. Suppose we invent a new architecture, new paradigm or just tweak the transformer which makes it so much more sample efficient and also cheaper compute wise, we can just use these automated researchers to make the next generation even more cheaper, and they could be used explicitly at OpenAI or other research labs to speed up ai research, there isn’t really broad deployment needed.
I posted some of my thoughts on their website, might aswell share it on here:
What I don’t understand is why they would need as much inference compute as a human? Maybe future architecture will make it way more inference cheaply compared to the human brain. And I don’t know how you can compare the human amount of inference and the amount of inference ai needs in order to automate remote work. Also, sample efficiency doesn’t apply to inference since weights are not updated at this stage (yet, some research suggest we should do that) and maybe it’ll end up more sparse (like transformers), in the sense that we can reduce the amount of compute we need for that. I also think you exaggerate how compute bound we are. Suppose we invent a new architecture, new paradigm or just tweak the transformer which makes it so much more sample efficient and also cheaper compute wise, we can just use these automated researchers to make the next generation even more cheaper, and they could be used explicitly at OpenAI or other research labs to speed up ai research, there isn’t really broad deployment needed.