The point I was trying to make is not that there weren’t fundamental advances in the past. There were decades of advances in fundamentals that rocketed forward development at an unsustainable pace. The effect of this can be seen with sheer amount of computation that is being used for SOTA models. I don’t forsee that same leap happening twice.
phdead
Karma: 137
The summary is spot on! I would add that the compute overhang was not just due to scaling, but also due to 30 years of Moore’s law and NVidia starting to optimize their GPUs for DL workloads.
The rep range idea was to communicate that despite AlphaStar being much smaller than GPT as a model, the training costs of both were much closer due to the way AlphaStar was trained. Reading it now it does seem confusing.
I meant progress of research innovations. You are right though, from an application perspective the plethora of low hanging fruit will have a lot of positive effects on the world at large.
Out of curiosity, what is your reasoning behind believing that DL has enough momentum to reach AGI?
We have achieved Noob Gains in AI
My thoughts for each question:
Depending on context, there are a few ways I would communicate this. Take the phrase “We are quiet here.” Said to prospective tenants at an apartment complex, it is “communicating group norms.” Said to a friend who is talking during a funeral, it is “enforcing group norms”. Telling yourself you will do this before you sleep is “enforcing identity norms”. You are sharing information, just local information about the group instead of global information about the world. All the examples given are information sharing.
Believing in an opinion can be a group norm, and this can be useful or harmful. For example, “We believe victims” may not be bulletproof life advice, but groups which have that group norm often are more useful to survivors of sexual assault than groups which try to figure out if they believe the story first.
I think motivations are complex and impossible to fully vocalize. Often I don’t realize why I really want or don’t want to do something until after I’ve had the instinctual response. Its possible narratives that obscure the full depth but feel true are good temporary stand ins in these cases.
Communication and enforcement of group and identity norms happens everywhere all the time, but increases the more status is associated with the group or identity.
I am a young bushy eyed first year PhD. I imagine if you knew how much of a child of summer I was you would sneer on sheer principle, and it would be justified. I have seen a lot of people expecting eternal summer, and this is why I predict a chilly fall. Not a full winter, but a slowdown as expectations come back to reality.