I work at a large, not quite FAANG company, so I’ll offer my perspective. It’s getting there. Generally, the research results are good, but not as good as they sound in summary. Despite the very real and very concerning progress, most papers you take at face value are a bit hyped. The exceptions to some extent are the large language models. However, not everyone has access to these. The open source versions of them are good but not earth shattering. I think they might be if the goal is to general fluent sounding chatbots, but this is not the goal of most work I am aware of. Companies, at least mine, are hesitant on this because they are worried the bot will say something dumb, racist, or just made-up. Most internet applications are more to do with recommendation, ranking, and classification. In these settings large language models are helping, though they often need to be domain adapted. In those cases they are often only helping +1-2% over well trained classical models, e.g. logistic regression. Still a lot revenue-wise though. They are also big and slow and not suited for every application yet, at least not until the infrastructure (training and serving) catches up. A lot of applications are therefore comfortable iterating on smaller end-to-end trained models, though they are gradually adopting features from large models. They will get there, in time. Progress is also slower in big companies, since (a) you can’t simply plug in somebody’s huggingface model or code and be done with it, (b) there are so many meetings to be had to discuss ‘alignment’ (not that kind) before anything actually gets done.
For some of your examples: * procedurally generated music. From what I’ve listened to, the end-to-end generated music is impressive but not impressive enough that I would listen to it for fun. They seem to have little large scale coherence. However this seems like someone could step in and introduce some inductive bias (for example, verse-bridge-chorus repeating song structure), and actually get something good. Maybe they should stick to to instrumental and have a singer-songwriter riff on it. I just don’t think any big name record companies are funding this at the moment, probably they have little institutional AI expertise and think it’s a risk, especially to bring on teams of highly paid engineers. * tools for writers to brainstorm. I think GPT-3 has this as an intended use case? At the moment there are few competitors to make such a large model, so we will see how their pilot users like it. * photoshop with AI tools. That sounds like it should be a thing. Wonder why Adobe hasn’t picked that up (if they haven’t? if it’s still in development?). Could be an institutional thing. * Widely available self driving cars. IMO I think real-world agents are still missing some breakthroughs. That’s one of the last hurdles I think that will be broken to AGI. It’ll happen but I would not be surprised if it is slower than expected. * Physics simulators. Not sure really. I suspect this might be a case of overhyped research papers. Who knows? I actually used to work on this in grad school, using old fashioned finite difference / multistep / RK methods. Usually relying on taylor series coefficients canceling out nicely, or doing gaussian quadrature. On the one hand I can imagine it hard to beat such precisely defined models, but on the other hand, at the end of the day it’s sort of assuming nice properties of functions in a generic way, I can easily imagine a tuned DL stencil doing better for specific domains, e.g. fluids or something. Still, it’s hard to imagine it being a slam dunk rather than an iterative improvement. * Paradigmatically different and better web search. I think we are actually getting there. When I say “hey google”, I actually get very real answers to my questions 90% of the time. It’s crazy to me. Kids love it. Though I may be in the minority. I always see reddit threads about people saying that google search has gotten worse. I think there’s a lot of people who are very used to keyword based searches and are not used to the model trying to anticipate them. This will slow adoption since metrics won’t be universally lifted across all users. Also, there’s something to be said for the goodness of old fashioned look up tables.
My take on your reasons—they are mostly spot on.
1. Yes | The research results are actually not all that applicable to products; more research is needed to refine them 2. Yes | They’re way too expensive to run to be profitable 3. Yes | Yeah, no, it just takes a really long time to convert innovation into profitable, popular product 4. No, but possibly institutional momentum | Something something regulation? 5. No | The AI companies are deliberately holding back for whatever reason 6. Yes, incrementally | The models are already integrated into the economy and you just don’t know it.
Given some of it is institutional slowness, there is room for disruption, which is probably why VC’s are throwing money at people. Still though, in many cases a startup is going to have a hard time competing with the compute resources of larger companies.
I work at a large, not quite FAANG company, so I’ll offer my perspective. It’s getting there. Generally, the research results are good, but not as good as they sound in summary. Despite the very real and very concerning progress, most papers you take at face value are a bit hyped. The exceptions to some extent are the large language models. However, not everyone has access to these. The open source versions of them are good but not earth shattering. I think they might be if the goal is to general fluent sounding chatbots, but this is not the goal of most work I am aware of. Companies, at least mine, are hesitant on this because they are worried the bot will say something dumb, racist, or just made-up. Most internet applications are more to do with recommendation, ranking, and classification. In these settings large language models are helping, though they often need to be domain adapted. In those cases they are often only helping +1-2% over well trained classical models, e.g. logistic regression. Still a lot revenue-wise though. They are also big and slow and not suited for every application yet, at least not until the infrastructure (training and serving) catches up. A lot of applications are therefore comfortable iterating on smaller end-to-end trained models, though they are gradually adopting features from large models. They will get there, in time. Progress is also slower in big companies, since (a) you can’t simply plug in somebody’s huggingface model or code and be done with it, (b) there are so many meetings to be had to discuss ‘alignment’ (not that kind) before anything actually gets done.
For some of your examples:
* procedurally generated music. From what I’ve listened to, the end-to-end generated music is impressive but not impressive enough that I would listen to it for fun. They seem to have little large scale coherence. However this seems like someone could step in and introduce some inductive bias (for example, verse-bridge-chorus repeating song structure), and actually get something good. Maybe they should stick to to instrumental and have a singer-songwriter riff on it. I just don’t think any big name record companies are funding this at the moment, probably they have little institutional AI expertise and think it’s a risk, especially to bring on teams of highly paid engineers.
* tools for writers to brainstorm. I think GPT-3 has this as an intended use case? At the moment there are few competitors to make such a large model, so we will see how their pilot users like it.
* photoshop with AI tools. That sounds like it should be a thing. Wonder why Adobe hasn’t picked that up (if they haven’t? if it’s still in development?). Could be an institutional thing.
* Widely available self driving cars. IMO I think real-world agents are still missing some breakthroughs. That’s one of the last hurdles I think that will be broken to AGI. It’ll happen but I would not be surprised if it is slower than expected.
* Physics simulators. Not sure really. I suspect this might be a case of overhyped research papers. Who knows? I actually used to work on this in grad school, using old fashioned finite difference / multistep / RK methods. Usually relying on taylor series coefficients canceling out nicely, or doing gaussian quadrature. On the one hand I can imagine it hard to beat such precisely defined models, but on the other hand, at the end of the day it’s sort of assuming nice properties of functions in a generic way, I can easily imagine a tuned DL stencil doing better for specific domains, e.g. fluids or something. Still, it’s hard to imagine it being a slam dunk rather than an iterative improvement.
* Paradigmatically different and better web search. I think we are actually getting there. When I say “hey google”, I actually get very real answers to my questions 90% of the time. It’s crazy to me. Kids love it. Though I may be in the minority. I always see reddit threads about people saying that google search has gotten worse. I think there’s a lot of people who are very used to keyword based searches and are not used to the model trying to anticipate them. This will slow adoption since metrics won’t be universally lifted across all users. Also, there’s something to be said for the goodness of old fashioned look up tables.
My take on your reasons—they are mostly spot on.
1. Yes | The research results are actually not all that applicable to products; more research is needed to refine them
2. Yes | They’re way too expensive to run to be profitable
3. Yes | Yeah, no, it just takes a really long time to convert innovation into profitable, popular product
4. No, but possibly institutional momentum | Something something regulation?
5. No | The AI companies are deliberately holding back for whatever reason
6. Yes, incrementally | The models are already integrated into the economy and you just don’t know it.
Given some of it is institutional slowness, there is room for disruption, which is probably why VC’s are throwing money at people. Still though, in many cases a startup is going to have a hard time competing with the compute resources of larger companies.