Interesting argumentation and viewpoints! You argue mainly from a technical and regulatory perspective and address the size of the incentive to automate from an economics perspective. One addition that will matter a lot and that you have not covered is the about the cost of automating.
A job, a project, or an autonomous lab is a bundle of tasks, be it drafting a proposal (e.g. as a .md file), getting physical and digital resources such as chemical fluids and cups, doing the experimentation, and then analyzing the results and putting them into a comprehensive form for either humans or other autonomous labs to use as documentation. What matters to answer your question is not only the technical feasibility of each of those components, and not only whether they obey the regulatory conditions, but also whether each one is economically attractive to do, because if the cost of the AI system doing a task is higher than the wage of the human doing it, the incentive to automate may not be there even where the technology works. This point has also been studied in e.g. computer vision and finds that the share of what is technically feasibly is often not cost-effective to automate (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4700751). Therefore I think the key and open question becomes what the cost of an existing human-run setup is against a fully autonomous one.
P.S.: I also wonder what the upper limit of the exponential of autonomous research will be. Is there a ceiling and if yes, might growth be limited by the speed of human adoption capability? Already today we see that adoption curves trail capability improvement curves. For instance, older generations are still very hesitant to use LLMs and might never use it, regardless how well the LLM performs.
I think you’re thinking about “automation” in a very different way from me. Copying from something I wrote here:
…what exactly is our “AGI Jeff Bezos” supposed to be doing at any given time?
Nobody knows!
Indeed, the fact that nobody knows is the whole point! That’s the very reason that an AGI Jeff Bezos can create so much value!
When Human Jeff Bezos started Amazon in 1994, he was obviously not handed a detailed spec for what to do in any possible situation, where following that spec would lead to the creation of a wildly successful e-commerce / cloud computing / streaming / advertising / logistics / smart speaker / Hollywood studio / etc. business. For example, in 1994, nobody, not Jeff Bezos himself, nor anyone else on Earth, knew how to run a modern cloud computing business, because indeed the very idea of “modern cloud computing business” didn’t exist yet! That business model only came to exist when Jeff Bezos (and his employees) invented it, years later.
By the same token, on any given random future day…
Our AGI Jeff Bezos will be trying to perform a task that we can’t currently imagine, using ideas and methods that don’t currently exist.
It will have an intuitive sense of what constitutes success (on this micro-task) that it learned from extensive idiosyncratic local experience, intuitions that a human would need years to replicate.
The micro-task will advance some long-term plan that neither we nor even the AGI can yet dream of.
This will be happening in the context of a broader world that may be radically different from what it is now.
And our AGI Jeff Bezos (along with other AGIs around the world) will be making these kinds of decisions at a scale and speed that makes it laughably unrealistic for humans to be keeping tabs on whether these decisions are good or bad.
Thus, if you automate all the tasks that Jeff Bezos has ever done, then you still have not automated Jeff Bezos. Indeed, I would say that you haven’t yet even begun to automate Jeff Bezos. Because the key value of Jeff Bezos is his ability to figure out how to do new things that nobody has ever done before.
For further discussion, see also: What do I mean by “Artificial General Intelligence”? E.g.: “…Many copies of one human brain design, barely changed since the African savannah, built the global economy from scratch … If you want a human to do [a new task], you don’t need to do R&D to breed a new subspecies of human!”
Interesting argumentation and viewpoints! You argue mainly from a technical and regulatory perspective and address the size of the incentive to automate from an economics perspective. One addition that will matter a lot and that you have not covered is the about the cost of automating.
A job, a project, or an autonomous lab is a bundle of tasks, be it drafting a proposal (e.g. as a .md file), getting physical and digital resources such as chemical fluids and cups, doing the experimentation, and then analyzing the results and putting them into a comprehensive form for either humans or other autonomous labs to use as documentation. What matters to answer your question is not only the technical feasibility of each of those components, and not only whether they obey the regulatory conditions, but also whether each one is economically attractive to do, because if the cost of the AI system doing a task is higher than the wage of the human doing it, the incentive to automate may not be there even where the technology works. This point has also been studied in e.g. computer vision and finds that the share of what is technically feasibly is often not cost-effective to automate (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4700751). Therefore I think the key and open question becomes what the cost of an existing human-run setup is against a fully autonomous one.
P.S.: I also wonder what the upper limit of the exponential of autonomous research will be. Is there a ceiling and if yes, might growth be limited by the speed of human adoption capability? Already today we see that adoption curves trail capability improvement curves. For instance, older generations are still very hesitant to use LLMs and might never use it, regardless how well the LLM performs.
I think you’re thinking about “automation” in a very different way from me. Copying from something I wrote here:
Thus, if you automate all the tasks that Jeff Bezos has ever done, then you still have not automated Jeff Bezos. Indeed, I would say that you haven’t yet even begun to automate Jeff Bezos. Because the key value of Jeff Bezos is his ability to figure out how to do new things that nobody has ever done before.
For further discussion, see also: What do I mean by “Artificial General Intelligence”? E.g.: “…Many copies of one human brain design, barely changed since the African savannah, built the global economy from scratch … If you want a human to do [a new task], you don’t need to do R&D to breed a new subspecies of human!”