Hi Paul. I’ve reflected carefully on your post. I have worked for several years on a SDC software infrastructure stack and have also spent a lot of time comparing the two situations.
Update: since commentators and downvoters demand numbers: I would say the odds of criticality are 90% by July 2033. The remaining 10% is that there is a possibility of a future AI winter (investors get too impatient) and there is the possibility that revenue from AI services will not continue to scale.
I think you’re badly wrong, again, and the consensus of experts are right, again.
First, let’s examine your definition for transformative. This may be the first major error:
(By which I mean: systems as economically impactful as a low-cost simulations of arbitrary human experts, which I think is enough to end life as we know it one way or the other.)
This is incorrect, and you’re a world class expert in this domain.
Transformative is a subclass of the problem of criticality. Criticality, as you must know, means a system produces self gain larger than it’s self losses. For AGI, there are varying stages of criticality, which each do settle on an equlibria:
Investment criticality : This means that each AI system improvement or new product announcement or report of revenue causes more financial investment into AI than the industry as a whole burned in runway over that timestep.
Equilibrium condition: either investors run out of money, globally, to invest or they perceive that each timestep the revenue gain is not worth the amount invested and choose to invest in other fields. The former equilibrium case settles on trillions of dollars into AI and a steady ramp of revenue over time, the later is an AI crash, similar to the dotcom crash of 2000.
Economic Criticality: This means each timestep, AI systems are bringing in more revenue than the sum of costs [amortized R&D, inference hardware costs, liability, regulatory compliance, …]
Equilibrium condition: growth until there is no more marginal tasks an AI system can perform cheaper than a human being. Assuming a large variety of powerful models and techniques, it means growth continues until all models and all techniques cannot enter any new niches. The reason why this criticality is not exponential, while the next ones are, is because the marginal value gain for AI services drops with scale. Notice how Microsoft charges just $30 a month for Copilot, which is obviously able to save far more than $30 worth of labor each month for the average office worker.
Physical Criticality: This means AI systems, controlling robotics, have generalized manufacturing, mining, logistics, and complex system maintenance and assembly. The majority, but not all, of labor to produce more of all of the inputs into an AI system can be produced by AI systems.
Equilibrium condition: Exponential growth until the number of human workers on earth is again rate limiting. If humans must still perform 5% of the tasks involved in the subdomain of “build things that are inputs into inference hardware, robotics”, then the equilibria is when all humans willing, able to work on earth are doing those 5% of tasks.
AGI criticality: True AGI can learn automatically to do any task that has clear and objective feedback. All tasks involved in building computer chips, robotic parts (and all lower level feeder tasks and power generation and mining and logistics) have objective and measurable feedback. Bolded because I think this is a key point and a key crux, you may not have realized this. Many of your “expert” domain tasks do not get such feedback, or the feedback is unreliable. For example an attorney who can argue 1 case in front of a jury every 6 months cannot reliably refine their policy based on win/loss because the feedback is so rare and depends on so many uncontrolled variables.
AGI may still be unable to perform as well as the best experts in many domains. This is not relevant. It only has to perform well enough for machines controlled by the AI to collect more resources/build more of themselves than their cost.
A worker pool of AI systems like this can be considerably subhuman across many domains, or rely heavily on using robotic manipulators that are each specialized for a task, being unable to control general purpose hands, relying heavily on superior precision and vision to complete tasks in a way different than how humans perform it. They can make considerable mistakes, so long as the gain is positive—miswire chip fab equipment, dropped parts in the work area cause them to flush clean entire work areas, wasting all the raw materials—etc. I am not saying the general robotic agents will be this inefficient, just that they could be.
Equilibrium Condition: exponential growth until exhaustion of usable elements in Sol. Current consensus is earth’s moon has a solid core, so all of it could potentially be mined for useful elements A large part of Mars, it’s moons, the asteroid belt, and Mercury are likely mineable. Large areas of the earth via underground tunnel and ocean floor mining. The Jovian moons. Other parts of the solar system become more speculative but this is a natural consequence of machinery able to construct more of itself.
Crux: AGI criticality seems to fall short of your requirement for “human experts” to be matched by artificial systems. Conversely, if you invert the problem: AGI cannot control robots well, creating a need for billions of technician jobs, you do not achieve criticality, you are rate limited on several dimensions. AI companies collect revenue more like consulting companies in such a world, and saturate when they cannot cheaply replace any more experts, or the remaining experts enjoy legal protection.
Requirement to achieve full AGI criticality before 2033: You would need a foundation model trained on all the human manipulation you have the licenses for the video. You would need a flexible, real time software stack, that generalizes to many kinds of robotic hardware and sensor stack. You would need an “app store” license model where thousands of companies could contribute, instead of just 3, to the general pool of AI software, made intercompatible by using a base stack. You would need there to not be hard legal roadblocks stopping progress. You would need to automatically extend a large simulation of possible robotic tasks whenever surprising inputs are seen in the real world.
Amdahl’s law applies to the above, so actually, probably this won’t happen before 2033, but one of the lesser criticalities might. We are already in the Investment criticality phase of this.
Autonomous cars: I had a lot of points here, but it’s simple: (1) an autonomous robo taxi must collect more revenue than the total costs, or it’s subcritical, which is the situation now. If it were critical, Waymo would raise as many billions as required and would be expanding into all cities in the USA and Europe at the same time. (look at a ridesharing company’s growth trajectory for a historical example of this)
(2) It’s not very efficient to develop a realtime stack just for 1 form factor of autonomous car for 1 company. Stacks need to be general.
(3) There are 2 companies allowed to contribute. Anyone not an employee of Cruise or Waymo is not contributing anything towards autonomous car progress. There’s no cross licensing, and it’s all closed source except for comma.ai. This means only a small number of people are pushing the ball forward at all, and I’m pretty sure they each work serially on an improved version of their stack. Waymo is not exploring 10 different versions of n+1 “Driver” agent using different strategies, but is putting everyone onto a single effort, which may be the wrong approach, where each mistake costs linear time. Anyone from Waymo please correct me. Cruise must be doing this as they have less money.
This is incorrect, and you’re a world class expert in this domain.
This is a rather rude response. Can you rephrase that?
All tasks involved in building computer chips, robotic parts (and all lower level feeder tasks and power generation and mining and logistics) have objective and measurable feedback. Bolded because I think this is a key point and a key crux, you may not have realized this. Many of your “expert” domain tasks do not get such feedback, or the feedback is unreliable. For example an attorney who can argue 1 case in front of a jury every 6 months cannot reliably refine their policy based on win/loss because the feedback is so rare and depends on so many uncontrolled variables.
I don’t like this point. Many expert domain tasks have vast quantities of historical data we can train evaluators on. Even if the evaluation isn’t as simple to quantify, deep learning intuitively seems it can tackle it. Humans also manage to get around the fact that evaluation may be hard to gain competitive advantages as experts of those fields. Good and bad lawyers exist. (I don’t think it’s a great example as going to trial isn’t a huge part of a most lawyers’ jobs)
Having a more objective and immediate evaluation function, if that’s what you’re saying, doesn’t seem like an obvious massive benefit. The output of this evaluation function with respect to labor output over time can still be pretty discontinuous so it may not effectively be that different than waiting 6 months between attempts to know if success happened.
An example of this is it taking a long time to build and verify whether a new chip architecture improves speeds or having to backtrack and scrap ideas.
This is a rather rude response. Can you rephrase that?
If I were to rephrase I might say something like “just like historical experts Einstein and Hinton, it’s possible to be a world class expert but still incorrect. I think that focusing on the human experts at the top of the pyramid is neglecting what would cause AI to be transformative, as automating 90% of humans matters a lot more than automating 0.1%. We are much closer to automating the 90% case because...”
I don’t like this point. Many expert domain tasks have vast quantities of historical data we can train evaluators on. Even if the evaluation isn’t as simple to quantify, deep learning intuitively seems it can tackle it. Humans also manage to get around the fact that evaluation may be hard to gain competitive advantages as experts of those fields. Good and bad lawyers exist. (I don’t think it’s a great example as going to trial isn’t a huge part of a most lawyers’ jobs)
Having a more objective and immediate evaluation function, if that’s what you’re saying, doesn’t seem like an obvious massive benefit. The output of this evaluation function with respect to labor output over time can still be pretty discontinuous so it may not effectively be that different than waiting 6 months between attempts to know if success happened.
For lawyers: the confounding variables means a robust, optimal policy is likely not possible. A court outcome depends on variables like [facts of case, age and gender and race of the plaintiff/defendant, age and gender and race of the attorneys, age and gender and race of each juror, who ends up the foreman, news articles on the case, meme climate at the time the case is argued, the judge, the law’s current interpretation, scheduling of the case, location the trial is held...]
It would be difficult to develop a robust and optimal policy with this many confounding variables. It would likely take more cases than any attorney can live long enough to argue or review.
Contrast this to chip design. Chip A, using a prior design, works. Design modification A’ is being tested. The universe objectively is analyzing design A’ and measurable parameters (max frequency, power, error rate, voltage stability) can be obtained.
The problem can also be subdivided. You can test parts of the chip, carefully exposing it to the same conditions it would see in the fully assembled chip, and can subdivide all the way to the transistor level. It is mostly path independent—it doesn’t matter what conditions the submodule saw yesterday or an hour ago, only right now. (with a few exceptions)
Delayed feedback slows convergence to an optimal policy, yes.
You cannot stop time and argue a single point to a jury, and try a different approach, and repeatedly do it until you discover the method that works. {note this does give you a hint as to how an ASI could theoretically solve this problem}
I say this generalizes to many expert tasks like [economics, law, government, psychology, social sciences, and others]. Feedback is delayed and contains many confounding variables independent of the [expert’s actions].
While all tasks involved with building [robots, compute], with the exception of tasks that fit into the above (arguing for the land and mineral permits to be granted for the ai driven gigafactories and gigamines), offer objective feedback.
the confounding variables means a robust, optimal policy is likely not possible. A court outcome depends on variables like [facts of case, age and gender and race of the plaintiff/defendant, age and gender and race of the attorneys, age and gender and race of each juror, who ends up the foreman, news articles on the case, meme climate at the time the case is argued, the judge, the law’s current interpretation, scheduling of the case, location the trial is held...]
I don’t see why there is no robust optimal policy. A robust optimal policy doesn’t have to always win. The optimal chess policy can’t win with just a king on the board. It just has to be better than any alternative to be optimal as per the definition of optimal. I agree it’s unlikely any human lawyer has an optimal policy, but this isn’t unique to legal experts.
There are confounding variables, but you could also just restate evaluation as trial win-rate (or more succinctly trial elo) instead of as a function of those variables. Likewise you can also restate chip evaluation’s confounding variables as being all the atoms and forces that contribute to the chip. The evaluation function for lawyers, and many of your examples is objective. The case gets won, lost, settled, dismissed, etc.
The only difference is it takes longer to verify generalizations are correct if we go out of distribution with a certain case. In the case of a legal-expert-AI, we can’t test hypotheses as easily. But this still may not be as long as you think. Since we will likely have jury-AI when we approach legal-expert-AI, we can probably just simulate the evaluations relatively easily (as legal-expert-AI is probably capable of predicting jury-AI). In the real world, a combination of historical data and mock trials help lawyers verify their generalizations are correct, so it wouldn’t even be that different as it is today (just much better). In addition, process based evaluation probably does decently well here, which wouldn’t need any of these more complicated simulations.
You cannot stop time and argue a single point to a jury, and try a different approach, and repeatedly do it until you discover the method that works. {note this does give you a hint as to how an ASI could theoretically solve this problem}
Maybe not, but you can conduct mock trials and look at billions of historical legal cases and draw conclusions from that (human lawyers already read a lot). You can also simulate a jury and judge directly instead of doing a mock trial. I don’t see why this won’t be good enough for both humans and an ASI. The problem has high dimensionality as you stated, with many variables mattering, but a near optimal policy can still be had by capturing a subset of features. As for chip-expert-AI, I don’t see why it will definitely converge to a globally optimal policy.
All I can see is that initially legal-expert-AI will have to put more work in creating an evaluation function and simulations. However, chip-expert-AI has its own problem where it’s almost always working out of distribution, unlike many of these other experts. I think experts in other fields won’t be that much slower than chip-expert-AI. The real difference I see here is that the theoretical limits of output of chip-expert-AI are much higher and legal-expert-AI or therapist-expert-AI will reach the end of the sigmoid much sooner.
I say this generalizes to many expert tasks like [economics, law, government, psychology, social sciences, and others]. Feedback is delayed and contains many confounding variables independent of the [expert’s actions].
Is there something significantly different between a confounding variable that can’t be controlled like scheduling and unknown governing theoretical frameworks that are only found experimentally? Both of these can still be dealt with. For the former, you may develop different policies for different schedules. For the latter, you may also intuit the governing theoretical framework.
So in this context, I was referring to criticality. AGI criticality is a self amplifying process where the amount of physical materials and capabilities increases exponentially with each doubling time. Note it is perfectly fine if humans continue to supply as inputs the network of isolated AGI instances are unable to produce. (Vs others who imagine a singleton AGI on its own. Obviously eventually the system will be rate limited by available human labor if its limited this way, but will see exponential growth until then)
I think the crux here is that all is required is for AGI to create and manufacture variants on existing technology. At no point does it need to design a chip outside of current feature sizes, at no point does any robot it designs look like anything but a variation of robots humans designed already.
This is also the crux with Paul. He says the AGI needs to be as good as the 0.1 percent human experts at the far right side of the distribution. I am saying that doesn’t matter, it is only necessary to be as good as the left 90 percent of humans. Approximately , I go over how the AGI doesn’t even need to be that good, merely good enough there is net gain.
This means you need more modalities on existing models but not necessarily more intelligence.
It is possible because there are regularities in how the tree of millions of distinct manufacturing tasks that humans do now use common strategies. It is possible because each step and substep has a testable and usually immediately measurable objective. For example : overall goal. Deploy a solar panel. Overall measurable value : power flows when sunlight available. Overall goal. Assemble a new robot of design A5. Overall measurable objective: new machinery is completing tasks with similar Psuccess. Each of these problems is neatly dividable into subtasks and most subtasks inherit the same favorable properties.
I am claiming more than 99 percent of the sub problems of “build a robot, build a working computer capable of hosting more AGI” work like this.
What robust and optimal means is that little human supervision is needed, that the robots can succeed again and again and we will have high confidence they are doing a good job because it’s so easy to measure the ground truth in ways that can’t be faked. I didn’t mean the global optimal, I know that is an NP complete problem.
I was then talking about how the problems the expert humans “solve” are nasty and it’s unlikely humans are even solving many of them at the numerical success levels humans have in manufacturing and mining and logistics, which are extremely good at policy convergence. Even the most difficult thing humans do—manufacture silicon ICs—converges on yields above 90 percent eventually.
How often do lawyers unjustly lose, economists make erroneous predictions, government officials make a bad call, psychologists fail and the patient has a bad outcome, or social science uses a theory that fails to replicate years later.
Early AGI can fail here in many ways and the delay until feedback slows down innovation. How many times do you need to wait for a jury verdict to replace lawyers with AI. For AI oncologists how long does it take to get a patient outcome of long term survival. You’re not innovating fast when you wait weeks to months and the problem is high stakes like this. Robots deploying solar panels are low stakes with a lot more freedom to innovate.
This is incorrect, and you’re a world class expert in this domain.
What’s incorrect? My view that a cheap simulation of arbitrary human experts would be enough to end life as we know it one way or the other?
(In the subsequent text it seems like you are saying that you don’t need to match human experts in every domain in order to have a transformative impact, which I agree with. I set the TAI threshold as “economic impact as large as” but believe that this impact will be achieved by systems which are in some respects weaker than human experts and in other respects stronger/faster/cheaper than humans.)
Do you think 30% is too low or too high for July 2033?
Do you think 30% is too low or too high for July 2033?
This is why I went over the definitions of criticality. Once criticality is achieved the odds drop to 0. A nuclear weapon that is prompt critical is definitely going to explode in bounded time because there are no futures where sufficient numbers of neutrons are lost to stop the next timestep releasing even more.
What’s incorrect? My view that a cheap simulation of arbitrary human experts would be enough to end life as we know it one way or the other?
Your cheap expert scenario isn’t necessarily critical. Think of how it could quench, where you simply exhaust the market for certain kinds of expert services and cannot expand to any more because of lack of objective feedback and legal barriers.
An AI system that has hit the exponential criticality phase in capability is the same situation as the nuclear weapon. It will not quench, that is not a possible outcome in any future timeline [except timelines with immediate use of nuclear weapons on the parties with this capability]
So your question becomes : what is the odds that economic or physical criticality will be reached by 2033? I have doubts myself, but fundamentally the following has to happen for robotics:
A foundation model that includes physical tasks, like this.
Sufficient backend to make mass usage across many tasks possible, and convenient licensing and usage. Right now Google and a few startups exist and have anything using this approach. Colossal scale is needed. Something like ROS 2 but a lot better.
No blocking legal barriers. This is going to require a lot of GPUs to learn from all the video in the world. Each robot in the real world needs a rack of them just for itself.
Generative physical sims. Similar to generative video, but generating 3d worlds where short ‘dream’ like segments of events happening in the physical world can be modeled. This is what you need to automatically add generality to go from 60% success rate to 99%+. Tesla has demoed some but I don’t know of good, scaled, readily licensed software that offers this.
For economics:
1. Revenue collecting AI services good enough to pay for at scale
2. Cheap enough hardware, such as from competitors to Nvidia, that make the inference hardware cheap even for powerful models
You speak with such a confident authoritative tone, but it is so hard to parse what your actual conclusions are.
You are refuting Paul’s core conclusion that there’s a “30% chance of TAI by 2033,” but your long refutation is met with: “wait, are you trying to say that you think 30% is too high or too low?” Pretty clear sign you’re not communicating yourself properly.
Even your answer to his direct follow-up question: “Do you think 30% is too low or too high for July 2033?” was hard to parse. You did not say something simple and easily understandable like, “I think 30% is too high for these reasons: …” you say “Once criticality is achieved the odds drop to 0 [+ more words].” The odds of what drop to zero? The odds of TAI? But you seem to be saying that once criticality is reached, TAI is inevitable? Even the rest of your long answer leaves in doubt where you’re really coming down on the premise.
By the way, I don’t think I would even be making this comment myself if A) I didn’t have such a hard time trying to understand what your conclusions were myself and B) you didn’t have such a confident, authoritative tone that seemed to present your ideas as if they were patently obvious.
I’m confident about the consequences of criticality. It is a mathematical certainty, it creates a situation where all future possible timelines are affected. For example, covid was an example of criticality. Once you had sufficient evidence to show the growth was exponential, which was available in January 2020, you could be completely confident all future timelines would have a lot of covid infections in them and it would continue until quenching, which turned out to be infection of ~44% of the population of the planet. (and you can from the Ro estimate that final equilibrium number)
Once AI reaches a point where critical mass happens, it’s the same outcome. No futures exist where you won’t see AI systems in use everywhere for a large variety of tasks (economic criticality) or billions or scientific notation numbers of robots in use (physical criticality, true AGI criticality cases).
July 2033 thus requires the “January 2020” data to exist. There don’t have to be billions of robots yet, just a growth rate consistent with that.
I do not know precisely when the minimum components needed to reach said critical mass will exist.
I gave the variables of the problem. I would like Paul, who is a world class expert, to take the idea seriously and fill in estimates for the values of those variables. I think his model for what is transformative and what the requirements are for transformation is completely wrong, and I explain why.
If I had to give a number I would say 90%, but a better expert could develop a better number.
Update: edited to 90%. I would put it at 100% because we are already past investor criticality, but the system can still quench if revenue doesn’t continue to scale.
My view that a cheap simulation of arbitrary human experts would be enough to end life as we know it one way or the other?
Just to add to this : many experts are just faking it. Simulating them is not helping. By faking it, because they are solving as humans an RL problem that can’t be solved, their learned policy is deeply suboptimal and in some cases simply wrong. Think expert positions like in social science, government, law, economics, business consulting, and possibly even professors who chair computer science departments but are not actually working on scaled cutting edge AI. Each of these “experts” cannot know a true policy that is effective, most of their status comes from various social proofs and finite Official Positions. The “cannot” because they will not in their lifespan receive enough objective feedback to learn a policy that is definitely correct. (they are more likely to be correct than non experts, however)
(In the subsequent text it seems like you are saying that you don’t need to match human experts in every domain in order to have a transformative impact, which I agree with. I set the TAI threshold as “economic impact as large as” but believe that this impact will be achieved by systems which are in some respects weaker than human experts and in other respects stronger/faster/cheaper than humans.)
I pointed out that you do not need to match human experts in any domain at all. Transformation depends on entirely different variables.
This is incorrect, and you’re a world class expert in this domain.
The proximity of the subparts of this sentence read, to me, on first pass, like you are saying that “being incorrect is the domain in which you are a world class expert.”
After reading your responses to O O I deduce that this is not your intended message, but I thought it might be helpful to give an explanation about how your choice of wording might be seen as antagonistic. (And also explain my reaction mark to your comment.)
For others who have not seen the rephrasing by Gerald, it reads
just like historical experts Einstein and Hinton, it’s possible to be a world class expert but still incorrect. I think that focusing on the human experts at the top of the pyramid is neglecting what would cause AI to be transformative, as automating 90% of humans matters a lot more than automating 0.1%. We are much closer to automating the 90% case because...
I share the quote to explain why I do not believe that rudeness was intended.
Hi Paul. I’ve reflected carefully on your post. I have worked for several years on a SDC software infrastructure stack and have also spent a lot of time comparing the two situations.
Update: since commentators and downvoters demand numbers: I would say the odds of criticality are 90% by July 2033. The remaining 10% is that there is a possibility of a future AI winter (investors get too impatient) and there is the possibility that revenue from AI services will not continue to scale.
I think you’re badly wrong, again, and the consensus of experts are right, again.
First, let’s examine your definition for transformative. This may be the first major error:
(By which I mean: systems as economically impactful as a low-cost simulations of arbitrary human experts, which I think is enough to end life as we know it one way or the other.)
This is incorrect, and you’re a world class expert in this domain.
Transformative is a subclass of the problem of criticality. Criticality, as you must know, means a system produces self gain larger than it’s self losses. For AGI, there are varying stages of criticality, which each do settle on an equlibria:
Investment criticality : This means that each AI system improvement or new product announcement or report of revenue causes more financial investment into AI than the industry as a whole burned in runway over that timestep.
Equilibrium condition: either investors run out of money, globally, to invest or they perceive that each timestep the revenue gain is not worth the amount invested and choose to invest in other fields. The former equilibrium case settles on trillions of dollars into AI and a steady ramp of revenue over time, the later is an AI crash, similar to the dotcom crash of 2000.
Economic Criticality: This means each timestep, AI systems are bringing in more revenue than the sum of costs [amortized R&D, inference hardware costs, liability, regulatory compliance, …]
Equilibrium condition: growth until there is no more marginal tasks an AI system can perform cheaper than a human being. Assuming a large variety of powerful models and techniques, it means growth continues until all models and all techniques cannot enter any new niches. The reason why this criticality is not exponential, while the next ones are, is because the marginal value gain for AI services drops with scale. Notice how Microsoft charges just $30 a month for Copilot, which is obviously able to save far more than $30 worth of labor each month for the average office worker.
Physical Criticality: This means AI systems, controlling robotics, have generalized manufacturing, mining, logistics, and complex system maintenance and assembly. The majority, but not all, of labor to produce more of all of the inputs into an AI system can be produced by AI systems.
Equilibrium condition: Exponential growth until the number of human workers on earth is again rate limiting. If humans must still perform 5% of the tasks involved in the subdomain of “build things that are inputs into inference hardware, robotics”, then the equilibria is when all humans willing, able to work on earth are doing those 5% of tasks.
AGI criticality: True AGI can learn automatically to do any task that has clear and objective feedback. All tasks involved in building computer chips, robotic parts (and all lower level feeder tasks and power generation and mining and logistics) have objective and measurable feedback. Bolded because I think this is a key point and a key crux, you may not have realized this. Many of your “expert” domain tasks do not get such feedback, or the feedback is unreliable. For example an attorney who can argue 1 case in front of a jury every 6 months cannot reliably refine their policy based on win/loss because the feedback is so rare and depends on so many uncontrolled variables.
AGI may still be unable to perform as well as the best experts in many domains. This is not relevant. It only has to perform well enough for machines controlled by the AI to collect more resources/build more of themselves than their cost.
A worker pool of AI systems like this can be considerably subhuman across many domains, or rely heavily on using robotic manipulators that are each specialized for a task, being unable to control general purpose hands, relying heavily on superior precision and vision to complete tasks in a way different than how humans perform it. They can make considerable mistakes, so long as the gain is positive—miswire chip fab equipment, dropped parts in the work area cause them to flush clean entire work areas, wasting all the raw materials—etc. I am not saying the general robotic agents will be this inefficient, just that they could be.
Equilibrium Condition: exponential growth until exhaustion of usable elements in Sol. Current consensus is earth’s moon has a solid core, so all of it could potentially be mined for useful elements A large part of Mars, it’s moons, the asteroid belt, and Mercury are likely mineable. Large areas of the earth via underground tunnel and ocean floor mining. The Jovian moons. Other parts of the solar system become more speculative but this is a natural consequence of machinery able to construct more of itself.
Crux: AGI criticality seems to fall short of your requirement for “human experts” to be matched by artificial systems. Conversely, if you invert the problem: AGI cannot control robots well, creating a need for billions of technician jobs, you do not achieve criticality, you are rate limited on several dimensions. AI companies collect revenue more like consulting companies in such a world, and saturate when they cannot cheaply replace any more experts, or the remaining experts enjoy legal protection.
Requirement to achieve full AGI criticality before 2033: You would need a foundation model trained on all the human manipulation you have the licenses for the video. You would need a flexible, real time software stack, that generalizes to many kinds of robotic hardware and sensor stack. You would need an “app store” license model where thousands of companies could contribute, instead of just 3, to the general pool of AI software, made intercompatible by using a base stack. You would need there to not be hard legal roadblocks stopping progress. You would need to automatically extend a large simulation of possible robotic tasks whenever surprising inputs are seen in the real world.
Amdahl’s law applies to the above, so actually, probably this won’t happen before 2033, but one of the lesser criticalities might. We are already in the Investment criticality phase of this.
Autonomous cars: I had a lot of points here, but it’s simple:
(1) an autonomous robo taxi must collect more revenue than the total costs, or it’s subcritical, which is the situation now. If it were critical, Waymo would raise as many billions as required and would be expanding into all cities in the USA and Europe at the same time. (look at a ridesharing company’s growth trajectory for a historical example of this)
(2) It’s not very efficient to develop a realtime stack just for 1 form factor of autonomous car for 1 company. Stacks need to be general.
(3) There are 2 companies allowed to contribute. Anyone not an employee of Cruise or Waymo is not contributing anything towards autonomous car progress. There’s no cross licensing, and it’s all closed source except for comma.ai. This means only a small number of people are pushing the ball forward at all, and I’m pretty sure they each work serially on an improved version of their stack. Waymo is not exploring 10 different versions of n+1 “Driver” agent using different strategies, but is putting everyone onto a single effort, which may be the wrong approach, where each mistake costs linear time. Anyone from Waymo please correct me. Cruise must be doing this as they have less money.
This is a rather rude response. Can you rephrase that?
I don’t like this point. Many expert domain tasks have vast quantities of historical data we can train evaluators on. Even if the evaluation isn’t as simple to quantify, deep learning intuitively seems it can tackle it. Humans also manage to get around the fact that evaluation may be hard to gain competitive advantages as experts of those fields. Good and bad lawyers exist. (I don’t think it’s a great example as going to trial isn’t a huge part of a most lawyers’ jobs)
Having a more objective and immediate evaluation function, if that’s what you’re saying, doesn’t seem like an obvious massive benefit. The output of this evaluation function with respect to labor output over time can still be pretty discontinuous so it may not effectively be that different than waiting 6 months between attempts to know if success happened.
An example of this is it taking a long time to build and verify whether a new chip architecture improves speeds or having to backtrack and scrap ideas.
This is a rather rude response. Can you rephrase that?
If I were to rephrase I might say something like “just like historical experts Einstein and Hinton, it’s possible to be a world class expert but still incorrect. I think that focusing on the human experts at the top of the pyramid is neglecting what would cause AI to be transformative, as automating 90% of humans matters a lot more than automating 0.1%. We are much closer to automating the 90% case because...”
I don’t like this point. Many expert domain tasks have vast quantities of historical data we can train evaluators on. Even if the evaluation isn’t as simple to quantify, deep learning intuitively seems it can tackle it. Humans also manage to get around the fact that evaluation may be hard to gain competitive advantages as experts of those fields. Good and bad lawyers exist. (I don’t think it’s a great example as going to trial isn’t a huge part of a most lawyers’ jobs)
Having a more objective and immediate evaluation function, if that’s what you’re saying, doesn’t seem like an obvious massive benefit. The output of this evaluation function with respect to labor output over time can still be pretty discontinuous so it may not effectively be that different than waiting 6 months between attempts to know if success happened.
For lawyers: the confounding variables means a robust, optimal policy is likely not possible. A court outcome depends on variables like [facts of case, age and gender and race of the plaintiff/defendant, age and gender and race of the attorneys, age and gender and race of each juror, who ends up the foreman, news articles on the case, meme climate at the time the case is argued, the judge, the law’s current interpretation, scheduling of the case, location the trial is held...]
It would be difficult to develop a robust and optimal policy with this many confounding variables. It would likely take more cases than any attorney can live long enough to argue or review.
Contrast this to chip design. Chip A, using a prior design, works. Design modification A’ is being tested. The universe objectively is analyzing design A’ and measurable parameters (max frequency, power, error rate, voltage stability) can be obtained.
The problem can also be subdivided. You can test parts of the chip, carefully exposing it to the same conditions it would see in the fully assembled chip, and can subdivide all the way to the transistor level. It is mostly path independent—it doesn’t matter what conditions the submodule saw yesterday or an hour ago, only right now. (with a few exceptions)
Delayed feedback slows convergence to an optimal policy, yes.
You cannot stop time and argue a single point to a jury, and try a different approach, and repeatedly do it until you discover the method that works. {note this does give you a hint as to how an ASI could theoretically solve this problem}
I say this generalizes to many expert tasks like [economics, law, government, psychology, social sciences, and others]. Feedback is delayed and contains many confounding variables independent of the [expert’s actions].
While all tasks involved with building [robots, compute], with the exception of tasks that fit into the above (arguing for the land and mineral permits to be granted for the ai driven gigafactories and gigamines), offer objective feedback.
I don’t see why there is no robust optimal policy. A robust optimal policy doesn’t have to always win. The optimal chess policy can’t win with just a king on the board. It just has to be better than any alternative to be optimal as per the definition of optimal. I agree it’s unlikely any human lawyer has an optimal policy, but this isn’t unique to legal experts.
There are confounding variables, but you could also just restate evaluation as trial win-rate (or more succinctly trial elo) instead of as a function of those variables. Likewise you can also restate chip evaluation’s confounding variables as being all the atoms and forces that contribute to the chip.
The evaluation function for lawyers, and many of your examples is objective. The case gets won, lost, settled, dismissed, etc.
The only difference is it takes longer to verify generalizations are correct if we go out of distribution with a certain case. In the case of a legal-expert-AI, we can’t test hypotheses as easily. But this still may not be as long as you think. Since we will likely have jury-AI when we approach legal-expert-AI, we can probably just simulate the evaluations relatively easily (as legal-expert-AI is probably capable of predicting jury-AI). In the real world, a combination of historical data and mock trials help lawyers verify their generalizations are correct, so it wouldn’t even be that different as it is today (just much better). In addition, process based evaluation probably does decently well here, which wouldn’t need any of these more complicated simulations.
Maybe not, but you can conduct mock trials and look at billions of historical legal cases and draw conclusions from that (human lawyers already read a lot). You can also simulate a jury and judge directly instead of doing a mock trial. I don’t see why this won’t be good enough for both humans and an ASI. The problem has high dimensionality as you stated, with many variables mattering, but a near optimal policy can still be had by capturing a subset of features. As for chip-expert-AI, I don’t see why it will definitely converge to a globally optimal policy.
All I can see is that initially legal-expert-AI will have to put more work in creating an evaluation function and simulations. However, chip-expert-AI has its own problem where it’s almost always working out of distribution, unlike many of these other experts. I think experts in other fields won’t be that much slower than chip-expert-AI. The real difference I see here is that the theoretical limits of output of chip-expert-AI are much higher and legal-expert-AI or therapist-expert-AI will reach the end of the sigmoid much sooner.
Is there something significantly different between a confounding variable that can’t be controlled like scheduling and unknown governing theoretical frameworks that are only found experimentally? Both of these can still be dealt with. For the former, you may develop different policies for different schedules. For the latter, you may also intuit the governing theoretical framework.
So in this context, I was referring to criticality. AGI criticality is a self amplifying process where the amount of physical materials and capabilities increases exponentially with each doubling time. Note it is perfectly fine if humans continue to supply as inputs the network of isolated AGI instances are unable to produce. (Vs others who imagine a singleton AGI on its own. Obviously eventually the system will be rate limited by available human labor if its limited this way, but will see exponential growth until then)
I think the crux here is that all is required is for AGI to create and manufacture variants on existing technology. At no point does it need to design a chip outside of current feature sizes, at no point does any robot it designs look like anything but a variation of robots humans designed already.
This is also the crux with Paul. He says the AGI needs to be as good as the 0.1 percent human experts at the far right side of the distribution. I am saying that doesn’t matter, it is only necessary to be as good as the left 90 percent of humans. Approximately , I go over how the AGI doesn’t even need to be that good, merely good enough there is net gain.
This means you need more modalities on existing models but not necessarily more intelligence.
It is possible because there are regularities in how the tree of millions of distinct manufacturing tasks that humans do now use common strategies. It is possible because each step and substep has a testable and usually immediately measurable objective. For example : overall goal. Deploy a solar panel. Overall measurable value : power flows when sunlight available. Overall goal. Assemble a new robot of design A5. Overall measurable objective: new machinery is completing tasks with similar Psuccess. Each of these problems is neatly dividable into subtasks and most subtasks inherit the same favorable properties.
I am claiming more than 99 percent of the sub problems of “build a robot, build a working computer capable of hosting more AGI” work like this.
What robust and optimal means is that little human supervision is needed, that the robots can succeed again and again and we will have high confidence they are doing a good job because it’s so easy to measure the ground truth in ways that can’t be faked. I didn’t mean the global optimal, I know that is an NP complete problem.
I was then talking about how the problems the expert humans “solve” are nasty and it’s unlikely humans are even solving many of them at the numerical success levels humans have in manufacturing and mining and logistics, which are extremely good at policy convergence. Even the most difficult thing humans do—manufacture silicon ICs—converges on yields above 90 percent eventually.
How often do lawyers unjustly lose, economists make erroneous predictions, government officials make a bad call, psychologists fail and the patient has a bad outcome, or social science uses a theory that fails to replicate years later.
Early AGI can fail here in many ways and the delay until feedback slows down innovation. How many times do you need to wait for a jury verdict to replace lawyers with AI. For AI oncologists how long does it take to get a patient outcome of long term survival. You’re not innovating fast when you wait weeks to months and the problem is high stakes like this. Robots deploying solar panels are low stakes with a lot more freedom to innovate.
What’s incorrect? My view that a cheap simulation of arbitrary human experts would be enough to end life as we know it one way or the other?
(In the subsequent text it seems like you are saying that you don’t need to match human experts in every domain in order to have a transformative impact, which I agree with. I set the TAI threshold as “economic impact as large as” but believe that this impact will be achieved by systems which are in some respects weaker than human experts and in other respects stronger/faster/cheaper than humans.)
Do you think 30% is too low or too high for July 2033?
Do you think 30% is too low or too high for July 2033?
This is why I went over the definitions of criticality. Once criticality is achieved the odds drop to 0. A nuclear weapon that is prompt critical is definitely going to explode in bounded time because there are no futures where sufficient numbers of neutrons are lost to stop the next timestep releasing even more.
What’s incorrect? My view that a cheap simulation of arbitrary human experts would be enough to end life as we know it one way or the other?
Your cheap expert scenario isn’t necessarily critical. Think of how it could quench, where you simply exhaust the market for certain kinds of expert services and cannot expand to any more because of lack of objective feedback and legal barriers.
An AI system that has hit the exponential criticality phase in capability is the same situation as the nuclear weapon. It will not quench, that is not a possible outcome in any future timeline [except timelines with immediate use of nuclear weapons on the parties with this capability]
So your question becomes : what is the odds that economic or physical criticality will be reached by 2033? I have doubts myself, but fundamentally the following has to happen for robotics:
A foundation model that includes physical tasks, like this.
Sufficient backend to make mass usage across many tasks possible, and convenient licensing and usage. Right now Google and a few startups exist and have anything using this approach. Colossal scale is needed. Something like ROS 2 but a lot better.
No blocking legal barriers. This is going to require a lot of GPUs to learn from all the video in the world. Each robot in the real world needs a rack of them just for itself.
Generative physical sims. Similar to generative video, but generating 3d worlds where short ‘dream’ like segments of events happening in the physical world can be modeled. This is what you need to automatically add generality to go from 60% success rate to 99%+. Tesla has demoed some but I don’t know of good, scaled, readily licensed software that offers this.
For economics:
1. Revenue collecting AI services good enough to pay for at scale
2. Cheap enough hardware, such as from competitors to Nvidia, that make the inference hardware cheap even for powerful models
Either criticality is transformative.
You speak with such a confident authoritative tone, but it is so hard to parse what your actual conclusions are.
You are refuting Paul’s core conclusion that there’s a “30% chance of TAI by 2033,” but your long refutation is met with: “wait, are you trying to say that you think 30% is too high or too low?” Pretty clear sign you’re not communicating yourself properly.
Even your answer to his direct follow-up question: “Do you think 30% is too low or too high for July 2033?” was hard to parse. You did not say something simple and easily understandable like, “I think 30% is too high for these reasons: …” you say “Once criticality is achieved the odds drop to 0 [+ more words].” The odds of what drop to zero? The odds of TAI? But you seem to be saying that once criticality is reached, TAI is inevitable? Even the rest of your long answer leaves in doubt where you’re really coming down on the premise.
By the way, I don’t think I would even be making this comment myself if A) I didn’t have such a hard time trying to understand what your conclusions were myself and B) you didn’t have such a confident, authoritative tone that seemed to present your ideas as if they were patently obvious.
I’m confident about the consequences of criticality. It is a mathematical certainty, it creates a situation where all future possible timelines are affected. For example, covid was an example of criticality. Once you had sufficient evidence to show the growth was exponential, which was available in January 2020, you could be completely confident all future timelines would have a lot of covid infections in them and it would continue until quenching, which turned out to be infection of ~44% of the population of the planet. (and you can from the Ro estimate that final equilibrium number)
Once AI reaches a point where critical mass happens, it’s the same outcome. No futures exist where you won’t see AI systems in use everywhere for a large variety of tasks (economic criticality) or billions or scientific notation numbers of robots in use (physical criticality, true AGI criticality cases).
July 2033 thus requires the “January 2020” data to exist. There don’t have to be billions of robots yet, just a growth rate consistent with that.
I do not know precisely when the minimum components needed to reach said critical mass will exist.
I gave the variables of the problem. I would like Paul, who is a world class expert, to take the idea seriously and fill in estimates for the values of those variables. I think his model for what is transformative and what the requirements are for transformation is completely wrong, and I explain why.
If I had to give a number I would say 90%, but a better expert could develop a better number.
Update: edited to 90%. I would put it at 100% because we are already past investor criticality, but the system can still quench if revenue doesn’t continue to scale.
It seems like criticality is sufficient, bot not necessary, for TAI, and so only counting criticality scenarios causes underestimation.
This was a lot clearer, thank you.
My view that a cheap simulation of arbitrary human experts would be enough to end life as we know it one way or the other?
Just to add to this : many experts are just faking it. Simulating them is not helping. By faking it, because they are solving as humans an RL problem that can’t be solved, their learned policy is deeply suboptimal and in some cases simply wrong. Think expert positions like in social science, government, law, economics, business consulting, and possibly even professors who chair computer science departments but are not actually working on scaled cutting edge AI. Each of these “experts” cannot know a true policy that is effective, most of their status comes from various social proofs and finite Official Positions. The “cannot” because they will not in their lifespan receive enough objective feedback to learn a policy that is definitely correct. (they are more likely to be correct than non experts, however)
(In the subsequent text it seems like you are saying that you don’t need to match human experts in every domain in order to have a transformative impact, which I agree with. I set the TAI threshold as “economic impact as large as” but believe that this impact will be achieved by systems which are in some respects weaker than human experts and in other respects stronger/faster/cheaper than humans.)
I pointed out that you do not need to match human experts in any domain at all. Transformation depends on entirely different variables.
Gentle feedback is intended
The proximity of the subparts of this sentence read, to me, on first pass, like you are saying that “being incorrect is the domain in which you are a world class expert.”
After reading your responses to O O I deduce that this is not your intended message, but I thought it might be helpful to give an explanation about how your choice of wording might be seen as antagonistic. (And also explain my reaction mark to your comment.)
For others who have not seen the rephrasing by Gerald, it reads
I share the quote to explain why I do not believe that rudeness was intended.