I think you are assuming the above will happen. (the line in blue).
I am assuming the red line, and obviously by building on what we have incrementally.
If you were somehow a significant way up the blue line and trying to get robots to do anything useful, yes, you might get goodheart optimized actions that achieve the instructed result, maybe (if the ASI hasn’t chosen to betray this time since it can do so), but not satisfying all the thousands of constraints you implied but didn’t specify.
In more “slow takeoff” scenarios. Your approach can probably be used to build something that is fairly useful at moderate intelligence. So for a few years in the middle of the red curve, you can get your factories built for cheap. Then it hits the really steep part, and it all fails.
I think the “slow” and “fast” models only disagree in how much time we spend in the orange zone before we reach the red zone. Is it enough time to actually build the robots?
I assign fairly significant probabilities to both “slow” and “fast” models.
I think the “slow” and “fast” models only disagree in how much time we spend in the orange zone before we reach the red zone. Is it enough time to actually build the robots?
I assign fairly significant probabilities to both “slow” and “fast” models.
Well how do these variables interact?
the g factor(intelligence) of the ASI depends on the algorithm of the ASI times the log of compute assigned to it, C.
The justification for it not being linear is because non search NN lookups already capture most of the possible gain on a task. You can do more samples with more compute, increasing the probability of a higher score. I’m having trouble finding the recent paper where it turns out if you sample GPT-4 you do get higher score, with it scaling with the log of the number of samples.
Then at any given moment in time, the rate of improvement of the ASI’s algorithm is going to scale with current intelligence times a limit based on S.
What is S? I sensed you were highly skeptical of my “neural sim” variable until 2 days ago. It’s Sora + you can get collidable geometry, not just images as output. At the time, the only evidence i had of a neural sim was nvidia papers. Note the current model is likely capable of this:
S is the suite of neural sim situations. Once the ASI solves all situations in the suite, self improvement goes to 0. (there is no error derivative to train on)
This also come to mind naturally. Have you, @Donald Hobson , ever played a simulation game that models a process that has a real world equivalent? You might notice you get better and better at the game until you start using solutions that are not possible in the game, but just exploit glitches in the game engine. If an ASI is doing this, it’s improvement becomes negative once it hits the edges of the sim and starts training on false information. This is why you need neural sims, as they can continue to learn and add complexity to the sim suite (and they need to output a ‘confidence’ metric so you lower the learning rate when the sim is less confident the real world estimation is correct). How do you do this?
Well at time 0, humans make a big test suite with everything they know how to test for (“take all these exams in every college major, play all these video games and beat them”). Technically it’s not a fixed process, but an iterative one, but the benefit of that suite is limited. System already read all the books and watched all the videos and has taken every test humans can supply, at a certain point new training examples are redundant. (and less than perfect scores on things like college exams are because the answer key has errors, so there’s negative improvement if the AI memorizes the wrong answers)
The second part is the number of robots providing correct data. (because the real world isn’t biased like a game or exam, if a specific robot fails an achievable task it’s because the robot policy needs improvement)
Note the log here : this comes from intuition. In words, the justification is that immediately when a robot does a novel task, there will be lots of mistakes and rapid learning. But then the mistakes take increasingly larger lengths of time and task iterations to find them, it’s a logistic growth curve approaching an asymptote for perfect policy. This is also true for things like product improvement. Gen1->Gen2 you get the most improvement, and then over time you have to wait increasingly longer times to get feedback from the remaining faults, especially if your goal is to increase reliability. (example task: medical treatments for humans. You’ll figure out the things that kill humans immediately very rapidly, while the things that doom them to die in 10 years you have to wait 10 years for. more specific example: cancer treatments)
How fast do you get more robots?
where,
R(t) represents the number of robots at time t.
R0 is the initial number of robots at time = 0
tau is the time constant in years
@Daniel Kokotajlo and I estimated this time constant at 2-3 years. This is a rough guess, but note there are strong real world factors that make it difficult to justify a lower number.
First, what is a robot?
+
+
and it lives in a setup like this, just now with AI driven arms instead of the older limited control logic the bots pictured use.
1.0 robot is: (robotic arms) (with arm placement similar to humans so you can learn how to do tasks the human way initially) + sensors + compute racks + support infrastructure. (cages, power conveyers, mounts for the sensors)
Robot growth means you double the entire industrial chain. Obviously, in a factory somewhere, robots are making robots. That would take like, a few hours right?
But then you also copy the CNC machines to make the robot’s parts:
And the steel forges, chip fabs, mines, power sources, buildings, aluminum smelter...everything in the entire industrial chain you must duplicate or the logistic growth bottlenecks on the weak link.
At the very start, you can kick-start the process by getting together 7 trillion or so, and employing a substantial fraction of the working population of the planet to serve as your first generation of ‘robots’. You also can have the ASI optimize existing equipment and supply chains to get more productivity out of it.
In the later future, you can probably get nanotechnology:
Note this helps but it isn’t a magic pass to overnight exponential growth. Nanotechnology still requires power, cooling, raw materials processing, it needs a warehouse and a series of high vacuum chambers for the equipment to run in. It does simplify things a lot and has other advantages.
Another key note : probably to build the above, which is all the complexity of earth’s industry crammed into a small box, you will need some initial number of robots, on the order of hundreds of millions to billions, to support the bootstrapping process to develop and build the first nanoforge.
Finally, obviously,
Where you have your compute at time 0 (when you got to ASI) and it scales over time with respect to the number of robots you have.
Wait, don’t humans build compute? Nope. They haven’t for years.
Everything is automated. Humans are in there for maintenance and recipe improvement.
Another thing that’s special about compute is the “robots” aren’t general purpose. You need specific bottlenecked equipment that is likely the hardest task for robots to learn to build and maintain. (I suspect harder than learning to do surgery with a high patient survival rate, because bodies can heal from small mistakes, while this equipment will fail on most tiny mistakes.
Discussion: We can model the above equation and get a timeline in for the orange area, for different assumptions for the variables. Perhaps you and I could have a dialogue where we develop the model. I would like feedback on any shaky assumptions made.
I sensed you were highly skeptical of my “neural sim” variable until 2 days ago.
No. Not really. I wasn’t claiming that things like SORA couldn’t exist. I am claiming that it’s hard to turn them towards the task of engineering a bridge say.
Current SORA is totally useless for this. You ask it for a bridge, and it gives you some random bridge looking thing, over some body of water. SORA isn’t doing the calculations to tell if the bridge would actually hold up. But lets say a future much smarter version of SORA did do the calculations. A human looking at the video wouldn’t know what grade of steel SORA was imagining. I mean existing SORA probably isn’t thinking of a particular grade of steel, but this smarter version would have picked a grade, and used that as part of it’s design. But it doesn’t tell the human that, the knowledge is hidden in it’s weights.
Ok, suppose you could get it to show a big pile of detailed architectural plans, and then a bridge. All with super-smart neural modeling that does the calculations. Then you get something that ideally is about as good at looking at the specs of a random real world bridge. Plenty of random real world bridges exist, and I presume bridge builders look at their specs. Still not that useful. Each bridge has different geology, budget, height requirements etc.
Ok, well suppose you could start by putting all that information in somehow, and then sampling from designs that fit the existing geology, roads etc.
Then you get several problems.
The first is that this is sampling plausible specs, not good specs. Maybe it shows a few pictures at the end to show the bridge not immediately collapsing. But not immediately collapsing is a low bar for a bridge. If the Super-SORA chose a type of paint that was highly toxic to local fish, it wouldn’t tell you. If the bridge had a 10% chance of collapsing, it’s randomly sampling a plausible timeline. So 90% of the time, it shows you the bridge not collapsing. If it only generates 10 minutes of footage, you don’t know what might be going on in it’s sim while you weren’t watching. If it generates 100 years of footage from every possible angle, it’s likely to record predictions of any problems, but good luck finding the needle in the haystack. Like imagine this AI has just given you 100 years of footage. How do you skim through it without missing stuff.
Another problem is that SORA is sampling in the statistical sense. Suppose you haven’t done the geology survey yet. SORA will guess at some plausible rock composition. This could lead to you building half the bridge, and then finding that the real rock composition is different.
You need a system that can tell you “I don’t know fact X, go find it out for me”.
If the predictions are too good, well the world it’s predicting contains Super-SORA. This could lead to all sorts of strange self fulfilling prophecy problems.
No. Not really. I wasn’t claiming that things like SORA couldn’t exist. I am claiming that it’s hard to turn them towards the task of engineering a bridge say.
I would agree with that.
Let me give you a summary of my overall position: “Trust an ASI with a complex task it needs context awareness to complete? Not even once”.
Everything is about how to get a large amount of benefits in the orange area, and yes we should stay there for a prolonged period.
What benefits specifically? Vast amounts of material manufactured goods, robots, robots are doing repetitive tasks that can be clearly defined. ASI models in use are only used in limited duration sessions, and you strip away context awareness.
Context awareness is the bits that tell the ASI this is a real bridge, humans are really going to build it, and not just another task in the training sim. “Always bluepill”. It should not be possible for the ASI to know when the task is real vs sim. (which you can do by having an image generator convert real frames to a descriptor, and then regenerate them so they have the simulation artifacts...)
The architecture fees for a new bridge design are about 10% of the cost.
Other 90% is, well, all the back breaking labor to build one. Not just at the construction site, but where the concrete was made, building/maintaining the trucks to haul the materials, etc.
Sora’s role: For autonomous cars and general robots, this is used for training. General robots means one robot at a time, and for short task sessions. For example : “remove the part from the CNC machine and place it on the output table”.
What you do is record video from the human workers doing the task, many hours of it. You train a model using techniques similar Sora to classify what it sees and to predict frame for frame the next frame. This is 5-30 hz sampling rate, so 33 ms into the future. Then do this recursively.
Since this is an RL problem you go further and may model a few seconds ahead.
There are ways to reproject in a robot instead.
Then the robot tries to do the task in the real world once it has settled on a policy that covers it in the simulated world.
The robot may fail, but you train the model, and you policy iterate with each failure. This is where the log(real world episodes) comes in. It’s correct and aligns very well with real world experience.
Note on Sora : Tesla and probably Waymo already use a neural sim that is similar to training their robotic cars. I found the tweet last night from an insider, and I’ve seen demo videos before, i can produce evidence. But it is accurate to say it’s already in use for the purpose I am talking about.
Bridges : at least in the near future, you want to automate making the parts for repairing a bridge, doing the dangerous work of inspection via drone, converting a bunch of pictures from many places from the drones to an overall model of the bridge compatible with load testing software, and so on. <https://artbabridgereport.org/reports/2021-ARTBA-Bridge-Report.pdf> current USA bridges need 7% replaced and work done on about 40%.
Key note : I would not trust an ASI or AGI with designing a new bridge, the goal is to make it cheaper to replace or repair the designs you already have. If AI + robots can contribute between 10% and 95% of the labor to replace a bridge (starting at 10, gradually ramping...), that lets you swap a lot more bridges for the same annual budget.
If we have the technical capacity to get into the red zone, and enough chips to make getting there easy. Then hanging out in the orange zone, coordinating civilization not to make any AI too powerful, when there are huge incentives to ramp the power up, and no one is quite sure where the serious dangers kick in...
That is, at least, an impressive civilization wide balancing act. And one I don’t think we have the competence to pull off.
It should not be possible for the ASI to know when the task is real vs sim. (which you can do by having an image generator convert real frames to a descriptor, and then regenerate them so they have the simulation artifacts...)
This is something you want, not a description of how to get it, and one that is rather tricky to achieve. That converting and then converting back trick is useful. But sure isn’t automatic success either. If there are patterns about reality that the ASI understands, but the simulator doesn’t, then the ASI can use those patterns.
Ie if the ASI understands seasons, and the simulator doesn’t, then if it’s scorching sunshine one day and snow the next, that suggests it’s the simulation. Otherwise, that suggests reality.
And if the simulation knows all patterns that the ASI does, the simulator itself is now worryingly intelligent.
robots are doing repetitive tasks that can be clearly defined.
If the task is maximally repetitive, then the robot can just follow the same path over and over.
If it’s nearly that repetitive, the robot still doesn’t need to be that smart.
I think you are trying to get a very smart AI to be so tied down and caged up that it can do a task without going rouge. But the task is so simple that current dumb robots can often do it.
For example : “remove the part from the CNC machine and place it on the output table”.
Economics test again. Minimum wage workers are easily up to a task like that. But most engineering jobs pay more than minimum wage. Which suggests most engineering in practice requires more skill than that.
I mean yes engineers do need to take parts out of the CNC machine. But they also need to be able to fix that CNC machine when a part snaps off inside it and starts getting jammed in the workings. And the latter takes up more time in practice. Or noticing that the toolhead is loose, and tightning and recalibrating it.
The techniques you are describing seem to be next level in fairly dumb automation. The stuff that some places are already doing (like boston dynamics robot dog level hardware and software), but expanded to the whole economy. I agree that you can get a moderate amount of economic growth out of that.
I don’t see you talking about any tasks that require superhuman intelligence.
By the way, these comment boxes have built in maths support.
Press Ctrl M for full line or Ctrl 4 for inline
LikeThisCtrlMusedhere
You might notice you get better and better at the game until you start using solutions that are not possible in the game, but just exploit glitches in the game engine. If an ASI is doing this, it’s improvement becomes negative once it hits the edges of the sim and starts training on false information. This is why you need neural sims, as they can continue to learn and add complexity to the sim suite
Neural sims probably have glitches too. Adversarial examples exist.
Note the log here : this comes from intuition. In words, the justification is that immediately when a robot does a novel task, there will be lots of mistakes and rapid learning. But then the mistakes take increasingly larger lengths of time and task iterations to find them, it’s a logistic growth curve approaching an asymptote for perfect policy.
This sounds iffy. Like you are eyeballing and curve fitting, when this should be something that falls out of a broader world model.
Every now and then, you get a new tool. Like suppose your medical bot has 2 kinds of mistakes, ones that instant kill, and ones that mutate DNA. It quickly learns not to do the first one. And slowly learns not to do the second when it’s patients die of cancer years later. Except one day it gets a gene sequencer. Now it can detect all those mutations quickly.
I find it interesting that most of this post is talking about the hardware.
Isn’t this supposed to be about AI? Are you expecting a regieme where
Most of the worlds compute is going into AI.
Chip production increases by A LOT (at least 10x) within this regieme.
Most of the AI progress in this regieme is about throwing more compute at it.
.everything in the entire industrial chain you must duplicate or the logistic growth bottlenecks on the weak link.
Everything is automated. Humans are in there for maintenance and recipe improvement.
Ok. And there is our weak link. All our robots are going to be sitting around broken. Because the bottleneck is human repair people.
It is possible to automate things. But what you seem to be describing here is the process of economic growth in general.
Each specific step in each specific process is something that needs automating.
You can’t just tell the robot “automate the production of rubber gloves”. You need humans to do a lot of work designing a robot that picks out the gloves and puts them on the hand shaped metal molds to the rubber can cure.
Yes economic growth exists. It’s not that fast. It really isn’t clear how AI fits into your discussion of robots.
Neural sims probably have glitches too. Adversarial examples exist.
Yes. That’s why I specifically mentioned :
and they need to output a ‘confidence’ metric so you lower the learning rate when the sim is less confident the real world estimation is correct
Confidence is a trainable parameter, and you scale down learning rate when confidence is low.
Ok. And there is our weak link. All our robots are going to be sitting around broken. Because the bottleneck is human repair people.
This is a lengthy discussion but the simple answer is that what a human ‘repair person’ does can be described as a simple algorithm that you can write in ordinary software. I’ve repaired a few modern things, this is from direct knowledge and watching videos of someone repairing a Tesla.
The algorithm in essence is every module is self diagnosing, and there is a graph flow of relationships between modules. There are simple experiments you do, it tells you in the manual many of them, to get better evidence.
Then you disassemble the machine partially—if you were a robot and had the right licenses you could download the assembly plan for this machine and reverse it—remove the suspect module, and replace. If the issues don’t resolve, you remove the module that said the suspect module was bad or was related to it.
For PCs, this is really easy. Glitches on your screen from your GPU? Replace the cable. Observe if the glitches go away. Try a different monitor. Still broken? Put in a different GPU. That doesn’t resolve it? Go and memtest86 the RAM. Does that pass? It’s either the motherboard or the processors.
This comes from simply understanding how the components interconnect, and obviously current AI can do this easily better than humans.
The hard part is the robotics.
The ‘simple’ parts like “connect a multimeter to <point 1>, <point 2>. “sand off the corrosion, wipe off the grease”, “does the oil have metal shavings in it”, “remove that difficult to reach screw” are what has been a bottleneck for 60 years.
You can’t just tell the robot “automate the production of rubber gloves”. You need humans to do a lot of work designing a robot that picks out the gloves and puts them on the hand shaped metal molds to the rubber can cure.
Yes economic growth exists. It’s not that fast. It really isn’t clear how AI fits into your discussion of robots.
Because it’s what humans want AI for, and due to the relationships between the variables, it is possible we will not ever get uncontrollable superintelligence before first building a lot of robots, ICs, collecting revenue, and so on.
Isn’t this supposed to be about AI? Are you expecting a regieme where
Most of the worlds compute is going into AI.
Chip production increases by A LOT (at least 10x) within this regieme.
Most of the AI progress in this regieme is about throwing more compute at it.
yes I think AI and robotics and compute construction are all interrelated. That log(Compute) means actually 10x probably is nowhere near enough for strong ASI.
I also personally think it is an...interesting...world model to imagine this ASI that can design a bridge or DNA editor, people are stupid enough to trust it, yet it cannot replace a rusty bolt on the underside of that same bridge or manipulate basic glassware in a lab.
lower the learning rate when the sim is less confident the real world estimation is correct
Adversarial examples can make an image classifier be confidently wrong.
Because it’s what humans want AI for, and due to the relationships between the variables, it is possible we will not ever get uncontrollable superintelligence before first building a lot of robots, ICs, collecting revenue, and so on.
You are talking about robots, and a fairly specific narrow “take the screws out” AI.
Quite a few humans seem to want AI for generating anime waifus. And that is also a fairly narrow kind of AI.
Your “log(compute)” term came from a comparison which was just taking more samples. This doesn’t sound like an efficient way to use more compute.
Someone, using a pretty crude algorithmic approach, managed to get a little more performance for a lot more compute.
I think you are assuming the above will happen. (the line in blue).
I am assuming the red line, and obviously by building on what we have incrementally.
If you were somehow a significant way up the blue line and trying to get robots to do anything useful, yes, you might get goodheart optimized actions that achieve the instructed result, maybe (if the ASI hasn’t chosen to betray this time since it can do so), but not satisfying all the thousands of constraints you implied but didn’t specify.
In more “slow takeoff” scenarios. Your approach can probably be used to build something that is fairly useful at moderate intelligence. So for a few years in the middle of the red curve, you can get your factories built for cheap. Then it hits the really steep part, and it all fails.
I think the “slow” and “fast” models only disagree in how much time we spend in the orange zone before we reach the red zone. Is it enough time to actually build the robots?
I assign fairly significant probabilities to both “slow” and “fast” models.
Well how do these variables interact?
the g factor(intelligence) of the ASI depends on the algorithm of the ASI times the log of compute assigned to it, C.
The justification for it not being linear is because non search NN lookups already capture most of the possible gain on a task. You can do more samples with more compute, increasing the probability of a higher score. I’m having trouble finding the recent paper where it turns out if you sample GPT-4 you do get higher score, with it scaling with the log of the number of samples.
Then at any given moment in time, the rate of improvement of the ASI’s algorithm is going to scale with current intelligence times a limit based on S.
What is S? I sensed you were highly skeptical of my “neural sim” variable until 2 days ago. It’s Sora + you can get collidable geometry, not just images as output. At the time, the only evidence i had of a neural sim was nvidia papers. Note the current model is likely capable of this:
https://x.com/BenMildenhall/status/1758224827788468722?s=20
S is the suite of neural sim situations. Once the ASI solves all situations in the suite, self improvement goes to 0. (there is no error derivative to train on)
This also come to mind naturally. Have you, @Donald Hobson , ever played a simulation game that models a process that has a real world equivalent? You might notice you get better and better at the game until you start using solutions that are not possible in the game, but just exploit glitches in the game engine. If an ASI is doing this, it’s improvement becomes negative once it hits the edges of the sim and starts training on false information. This is why you need neural sims, as they can continue to learn and add complexity to the sim suite (and they need to output a ‘confidence’ metric so you lower the learning rate when the sim is less confident the real world estimation is correct). How do you do this?
Well at time 0, humans make a big test suite with everything they know how to test for (“take all these exams in every college major, play all these video games and beat them”). Technically it’s not a fixed process, but an iterative one, but the benefit of that suite is limited. System already read all the books and watched all the videos and has taken every test humans can supply, at a certain point new training examples are redundant. (and less than perfect scores on things like college exams are because the answer key has errors, so there’s negative improvement if the AI memorizes the wrong answers)
The second part is the number of robots providing correct data. (because the real world isn’t biased like a game or exam, if a specific robot fails an achievable task it’s because the robot policy needs improvement)
Note the log here : this comes from intuition. In words, the justification is that immediately when a robot does a novel task, there will be lots of mistakes and rapid learning. But then the mistakes take increasingly larger lengths of time and task iterations to find them, it’s a logistic growth curve approaching an asymptote for perfect policy. This is also true for things like product improvement. Gen1->Gen2 you get the most improvement, and then over time you have to wait increasingly longer times to get feedback from the remaining faults, especially if your goal is to increase reliability. (example task: medical treatments for humans. You’ll figure out the things that kill humans immediately very rapidly, while the things that doom them to die in 10 years you have to wait 10 years for. more specific example: cancer treatments)
How fast do you get more robots?
where,
R(t) represents the number of robots at time t.
R0 is the initial number of robots at time = 0
tau is the time constant in years
@Daniel Kokotajlo and I estimated this time constant at 2-3 years. This is a rough guess, but note there are strong real world factors that make it difficult to justify a lower number.
First, what is a robot?
+
+
and it lives in a setup like this, just now with AI driven arms instead of the older limited control logic the bots pictured use.
1.0 robot is: (robotic arms) (with arm placement similar to humans so you can learn how to do tasks the human way initially) + sensors + compute racks + support infrastructure. (cages, power conveyers, mounts for the sensors)
Robot growth means you double the entire industrial chain. Obviously, in a factory somewhere, robots are making robots. That would take like, a few hours right?
But then you also copy the CNC machines to make the robot’s parts:
And the steel forges, chip fabs, mines, power sources, buildings, aluminum smelter...everything in the entire industrial chain you must duplicate or the logistic growth bottlenecks on the weak link.
At the very start, you can kick-start the process by getting together 7 trillion or so, and employing a substantial fraction of the working population of the planet to serve as your first generation of ‘robots’. You also can have the ASI optimize existing equipment and supply chains to get more productivity out of it.
In the later future, you can probably get nanotechnology:
Note this helps but it isn’t a magic pass to overnight exponential growth. Nanotechnology still requires power, cooling, raw materials processing, it needs a warehouse and a series of high vacuum chambers for the equipment to run in. It does simplify things a lot and has other advantages.
Another key note : probably to build the above, which is all the complexity of earth’s industry crammed into a small box, you will need some initial number of robots, on the order of hundreds of millions to billions, to support the bootstrapping process to develop and build the first nanoforge.
Finally, obviously,
Where you have your compute at time 0 (when you got to ASI) and it scales over time with respect to the number of robots you have.
Wait, don’t humans build compute? Nope. They haven’t for years.
Everything is automated. Humans are in there for maintenance and recipe improvement.
Another thing that’s special about compute is the “robots” aren’t general purpose. You need specific bottlenecked equipment that is likely the hardest task for robots to learn to build and maintain. (I suspect harder than learning to do surgery with a high patient survival rate, because bodies can heal from small mistakes, while this equipment will fail on most tiny mistakes.
Discussion: We can model the above equation and get a timeline in for the orange area, for different assumptions for the variables. Perhaps you and I could have a dialogue where we develop the model. I would like feedback on any shaky assumptions made.
First of all. SORA.
No. Not really. I wasn’t claiming that things like SORA couldn’t exist. I am claiming that it’s hard to turn them towards the task of engineering a bridge say.
Current SORA is totally useless for this. You ask it for a bridge, and it gives you some random bridge looking thing, over some body of water. SORA isn’t doing the calculations to tell if the bridge would actually hold up. But lets say a future much smarter version of SORA did do the calculations. A human looking at the video wouldn’t know what grade of steel SORA was imagining. I mean existing SORA probably isn’t thinking of a particular grade of steel, but this smarter version would have picked a grade, and used that as part of it’s design. But it doesn’t tell the human that, the knowledge is hidden in it’s weights.
Ok, suppose you could get it to show a big pile of detailed architectural plans, and then a bridge. All with super-smart neural modeling that does the calculations. Then you get something that ideally is about as good at looking at the specs of a random real world bridge. Plenty of random real world bridges exist, and I presume bridge builders look at their specs. Still not that useful. Each bridge has different geology, budget, height requirements etc.
Ok, well suppose you could start by putting all that information in somehow, and then sampling from designs that fit the existing geology, roads etc.
Then you get several problems.
The first is that this is sampling plausible specs, not good specs. Maybe it shows a few pictures at the end to show the bridge not immediately collapsing. But not immediately collapsing is a low bar for a bridge. If the Super-SORA chose a type of paint that was highly toxic to local fish, it wouldn’t tell you. If the bridge had a 10% chance of collapsing, it’s randomly sampling a plausible timeline. So 90% of the time, it shows you the bridge not collapsing. If it only generates 10 minutes of footage, you don’t know what might be going on in it’s sim while you weren’t watching. If it generates 100 years of footage from every possible angle, it’s likely to record predictions of any problems, but good luck finding the needle in the haystack. Like imagine this AI has just given you 100 years of footage. How do you skim through it without missing stuff.
Another problem is that SORA is sampling in the statistical sense. Suppose you haven’t done the geology survey yet. SORA will guess at some plausible rock composition. This could lead to you building half the bridge, and then finding that the real rock composition is different.
You need a system that can tell you “I don’t know fact X, go find it out for me”.
If the predictions are too good, well the world it’s predicting contains Super-SORA. This could lead to all sorts of strange self fulfilling prophecy problems.
I would agree with that.
Let me give you a summary of my overall position: “Trust an ASI with a complex task it needs context awareness to complete? Not even once”.
Everything is about how to get a large amount of benefits in the orange area, and yes we should stay there for a prolonged period.
What benefits specifically? Vast amounts of material manufactured goods, robots, robots are doing repetitive tasks that can be clearly defined. ASI models in use are only used in limited duration sessions, and you strip away context awareness.
Context awareness is the bits that tell the ASI this is a real bridge, humans are really going to build it, and not just another task in the training sim. “Always bluepill”. It should not be possible for the ASI to know when the task is real vs sim. (which you can do by having an image generator convert real frames to a descriptor, and then regenerate them so they have the simulation artifacts...)
The architecture fees for a new bridge design are about 10% of the cost.
Other 90% is, well, all the back breaking labor to build one. Not just at the construction site, but where the concrete was made, building/maintaining the trucks to haul the materials, etc.
Sora’s role: For autonomous cars and general robots, this is used for training. General robots means one robot at a time, and for short task sessions. For example : “remove the part from the CNC machine and place it on the output table”.
What you do is record video from the human workers doing the task, many hours of it. You train a model using techniques similar Sora to classify what it sees and to predict frame for frame the next frame. This is 5-30 hz sampling rate, so 33 ms into the future. Then do this recursively.
Since this is an RL problem you go further and may model a few seconds ahead.
There are ways to reproject in a robot instead.
Then the robot tries to do the task in the real world once it has settled on a policy that covers it in the simulated world.
The robot may fail, but you train the model, and you policy iterate with each failure. This is where the log(real world episodes) comes in. It’s correct and aligns very well with real world experience.
Note on Sora : Tesla and probably Waymo already use a neural sim that is similar to training their robotic cars. I found the tweet last night from an insider, and I’ve seen demo videos before, i can produce evidence. But it is accurate to say it’s already in use for the purpose I am talking about.
Bridges : at least in the near future, you want to automate making the parts for repairing a bridge, doing the dangerous work of inspection via drone, converting a bunch of pictures from many places from the drones to an overall model of the bridge compatible with load testing software, and so on. <https://artbabridgereport.org/reports/2021-ARTBA-Bridge-Report.pdf> current USA bridges need 7% replaced and work done on about 40%.
Key note : I would not trust an ASI or AGI with designing a new bridge, the goal is to make it cheaper to replace or repair the designs you already have. If AI + robots can contribute between 10% and 95% of the labor to replace a bridge (starting at 10, gradually ramping...), that lets you swap a lot more bridges for the same annual budget.
If we have the technical capacity to get into the red zone, and enough chips to make getting there easy. Then hanging out in the orange zone, coordinating civilization not to make any AI too powerful, when there are huge incentives to ramp the power up, and no one is quite sure where the serious dangers kick in...
That is, at least, an impressive civilization wide balancing act. And one I don’t think we have the competence to pull off.
This is something you want, not a description of how to get it, and one that is rather tricky to achieve. That converting and then converting back trick is useful. But sure isn’t automatic success either. If there are patterns about reality that the ASI understands, but the simulator doesn’t, then the ASI can use those patterns.
Ie if the ASI understands seasons, and the simulator doesn’t, then if it’s scorching sunshine one day and snow the next, that suggests it’s the simulation. Otherwise, that suggests reality.
And if the simulation knows all patterns that the ASI does, the simulator itself is now worryingly intelligent.
If the task is maximally repetitive, then the robot can just follow the same path over and over.
If it’s nearly that repetitive, the robot still doesn’t need to be that smart.
I think you are trying to get a very smart AI to be so tied down and caged up that it can do a task without going rouge. But the task is so simple that current dumb robots can often do it.
Economics test again. Minimum wage workers are easily up to a task like that. But most engineering jobs pay more than minimum wage. Which suggests most engineering in practice requires more skill than that.
I mean yes engineers do need to take parts out of the CNC machine. But they also need to be able to fix that CNC machine when a part snaps off inside it and starts getting jammed in the workings. And the latter takes up more time in practice. Or noticing that the toolhead is loose, and tightning and recalibrating it.
The techniques you are describing seem to be next level in fairly dumb automation. The stuff that some places are already doing (like boston dynamics robot dog level hardware and software), but expanded to the whole economy. I agree that you can get a moderate amount of economic growth out of that.
I don’t see you talking about any tasks that require superhuman intelligence.
Response to the rest of your post.
By the way, these comment boxes have built in maths support.
Press Ctrl M for full line or Ctrl 4 for inline
LikeThisCtrlMusedhereNeural sims probably have glitches too. Adversarial examples exist.
This sounds iffy. Like you are eyeballing and curve fitting, when this should be something that falls out of a broader world model.
Every now and then, you get a new tool. Like suppose your medical bot has 2 kinds of mistakes, ones that instant kill, and ones that mutate DNA. It quickly learns not to do the first one. And slowly learns not to do the second when it’s patients die of cancer years later. Except one day it gets a gene sequencer. Now it can detect all those mutations quickly.
I find it interesting that most of this post is talking about the hardware.
Isn’t this supposed to be about AI? Are you expecting a regieme where
Most of the worlds compute is going into AI.
Chip production increases by A LOT (at least 10x) within this regieme.
Most of the AI progress in this regieme is about throwing more compute at it.
Ok. And there is our weak link. All our robots are going to be sitting around broken. Because the bottleneck is human repair people.
It is possible to automate things. But what you seem to be describing here is the process of economic growth in general.
Each specific step in each specific process is something that needs automating.
You can’t just tell the robot “automate the production of rubber gloves”. You need humans to do a lot of work designing a robot that picks out the gloves and puts them on the hand shaped metal molds to the rubber can cure.
Yes economic growth exists. It’s not that fast. It really isn’t clear how AI fits into your discussion of robots.
Yes. That’s why I specifically mentioned :
Confidence is a trainable parameter, and you scale down learning rate when confidence is low.
This is a lengthy discussion but the simple answer is that what a human ‘repair person’ does can be described as a simple algorithm that you can write in ordinary software. I’ve repaired a few modern things, this is from direct knowledge and watching videos of someone repairing a Tesla.
The algorithm in essence is every module is self diagnosing, and there is a graph flow of relationships between modules. There are simple experiments you do, it tells you in the manual many of them, to get better evidence.
Then you disassemble the machine partially—if you were a robot and had the right licenses you could download the assembly plan for this machine and reverse it—remove the suspect module, and replace. If the issues don’t resolve, you remove the module that said the suspect module was bad or was related to it.
For PCs, this is really easy. Glitches on your screen from your GPU? Replace the cable. Observe if the glitches go away. Try a different monitor. Still broken? Put in a different GPU. That doesn’t resolve it? Go and memtest86 the RAM. Does that pass? It’s either the motherboard or the processors.
This comes from simply understanding how the components interconnect, and obviously current AI can do this easily better than humans.
The hard part is the robotics.
The ‘simple’ parts like “connect a multimeter to <point 1>, <point 2>. “sand off the corrosion, wipe off the grease”, “does the oil have metal shavings in it”, “remove that difficult to reach screw” are what has been a bottleneck for 60 years.
Because it’s what humans want AI for, and due to the relationships between the variables, it is possible we will not ever get uncontrollable superintelligence before first building a lot of robots, ICs, collecting revenue, and so on.
yes I think AI and robotics and compute construction are all interrelated. That log(Compute) means actually 10x probably is nowhere near enough for strong ASI.
I also personally think it is an...interesting...world model to imagine this ASI that can design a bridge or DNA editor, people are stupid enough to trust it, yet it cannot replace a rusty bolt on the underside of that same bridge or manipulate basic glassware in a lab.
Adversarial examples can make an image classifier be confidently wrong.
You are talking about robots, and a fairly specific narrow “take the screws out” AI.
Quite a few humans seem to want AI for generating anime waifus. And that is also a fairly narrow kind of AI.
Your “log(compute)” term came from a comparison which was just taking more samples. This doesn’t sound like an efficient way to use more compute.
Someone, using a pretty crude algorithmic approach, managed to get a little more performance for a lot more compute.