it’s not obvious to me that strong takeoff is possible. I’d give it 80% probability that it is right now, but there’s significant weight on logarithmically diminishing returns such that the things that are strong than us
Other than compute requirements, have you considered what kinda of cognitive tasks you would assign a model to complete that would lead to developing this kind of superintelligence?
Remember you started with random numbers. In a sense, the annealing and SGD is saying in words “find the laziest function that will solve this regression robustly”. For LLMs the regression is between a calculated number of input tokens and the next token continuation.
The input set is so large that the “laziest” algorithm that has the least error seems to mimic the cognitive process humans use to generate words, which then guesses the most tokens.
And then RL etc after that.
So when I think of the kinds of things a superintelligence is supposed to be able to do, I ask myself how you would build a valid test where the model will be able to do this in the real world.
This is harder than it sounds because in many cases, if something is “correct” or not depends on information current sims can’t model well. For example particle sims don’t model armor perfectly, FSM sims don’t model the wear of mechanical parts correctly, and we have only poor models for how a human would respond to a given set of words said a specific way.
So like the flaw here is say you come up with tasks humans can’t quite do on their own, but the sim can check if it were done right. “Design an airliner” for instance. So the sim models every bolt and the cockpit avionics and so on. All of it.
And an ASI model is trained and it can do this task. Humans cannot beat this “game” because there are millions of discrete parts.
But any aircraft the ASI designs have horrific failure modes and crash eventually. Because the sim is just a little off. And the cockpit avionics HMI is unusable because the model of what humans can perceive is slightly off.
So you collect more data and make the model better and so on, but see it’s functionally “ceilinged” at just a bit better than humans because the model becomes just superintelligent enough to max out the airline design task and no more, or just smart enough to max out the suite of similar tasks.
It’s also not going to be able to ever 1 shot a real airplane, just get increasingly close.
yeah this sounds like a reasonable description of the importance of extremely high quality data. that training data limit I ended on is not trivial by any means.
80 percent confidence seems unsupported by evidence. With the evidence that human data is very poor—for clear and convincing evidence look at all the meta analysis of prior studies, or the constant “well actually” rebuttals to facts people thought they knew. (And then a rebuttal rebuttal and in the end nobody knows anything) A world where analyzing all the data humans know on a subject leaves most people less confident about anything than before they started is not one where we have the data to train a superintelligence.
Such a machine will be as confused as we are even if it has the memory to simultaneously assume every single assumption is both true and not true, and keep track of the combinatorial explosion of possibilities.
To describe the problem succinctly: if you have a problem that only a superintelligence can solve in front of you, and your beliefs about all the variables form a tree with hundreds of millions of possibilities (medical problems will be this way), you may have the cognitive capacity of a superintelligence but in actual effectiveness your actions will be barely better than humans. As in functionally not an ASI.
Getting the data is straightforward. You just need billions of robots. You replicate every study and experiment humans ever did with robots this time, you replicate human body failures with “reference bodies” that are consistent in behavior and artificial. All data analysis is done from raw data, all conclusions always take into account all prior experiments data, no p-hacking.
We don’t have the robots yet, though apparently Amazon robotics is on an exponential trajectory, having added 750k in the last 2 years, which is more than all prior years combined.
Assuming the trajectory continues, it will be 22 years until 1 billion robots. Takeoff but not foom.
Other than compute requirements, have you considered what kinda of cognitive tasks you would assign a model to complete that would lead to developing this kind of superintelligence?
Remember you started with random numbers. In a sense, the annealing and SGD is saying in words “find the laziest function that will solve this regression robustly”. For LLMs the regression is between a calculated number of input tokens and the next token continuation.
The input set is so large that the “laziest” algorithm that has the least error seems to mimic the cognitive process humans use to generate words, which then guesses the most tokens.
And then RL etc after that.
So when I think of the kinds of things a superintelligence is supposed to be able to do, I ask myself how you would build a valid test where the model will be able to do this in the real world.
This is harder than it sounds because in many cases, if something is “correct” or not depends on information current sims can’t model well. For example particle sims don’t model armor perfectly, FSM sims don’t model the wear of mechanical parts correctly, and we have only poor models for how a human would respond to a given set of words said a specific way.
So like the flaw here is say you come up with tasks humans can’t quite do on their own, but the sim can check if it were done right. “Design an airliner” for instance. So the sim models every bolt and the cockpit avionics and so on. All of it.
And an ASI model is trained and it can do this task. Humans cannot beat this “game” because there are millions of discrete parts.
But any aircraft the ASI designs have horrific failure modes and crash eventually. Because the sim is just a little off. And the cockpit avionics HMI is unusable because the model of what humans can perceive is slightly off.
So you collect more data and make the model better and so on, but see it’s functionally “ceilinged” at just a bit better than humans because the model becomes just superintelligent enough to max out the airline design task and no more, or just smart enough to max out the suite of similar tasks.
It’s also not going to be able to ever 1 shot a real airplane, just get increasingly close.
yeah this sounds like a reasonable description of the importance of extremely high quality data. that training data limit I ended on is not trivial by any means.
80 percent confidence seems unsupported by evidence. With the evidence that human data is very poor—for clear and convincing evidence look at all the meta analysis of prior studies, or the constant “well actually” rebuttals to facts people thought they knew. (And then a rebuttal rebuttal and in the end nobody knows anything) A world where analyzing all the data humans know on a subject leaves most people less confident about anything than before they started is not one where we have the data to train a superintelligence.
Such a machine will be as confused as we are even if it has the memory to simultaneously assume every single assumption is both true and not true, and keep track of the combinatorial explosion of possibilities.
To describe the problem succinctly: if you have a problem that only a superintelligence can solve in front of you, and your beliefs about all the variables form a tree with hundreds of millions of possibilities (medical problems will be this way), you may have the cognitive capacity of a superintelligence but in actual effectiveness your actions will be barely better than humans. As in functionally not an ASI.
Getting the data is straightforward. You just need billions of robots. You replicate every study and experiment humans ever did with robots this time, you replicate human body failures with “reference bodies” that are consistent in behavior and artificial. All data analysis is done from raw data, all conclusions always take into account all prior experiments data, no p-hacking.
We don’t have the robots yet, though apparently Amazon robotics is on an exponential trajectory, having added 750k in the last 2 years, which is more than all prior years combined.
Assuming the trajectory continues, it will be 22 years until 1 billion robots. Takeoff but not foom.