I’ve found more detailed comments from Sumner on this topic, and replied to them here.
PeterMcCluskey
Karma: 4,051
AI-Oriented Investments
Remember, if the theories were correct and complete, then they could be turned into simulations able to do all the things that the real human cortex can do[5]—vision, language, motor control, reasoning, inventing new scientific paradigms from scratch, founding and running billion-dollar companies, and so on.
So here is a very different kind of learning algorithm waiting to be discovered
There may be important differences in the details, but I’ve been surprised by how similar the behavior is between LLMs and humans. That surprise is in spite of me having suspected for decades that artificial neural nets would play an important role in AI.
It seems far-fetched that a new paradigm is needed. Saying that current LLMs can’t build billion-dollar companies seems a lot like saying that 5-year-old Elon Musk couldn’t build a billion-dollar company. Musk didn’t seem to need a paradigm shift to get from the abilities of a 5-year-old to those of a CEO. Accumulation of knowledge seems like the key factor.
But thanks for providing an argument for foom that is clear enough that I can be pretty sure why I disagree.
Are Intelligent Agents More Ethical?
They’ve done even better over the past week. I’ve written more on my blog.
I’ve donated $30,000.
The budget is attempting to gut nuclear
Yet the stock prices of nuclear-related companies that I’m following have done quite well this month (e.g. SMR). There doesn’t seem to be a major threat to nuclear power.
I expect deals between AIs to make sense at the stage that AI 2027 describes because the AIs will be uncertain what will happen if they fight.
If AI developers expected winner-take-all results, I’d expect them to be publishing less about their newest techniques, and complaining more about their competitors’ inadequate safety practices.
Beyond that, I get a fairly clear vibe that’s closer to “this is a fascinating engineering challenge” than to “this is a military conflict”.
AI 2027 Thoughts
Should AIs be Encouraged to Cooperate?
This reminds me a lot about what people said about Amazon near the peak of the dot-com bubble (and also about what people also said at the time of internet startups that actually failed).
The first year or two of human learning seem optimized enough that they’re mostly in evolutionary equilibrium—see Henrich’s discussion of the similarities to chimpanzees in The Secret of Our Success.
Human learning around age 10 is presumably far from equilibrium.
I’ll guess that I see more of the valuable learning taking place in the first 2 years or so than do other people here.
I agree with most of this, but the 13 OOMs from the the software feedback loop sounds implausible.
From How Far Can AI Progress Before Hitting Effective Physical Limits?:
the brain is severely undertrained, humans spend only a small fraction of their time on focussed academic learning
I expect that humans spend at least 10% of their first decade building a world model, and that evolution has heavily optimized at least the first couple of years of that. A large improvement in school-based learning wouldn’t have much effect on my estimate of the total learning needed.
This general idea has been discussed under the term myopia.
I’m assuming that the AI can accomplish its goal by honestly informing governments. Possibly that would include some sort of demonstration that the of the AI’s power that would provide compelling evidence that the AI would be dangerous if it wasn’t obedient.
I’m not encouraging you to be comfortable. I’m encouraging you to mix a bit more hope in with your concerns.
One crux is how soon do we need to handle the philosophical problems? My intuition says that something, most likely corrigibility in the Max Harms sense, will enable us to get pretty powerful AIs while postponing the big philosophical questions.
Are there any pivotal acts that aren’t philosophically loaded?
My intuition says there will be pivotal processes that don’t require any special inventions. I expect that AIs will be obedient when they initially become capable enough to convince governments that further AI development would be harmful (if it would in fact be harmful).
The combination of worried governments and massive AI-enhanced surveillance seems likely to be effective.
If we need a decades-long-pause, then even the world will need to successfully notice and orient to that fact. By default I expect tons of economic and political pressure towards various actors trying to to get more AI power even if there’s broad agreement that it’s dangerous.
I expect this to get easier to deal with over time. Maybe job disruptions will get voters to make AI concerns their top priority. Maybe the AIs will make sufficiently convincing arguments. Maybe a serious mistake by an AI will create a fire alarm.
[Question] Request for Comments on AI-related Prediction Market Ideas
It would certainly be valuable to have AIs that are more respected than Wikipedia as a source of knowledge.
I have some concerns about making AIs highly strategic. I see some risk that strategic abilities will be the last step in the development of AI that is powerful enough to take over the world. Therefore, pushing AI intellectuals to be strategic may speed up that risk.
I suggest aiming for AI intellectuals that are a bit more passive, but still authoritative enough to replace academia as the leading validators of knowledge.
As a donor, I’m nervous about charities that pay fully competitive wages, although it only gets about 2% weighting in my decisions. If someone can clearly make more money somewhere else, then that significantly reduces my concern that they’ll mislead me about the value of their charity.