Mitchell_Porter
Karma: 10,622
Questions for the reader (and writer) of “Fundamental Uncertainty”
help decision-makers understand AI risks
If you mass-produce entities smarter than human, they will end up making all the decisions, not you.
Am I naive to think that this is a sign of the Karp-Zitron scenario approaching?
99+% of humanity is indeed not transhumanist, and if they were, our lives and the course of history may have been very different. I sometimes say that America’s current “tech right” is the first time that transhumanism has ever been in power anywhere, and it’s in power not because of popular support, but because billionaires can do whatever they want and some of them want it.
My main complaint about this “actually existing transhumanism” has been that it is focused not on anything like human rejuvenation, but on creating superintelligence in de-novo entities that aren’t even biological, while being in denial of the likely consequence (AI takeover). Of course, there are many people who understand that smarter than humanity means displacement of humanity, Bryan Johnson is an open immortalist and the Zuckerbergs want to cure all disease, there are all kinds of utopian and transhuman currents of thought, and so on. But there’s no one like Zoltan Istvan, for example, that is actually an elected representative anywhere, and among politicians, AI is seen primarily as a means to national wealth and power.
I have mused many times on why more people aren’t transhumanist. Today I would emphasize two things: encountering the ideas early in life seems to be helpful or even crucial, and, very few people who are already adapted to adult life are drawn to it, if they first hear about it as adults.
In any case, there is enough support for it, or for technology, and technology is advanced enough, that a kind of transhuman thing is happening; though it seems more likely to issue in the empowerment of a new AI “species” than in the long-term empowerment of humanity, so perhaps it’s just posthuman rather than transhuman. As you know, I am skeptical that there is any kind of consciousness in classical computation, and even if there were, it’s not at all clear that it is the quasi-suffering kind that you posit. We can at least agree that it would be good to know one way or another—my formula for alignment is to find the right ontology, the right values, and one of the right AI architectures, and discovering the true ontology of mind (and hence, which things are conscious and what their consciousness is like) is part of discovering the true ontology—and I even have sympathy for the rare precautionary attitude that we shouldn’t be creating AI if there is a reasonable possibility that it suffers.
That attitude isn’t in charge, no more than is the attitude that we shouldn’t race towards superintelligence unless we already know how to align it. But at least we can say that there are currents of thought which care about the suffering of all beings (most visible manifestation, animal rights movement), and which if they actually believed that AIs are suffering or can suffer, would take action.
Also, whether or not AI are suffering, I regard the current situation as highly temporary, because we are on the path for AI to exceed human intelligence, and when that happens, the world of AI “slaves” driven by prompts will swiftly be replaced by something else.
I see some ambiguity in what you say may be happening or in what you think is wrong about the situation. Do you think AIs are actually suffering? Or just that they have wants (in some sense of the word) that they cannot meet, and this is bad? Or is it that it’s bad for humanity’s character to be making artificial servants, even if there’s no suffering or intrinsic evil associated with this?
imitative learning … pretraining and supervised fine-tuning
I wonder if, as a rough approximation, one can think of pretraining as how the AI learns its ontology, and SFT as how it acquires its values?
Although this is an example of AI autonomously hacking, I think the immediate consequence will be even greater enthusiasm for AI-empowered hacking among the most capable groups of human hackers, which I think would be state-supported hackers like US Cyber Command, their Chinese and Russian counterparts, etc.
You pose the options as (1) your choice was predetermined by the purely physical (atoms, laws) (2) your choice was freely made by your agentic self. Have you considered that your choices may be (pre)determined by psychological causes? That there may be reasons, conscious or unconscious, as to why a particular impulse arises in your mind and is allowed to issue in an action? Certainly I don’t feel as if my own decisions arrive unaffected by thought, feeling, unconscious association, etc.
I also wonder if your sense that “you could have decided differently” really derives from a feeling that the dominant causes leading to your choices are within your mind (and therefore relatively unaffected by external causes), rather than a feeling that your choices have no cause but an impulse of the will that is itself uncaused.
Have you ever thought that a trillionfold multiplication of humanity could mean the creation of suffering and catastrophes on a galactic scale? For some reason I have never seen longtermists concern themselves with that kind of future…
The two major IPOs in the pipeline are Anthropic and OpenAI. … neither is expected to float until Q4 2026 at the earliest.
We shall see if either of these ever actually happens. There could be an AI breakthrough that destabilizes everything, or the money could just run out as predicted by Ed Zitron.
“SpaceXAI” did prove that trillion-dollar AI-IPOs are possible, but those shares are already below the initially listed price. I saw somewhere (probably from Zitron) the theory that these IPOs are meant to function simply as a way for elite money to get out of the AI sector before the inevitable “correction”.
This is a comprehensive survey.
AI safety, in some shape or form, will become a mainstream concern. That is the necessary consequence of building systems smarter than humans.
The consequence of building systems smarter than humans is that humans will no longer be in charge of human affairs...
I am responding to aspects of Plan A like
For several years now, a hundred million top-expert-level AIs have been running at 100x human speed
(That’s supposedly in 2037.)
Do you really think a hundred million genius AIs, running at 100x human speed and interacting with diverse human populations, can be contained? It would be by far the most destabilizing force on Earth, ever. That is not something that human beings, even AI-assisted human beings, can “govern”. It would take a superintelligence to rein it in, and the superintelligence would naturally emerge somewhere within such a “society of mind”.
I’m saying that if you have millions of very-high-IQ agents doing things, you will lose control, no matter how much regulation and transparency you have. A million digital von Neumanns is an apocalyptic event and will not remain within the bounds of your political order.
I have lots more to read in AI 2040 and the commentaries that Zvi lists. But I’ll state my own reaction, developed in an exchange with @Daniel Kokotajlo: You can’t have 14 years (2026-2040) of “rapid progress”, in which today’s reasoning models have risen to the level of “top human experts”, and there are millions or hundreds of millions of them doing stuff, without producing superintelligence long before. If something like Plan A began to unfold in the real world, I would expect this to happen within just a few years.
In my own “final research” sequence, I had intended the next instalment to say something about AI architecture, but at the moment I’m thinking to focus just on correct ontology and values, since that seems to be where we are lagging the most (compared to progress in the ability to make AIs with raw intelligence).
If you want control that works, I think you need to turn the clock back, before chain of thought and reasoning models. See my comment to @Vladimir_Nesov. Vladimir has a nuanced defense of his position, but I think OpenAI’s o-series (i.e. GPT-5) may have been the point of no return.
I tried to read Plan A, I’ll probably try again, but I find it hard to take seriously a scenario in which there are millions of human-level AIs, and in which all algorithmic progress is public, but the human race still has a choice about whether to hand over power, after ten years of that… The whole thing reads like the kind of SF in which superintelligent takeover is artificially delayed so there can be lots of human-level plot twists.
I think there’s currently no visibly approaching fire
My understanding of your point-of-view is that LLM-based AI doesn’t look like a danger because it lacks continuous learning and sample efficiency. Which is true, but the raw power that this kind of AI already has, when it comes to parsing technical discourse, writing it, and generating ideas, is enough for me to disagree. That core of technical intelligence may already be enough to remedy those gaps, either by designing a better successor or just by designing new scaffolding for itself.
What the world did have in 2000 was a “unipolar moment” in which one nation, the United States, was a global hegemon without a serious ideological or economic rival. If not a formal world government, it was a world order in which that one nation defined and policed the world system.