TAG

Karma: 1,487

About Me

Scientist by training, coder by previous session,philosopher by inclination, musician against public demand.

https://theancientgeek.substack.com/?utm_source=substack&utm_medium=web&utm_campaign=substack_profile

Why I am not a Doomer

I’m specifically addressing the argument for a high probability of near extinction (doom) from AI...

Eliezer Yudkowsky: “Many researchers steeped in these issues, including myself, expect that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. ”

....not whether it is barely possible, or whether other, less bad outcomes (dystopias) are probable. I’m coming from the centre, not the other extreme

Doom, complete or almost complete extinction of humanity, requires a less than superintelligent AI to become superintelligent either very fast , or very surreptitiously … even though it is starting from a point where it does not have the resources to do either.

The “very fast” version is foom doom...Foom is rapid recursive self improvement (FOOM is supposed to represent a nuclear explosion)

The classic Foom Doom argument (https://www.greaterwrong.com/posts/kgb58RL88YChkkBNf/the-problem) involves an agentive AI that quickly becomes powerful through recursive self improvement, and has a value/goal system that is unfriendly and incorrigible.

The complete argument for Foom Doom is that:-

The AI will have goals/values in the first place (it wont be a passive tool like GPT*),.
The values will be misaligned, however subtly, to be unfavorable to humanity.
That the misalignment cannot be detected or corrected.
That the AI can achieve value stability under self modification.
That the AI will self modify in way too fast to stop.
That most misaligned values in the resulting ASI are highly dangerous (even goals that aren’t directly inimical to humans can be a problem for humans, because the AS I might want to director sources away from humans.
And that the AI will have extensive opportunities to wreak havoc: biological warfare (custom DNA can be ordered by email), crashing economic systems (trading can be done online), taking over weapon systems, weaponing other technology and so on.

It’s a conjunction of six or seven claims, not just one. ( I say “complete argument ” because pro doomers almost always leave out some stages. I am not convinced that rapid self improvement and incorrigibility are both needed, both needed, but I am sure that one or the other is. Doomers need to reject the idea that misalignment can be fixed gradually, as you go along. . A very fast-growing ASI, foom, is way of doing that; and assumption that AI’s will resist having their goals changed is another).

Obviously the problem is that to claim a high overall probability of doom, each claim in the chain needs to have a high probability. It is not enough for some of the stages to be highly probable, all must be.

There are some specific weak points.

Goal stability under self improvement is not a given: it is not possessed by all mental architectures, and may not be possessed by any, since noone knows how to engineer it, and humans appear not to have it.

The Orthogonality Thesis (https://www.lesswrong.com/w/orthogonality-thesis)is sometimes mistakenly called on to support to support goal stability. It implies that a lot of combinations of goals and intelligence levels are possible, but doesn’t imply that all possible minds have goals, or that all goal driven agents have fixed, incorrigible goals. There are goalless and corrigible agents in mindspace, too. That’s not just an abstract possibility. At the time of writing, 2025, our most advanced AI’s, the Large Language Models, are non agentive and corrigible.

It is plausible that an agent would desire to preserve its goals, but the desire to preserve goals does not imply the ability to preserve goals. Therefore, no goal stable system of any complexity exists on this planet, and goal instability cannot be assumed as a default or given. So the orthogonality thesis is true of momentary combinations of goal and intelligence, given the provisos above, but not necessarily true of stable combinations.

Another thing that doesn’t prove incorrigibility or goal stability is von Neumann rationality. Frequently appealed to in MIRI ’s early writings , it is an idealised framework for thinking about rationality , that doesn’t app!y to humans, and therefore doesn’t have to apply to any given mind.

There are arguments that AI’s will become agentive because that”s what humans want. Gwerns Branwen’s confusingly titled “Why Tool AIs Want to Be Agent AIs” ( https://gwern.net/tool-ai) is an example. This is true, but in more than one sense:-

The basic idea is that humans want agentive AI’s because they are more powerful. And people want power, but not at the expense of control. Power that you can’t control is no good to you. Taking the brakes off a car makes it more powerful, but more likely to kill you. No army wants a weapon that will kill their own soldiers, no financial organisation wants a trading system that makes money for someone else, or gives it away to charity, or causes stick market crashes. The maximum amount of power and the minimum of control is an explosion.

One needs to look askance at what “agent” means as well. Among other things, it means an entity that acts on behalf of a human—as in principal/agent.(https://en.m.wikipedia.org/wiki/Principal–agent_problem) An agent is no good to its principal unless it has a good enough idea of its principal’s goals. So while people will want agents, they wont want misaligned ones—misalgined with themselves, that is. Like the Orthogonality Thesis, the argument is not entirely bad news.

Of course, evil governments and corporations controlling obedient superintelligences isn’t a particularly optimistic scenario, but it’s dystopia, not doom.

Yudkowsky’s much repeated argument that safe , well-aligned behaviour is a small target to hit … could actually be two arguments.

One would be the random potshot version of the Orthogonality Thesis, where there is an even chance of hitting any mind, and therefore a high chance ideas of hitting an eldritch, alien mind. But equiprobability is only one way of turning possibilities into probabilities, and not particularly realistic. Random potshots aren’t analogous to the probability density for action of building a certain type of AI, without knowing much about what it would be.

While, many of the minds in mindpsace are indeed weird and unfriendly to humans, that does not make it likely that the AIs we will construct will be. we are deliberately seeking to build certainties of mind for one thing, and have certain limitations, for another. Current LLM ’s are trained in vast copora of human generated content, and inevitably pick up a version of human values from them.

Another interpretation of the Small Target Argument is, again , based on incorrigibility. Corrigibility means you can tweak an AI’s goals gradually, as you go on, so there s no need to get them exactly right on the first try.

TAG 20 Oct 2025 16:23 UTC
2 points
2
on: Contra-Zombies? Contra-Zombies!: Chalmers as a parallel to Hume

Hume points out that nothing guarantees causal induction

The success of that argument depends on how strongly “guarantee” is taken.

Induction, considered as a form of deduction takes a finite list of similar observations, (A ^ B ^ C.) and models.the general law it leads to as ( A^ B ^ C ^ D ..). Since even the D is not deductively valid, the whole thing is invalid. Or is it?

Inductivists aren’t persuaded. The traditional view is that induction is a separate magisterium anyway. The problem with that approach is ..where does it end? Is tea leaf reading a separate epistemic magisterium?

The problem of induction can be rendered easier to solve by aiming lower: one can regard a set of observations as rendering a continuation of the pattern more likely, without rendering it certain. In further good news, that can be fast as kind of deductive justification, because it can be justified by probabilistic loguc, which is An extension if deductive logic.

I think Eliezer Yudkowsky and Chalmers agree in a similar way to how Hume agreed with The Enlightenment. Both Yudkowsky and Chalmers see the issue of p-zombies as unimportant, something to be assumed away

But it is not unimportant to Chalmers who needs it to argue against materialism.

And avoidance of Epiphenomenality is still important to Yudkowsky. I don’t see how the Humean argument addresses it.

We do know how Chalmers addresses it …

I endorse Z[ombies], but I don’t endorse E[piphenomenalism].

...up to a point. On the other hand, he’s pretty ambiguous about the exact solution …

I think that the correct conclusion of zombie-style arguments is the disjunction of the type-D, type-E, and type-F views,

...making difficult to say with certainty that

Chalmers is a minimal dualist, arguably practically a physicalist

TAG 18 Oct 2025 10:14 UTC
2 points
0
on: Nontrivial pillars of IABIED
The cruxes you have picked out are not the ones I would have.

The complete argument for complete extinction rests on assumptions...assumptions about the nature of intelligence, the motivations of an artificial intelligence and the means of bringing about extinction. And the conjunctive part of the argument consists of claims which need to be of high probability individually for the conclusion to be of high probability.
1. Artificial Intelligence greater than human intelligence is possible.
2. The AI will be an agent, it have goals/values in the first place.
3. The goals will be misaligned, however subtly, to be unfavorable to humanity.
4. That the misalignment between the AI’s goals, and what we want, cannot be corrected incrementally (incorrigibility), because
5a. …the AI will self modify in way too fast to stop.(With a sub assumption that the AI can achieve value stability under self modification.)

Or

5b. …the AI will.The engage in deception about its powers or motivations.
1. That most misaligned values in the resulting ASI are highly dangerous (even goals that aren’t directly inimical to humans can be a problem for humans)
2. And that the AI will have extensive opportunities to wreak havoc: biological warfare (custom DNA can be ordered by email), crashing economic systems (trading can be done online), taking over weapon systems, weaponing other technology and so on.
Obviously the problem is that to claim a high overall probability of doom, each claim in the chain needs to have a high probability. It is not enough for some of the stages to be highly probable, all must be. In my opinion, the weakest parts of the argument are the ones dealing with the motivation, steps 2 to 6, not the ones dealing with the natures of intelligence and the means of destruction, (1 and 7). There’s an obvious problem in making specific high probability claims about systems that don’t yet exist.

@Mishka

It’s kind of strange. Why do we want to have a technical ability for any Mr.X from the defense department of a superpower Y to impart his goals and values to some ASI? It’s very easy to imagine how this could be detrimental

Yes, but not everybody’s-dead detrimental.

Doomers are concerned about imparting values because they believe that we are going to end up with an incorrigible Sovereign AI running everything, not a multipolar scenario with superpowers using aligned-with-themselves superintelligences as superweapons.

TAG 17 Oct 2025 13:09 UTC
2 points
0
in reply to: Nicolas Villarreal’s comment on: Materialist Semiotics and the Nature of Qualia

I don’t see why camp #1 requires dualism

Camp 1 is unable to account for self evident experience qua experience, it creates an outside to physical reality in saying “this isn’t worth addressing”, there’s what’s real and, perhaps uncharitably, what’s fiction. To create a truly material monist ontology you must create a material account of this fiction or belief or whatever experience.

Camp 1 is everyone who thinks that qualia are explicable within a materialist ontology. .. so it includes all attempted matetialist explanations, successful or not ..so it includes yours.

TAG 16 Oct 2025 14:42 UTC
2 points
0
in reply to: Sen’s comment on: Orthogonality Thesis

You cannot simultaneously assert that the agent is “sufficiently advanced” such that it can always think at or above human-level higher order reasoning, and also that it will not be able to re-prioritise its terminal goals (survivability, paperclips, etc).

Almost. A given agent is not necessary capable of changing its own goals … but it’s not necessarily goal stable either:-

Goal stability is an assumption behind recursive self improvement, which is an argument behind rapid takeoff, which is an argument behind getting goals/value right on the first attempt

It is plausible that an agent would desire to preserve its goals, but the desire to preserve goals does not imply the ability to preserve goals.

An argument for the desire to preserve goals is the “Ghandi Pill” argument.

“See the “Ghandi argument”—if you offer Ghandi a pill that makes him love murdering people, he won’t take the pill because right now he doesn’t want to murder people. A self-modifying AI that wants things will tend to avoid changing what it wants, because it can predict that would lead to things it doesn’t want”

Humans do not preserve the same goals across their lifetimes, and do things which are known to change values, such as getting educated, travelling, and getting married. So the Ghandi Pill argument only seems to to apply to extreme cases, in humans. And there a no reason to suppose that any given AI will b different to a human—the human mind is a possibility in mindspace.

The Orthogonality Thesis is also sometimes mistakenly called on to support goal stability. It does imply that a lot of combinations of goals and intelligence levels are possible, but doesn’t imply that all possible minds have goals, or that all goal driven agents have fixed, incorrigible goals. There are goalless and corrigible agents in mindspace, too.

No goal stable system of any complexity exists on this planet, and goal instability cannot be assumed as a default or given. So the orthogonality thesis is true of momentary combinations of goal and intelligence, given the provisos above, but not necessarily true of stable combinations.

There is, however, a puzzle about how an agent would rationally update its goals. AI doomers like to think in terms of von Neumann rationality.

According to the framework of von Neumann rationality, such an update must be guided by a set of terminal goals … but how can it be when the agents goals themselves are subject to change?

But the puzzle is rather artificial and inconsequential. For one thing, vNM is a highly artificial framework, with the advantage of functionally well defined, and the disadvantage of being inapplicable to almost any real world system, including most AI s. For another, rational people—in the informal sense—are capable of reflecting on , and changing, their goals and values. (https://www.lesswrong.com/posts/8A6wXarDpr6ckMmTn/another-argument-against-utility-centric-alignment-paradigms).?

Irrational goal-drift is possible, too. One thing the Orthogonality Thesis does show is that most possible minds aren’t artificial philosophers that will converge onto the One True Morality.

TAG 12 Oct 2025 10:21 UTC
2 points
0
on: Materialist Semiotics and the Nature of Qualia

A sign has meaning by comparison to things that it’s not. Saussure in his “Course in General Linguistics” shows effectively how the differentiation of a given spoken word from other words is what allows us to assign a meaning to it, if a word begins to be pronounced in two different ways in different contexts then this difference in pronunciation can take on a conceptual difference.[2]

Which amounts to the claim that difference is necessary for meaning. The more contentious claim is that it is sufficient.

tiated from each other in order for them to be connected back to signifiers. Philosophers obsessed with qualia often ask things like what gives the red of a rose its redness. From a semiotic perspective the answer is ready at hand: the redness of a rose comes from its differentiation from green and black and blue and all the other colors of all the other objects and their associated textures, smells, etc

You seem to be hinting at a theory of pure difference, that Qualia are not intrinsic properties with difference between , but constituted by the differences alone. That’s a very difficult theory to make work. If Red is constitutes by being differentiate blue and green , blue is constituted by being different to red and and green, etc, then each of them is an (unspecified) X that is different to an (also unspecified) Y and Z....so they are all the same! The situation is even worse dealing with normatively laden qualia like pleasure and pain. Does pain really have n o characteristic s other than being different from pleasure?

Having said that, I can’t be certain that you are putting forward a theory of pure difference. If instead you are just saying that qualia are [something unknown] that happens to have a structure of differences, then you are making a statement anyone could agree with and not solving the problem.

People might produce a more or less fuzzy and three dimensional projection of a rose in their mind, but no matter how sharp it is, the qualities the mental construct has are present on a spectrum where the other positions of that spectrum are other possible characteristics of objects. Green vs woody stems, red petals vs white ones, thorns vs fuzz, and so on.

Yes, qualia space has a structure. It doesn’t follow from that a quale is characterised entirely by its location in qualia space. Qualia space would still have a structure,.If Qualia had intrinsic properties instead of being pure differences.That’s the necessity/sufficiency distinction again.

Now that we’ve established that ideas and qualia are signs

That has been assumed, not established.

Any and all systems of signs that process an input according to a preexisting code necessarily have qualia, and share the sort of negativity which philosophers have so long restricted to the human subject. Your dog has qualia, your phone has qualia, your infection has qualia.

Obviously not, since humans have conditions in which qualia can go missing, or become hypertrophic, eg. colour blindness and synaesthesia.

I’m purposefully trying to avoid either camp as both camps require a type of dualism

I don’t see why camp #1 requires dualism.

TAG 7 Oct 2025 14:06 UTC
16 points
8
in reply to: Kaj_Sotala’s comment on: abramdemski’s Shortform

But brainwashing was never really a thing.

On the other hand, social conditioning does work. You can have societies where 98% of people believe in the same religion, and multiple societies who believe they are objectively the best, and so on. Social conditioning is the the thing that’s implemented by anthem-singing, flag waving, public prayer, rallies, marches and parades, and a host of other than things that are seen as perfectly normal … unlike the weird stuff cults get up to.

Brainwashing is a special or intensified form of conditioning … so why wouldn’t it work, when social conditioning generally does? One of the pieces of evidence against brainwashing is that US soldiers who had been “brainwashed” after being captured by communists reverted when they returned to the US. That could. be seen as brainwashing lacking a particular feature, the ability to lock-in permanently. It could also be seen as a success of the kind of social conditioning that’s unnoticed and in the water. Attempted cult brainwashing into minority beliefs has the Achille’s heal of attempting to instill minority beliefs, despite the fact that people generally want to fit in with majority beliefs. Cults try to get round this by separating their subjects from wider society, which doesn’t entirely work , because they need to proselytize. On the other hand, small religions are large cults, and they work just fine.

Depending on how you define it, brainwashing is ubiquitous, non existent , or underwhelming effective. ,

TAG 6 Oct 2025 12:49 UTC
2 points
0
in reply to: James Diacoumis’s comment on: Beyond the Zombie Argument
There’s is a reason I mentioned conceivability without contradiction. I don’t particularly have it but into the standard framework about conceivability, metaphysical possibility, etc, and I dont. If you ground conceivability in non contradiction, then it has an objective basis, otherwise it’s just an opinion. P-zombies don’t contradict physics , but do contradict physicalism, which is why physicalists don’t believe in them.

TAG 3 Oct 2025 20:29 UTC
−2 points
−2
on: Why Corrigibility is Hard and Important (i.e. “Whence the high MIRI confidence in alignment difficulty?”)

In 2014, we proposed that researchers try to find ways to make highly capable AIs “corrigible,” or “able to be corrected.” The idea would be to build AIs in such a way that they reliably want to *help *and cooperate with their programmers, rather than hinder them — even as they become smarter and more powerful, and even though they aren’t yet perfectly aligned.

Minimally, corrigibility just means present goals can be changed. There could be crude, brute force ways of doing that, like Pavlovian conditioning ,or overwriting an explicit coded UF.

The whole point of corrigibility is to scale to novel contexts and new capability regimes.

That can be done gradually, if everything is gradual.

@Roko

The reason that MIRI wasn’t able to make corrigibility work is that corrigibility is basically a silly thing to want, I can’t really think of any system in the (large) human world which needs perfectly corrigible parts, i.e. humans whose motivations can be arbitrarily reprogrammed.

That’s normal much an argument against corrigibility, as an argument against perfect corrigibillity.

TAG 2 Oct 2025 20:38 UTC
0 points
0
in reply to: Garrett Baker’s comment on: Beyond the Zombie Argument

here he argues directly against “To be able to determine the future in a way that is itself determined falls short of being able to steer to a future of your choosing.”

Not really, because he is only saying , in typical compatiblist fashion, that you can bring about future in accordance with your desires …which are determined. There is still no unpredictable turning point when you make a choice … and therefore no steering to one of a number of physically possible futures...just a bunch of clockwork that’s all inevitable.

TAG 2 Oct 2025 20:32 UTC
2 points
0
in reply to: Garrett Baker’s comment on: Beyond the Zombie Argument
I think he actually had different attitudes about different kinds of FW, although he isn’t fully explicit about it

TAG 2 Oct 2025 19:54 UTC
0 points
0
in reply to: Garrett Baker’s comment on: Beyond the Zombie Argument
Maybe you are referring to this passage.

My position might perhaps be called “Requiredism.” When agency, choice, control, and moral responsibility are cashed out in a sensible way, they require determinism—at least some patches of determinism within the universe. If you choose, and plan, and act, and bring some future into being, in accordance with your desire, then all this requires a lawful sort of reality; you cannot do it amid utter chaos.

But choice..choice between options, any of which could possibly occur..is incompatible with determinism, because determinism means there is only ever one possible (and therefore necessary) outcome … even if determinism is useful in other ways. You can still have a deterministic decision making process...but it is only some clockwork gets you to an outcome that was predetermined before you were born.

A libertarian choice is an actual fork in the road, and a Laplace’s Demon wouldn’t be able to see past it.

So, he thinks that compatibilist FW is as good as libertarian FW, because he thinks they both involve the same notion of choice...but they aren’t.

Also, “utter chaos” isn’t the only alternative to strict determinism. That would be false dichotomy. Lawful probabilistic causation is a thing.

TAG 2 Oct 2025 17:58 UTC
0 points
0
in reply to: Garrett Baker’s comment on: Beyond the Zombie Argument
He also argues against FW, in the sense of libertarian free will. Above , he argues for compatibilist free will. As usual, its not much of a consolation prize. To be able to determine the future in a way that is itself determined falls short of being able to steer to a future of your choosing.

ETA:

He also seems to think that merely being part of a wider system means you are determined. Nope.

TAG 2 Oct 2025 14:35 UTC
2 points
0
in reply to: Signer’s comment on: [Question] What the discontinuity is, if not FOOM?
All AI’s scheming co-operatively is less likely than on scheming.

TAG 2 Oct 2025 13:35 UTC
2 points
−1
in reply to: James Diacoumis’s comment on: Beyond the Zombie Argument

On standard physicalism zombies would be conceivable

On standard (non eliminative) physicalism, zombies cannot be conceived without contradiction , because physicalism holds that consciousness is entirely physical, and a physical duplicate is a duplicate simpliciter. So the inconceivability of zombies is not a USP of RM.

The bite of the zombie argument is that zombies are conceivable given physics … physics does not predict phenomenal properties, so absence of phenomenal properties does not lead to contradiction.

I could also imagine other flavours of Monism (which might be better labelled as property dualism?) for which the intrinsic categorical properties are contingent rather than necessary

Phenomenal “properties” are more necessary under DAT than RM, because they are not even separate properties, but arise from the subjective perspective on the underlying reality.

TAG 1 Oct 2025 18:17 UTC
2 points
0
in reply to: Felix C.’s comment on: [Question] What the discontinuity is, if not FOOM?

AI research might end up with similar problems

Might. IABIED requires a discontinuity to be almost certain.

TAG 1 Oct 2025 18:15 UTC
2 points
0
in reply to: Signer’s comment on: [Question] What the discontinuity is, if not FOOM?

Why would that be discontinuous?

Because incremental progress missed deception

I’m talking about the how of takeover. Could any AI, even one of many, take over successfully in its first attempt?

TAG 1 Oct 2025 14:32 UTC
2 points
0
in reply to: StanislavKrym’s comment on: The Basic Case For Doom

While we try to ensure that it will pursue the goals that we gave it, we are NOT sure that it won’t develop its own goals

“Not sure that not X” is very different to “sure that X”.

I’m not arguing for 0% p(doom) , I’m arguing against 99%.

The point about commercially viable AI being aligned well enough

It’s a tautology. It’s aligned well enough to be usable, because it’s usable. If it were unaligned in a binary sense, it wouldn’t follow instructions.

as dumb as claiming that the AIs cannot fake alignment, gather power and take over the world.

Possible is very far from certain.

TAG 1 Oct 2025 14:21 UTC
0 points
−6
on: Beyond the Zombie Argument

As Eliezer points out in his article here, a zombie duplicate of yourself would profess that it is conscious, when asked whether it had any conscious experience it would say “yes!” and when given tests designed to demonstrate its conscious experience it would pass

EY also thinks we are zombies wrt. free will—we say we have it for reasons other than having it.

TAG 1 Oct 2025 14:17 UTC
6 points
0
on: Beyond the Zombie Argument

So, returning to our thought experiment, spectrum inversion is actually impossible on Russellian Monism because the intrinsic categorical facts are also physical facts.

Not in the sense of the kind of facts that physics deals with.

Russelian monism struggles with Epiphenomenality: if the measurable, structural properties are sufficient to predict what happens, the the phenomenal properties are along for the ride.

Dual-aspect neutral monism is Russelian monism done right. This holds that the physical sciences are one possible map of territory which is not itself, intrinsically, physical (or, for that matter, mental). Consciousness is another map, or aspect.

This approach has the advantage of dualism, in that there is no longer a need to explain the mental in terms of the physical, to reduce it to the physical, because the physical is no longer regarded as fundamental (nor is the mental, hence the “neutral”). Although an ontological identity between the physical and mental is accepted, the epistemic irreducibility of the mental to the physical is also accepted. Physicalism, in the sense that the physical sciences have a unique and priveleged explanatory role, is therefore rejected. To take one example, since the a conscious mental state and physical brain state are ultimately the same thing, the expected correlations hold between them. For instance, mental states cannot vary without some change in the physical state (supervenience follows directly from identity, without any special apparatus); furthermore, since mental states are ultimately identical to physical brain states, they share the causal powers of brain states (again without the need to posit special explanatory apparatus such as “psychophysical laws”), and in that way epiphenomenalism is avoided.

The more familiar kinds of dualism are substance and property dualism. Both take a physical ontology “as is” and add something extra, and both have problems with explaining how the additional substances or properties interact with physical substances and properties, and both of course have problems with ontological parsimony (Occam’s Razor).

In contrast to a substance or property, an aspect is a relational kind of thing. In Dual Aspect theory, a conscious state is interpreted as being based on the kind of relationship and entity has with itself, and the kind of interaction it has with itself. The physical is reinterpreted as a kind of interaction with and relation to the external. It is not clear whether this theory adds anything fundamentally new, ontologically, since most people will accept the existence of some kind of inner/outer distinction, although the distinction may be made to do more work in Dual Aspect theory. Reinterpreting the physical is a genuine third alternative to accepting (only) the physical, denying the physical, and suplementing the physical.

TAG 1 Oct 2025 13:47 UTC
2 points
0
in reply to: Signer’s comment on: [Question] What the discontinuity is, if not FOOM?

It’s just that if after incremental progress AI takes over,

Why would that be discontinuous?

if you missed only one way for AI to hide it’s intentions, it will take over.

Assuming it has an intention, and a malign one. Deception depends on a chain of assumptions. They all have to be well over 90% to lead to a conclusion of near certain doom.

Again, I don’t see how you can be confident that future AI wouldn’t scheme.

I’m not arguing for 0% p(doom) , I’m arguing against 99%.