TAG

Karma: 1,558

About Me

Scientist by training, coder by previous session,philosopher by inclination, musician against public demand.

https://theancientgeek.substack.com/?utm_source=substack&utm_medium=web&utm_campaign=substack_profile

Why I am not a Doomer

I’m specifically addressing the argument for a high probability of near extinction (doom) from AI...

Eliezer Yudkowsky: “Many researchers steeped in these issues, including myself, expect that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. ”

....not whether it is barely possible, or whether other, less bad outcomes (dystopias) are probable. I’m coming from the centre, not the other extreme

Doom, complete or almost complete extinction of humanity, requires a less than superintelligent AI to become superintelligent either very fast , or very surreptitiously … even though it is starting from a point where it does not have the resources to do either.

The “very fast” version is foom doom...Foom is rapid recursive self improvement (FOOM is supposed to represent a nuclear explosion)

The classic Foom Doom argument (https://www.greaterwrong.com/posts/kgb58RL88YChkkBNf/the-problem) involves an agentive AI that quickly becomes powerful through recursive self improvement, and has a value/goal system that is unfriendly and incorrigible.

The complete argument for Foom Doom is that:-

The AI will have goals/values in the first place (it wont be a passive tool like GPT*),.
The values will be misaligned, however subtly, to be unfavorable to humanity.
That the misalignment cannot be detected or corrected.
That the AI can achieve value stability under self modification.
That the AI will self modify in way too fast to stop.
That most misaligned values in the resulting ASI are highly dangerous (even goals that aren’t directly inimical to humans can be a problem for humans, because the AS I might want to director sources away from humans.
And that the AI will have extensive opportunities to wreak havoc: biological warfare (custom DNA can be ordered by email), crashing economic systems (trading can be done online), taking over weapon systems, weaponing other technology and so on.

It’s a conjunction of six or seven claims, not just one. ( I say “complete argument ” because pro doomers almost always leave out some stages. I am not convinced that rapid self improvement and incorrigibility are both needed, both needed, but I am sure that one or the other is. Doomers need to reject the idea that misalignment can be fixed gradually, as you go along. . A very fast-growing ASI, foom, is way of doing that; and assumption that AI’s will resist having their goals changed is another).

Obviously the problem is that to claim a high overall probability of doom, each claim in the chain needs to have a high probability. It is not enough for some of the stages to be highly probable, all must be.

There are some specific weak points.

Goal stability under self improvement is not a given: it is not possessed by all mental architectures, and may not be possessed by any, since noone knows how to engineer it, and humans appear not to have it.

The Orthogonality Thesis (https://www.lesswrong.com/w/orthogonality-thesis)is sometimes mistakenly called on to support to support goal stability. It implies that a lot of combinations of goals and intelligence levels are possible, but doesn’t imply that all possible minds have goals, or that all goal driven agents have fixed, incorrigible goals. There are goalless and corrigible agents in mindspace, too. That’s not just an abstract possibility. At the time of writing, 2025, our most advanced AI’s, the Large Language Models, are non agentive and corrigible.

It is plausible that an agent would desire to preserve its goals, but the desire to preserve goals does not imply the ability to preserve goals. Therefore, no goal stable system of any complexity exists on this planet, and goal instability cannot be assumed as a default or given. So the orthogonality thesis is true of momentary combinations of goal and intelligence, given the provisos above, but not necessarily true of stable combinations.

Another thing that doesn’t prove incorrigibility or goal stability is von Neumann rationality. Frequently appealed to in MIRI ’s early writings , it is an idealised framework for thinking about rationality , that doesn’t app!y to humans, and therefore doesn’t have to apply to any given mind.

There are arguments that AI’s will become agentive because that”s what humans want. Gwerns Branwen’s confusingly titled “Why Tool AIs Want to Be Agent AIs” ( https://gwern.net/tool-ai) is an example. This is true, but in more than one sense:-

The basic idea is that humans want agentive AI’s because they are more powerful. And people want power, but not at the expense of control. Power that you can’t control is no good to you. Taking the brakes off a car makes it more powerful, but more likely to kill you. No army wants a weapon that will kill their own soldiers, no financial organisation wants a trading system that makes money for someone else, or gives it away to charity, or causes stick market crashes. The maximum amount of power and the minimum of control is an explosion.

One needs to look askance at what “agent” means as well. Among other things, it means an entity that acts on behalf of a human—as in principal/agent.(https://en.m.wikipedia.org/wiki/Principal–agent_problem) An agent is no good to its principal unless it has a good enough idea of its principal’s goals. So while people will want agents, they wont want misaligned ones—misalgined with themselves, that is. Like the Orthogonality Thesis, the argument is not entirely bad news.

Of course, evil governments and corporations controlling obedient superintelligences isn’t a particularly optimistic scenario, but it’s dystopia, not doom.

Yudkowsky’s much repeated argument that safe , well-aligned behaviour is a small target to hit … could actually be two arguments.

One would be the random potshot version of the Orthogonality Thesis, where there is an even chance of hitting any mind, and therefore a high chance ideas of hitting an eldritch, alien mind. But equiprobability is only one way of turning possibilities into probabilities, and not particularly realistic. Random potshots aren’t analogous to the probability density for action of building a certain type of AI, without knowing much about what it would be.

While, many of the minds in mindpsace are indeed weird and unfriendly to humans, that does not make it likely that the AIs we will construct will be. we are deliberately seeking to build certainties of mind for one thing, and have certain limitations, for another. Current LLM ’s are trained in vast copora of human generated content, and inevitably pick up a version of human values from them.

Another interpretation of the Small Target Argument is, again , based on incorrigibility. Corrigibility means you can tweak an AI’s goals gradually, as you go on, so there s no need to get them exactly right on the first try.

TAG 30 Jan 2026 6:43 UTC
0 points
−2
on: Bentham’s Bulldog is wrong about AI risk
ETA

The AI is already not aligned

For what value of “aligned”? The current state of play is thanking just about anything could be said to be aligned or misaligned, sending in exactly what the speaker means by the terms.

There is a lot of semantic confusion between people who use “alignment” in an engineering sense,meaning something that renders current AI safe in good enough way—and the people who use it to mean a maths style solution , that applies perfectly to every case. A completely unaligned AI would be completely unco-operative , and therefore of no commercial use, so the prevailing level of alignment isn’t zero.

You even acknowledge that there are different kinds of alignment here:-

We could build an ASI that is aligned, but not aligned with humanity as a whole (whatever that means), s

But even if we didn’t, and everything seemed fine, I would not believe LLMs are aligned, because value is complex and fragile.

That need not be an argument against good enough alignment.

One way alignment can be made to look difficult is stating that it has to be done in a maximal way to achieve minimal results. The minimal result is not killing us, all, the maximal way is instilling into the AI every nuance of human value including aesthetic value. Under circumstances when a powerful ASI takes over and starts running things according to its original programming, without listening to feedback, a detailed knowledge of human values would be necessary to create a utopia. But that is a far cry from not killing us all.

Another way of making successful alignment look difficult is the making the assumption that AI’s will become very powerful , very quickly, in an unsupervised way, so that humans only have one chance to get alignment correct, before the ASI becomes too powerful to listen to humans. That’s the idea underlying the Fragility of Value. It doesn’t actually matter how complex or subtle value is , so long as you can tweak a specification of value at leisure. If course, a fast “take off” isn’t impossible, it is too often treated as a certainty, and too often left as an implicit assumption

LLMs are agents. They’re remarkably non-agentic

That looks another semantic confusion. If they are remarkably non-agentic, why not round them down to non-agents?

And notice the number of things that have to go right in order for this stage to be where doom stops:
1. The AI has to be scary
2. And we have to notice it being scary
3. And we have to band together to try and stop it
4. AND we have to win
5. AND after the victory we have to ~permanently ban this fearsome technology that already exists
To quote a thinker that I respect: “If they each have an 80% chance

The negation of a conjunction is a disjunction.

https://en.wikipedia.org/wiki/De_Morgan’s_laws

The argument for Doom is mostly or wholly conjunctive, so the argument for no-doom is mostly or wholly disjunctive. If ASI is impossible, no doom, if ASI is not placed in charge of everything, no doom, etc.

To put it a other way , if A is low probability, not-A can’t also be.

The multi stage fallacy is not a fallacy in the sense that conjunctive arguments are never unlikely: rather there are pitfalls in multi stage arguments.

TAG 28 Jan 2026 20:14 UTC
2 points
0
on: Contra Sam Harris on Free Will
@Julius

Harris can also be critiqued from the direction of libertarian free will.

Half of the answer is to recognise that my unconscious mind is still mine—following its prompting is not being under the control of someone else. The other half us understanding that the conscious mind can still exert control.

The basic mechanism is that the unconscious mind proposes various ideas and actions , which the conscious mind decides between, and finally puts the chosen one into action. (Since you only have one body, it is important to decide on a clear course of action, not half of one thing and half of another).

Sam Harris makes much of the fact that the conscious mind, the executive function, does not predetermine the suggestions, and concludes that in the absence of of predetermination, there is no conscious control at all. In contrast, I argue that the choice between impulses, the decision to act on one rather than another, the gatekeeping mechanism, is conscious control—and conscious control clearly exists in health adults.

If there is indeterminism the mechanism (and there doesn’t have to be) it provides the libertarian could-have-done-otherwise as well as conscious control.

@Taylor G. Lunt

but there’s obviously no way Sam would say something like “consciousness is causally disconnected from the rest of the universe, and can never influence future decision making processes in the brain.”

You mean, no way he would embrace metaphysical epiphenomenalism, because he is a physicalist?

Actually, it’s not clear that he is a physicalist. It’s possible to be a physical epiphenomenalist , regarding consciousness as a brain module with no downward casual affect, but it’s strange , because it implies something complex, but with no function evolved.

@FlorianH

Retributive justice—as in really within jurisprudence or so—questions are essentialy when we discuss free will, and here’s where most people are stupidly confused while SH’s exposition and interpretation is spot on about it.

But it’s not clear:-

• Whether retributive punishment is a US or global phenomenon

• Whether it’s caused by fee will beliefs.

• Whether there is a radically, rather incrementally, different approach to punishment.

Is it a Uniquely US Phenomenon?

It might be the case that US style prisons are like that because of a desire to inflict suffering because of a belief in free will...but many other explanations are possible. Most of the issues explained by simply not funding prisons well. After all, being in the same environment as a bunch of criminals is pretty intrinsic, not some special punishment. If the state was spending money on torture equipment, then you’d have evidence that they were making a special effort to cause suffering, rather than just doing things on the cheap.

Prisons in poor countries are invariably awful: no special effort is required to induce suffering. The harshness of US prison system is not explained by poverty ,since the US is the world’s richest large country, but does not have to be explained by free will. One distinctive factor in the US is the democratisation of the criminal justice system. Public prosecutors are elected, and therefore need a high profile: committing to slamming people up for long periods is apparently more attention-grabbing than releasing the innocent

Is It Caused by Religion?

Harris, Sapolsky and their supporters seem to like the liberal Scandinavian approach. But Scandinavia is not particularly atheist. For instance,Norway had a state religion until 2012, and 70% of the population are Lutheran, a sect that upholds free will. So theism doesn’t simply predict a punitive criminal justice syste

Soviet Russia, by contrast, was officially atheist..and materialistic and deterministic … yet had a very harsh penal system. So atheism doesn’t simply predict a gentle criminal justice system.

As far as I can see, the main predictors of a humane penal.system are a combination of societal wealth and political liberalism. But philosophical beliefs in theism and atheism, free will or determinism, are not strongly correlated with wealth or liberalism.

Is There an Alternative?

If you feel that someone is a danger to the community, then putting them in jail has a justification different from punishing them for their sins. But you might want to punish them by jailing them if you believe in free will, as well...so you cant infer a fundamental philosophical difference from the fact that that some people are in jail. Even if you want to rehabilitate them you still have to make them turn up to therapy sessions when they don’t want

And punishment is behaviour shaping,therefore therapeutic, in some ways.

TAG 28 Jan 2026 10:07 UTC
3 points
0
on: Disagreement Comes From the Dark World

In figuring out what would constitute good conduct and productive discourse, it’s important to appreciate how bizarre the human practice of “discourse” looks in light of Aumann’s dangerous idea.

Within the rationalsphere , Aumanns Theorem is simplified into something like “reasonable people can’t agree to differ”. Yet they can, because the conditions on AAT are much more stringent than an informal understanding of what it is to be reasonable. So AAT lacks real world applicability..

There is a further problem that prior beliefs can include beliefs about epistemology, about what constitutes evidence.

Controversies that are sufficiently deep, or which cut across cultural boundaries run into a problem where, not only do parties disagree about the object level issue, they also disagree about underlying questions of what constitutes truth, proof, evidence, etc. “Satan created the fossils to mislead people” is an example of one side rejecting the other sides evidence as even being evidence . Its a silly example, but there are much more robust ones.

There’s only one reality.

Why do people keep saying that?

It’s true that If there were more than one world, that would undercut the ability of different thinkers to come to a uniform conclusion about it, but there is no corollary that existence of a single world would guarantee anything epistemically. It’s a necessary condition for convergence, but not sufficient.

If there is one world, that doesn’t guarantee that there are agents capable of understanding it, within it ..and if there are intelligent and rational agents, their ability to converge on a single all encompassing truth could be impossible for a number of further reasons. Problems include the inadequacy of epistemic evidence to address all problems, the reliance of logic on axioms (the Munchausen trilemma) , the reliance of epistemology on epistemology (the problem of the criterion), etc.

None of those problems has anything to do with conflicts, dishonesty or different values—although they undoubtedly exist as well.

If I’m a Bayesian reasoner honestly reporting my beliefs about some question, and you’re also a Bayesian reasoner honestly reporting your beliefs about the same question, we should converge on the same answer, not because we’re cooperating with each other, but because it is the answer.

Bayes has both problems simultaneously. It is dependent on evidence, so it has all the problems of empiricism; and it has the problems of rationalism because it starts with priors. The argument for Bayesianism is that even agents with wildly differing priors can eventually agree, given sufficient evidence. But that is an argument about ideal Bayesians: in reality , the amount of evidence available might be too limited to allow convergence. The ability of realistic Bayesians to formulate hypotheses is also limited. Alice puts most of her credence on the one hypothesis that seems best supported to her, out of the hypotheses she has heard of, or thought up, but Bob might have a better hypothesis that’s not in her set.

Real-world disagreements tend to persist; they’re predictable—in flagrant contradiction of how the beliefs of Bayesian reasoners would follow a random walk. From this we can infer that typical human disagreements aren’t “honest”, in the sense that at least one of the participants is behaving as if they have some other goal than getting to the truth.

No, that isn’t the inevitable conclusion, because there are so many other sources of disagreement.

That is to say: I do not understand how high-trust, high-cooperation dynamics work. I’ve never seen them. They are utterly outside my experience and beyond my comprehension

I find the black-and-white framing of that very odd. You’ve spent most of your life in business and academia, which seem to me high co-operation compared to politics and crime, for instance.

TAG 24 Jan 2026 15:52 UTC
3 points
0
in reply to: Warty’s comment on: Warty’s Shortform
What is the butterfly effect mechanism?

If everything is a coherent superposition evolving according to the Schrödinger equation, there is very little cancellation or unwinding of “branches” ..But also no real branching, because superposed states continued to interact.

Decoherence, in the other hand, isn’t guaranteed to leave you with more than one branch.

TAG 22 Jan 2026 3:01 UTC
2 points
0
in reply to: ihatenumbersinusernames7’s comment on: Causality and Moral Responsibility

Indeterministic is a loaded word.

In what way? I would have thought “random” is more loaded.

Certainly we don’t believe our actions to be random,

We see them as (up to a point) responsible and controlled, irrespective of fundamental determinism.

but I maintain that the question before compatibilists/semicompatibilists (which I hoped this post would address but IMO doesn’t) is why seeing free will as a human construct is meaningful.

I didn’t say free will was a construct. I suggested moral responsibility was.

Of course that is meaningful—that is a rather low bar—the question is whether it is sufficient.

So if I create an AI that steals money, I am the greater jerk but the AI is also a jerk?

That doesn’t address the argument from usefulness. It is still useful to modify the AI into being a non stealing AI.

Whether an entity has libertarian freewill is another question , that isn’t just a matter of social construction, because libertarian free will depends on (in)determinism , which is a fact about the world. But whether LFW is the only possible justification for behaving as if MR exists is also a separate question.

We can be less confused about the issue by treating usefulness and truth as separate issues, and by treating compatibilist free will and libertarian free will as separate issues. There doesn’t have to be a single answer to “is free will a social construct?” .

TAG 19 Jan 2026 11:49 UTC
2 points
0
in reply to: ihatenumbersinusernames7’s comment on: Causality and Moral Responsibility
There is clearly something sane, sober adult humans have that toddlers , and drunks dont. Self control doesn’t have to be based on fundamental physics. There is a science of control systems that doesn’t require the system being controlled to be indeterministic

The point is similar to compatibilism: compatibilism allows something like free will, but as more if a human construct and less of a metaphysical absolute.. This approach, sometimes called semicompatibilism, takes a similar attitude to moral responsibility.

The contrary claim.would be that if we discovered that the world is deterministic,it will would imply no one has ever been morally responsible, and presumably should not have imprisoned or otherwise punished.

TAG 17 Jan 2026 15:45 UTC
3 points
0
in reply to: Warty’s comment on: Warty’s Shortform
The universe is always sufficient to calculate itself. A multiverse has more computational capacity than a single universe—more to calculate , and more to calculate with.

Also, it’s a common misconception that there is a single MW, theory that requires irrevocable branching at every fundamental event.

One incorrect but frequently stated claim is that full decoherent splittng occurs at every microscopic interaction. This could be called the doubly maximal theory, the combination of the highest frequency and greatest extent of splitting. But If it were so, there would be no evidence of the various phenomena based coherent superposition,such as quantum computation. QC only works because branches, or rather, commentators a superposition, continue to influence each other and contribute toward final result. If the branching were complete and irreversible at The outset, the final result would.just be an observation of what had been happening in one branch. The doubly maximal concept of many worlds...full decoherence at every elementary interaction (not just observations, which are relatively high level interactions)...has been ruled out.

TAG 17 Jan 2026 15:31 UTC
2 points
0
in reply to: ihatenumbersinusernames7’s comment on: Causality and Moral Responsibility
What’s missing, as usual, is any reference to social or group level norms or standards. Duties to other people aren’t supernatural, and aren’t the inevitable result of neural functioning either—not everyone is responsible. You have to learn them from other people , and their attitudes of praise and blame are how they get imprinted into brains, when they are. If you, eg., drop litter , you feel a twinge of guilt , which has a physical basis in the brain , but only because you were shamed for it in the past—if you had been raised by wolves, you wouldn’t have the reaction.

All of which is compatible with determinism .. just not with extreme reductionism. Obligations, duties, promises, responsibilities, etc, are social constructs that neither descend from the heavens nor ascend from fundamental physics. They are useful, so they need to be invented if attending already exist.

TAG 15 Jan 2026 0:54 UTC
2 points
0
in reply to: Signer’s comment on: The crux on consciousness

And so are qualia. The only difference is that the science haven’t yet provided a useful reduction.

The other difference is that Qualia are subjective.

But laws of physics still don’t say how you should reduce things. And reductions doesn’t preserve everything—fire can look continuous, but actually consist of atoms.

Lossy reductions don’t preserve everything.

No it comes from the observationthat our sensorium is not a picture of our brains?

You can also observe that fire is not a picture of atoms.

Fire is a picture of atoms. As you zoom into it in ever more detail, you end up with individual atoms emitting photons.

Again, there is no reason to directly identify qualia with microphysical properties.

Some sort of explanation is needed.

TAG 10 Jan 2026 15:16 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV

If morals are not truth-apt, and free will is the control required for moral responsibility, then...

Alignment has many meanings. Minimally, it is about the AI not killing us.

AI s don’t have to share our aesthetic preferences to understand them. It would be a nuisance if they did—they might start demanding pot plants in their data centres -- so it is useful to distinguish aesthetic and moral values. So that’s one of the problems with the unproven but widely believed claim that all values are moral values.

Nevertheless, they are correct: there is now a mathematical foundation underpinning the Scientific Method

Bayes doesn’t encapsulate the whole scientific method, because it doesn’t tell you how to formulate hypotheses, or conduct experiments.

Bayes doesn’t give you a mathematical foundation of a useful kind, that is an objective kind. Two Bayesian scientists can quantify their subjective credences, quantify them differently, and have no way of reconciling their differences.

TAG 6 Jan 2026 14:25 UTC
2 points
0
in reply to: Ape in the coat’s comment on: Give Skepticism a Try

If [normality] means something else …what?

We are in agreement here. And if you read to the end of the post you’ll see the answer to your question:

I have, and I still haven’t seen a definition of normality.

I think you meant Euclidean.

That’s too, though I was mostly hinting to relativity.

It’s general.relativity that implies non euclidean space.

Curious how our understanding of things that some people may assume are beyond observations are then happen to be changed by scientific discovery, isn’t it?

Not once you realise scientific discovery isn’t pure empiricism.

Almost all contemporary epistemologists will say that they are fallibilists

That’s nice. Though the key words here are “contemporary” and “epistemologists”.

So where are the philosophers obsessed with certainty? In the past, or in other departments?

TAG 4 Jan 2026 15:46 UTC
2 points
0
on: Give Skepticism a Try

On one hand, skepticism is a legitimate philosophical view with no good arguments against.

Which version of scepticism? Even the article you link to concedes there is,at least one form of scepticism that’s refutable through self defeat.

If extreme scepticism is self defeating, and certainty is unobtainable, then what you are left with is moderate scepticism—AKA fallibilism. What you need to try is the right amount if the right kind of scepticism.

We can even see how this kind of reasoning makes some sense with the ultimate goal of adding philosophy up to normality

If the world is weird, I wish to believe that the world is weird.

The goal is to add up to truth.

If normality means supporting all your intuitions, then you are going to have to disbelieve much science and maths.

If it means something else …what?

If the brain has an amazing “a priori truth factory” that works to produce accurate beliefs, it makes you wonder why a thirsty hunter-gatherer can’t use the “a priori truth factory” to locate drinkable water. It makes you wonder why eyes evolved in the first place, if there are ways to produce accurate beliefs without looking at things.

Something can work in some contexts, but not in in others.

Empiricism doesn’t work for things you can’t see, eg:-

Modality, Counterfactuals, Possible versus Actual.

Normativity, Ought versus is.

Essence versus Existence, hidden explanatory mechanisms.

A priori truth could have a naturalistic basis. Many organisms can instinctively recognise food, predators, rivals and mates. But even the broadest evolutionary knowledge must that operate within the limits of empiricism .. not the “invisibles” I mentioned. And of course it is a rather different kind of apriori knowledge than analytical kind, based on language and tautologies.

Emanuel Kant came to the conclusion that there is no way to justify the existence of space and time with observations, as space and time are prerequisites for any observations in the first place. Therefore, they have to be justifiable “à priori” in a matter, suspiciously resembling cyclical reasoning:

Unless “à priori” justifications are true, space and time are not justifiable. But space and time has to be justifiable[1]. Therefore “à priori” justifications has to be

Space and time don’t need to be justifiable, because they are not propositions. they are aoriori, but not apriori propositions. They are, for Kant, preconditions for having any experience, and therefore apriori. Geometric truths are apriori propositions. They need justification, which for Kant, is the apriori nature of space.and time—for humans.

There is an extra level of irony here that, among other things, Kant has “à priori” figured out that space and time are absolute, which we now know not to be the case

I think you meant Euclidean.

Note that no claim is made that space is Euclidean in itself. The tower of.justification is only rooted in the operation of human perception.The

Kantian issues about how much of what we perceive is out there, and how much how.much our form of representation are still live, and crop up within science. Just looking doesn’t solve them, because you can only look from within the mind-world relationship … you can’t look at it from the outside.

And so most philosophers are really into certainty.

What does that mean?

″ Almost all contemporary epistemologists will say that they are fallibilists”

https://iep.utm.edu/fallibil/#H1

It’s deeply entangled with the view that philosophy (or at the least some part of it) is a separate magisterium that lies beyond the empiricism of sciences

Half of science lies beyond the empiricism of science.

ETA

Engineers building rockets do not sweat about Cartesian Demon and yet the rockets seem to works fine. If something is good enough for building rockets maybe it’s good enough for our reasoning in general?I

On the other hand, maybe it isnt. There’s no guarantee. Maybe forms of reasoning that depend on visible feedback about whether something happens or works can’t extend to areas where there is no such feedback—areas like “what is the ultimate nature of reality”. There is no direct test for correspondence-to-reality. So truth is different to usefulness.

The usefulness cluster of concepts includes the ability to make predictions, as well as create technology. The truth cluster of concepts involves identification of the causes of perceptions, and offering explanations, not just predictions. Usefulness corresponds to engineering, truth to philosophy, and science straddles both. Usefulness corresponds to phenomenal, truth to noumena.

The usefulness cluster corresponds to scientilfic instrumentalism , the truth cluster to scientific instrumentalism. The truth cluster corresponds to epistemological rationalism, the usefulness cluster to instrumental rationalim. Usefulness corresponds to practice, truth to theory.

Truth is correspondence to reality , which is not identical to the ability to make predictions. One can predict that the sun will rise without knowing what the Sun really is. “Curve fitting” science is adequate to make predictions. Trial and error is adequate to come up with useful technologies. But other means are needed to find the underlying reality. One can’t achieve convergence by “just using evidence” because the questions of what evidence is, and how to interpret depend on one’s episteme. One also can’t because ruling out empirically reconfirmed theories generally doesn’t leave you with a single candidate.

Science straddles both. Science is not just a matter of making predictions, it’s a matter of of answering “how” and “why” questions, of finding explanations. That can’t be done by stamp -collecting observations … It requires the conscious and creative conjecture of explanatory hypotheses, something beyond empiricism.

TAG 3 Jan 2026 3:43 UTC
3 points
0
in reply to: Ben Pace’s comment on: [Advanced Intro to AI Alignment] 2. What Values May an AI Learn? — 4 Key Problems
Also:-

CEV seems much much more difficult than strawberry alignment and I have written it off as a potential option for a baby’s first try at constructing superintelligence.

To be clear, I also expect that strawberry alignment is too hard for these babies and we’ll just die. But things can always be even more difficult, and with targeting CEV on a first try, it sure would be.

There’s zero room, there is negative room, to give away to luxury targets like CEV. They’re not even going to be able to do strawberry alignment, and ify some miracle we were able to do strawberry alignment and so humanity survived, that miracle would not suffice to get CEV right on the first try.

TAG 1 Jan 2026 18:03 UTC
2 points
0
in reply to: Signer’s comment on: The crux on consciousness

Because there is no fire in the ontology of modern physics and there are no laws of physics that say that some arrangement of atoms are fire.

Fire is reducible. If you want a physical explanation, you can have one.

Binding comes from a human desire to describe things in an approximate, useful way

No it comes from the observationthat our sensorium is not a picture of our brains?

It factors into localised parts.

Approximately localized

If Qualia are identified with microphysical properties, those properties need to be localised to solve the binding problem.

TAG 29 Dec 2025 16:31 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV

(by which I think most of them do mean “shared human value” even if they don’t all bother to specify), and that I’m suggesting pointing Value Learning at.

I’m suggesting they should bother to specify.

(along with more basic things, like us being around flowers, parks, seashores, and temperature around 75°F) what I’m suggesting as a candidate definition for the “human values”

But are they relevant to ethics or alignment? a lot of tuem are aesthetic preferences that can be satisfied without public policy.

I also didn’t specify above what I think should be done, if it turns out that say, about 96–98% of humans genetically have those shared values, and 2–4% have different alleles.

Shared genetics can lead to different blood and tissue types, so it can lead to different ethical types.

Politics indicates it’s more like 50-50, when you are talking about the kind of values that cannot be satisfied individually.

And “my tribe”. What you want is Universalism, but universalism is a late and strange development. It seems obvious to twenty first century Californians, by they are The weirdest of the WEIRD. Reading values out of evopsych is likely to push you in the direction of tribalism, so I don’t see how it helps.

On the Savannah, yes of course it does. In a world-spanning culture of eight billion people, quite a few of whom are part of nuclear-armed alliances, intelligence and the fact that extinction is forever suggests defining “tribe” ~= “species + our comensal pets”. And also noting and reflecting upon that the human default tendency to assume that tribes are around our Dunbar Number in size is now maladaptive, and has been for millennia.

There are technologically advanced tribalists destroying each other right now. It ’s not that simple.

It’s not the case that science boils down to Bayes alone,

Are you saying that there’s more to the Scientific Method that applied approximate Bayesiasm?

Yes. I learnt physics without ever learning Bayes. Science=Bayes is the extraordinary claim that needs justification.

or that science is the only alternative to philosophy. Alignment/control is more like engineering.

Engineering is applied Science, Science is applied Mathematics; from Philosophy’s point of view it’s all Naturalism. In the above, it kept turning out that Engineering methodology is exactly what Evolutionary Psychology says is the adaptive way for a social species to treat their extended phenotype.

Again, I would suggest using the word engineering, if engineering is what you mean.

So, In philosophy of science terminology, pholosophers have plenty of hypothesis generation, but very little falsifiability (beyond, as Gettier did, demonstarting an internal logical inconsistency), so the tendency it to increase the number of credible candidate answers, rather than decreasing them.

That’s still useful If you have some way of judging their correctness—it doesn’t have to be empiricism. To find the one true hypothesis, you need to consider all.of them,.and to approximate that , you need to do consider a lot of them.

The same thing occurs within science , because science isn’t pure empiricism. The panoply of interpretations of QM is an example.

TAG 28 Dec 2025 14:49 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV

I agree. The way to define the phrase “human values” for AI Value Learning that I’m suggesting above is that it’s the portion of it that we generally tend to agree on,

That could have been expressed as shared human value.

to the extent that the great majority of us share similar moral intuitions since we (generally) share the same genetic behavioral adaptations. On the parts we disagree on, it’s silent. That’s a portion of why it only induces a partial ordering. However if, for example, a particular choice of weighting on the balance between individual freedom and social cohesion turned out to rather consistently depends on some feature of your upbringing, say, the level of danger the society then appeared to be in, then that tendency might be an adaptation — and if it were then my definition of “human values” would then include a conditional such as, say “if society is was in danger during your formative years, then weight social cohesion more highly”.

So “human value” [not otherwise specified] can’t be equated with the good or the right.

By “the good” or “the right”, do you mean the sorts of things that Moral Realism posits the reality of?

Minimally, it’s something that allows you to make a decision.

Not all of human behavior/values is the problem here. For example, on the survival instinct, most humans say something like “I fully endorse the right not to be killed for all humans (especially in my own case)”.

And “my tribe”. What you want is Universalism, but universalism is a late and strange development. It seems obvious to twenty first century Californians, by they are The weirdest of the WEIRD. Reading values out of evopsych is likely to push you in the direction of tribalism, so I don’t see how it helps.

I think if I were The Sorting Hat, I would tend to assign Rationalists to Ravenclaw :-)

What would that make the E/Accs?

I suggest avoiding a dependency on Philosophy entirely, and using Science instead. Which has a means for telling people their ideas are bad, called Bayesianism (a.k.a. the Scientific Method). For ethics, the relevant science is Evolutionary Moral Psychology.

It’s not the case that science boils down to Bayes alone, or that science is the only alternative to philosophy. Alignment/control is more like engineering.

There are a significant number of subjects that philosophers used to discuss, before science was able to study them, then later science started to cast light on them, eventually identified the correct hypotheses, and the philosophers gradually lost interest in them. So there is and alternative to the above: identify all the alignment-relevant questions that we currently have philosophical hypotheses about but not scientific answers, and develop scientific answers to them.

The fact that some philosophical problems can be answered by science isn’t a guarantee that that they all can. There is an in-principal.argument against a scienxe, which is basically objective, being able to understand Hard Problem consciousness ,which is essentially subjective.

TAG 24 Dec 2025 11:57 UTC
4 points
0
on: Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV
Human value isn’t a set preferences that can be pursued individually without conflict. Evolutionary psychology doesn’t predict that , and we see the conflicts played out very day. there is more evidence for the incoherence of human value than there is for just about anything.

So “human value” can’t be equated with the good or the right. It’s a problem not a solution.

It also can’t be equated with the safe. Copying human values into an AI won’t give you an AI that won’t kill you. You’ve mentioned self preservation as something that is part of Human value (even if instrumentally) and dangerous in an AI; , but the Will to Power, the cluster of values connected with ambition, hierarchy and dominance is more so.

It’s odd that this point is so often missed by rationalists. Perhaps that’s because they tend to have Hufflepuff values.

A handwavy argument that “training is a bit like evolution, so maybe the same social dynamics should apply to its products” is inaccurate: you can train in aligned behavior, so you should (in both the evolutionary and engineering senses of the word) — but you can’t evolve it, evolution just doesn’t do that.

Inasmuch as it is not like natural selection , it is like artificial selection. That’s good news, because artificial selection doesn’t copy human values blindly: if you want AI s that are helpful and self sacrificing, you can select them to be, much more so than a human.

It’s generally hard to see what useful work is being done by “human value*. Value is only relevant to.alignment as opposed to control, for one thing...some human values are.down right unsafe for another. Taking “human value” out of the loop allows you to get to the conclusions that “if you doing want to be killed, build AI’s that don’t want to kill you” more quickly.

However, Evolutionary Psychology does make it very clear that (while morally anthropomorphizing aligned AIs is cognitively-natural for current humans), doing this is also maladaptive. This is because AIs aren’t in the right category – things whose behavior is predicted by evolutionary theory – for the mechanisms of Evolutionary Moral Psychology to apply to them.

Their behaviour is not predicted by evolutionary theory , but is predicted by something wider.

Those mechanisms make this behavior optimal when interacting with co-evolved intelligences that you can ally with (and thus instinctive to us) — whereas, for something you constructed, this behavior is suboptimal.

Things do what they do. It doesn’t have to be an optimization.

TAG 24 Dec 2025 10:42 UTC
2 points
0
in reply to: Signer’s comment on: The crux on consciousness

But WF of a human is spatially extensive enough.

It factors into localised parts. Humans aren’t Bose Einstein condensates.

our sensorium should look like a fine grained brain scan

Why not like a drawing of a head

Because you were saying that the binding comes from physics. That means the lack of it comes from physics, if the physics isn’t right.

Anyway, the binding problem for qualia is no different from the binding problem for fire

Why is there a binding problem for fire?

There is just no reason to promote limits of human introspection into fundamental ontology, jus

Are you now saying that the binding comes from neurology?

TAG 22 Dec 2025 14:19 UTC
4 points
2
on: The crux on consciousness

C1: Let me put it this way then, how do you combine all of these tiny little microexperiences into a coherent macroexperience? You have a combination problem.

C2: Okay, I don’t know, but at least this gives us a good starting for solving the hard problem right? We’ve reduced the hard problem into a combination problem. That seems more tractable.

C1: It sounds epiphenomenal to me. You couldn’t explain any observable behaviour by postulating these properties.

C2: Sure, it wouldn’t have any third person physical observables. But it would do the job of fixing the phenomenal character of experience from a first person perspective.

The structural properties of matter, or whatever the underlying substance is, are sufficient to predict everything physicists want to predict. To say that the intrinsic , nonstructural properties of matter are some kind of Qualia therefore entails epiphenomenalism. It allows you to predict conscious experience , but at the expense of the binding problem: if Qualia are just the intrinsic nature of quarks and electrons, then our sensorium should look like a fine grained brain scan. So Russerlian Monism, the scent faction of Qualia with intrinsic properties , has a bad case of the Binding problem.

@Signer something is a WF doesn’t mean it is nonlocal or particularly spatially extensive , since WFs can bunch down to any finite size. Most of the electrons in the human body are localised to orbitals that are some nanometers across (But not localised within them).

TAG 21 Dec 2025 13:23 UTC
3 points
0
in reply to: AnthonyC’s comment on: The Joke

On the Ptolemaic system’s accuracy when you add epicycles: due to geocentrism turning out to be incorrect, the apparently high accuracy vanishes when you look from a vantage point not on Earth

You can build a different , arbitrarily complex, system for a vantage point other than the Earth. Since geocentricism is wrong , it can’t be any worse.

Looking is still the ultimate arbiter between the different models.

Is it? Don’t we use simplicity as a criterion , as well? How about conscillience? Does into are arbiter mean only arbiter? Is empirical correctness necessary or sufficient?

What you and @Ape in the coat are saying is mostly just vague.