Signer

Karma: 595

Signer 2 Oct 2025 7:13 UTC
1 point
0
in reply to: TAG’s comment on: [Question] What the discontinuity is, if not FOOM?
If all AIs are scheming, they can take over together. If a world with a powerful AI that is actually on humanity’s side is assumed instead, then at some level of power of friendly AI you probably can run unaligned AI and it will not be able to do much harm. But just assuming there being many AIs doesn’t solve scheming by itself—if training actually works as bad as predicted, then no AI of many would be aligned enough.

Signer 1 Oct 2025 15:51 UTC
1 point
0
in reply to: TAG’s comment on: Beyond the Zombie Argument

Russelian monism struggles with Epiphenomenality: if the measurable, structural properties are sufficient to predict what happens, the the phenomenal properties are along for the ride.

I mean, it’s monism—it supposed to only has one type of stuff, obviously structural properties only work, because of underlying phenomenal/physical substrate.

furthermore, since mental states are ultimately identical to physical brain states, they share the causal powers of brain states (again without the need to posit special explanatory apparatus such as “psychophysical laws”), and in that way epiphenomenalism is avoided.

I don’t see how having two special maps has anything to do with monistic ontology, that enables casual closure. What’s the problem with just having neutral-monistic ontology, like you say Dual-aspect neutral monism has, and use normal physical epistemology?

the epistemic irreducibility of the mental to the physical is also accepted.

Why? If ontologically there is only one type of stuff, then you can reduce mental description to physical, because they describe one reality. Same way you reduce old physical theory to a new one.

Signer 1 Oct 2025 15:21 UTC
1 point
0
in reply to: TAG’s comment on: [Question] What the discontinuity is, if not FOOM?

Why would that be discontinuous?

Because incremental progress missed deception.

I’m arguing against 99%

I agree such confidence lacks justification.

Signer 1 Oct 2025 13:33 UTC
1 point
−9
on: Beyond the Zombie Argument
I don’t think there is a need to qualify it as a potential solution—Russellian Monism just solves the Hard Problem.

Signer 1 Oct 2025 13:02 UTC
1 point
0
in reply to: TAG’s comment on: [Question] What the discontinuity is, if not FOOM?
I don’t think anyone is against incremental progress. It’s just that if after incremental progress AI takes over, then it’s not good enough alignment. And what’s the source of confidence in it being enough?

“Final or nonexistent” seems to be appropriate for scheming detection—if you missed only one way for AI to hide it’s intentions, it will take over. So yes, degree of scheming in broad sense and how much you can prevent it is a crux and other things depend on it. Again, I don’t see how you can be confident that future AI wouldn’t scheme.

Signer 1 Oct 2025 2:29 UTC
2 points
0
in reply to: boazbarak’s comment on: A non-review of “If Anyone Builds It, Everyone Dies”

I just think that it wouldn’t be the case that we had one shot but we missed it, but rather had many shots and missed them all.

This interpretation only works if by missed shots you mean “missed opportunities to completely solve alignment”. Otherwise you can observe multiple failures along the way and fix observable scheming, but you only need to miss one alignment failure on the last capability level. The point is just that your monitoring methods, even improved after many failures to catch scheming in pre-takeover regime, are finally tested only when AI is really can take over. Because real ability to take over is hard to fake. And you can’t repeat this test after you improved your monitoring, if you failed. Maybe your alignment training after previous observed failure in pre-takeover regime really made AI non-scheming. But maybe you just missed some short thought where AI decided to not think about takeover since it can’t win yet. And you’ll need to rely on your monitoring without actually testing whether it can catch all such possibilities that depend on actual environment that allows takeover.

Signer 1 Oct 2025 0:32 UTC
9 points
0
on: [Question] What the discontinuity is, if not FOOM?

if ASI is developed gradually , alignment can be tweaked as you go along.

The whole problem is that alignment, as in “AI doesn’t want to take over in a bad way” is not assumed to be solved. So you think your alignment training works for your current version of pre-takeover ASI, but actually previous versions already schemed for a long time, so running a version capable of takeover suddenly for you creates a discontinuity, where ASI takes over because it now can. It means all your previous alignment work and scheming detection is finally tested when you run a version capable of takeover and you can only fail once on this test. And training against scheming is predicted to not work and just create stealthier schemers. And “AI can take over” is predicted to be hard to fake for AI so you can’t confidently check for scheming just by observing what it would do in fake environment.

Signer 30 Sep 2025 22:37 UTC
8 points
8
on: Why Corrigibility is Hard and Important (i.e. “Whence the high MIRI confidence in alignment difficulty?”)

The technical intuitions we gained from this process, is the real reason for our particularly strong confidence in this problem being hard.

I don’t understand why anyone would expect such reason to be persuasive to other people. Like, to rely on illegible intuitions in the matters of human extinction just feels crazy. Yes, certainty doesn’t matter, we need to stop either way. But still—is it even rational to be so confident when you rely on illegible intuitions? Why don’t check yourself with something more robust, like actually writing your hypotheses, reasoning, and counting evidence? Sure there is something better than saying “I base my extreme confidence on intuitions”.

And it’s not only about corrigibility—“you don’t get what you train for” being universal law of intelligence in the real world, or utility maximization, especially in the limit, being good model of real things, or pivotal real world science being definitely so hard you can’t possibly be distracted even once and still figure it out—everything is insufficiently justified.

Signer 21 Sep 2025 11:20 UTC
2 points
1
on: The title is reasonable

Once you do that, it’s a fact of the universe, that the programmers can’t change, that “you’d do better at these goals if you didn’t have to be fully obedient”, and while programmers can install various safeguards, those safeguards are pumping upstream and will have to pump harder and harder as the AI gets more intelligent. And if you want it to make at least as much progress as a decent AI researcher, it needs to be quite smart.

Is there a place where this whole hypothesis about deep laws of intelligence is connected to reality? Like, how hard they have to pump? What’s exactly the evidence that they will have to pump harder? Why “quite smart” point can’t be when safeguards still work? Right now it’s not different from saying “world is NP-hard, so ASI will have to try harder and harder to solve problems, and killing humanity is quite hard”.

If there were a natural shape for AIs that let you fix mistakes you made along the way, you might hope to find a simple mathematical reflection of that shape in toy models. All the difficulties that crop up in every corner when working with toy models are suggestive of difficulties that will crop up in real life; all the extra complications in the real world don’t make the problem easier.

If there were a natural shape for AIs that don’t wirehead, you might hope to find a simple mathematical reflection of that shape in toy models. So MIRI failing to find such a model means NNs are anti-natural. Again, what’s the justification for significant update from MIRI failing to find a mathematical model?

Signer 25 Aug 2025 17:57 UTC
−5 points
−5
on: Arguments About AI Consciousness Seem Highly Motivated And At Best Overconfident
Personally, I’m against unqualified usage of “uncertainty” in relation to consciousness—it conflates factual uncertainty with ethical uncertainty. And it mostly ethical uncertainty that needs to be worked on: it’s not like there couldn’t be relevant questions about specifics of information processing in LLMs, but without consensus about what we value in humans empirical research in the name of helping with ethical questions is mostly a distraction motivated by moral-realistic thinking. We have enough knowledge about brain to decide on solutions to simple ethical questions now—you don’t need more neuroscience to decide why you wouldn’t value global workspace implemented in a couple of lines of code that takes 2 MB of RAM.

Signer 18 Aug 2025 17:00 UTC
1 point
0
on: Debugging for Mid Coders

Do people just read really fast? I think they have some heuristics for figuring out what parts to read and how to skim, which maybe involves something like binary search and tracking-abstraction-borders. But something about this still feels opaque to me.

Even if you have poor heuristics, it’s still may be worth it to google/open docs and walk obvious links. The point is not to have an algorithm that certainly finds everything relevant, but to try many things that may work.

How do you learn to replicate bugs, when they happen inconsistently in no discernable pattern? especially when the bug comes up, like, once every couple days or weeks, instead of once every 5 minutes.

You speed up time. Or more generally prepare an environment that increases reproduction frequency, like slow hardware/network or higher load. You spam clicks and interrupt every animation, because all bugs are about asynchronous things. You save state after reproduction, or better before and start from it. If all fails, you add logs/breakpoints to be ready next week. But usually you just look at code to figure out paths that may manifest as your bug and then try to reproduce promising paths.

Signer 7 Aug 2025 12:12 UTC
8 points
0
on: No, Rationalism Is Not a Cult
The truth was always before our eyes:

Signer 3 Jul 2025 12:01 UTC
2 points
0
in reply to: Kaj_Sotala’s comment on: Kaj’s shortform feed
Neuron count intuitively seems to be a better proxy for the variety/complexity/richness of positive experience. Then you can have an argument about how you wouldn’t want to just increase intensity of pleasure, that just a relative number. That what matters is that pleasure is interesting. And so you would assign lesser weights to less rich experience. You can also generalize this argument to negative experiences—maybe you don’t want to consider pain to be ten times worse just because someone multiplied some number by 10.

But I would think that the broader spectrum of potential sources for pleasure and pain would still not require a greater spectrum of intensity.

Isn’t pain in both wings worse than in one?

Signer 17 Jun 2025 0:10 UTC
1 point
0
on: Subjective experience is most likely physical
Russelian panpsychism doesn’t postulate a new force—physics already accepts casual role of existence: only existing neurons can fire.

And it explains epistemic link—it’s cogito ergo sum—you’re always right, when you think that universe exists.

Signer 16 Jun 2025 0:59 UTC
1 point
0
in reply to: Lorec’s comment on: The Boat Theft Theory of Consciousness
And rock’s perception belongs to a rock.

Would anyone describe it as theirs? That access is reflective. It’s pretty difficult to retrieve data in a format you didn’t store it in.

But what if there is no access or self-description or retrieval? You just appear fully formed, stare at a wall for a couple of years and then disappear. Are you saying that describing your experience makes them retroactively conscious?

Signer 15 Jun 2025 0:29 UTC
1 point
0
in reply to: Lorec’s comment on: The Boat Theft Theory of Consciousness

Even if I’m not thinking about myself consciously [ i.e., my self is not reflecting on itself ], I have some very basic perception of the wall as being perceived by me, a perceiver—some perception of the wall as existing in reference to me.

Is it you inspecting your experience or you making an inference from the “consciousness is self-awareness” theory? Because it doesn’t feel reflective to me? I think I just have a perception of a wall without anything being about me. It seems to be implementable by just forward pass streamed into short-term memory or something. If you just separated such a process and put it on repeat, just endlessly staring at a wall, I don’t see a reason why would anyone would describe it as reflective.

I mean, it is reflective in a sense that inner neurons observe outer neurons so in a sense it is a brain observing brain. But even rocks have connected inner layers.

Signer 14 Jun 2025 2:28 UTC
21 points
19
on: The Boat Theft Theory of Consciousness
The thing I don’t understand about claimed connection between self-model and phenomenal consciousness is that I don’t see much evidence for the necessity of self-model for conscious perception’s implementation—when I just stare at a white wall without internal dialog or other thoughts, what part of my experience is not implementable without self-model?

Signer 14 Jun 2025 2:09 UTC
0 points
−3
on: Do Not Tile the Lightcone with Your Confused Ontology
“Death is fine if AI doesn’t have self-preservation goal” or “suffering is bad” are also just human ethical assumptions.

Signer 8 Jun 2025 8:38 UTC
1 point
−1
in reply to: Isopropylpod’s comment on: Isopropylpod’s Shortform
You are talking about experience of certainty. I’m asking why do you trust it?
I know it’s beyond doubt because I am currently experiencing something at this exact moment.
That’s a description of a system, where your experience directly hijacks your feeling of certainty. You wouldn’t say that “I know it’s beyond doubt there is a blue sky, because blue light hits my eyes at this exact moment” is a valid justification for absolute certainty. Even if you feel certain about some part of reality, you can contemplate being wrong, right? Why don’t say “I’m feeling certain, but I understand the possibility of being wrong” the same way you can say about there being blue sky? The possibility is physically possible (I described it). It’s not even phenomenologically unimaginable—it would feel like misremembering.
Why insist on describing your experience as “knowledge”? It’s not like you have perfect evidence for a fact “experience is knowledge”, you just have a feeling of certainty.
And if seeing-neurons of someone’s brain are in the state of seeing red, but they are thinking and saying that they see blue, would you say they are right?

Signer 7 Jun 2025 15:33 UTC
2 points
0
in reply to: Isopropylpod’s comment on: Isopropylpod’s Shortform
How do you know it’s beyond doubt? Why is your experience of blue sky is not guaranteed to be right about the sky, but your experience of certainty of experience is always magically right?
What specifically is beyond doubt, if seeing-neurons of your brain are in the state of seeing red, but you are thinking and saying that you see blue?