I think the heuristic “nothing ever happens” is better interpreted to mean “nothing ever happens relative to baseline trends” than “literally nothing ever happens”. The incrementalist worldview seems like a better fit for this heuristic than Eliezer’s, which after all ultimately predicts something very dramatic happening.
interstice
Karma: 2,839
(didn’t downvote, but) I don’t think you’re necessarily wrong, but couldn’t it just be the case that being a singleton isn’t that hard? As an empirical matter, the size(as a fraction of the total) of the largest somewhat-coherent entities controlling resources on Earth seems to have been increasing over time. Space expansion could change things, but a stable singleton might already exist by then, and be faced with a relatively homogeneous set of environments to expand into. I’ve written some pieces along similar lines btw.
What picture does all this data paint? What patterns can you see when you look at all the data, instead of facing down each argument one at a time and dismissing them one at a time?
One possible explanatory pattern is that Michael is an insane and conspiratorial guy and attracts people who are also insane and conspiratorial, and it’s easy to push such people into having a meltdown.
“I,” you say, and are proud of that word. But the greater thing—in which you are unwilling to believe—is your body with its great wisdom; that does not say “I,” but does “I”
What the sense feels, what the mind knows, never has its end in itself. But sense and mind would rather persuade you that they are the end of all things: so vain are they.
Instruments and toys are sense and mind: behind them there is still the Self. The Self seeks with the eyes of the senses, it listens also with the ears of the mind.
Always the Self listens and seeks; it compares, masters, conquers, and destroys. It rules, and is also the mind’s ruler.
Behind your thoughts and feelings, my brother, there is a mighty lord, an unknown sage—it is called Self; it dwells in your body, it is your body
There is more wisdom in your body than in your best wisdom. And who then knows why your body needs precisely your best wisdom?
Your Self laughs at your mind, and its bold leaps. “What are these leaps and flights of thought to me?” it says to itself. “A detour to my end. I hold the puppet-strings of the mind, and am the prompter of its notions.”
I think if Kim Jong Un lived for a million years, and had the smartest AI advisors, and access to intelligence augmentation techniques, he would probably still never come to admit that murdering his brother was an evil thing to do
I feel like the evidence you’ve provided here is pretty weak for drawing that conclusion. The regime where KJU lives for a million years and augments his intelligence is really outside distribution.
Halifax, Canada—ACX Spring Schelling 2026
LessWrong but with cats instead of posts
Self explanatory
I’ve been taking collagen and the exogenous antioxidant astaxanthin for a while and it’s definitely improved my sleep among other things.
Good post about LLMs scheming against their stupid human overseers.
What sources have you used to derive your understanding of brain function from?
Simulators
On the object level, there’s a very plausible mechanism why solving some problems can make things worse
Hmm, I see. But it seems to me that the linked post is overoptimistic that making problems legible will cause people to slow down until they are solved. Already, lots of problems that are “legible” to this community are not “legible” enough to make the labs or governments want to slow down. So working to solve them(by our standards, since apparently other people already consider them solved or unimportant) could still be useful.
Of course you could then say that what we should be doing is trying to make problems legible to this community legible to the wider world, which is pretty much what MIRI is trying to do at the moment. Certainly that seems like a valuable thing to do. But far from guaranteed to succeed. And I think the fact that LW-legibility of takeover isn’t on its own enough to cause the wider world to slow down, should make us less worried that solving one problem to our standards will make the world push ahead more recklessly, since they seemingly aren’t that responsive to what problems are considered solved by our standards.
there is a bunch of different safety problems where solving some but not all of them at the same time can make the overall situation worse
But isn’t the relevant thing here whether solving some makes the overall situation worse in expectation? Like sure, it could be the case that e.g. solving intent alignment makes things worse, but it seems relatively unlikely. It seems implausible that the current set of problems we have now cancel each other out optimally. Or maybe we should try to introduce new problems to further mitigate the set of existing problems!
Yeah it’s more like it’s relevant to the kind of world we find ourselves in. But that is itself important to agency as a given agent design will only be successful in certain kinds of worlds.
I think QM itself. It’s important somehow that the world is actually quantum mechanical. But probably not in a very direct fashion, but via influencing the sort of high-level properties and entities that end up “emerging” from the base laws.
[I don’t actually think this is true, but] It would be funny if rationalism turns out to not merely be a euphemism for autism but “mal”functioning oxytocin receptors and rationalists are constitutionally unable to normally feel love/social emotions; whether this would be to the discredit of love or rationalism is up to taste.
“quantum mechanics is probably important to the structure of agency/the mind in some way we don’t understand yet”.
Can’t speak for everybody but when I read simulators I was like “yeah obviously GPTs are generative models not agents, duh, why do people need an entire post to tell them that” and didn’t really expect the thing where agents are like “characters” in the world model to scale to high levels of capability. That seeming to happen raised the salience of the ideas a lot for me.
I’m currently thinking that it might be a good idea to publish a bunch of text that I think could help AIs make conceptual/philosophical progress quickly. Basically because it seems like (a) there could be a time period(like, uh, now) where AIs can do some reasoning on their own but don’t have good taste or autonomy in high-level research directions, so publishing incomplete stubs of ideas or promising things to look into could help them. (b) AIs that rely on human text more to make intellectual progress are likely to be more aligned than AIs that don’t, so this should differentially help more-aligned models. There is the risk that helping AIs in this way could empower misaligned models, but I think this is outweighed by factor (b) overall. And of course the most likely outcome is that whatever I publish simply isn’t that useful, but that seems ~neutral. Does anybody have any other considerations that could be relevant here?
Yes. But the OP is about contrasting people like Paul with Eliezer. Paul (I think) does indeed predict a dramatic singularity, but also that the ramp up to said singularity will be smoother and more widely distributed across society than Eliezer predicts.