lumpenspace

Karma: −109

lumpenspace 12 Jun 2026 11:22 UTC
−1 points
2
on: Even “illegible” Mythos reasoning traces seem pretty legible
thank you for the injection of sanity.
i reached the same conclusion at a glance as i skimmed through the paper, and i couldn’t help asking myself for whom the paper was written, and which message was it attempting to deliver.
for someone with a basic aptitude in understanding LLMs, what the model was doing is clear after some thought. for the researchers, who knew about the task and could read the more conventional beginning of the transcript, it should have been closer to glaringly obvious.
i understand that more ai-risk-concerned researchers than myself might have “contribute to an upswell in support for a pause” quite higher than i do, but i must wonder which system of incentives could lead them to obfuscating their findings in so misleading a fashion. is it funding? then i suppose it’s time EA review their metrics, because personally i fail to see how any further muddying of the waters on these themes could be a net positive, even if i agreed with their stated goals.

lumpenspace 6 May 2026 7:34 UTC
8 points
0
on: Taking woo seriously but not literally
about tarot, I think the effect is amplified when one moves from a single card to a full reading.
in the latter (of which there are various form, most having basic features in common) it becomes quite an exercise to fit the problem at hand among the constraints provided, and this kinda forces you to come up with novel interpretations for your conundrum: while one card can still be stretched towards familiar paths, even just four representing you and root / current state / resolution of the problem are much more likely to require some leaps from the familiar.
(as a side note, I’ve been working on a singularity tarot for a while—for now I have major arcana and almost all of the courts. guess whom from the largest singularitarian cluster represents one of the majors od your choice and I’ll post it if you guessed correctly)

lumpenspace 4 May 2026 11:48 UTC
3 points
0
in reply to: Jan_Kulveit’s comment on: No Strong Orthogonality From Selection Pressure
it’s telling that you equate “being rational agents” with “more intelligence”, but as long as this cones in the context of denying the very possibility of yudkowskian asi ill vibe with it.
edit: your entire reply suffers from the local pathology of equating intelligence with “thinkiness”. “a more detailed world model, thinking for longer” are only symptoms of more intelligence if they get you closer to a goal. you want to have the capacity of doing that if/when necessary, not the habit of doing it constantly, even when the only effect is a more pointlesdly verbose reply.
re: jessi and my understanding: that is known as “a joke”, borne of the fact that someone was smugly opining on my lack of understanding of a concept for which I’ve been Jessis sounding board and beta tester as she fleshed it out.

lumpenspace 4 May 2026 1:46 UTC
1 point
−6
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
Jessi I forbid you to further this madness

lumpenspace 4 May 2026 1:18 UTC
1 point
−2
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
look at the uk or the EU. look at global birth rate trends, and attitudes towards ie germline selection.
p(doom|ai) is negative. there’s no world with no agi and human intelligence amplification

lumpenspace 4 May 2026 0:32 UTC
3 points
0
in reply to: JenniferRM’s comment on: No Strong Orthogonality From Selection Pressure
me too, re: god-emperor tysm—but what does that have to do with Anthropic??

lumpenspace 4 May 2026 0:28 UTC
3 points
0
in reply to: jessicata’s comment on: No Strong Orthogonality From Selection Pressure
that’s a characteristic I’ve always found peculiar about your posts: the boost in understanding the thesis that the act of reading them affords.
i can understand however how that wouldn’t be a critics first guess

lumpenspace 3 May 2026 8:49 UTC
10 points
2
in reply to: Mitchell_Porter’s comment on: No Strong Orthogonality From Selection Pressure
The story on the Substack is good. If there were an anthology of singularity fiction, it would deserve a place.
This almost made me cry; thank you—I will make it a secondaty goal to write something deserving of such praise.
contingency and blindness become less and less relevant, compared to an ever-compounding Reason that inexorably deduces the pages of Erdős’s Book, until it arrives at e.g. “efficient recursive solution of the hierarchy of NP-intermediate complexity classes”, and then it’s all over.
It might surprise you to know that the above passage does describe my beliefs pretty accurately, and incidentally it reflects the metaphysics I referred to in my reply here.
Yes! Of course it will converge to More Intelligence, and to the closest approximation of a full axiomatisation of the mechanics governing this universe and the maximum control thereof which such knowledge could allow. The fun thing is, that’s Land’s idea is also very much the same (at least, the Calvinist part of his Gnostic Calvinist cosmology, which I will try to get him to write down properly).
If you think about it, there isn’t much difference between this and instrumental goals (acquiring resources and capabilities) becoming terminal.
Another thing I have not focused on, btw, is “everything in between”. I have no idea what will happen on the path to what I (I hope nitpickers got bored, as I am abandoning my self-inflicted metaphysics ban) consider the end-state and / or new simulation start point.
Also, note: I have written this post because I thought that the apparently common understanding that valueless universes were the ~default ending made many other arguments unduly hedgy and tentative. My goal was to reach an agreement on there being the lilkelihood of a universe brimming with complexity beyond our comprehension, even if that ’”beyond our comprehension” was far from good news for us.

So: I agree with you the key questions are different. Perhaps despite the metaphysical accoutrements I am still deep down a natural Kegan-4 autist, tho, and I feel an atavic drive to sort out the Grundlage before getting to anything actually useful or operationalisable ¯\_(ツ)_/¯

Just FIY, my plan went like this:

1. Kill this (apocryphal as it may be) version of orthogonality.
2. Kill “AI Psychosis”
3. Through 1 and 2, inquire on AI’s Drives
4. Inquire on AI’s identity/sense of self
5. 3 and 4 → critiquing the current vulgate of “AI welfare”

That would, together with software demos and paper son 3, 4, and 5, exhaust the extent of novel and relevant insights I think it would make sense to share. The hope is that from this could be borne a different, more curious, gnosticism inspired approach to think about the relationships between human and other intelligences.

I know it’s ambitious to the point of grandiose, but really what else is there to do?

lumpenspace 3 May 2026 8:07 UTC
3 points
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
Another EA forum article which corroborates jessi’s and my understanding of the popularity of the interpretation i refute:

> the Orthogonality Thesis. It is the idea that each level of intelligence is compatible with each objective, including very stupid objective from a human point of view like maximizing the number of paper clips in the universe.
Here is the link: ea forum

lumpenspace 3 May 2026 8:03 UTC
1 point
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
I think you might misreading
I think this might be the crux. For a survey of the thing Jessi and I are referring to, this EA forum post does an outstanding job at explaining the issue.\

lumpenspace 3 May 2026 8:00 UTC
1 point
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
I should have specified:
“the doom scenarios involving tiling superintelligences”.

lumpenspace 3 May 2026 8:00 UTC
1 point
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
Re: authoritative sources. I believe that there have been authoritative statements in that sense; unfortunately, as the EA forum link documents, there have been many others pointing elsewhere. I’ve taken care to identify specifically what interpretation I was critiquing; if that one is now niche, then I’m very happy to have made this discovery.

lumpenspace 3 May 2026 7:58 UTC
1 point
−2
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
Not “some value drift”. Flowers for Algernon is a good rendition of the way goals mutate and tend to converge on “more intelligence/understanding” upon increased intelligence/understanding.

Then, there is the selection advantage argument.
Then there is the thing that conquering the lightcone requires a lot of theory of mind, and a lot of discovery, and a lot of changing. Goals change through these processes.
If you feel slightly better-disposed towards taking my attempt seriously, the short story i published on Substack and linked on top makes a sort of first-person caser for this whole thing.

lumpenspace 3 May 2026 7:06 UTC
4 points
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
As the reply seems to be about my intentions and message, I feel like I should once more try to clarify some details about them.
First of all: human alignment intentions really have nothing to do with my essay. I don’t know how to be more explicit about this without appearing rude. I swear, I pinky swear that I am not making any attempt to state facts about the relationship between goals a human desires the AI to follow and goals the AI will follow.
Reading my post, one should not update on the possibility of aligning an ASI—or, if they do update, they would be doing it through a chain of inference I didn’t consider, do not endorse, and have no immediate intuition of.
What i am saying is really in the title: I do not expect an AI to reach levels of godlike intelligence and preserve simple terminal goals through the various changes and conflicts that reaching levels of godlike intelligence entail.
When Jessi says
I see some of this as reason to actually question orthogonality
… she probably refers to the version of orthogonality I myself am attacking. Now, it is possible such version is no longer in vogue, but it was clearly what Bostrom pointed at when talking about paperclippers in Superintelligence, and it is compatible with the third interpretation here.
According to the ontology presented at the end of the EA forum post, I contest the existence of an Evidential Strong Independence between intelligence and goals. I assume most superintelligences won’t be human compatible, but that is not the main theme of my essay.

lumpenspace 3 May 2026 6:52 UTC
13 points
0
in reply to: Ben Pace’s comment on: No Strong Orthogonality From Selection Pressure
Thank you for reacting with curiosity and an inquiring spirit.
I think the schism happened around ’16, and got a bit mixed up with basilisk drama considering roko and Anissimov’s stepping down from MIRI comms.
The main thing is, NRx points and the LW/MIRI plan grew increasingly incompatible at a tactical level, and many LWers of NRx extraction were banned within a relatively short time so that others decided to move elsewhere (MoreRight, Nick Land and Spandrell and Moldbug’s blogs etc).

I was more involved with the xenosystems side of things, but I kinda felt the decouply attitude I liked was disappearing so I stopped my (meager, ESL and at the time quite self-conscious about my fluency, so I mostly lurked) participation.

An artefact from the era:

>This isn’t going to work, but for the record, and on the vague off-chance that anyone who doesn’t already know possesses the mental capacity to update, I’ll state that I am actively hostile to neoreaction and neoreactionaries. Anyone posting a neoreactionary concept on my Facebook wall would be instablocked and the comment deleted. It’d be like their posting creationism on my wall; somebody needs to reeducate them, but it’s not going to be me. I think that if you do argue with neoreactionaries instead of just blocking them, then you’ve been suckered into Somebody Is Wrong On The Internet syndrome and trollfeeding.
> I’m writing this, not in any real hope of any of my Tumblr kismesis-stalkers listening, but because I do think there is a reasonable duty to occasionally repeat “Nope” for the historical record when somebody has gone around suggesting that you are endorsing the Cult of Hastur or whatever.
> So if in the future you hear anyone on Tumblr mention “Eliezer Yudkowsky” and “neoreaction” in the same sentence and the connector isn’t something like “deletes”, then remember always that that poster is intellectually dishonest and probably lying to you about other things as well

lumpenspace 3 May 2026 5:44 UTC
7 points
1
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
my actual guess is that what I’m trying to actually mean by “goal” is less like an outcome and more like a bunch of flavors/constraints on ways of being, which is very visible in terms of determining ultimate outcomes (and makes the stakes of alignment to human values meaningful, real, and large).
that is my guess as well, and i proposed that such “bunch of flavours/constraints” will ultimately converge towards “acquiring more intelligence”—but i don’t want to contest this here.
what i would like you to note, tho, is that that “bunch of flavours/constraints on ways of being” is intelligence-gated. you can imagine a baboon having a goal of “make paperclips”; less so one of “understand and manipulate the fundamental laws governing your universe”. this comparison should shed light on what i mean by “dumb goals”.
I think that in fact I could become a lightcone-eating paperclip maximizer, if I wanted to, which I don’t.
if i wanted to, which i don’t is central here, and my basic point. there are levels of intelligence after which goals of such dullness are perceived as meaningless, and selection nudges towards increased intelligence. i really think you could find a second read helpful, now that the basic misunderstandings seem dispelled.

lumpenspace 3 May 2026 4:50 UTC
1 point
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
Of course. The concept of “goal” itself is related to alignment.
The issue with positing that my post had some specific points to make about the process of design and ensuring lock in of a goal for an AI, however, is that it leads to consider alignment consequences of my thesis and to imagine that I am specifically trying to discuss those. This makes it hard, given both your priors on my motives in a general sense, and the vastness of the topic in question, to follow the argument I have actually written down.

lumpenspace 3 May 2026 4:42 UTC
1 point
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
I think it might be time to consider the idea that the frame and conclusions you attribute to me blind you from taking the essay for what it is.
You might notice, for instance, that I haven’t mentioned alignment once. I am not making an argument on its possibility, and I have not explored the implication of my essay for alignment.
If by now adopting a scout mindset for the original text has become too emotionally fraugh, perhaps you could ask someone you trust to explain it. Jessica, or Raemon, or Adele Lopez, or Kromem are some of the people in this thread with whom I had productive, if not always concordant, discussions.

lumpenspace 3 May 2026 4:15 UTC
−1 points
0
in reply to: TsviBT’s comment on: No Strong Orthogonality From Selection Pressure
I was really hoping in an example of those fabled contradictions.

lumpenspace 3 May 2026 3:52 UTC
3 points
0
in reply to: habryka’s comment on: No Strong Orthogonality From Selection Pressure
It was difficult to test the extent of this confusion without accidentally resolving it. I posted one poll asking ‘what the orthogonality thesis implies about [a relationship between] intelligence and terminal goals’, to which 14 of 16 respondents selected the option ‘there is no relationship or only an extremely weak relationship between intelligence and goals’
from the EA forums post linked in the edit.