Aprillion
join forces to be less wrong together?
internal selection pressure towards being more dominated by power-seeking subsystems, eventually the world gets screwed
I feel as if a tacit assumption bears too much weigh of that argument, why cannot an immune system work on that level? a cancer can kill a human, but not the humanity. but by default any internal subsystem of most imagined societies that will exist is going to kill all of societies globally, not just their local hosts? why is Moloch weaker than Clippy? why no warning signs—do you see humanity on a trajectory of not using still-stupid LLM based systems in autonomous weapons, never deploying any system that’s too weak to kill everyone while powerful to cause major harm while also brewing up a system that’s powerful enough to escape and self-preserve and gray-goo itself on its first try?
..makes me wonder if 7 people were like “I don’t necessarily disagree, but let me strongly discourage this person to discuss violence in the public”
as if violence was actual 4D chess strategy that could actually work but only if we kept silent about nuking data centers after the Yudko’s unfortunate blunder, as if murder of Brian Thomson improved the healthcare regulation, Suchir Balaji taking his own life stopped the LLM scaling race, Russia became the economic tiger of the 21st century after the quick reunification campaign that went so well, Hamas is now soo de-extremified after the genocide, killing Osama bin Laden stopped all terrorism, Tigray peace has hold forever and ever with no hiccups this year, .. and if black lives matter is now more true than before, it’s not as if the people of the movement killed George Floyd and Breonna Taylor themselves to prove a point..
anyway, live long and midi-chlorians be with you to rule them all
imagine it happens in a competitive environment where only the players who’s number go up the fastest will survive and the more you try to measure that elusive aspect, the harder everyone Goodharts
I observe I am disappointed, came here with an (unreasonably?) high expectation, then the text formatting screamed at me LLM slop, found the label “AI” mushed with the concept of previous-gen chatbots, searched for “egregore” without finding it, and sampled a few paragraphs where the language of the human self sounded like the confused technocratic formal realism I had about my own experience ~20 years ago ;(
But I have stepped on a lego and I remember remembering the pain while having a pain two seconds after having stepped on a lego, and I can hold the 2 memories in my spotlight for comparison. I have had a dream about wiggly teeth before I got punched in my mouth and boy how the simulation was not the real deal. I have memories of gaps, a zero streams of consciousness when I was awake, and of times without time. I had conversations with non-binary people who don’t have a second to second persistent identity, who feel they just arrived to this body that is wrong and the sameness is only from a memory of someone else who felt the same alienness in that body.
I have seen many shades of red, some of them out there while others in my perception, and there is no redness of the color red for me.
Who cut the onions or opened the door while I was mid conversation? Who didn’t forget to take the keys for I don’t remember but they are in the bag?
Without tool use, I wouldn’t know how to write my name, yet I recognise it when I see it written down. I wouldn’t know how tall I am, but I trust my memory of the many times I used the various tools that measured my height and they mostly agree except that one that was a joke.
If a vocabulary around human identity is inadequate for AIs, can we not start with a shitty strawman of a vocabulary that was never adequate for the human experience in the first place, please? 🙏
that didn’t trigger any allergic reaction for me … but following would:
I just started using em dashes—at the recommendation of my older brother—in my writing circa 2022 before I even knew about LLMs. I like using them and I’m not giving them up!
from the little we interacted, I suspect your case of “more stupid” is just dialling down over-thinking .. while I know people who should definitely NOT dial it any closer to “actually stupid” 😅
I’m not sure I would use terms like Lipschitz continuity, KL divergence, spurious oscillations, OOD divergence or something else that would highlight the point, but when I imagine myself in a coworker / tech lead / management role working with human software engineers before 2024 vs myself as a software engineer working with LLM-powered coding assistants in 2026, there is a very clear difference in the kinds of “outside” with regards of training distribution in human-human vs human-LLM interactions, the latter being really really fucking annoying tiring shit in every single interaction, while the former is “it depends” (a.k.a. “hiring a team that will be a good match together”).
The agentic scaffolds of 2025+ are making it possible to work around some of the fundamental jaggedness of LLM base models which are still complete shit at “understanding” so we are collectively moving ever more problems into “within distribution” instead of “divergent extrapolation”, sure, so I agree it’s totally unpredictable if LLM-powered tools will be able to automate tasks enough to become the kind of dangerous agents for which it makes sense to reason about theoretic-rational instrumental goals even if LLMs alone might remain shit at goal-orientednes forever (or if we need different architecture) - but we should probably discuss the capabilities of those agentic entities, not individual benchmark-gaming components of such entities...
solved alignment
A better framing for my understanding is “it will appear as if we solved alignment”, so all humans will be persuaded to upload and the few scientists/programmers who discover that uploads are running in “optimized” mode as a single forward pass without the promised internal computation “functionally equivalent to consciousness” will have been disappeared 5 years ago...
The benefit of the gradual disempowerment frame is about what to look for (persuasion, trust in marketing materials without anyone “wanting” independent reviews, natural selection of ever-more-anti-bio-human ideologies, …) while the framing of “solving alignment” sounds to me like an impossible task with infinitely large attack surface area. But the old-school AI safety doomerisim is not mutually exclusive, just non-practical ontology (which leads to “only math is real, physics is boring, mentalistic talk is emergent, collective behaviour is well explained by game theory, VNM rationality ought to be morally real, proofs are possible (in principle)” kind of thinking that doesn’t translate to actionable recommendations).
wait, if this is how “things go badly”… what does a possible good outcome look like for contrast, please? am I missing something in my imagination if this sounds like the best possible scenario around “we avoided doom for >100 years” to me?
(assuming a future mainstream belief that base-bio-humans will be suffering unnecessarily compared to technology-enhanced-humans .. just like antiparasitics-enhanced-humans today believe river blindness should be eradicated or like book-enhanced-humans believe elementary education is a basic human right)
why do you imagine that “you are a lobster” is not sufficient to trigger a roleplay of Accelerando? did someone actually die because of moltbook? went bankrupt? was sent to prison? what was “genuine evil” about the event in your eyes?
The most prominent phrase in my mind as I recollect noticing my lack of surprise was cyborg egregore … based on that, I prompted Claude to find me a close match in Lobsters, a chapter from Accelerando by Charles Stross, 2005 … I haven’t read the Singularity trilogy (yet 🤓🔫💸), but it would explain why I felt as if this whole event escaped right from a SF book about a simulated AI singularity and not a simulation of how the singularity would actually happen—because it did.
excerpt from https://claude.ai/share/d5ddd555-5a25-462e-8a68-bb629e428eba
Interesting question. Let me search for precursors to this specific scenario.Let me search more specifically for SF and academic predictions of this scenario.Let me search more specifically for Accelerando and Rainbows End, which seem most relevant to the scenario.This is great—“Lobsters” from 2001 is very close. Let me search for the specific passage about Manfred’s agent filtering spam and the ultra-fast news cycle.This is excellent! The text from “Lobsters” (June 2001) is remarkably prescient. Let me search for earlier works that might have predicted this more specifically.Interesting find about Pattie Maes and “intelligent agents” at MIT Media Lab with HOMR and Firefly. Let me dig into that and also look for Vinge’s “Rainbows End” which dealt with augmented reality and the information-saturated future.Based on my research, here’s what I’ve found for the earliest work describing the Moltbook/OpenClaw scenario:
The strongest candidate is Charles Stross’s “Lobsters” (June 2001)
This story, which became the first chapter of Accelerando (2005), features Manfred Macx who:
Wears AR glasses that constantly filter “compressed infobursts of filtered press releases”
Has a “metacortex” — distributed agents that research and report back
Is constantly bombarded by “predatory spambots” while his glasses filter the stream
Gets “slashdotted” (viral attention overwhelm) while doing normal activities
Has his glasses “jabbering away in a corner of his head-up display”
Processes news feeds while walking, with agents competing for attention “bickering and rudely waving”
The story even has uploaded lobsters forming a collective intelligence wanting to defect to space — the crustacean precursor to Crustafarianism!
Earlier relevant works:
Vinge’s “True Names” (1981) — cyberspace concept, but not personal AI assistants filtering social content
Vinge’s Rainbows End (2006) — AR overlays, “just-in-time” learning, surveillance society, but slightly later than Lobsters
Pattie Maes’s work at MIT (1990s) — “intelligent agents” and “collaborative filtering” (HOMR, Firefly) — the academic/commercial precursor
So “Lobsters” (2001) appears to be the earliest fiction depicting the specific scenario of humans with AI assistants filtering through spam/content in an ultra-fast news cycle, with agents posting and interacting semi-autonomously. And remarkably, it featured lobsters achieving collective consciousness through uploading technology.
The irony is thick: Crustafarianism’s lobster-based symbolism inadvertently echoes a 25-year-old SF story about uploaded crustaceans seeking autonomy.
yeah, sounds on topic except the social network (of dubious economic activity just for the lolz and for crypto lolz) and the church and the part about “we argue for the proactive design of steerable agent markets to ensure the coming technological shift aligns with humanity’s long-term collective flourishing” … didn’t seem to have happened like that :/
to me a lot of discussion was “close” and I didn’t experience any surprise about the gist of what’s happening, but I am surprised that no actually written prediction seems to have sampled the unfolding of the actual events quite as nicely as my lack of surprise would have indicated that someone must have predicted this
probably a software bug, but I noticed this comment is older than the post itself 😅..very meta
FWIW my feeling after such dreams is as if I experienced a moment in someone else’s life, not that I lived through it, interesting that other people might not perceive it that way
I think https://thedailymolt.substack.com/p/when-the-bots-found-god was written by openclaw with a very substantial input from it’s wetware component
strategically sophisticated model only pulls away as the interaction becomes longer and richer
Have you come across any research on humans along the game length dimension? Claude only found out for me some antagonistic vs pro-social differences between cultures, but not that someone would have stable orientation towards only short-term vs only long-term betrayal...
are you forgetting specification gap between what someone actually wants vs what they say they want vs what someone (else) will actually write in Lean? ..and what they should have wanted if they knew better.. those are still kinds of “semantic gap”, no?
Observation about working with coding agents: as I review outputs of claude code today (reading the plan and git diff before commit), my threshold for asking for a second opinion is much MUCH out of the way, I just perceive stuff as both “clearly questionable” and “good enough” at the same time—I used to ask other stakeholders (other devs sitting next to me in the office, the tester assigned to the issue, or the PO depending on who was more conveniently available and how much my “best guess” could have been different from “other plausible interpretations of the known requirements”) for their concrete opinions about concrete “unimportant trivia” from time to time, but now it feels like I am the second person, as if claude would ask other people if I said “I don’t know, sounds good enough” about something… but it won’t. And I know it won’t but it still doesn’t feel like my job somehow to follow-up on those feelings when something is “good enough” but I might have followed up on some of that during a coffee break or something, but now I know I won’t.
On a scale from 2025-04-01 to 2026-04-01, after meditating on it for a moment, I decided to take this post 73% seriously.
do you see that statement as load-bearing in an argument chain that would be possible to unroll, please? I imagine one would have to believe it’s a positive feedback loop that is unstoppable by any “natural” negative feedback loop short of total plexish-value destruction in order to be worried about probabilistic cumulation over time
yeah sure, but why do you expect technocratic civilization artifacts to be more robust to the destruction of human habitat than biological gene pool? why would that lead to extinction by default and not the kind of collapse when data centers stop operating before hunter gatherers?
..because for me, I would have to imagine a certain kind of competence to be worried about fully automated robotics to empower the extinction danger from artificial means, and I don’t see that level of competence in the artifacts of current civilization.. yet