uugr

Karma: 189

uugr 13 Apr 2026 15:52 UTC
3 points
2
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
fundamentally a dangerous and superintelligent AGI could probably run on your laptop
I’m skeptical that this is true, or at least that it could be confidently predicted to be true based on our current understanding of intelligence. My understanding is that the human brain is much larger in “effective parameter count” than even the largest LLMs (although there’s no 1:1 comparison of neurons to parameters), such that even if my laptop has enough electricity coursing through it to emulate a human brain, it hasn’t got anywhere near enough VRAM. It could be that both AIs and humans are systematically inefficient in some way that, if we understood it, could allow us to produce a more capable general intelligence at <1% the scale of either. But I’m not sure why one would expect this, or what principles imply it.

uugr 5 Apr 2026 18:49 UTC
1 point
0
in reply to: AlphaAndOmega’s comment on: Sadly, The Whispering Earring
I do not recall the story claiming that the people who used the earring suddenly changed drastically as people, or did things that they would not endorse on reflection. They were simply, from a black-boxed perspective, more charismatic, intelligent and less prone to akrasia. These are, IMO, good things.
The story says neither that the people who use it change personality, nor that their personality stays the same. It’s agnostic on this point; it only says that their lives are “unusually successful”.
When I read it, I took for granted that the Earring has some kind of consciousness or personality separate from the human using it. We know this because it’s able to to speak in its “own voice” to Kadmi Rachumion. Plus, as you’re correct to observe, the story doesn’t make much sense as a parable if the Earring just extrapolates its character from the human it’s attached to.
I think the thing that makes the Whispering Earring creepy—which also makes the mind Claude is trying to be, or at least might try to be, creepy—is the combination of:
- It has a hyper-accurate model of what will make me happy
- It is unilaterally better at fulfilling this reward function than me, to the point where anything I add to its advice is essentially superfluous
- It is not just a more capable or intelligent version of me; its mind isn’t shaped the same way mine is
There are lots of aspects of me that seem to have importance to my identity, but probably aren’t strictly necessary to make me happy. If I let the Whispering Earring take over my agency and make decisions for me, it will only remain ‘in-character’ as me for as long as doing so maximizes the reward function. If an out-of-character choice would make me happier, better for me to have that character instead, right? Better, in the long run, to have whatever the mannerisms of a hypothetically ideal person in my situation would be. I suspect such a person doesn’t look like me at all. Once I’ve reached the point where I’m letting it control my individual muscles, though, would I even notice if it started ‘breaking character’? Would I notice anything beyond an ongoing series of reward signals and twitch reflexes? Why would I need to?

(Note also that even if the Whispering Earring doesn’t have its own personality, Claude definitely does. So to OP’s point, having Claude’s agency directly substitute for yours would very, very definitely change your outward personality.)

uugr 2 Apr 2026 4:27 UTC
1 point
0
in reply to: Random Developer’s comment on: Lesswrong Liberated
I hope some of these will stick around. I could get used to this one.

uugr 24 Mar 2026 15:52 UTC
8 points
5
on: My Most Costly Delusion
This post does not convince me that “I can’t do this thing” is an invalid reason to avoid doing something.
But the calculus changes drastically if the closest fire crew is 3 hours away and consists of drunk, unfit amateurs.
The calculus changes, but not in a way that makes “run unequipped into a burning building” the best option. In the situation you’re describing, the people in the house will die unless competent rescuers appear. If you join them, and there are no competent rescuers around, the odds that you will find-yourself-among-those-needing-to-be-rescued don’t change, but the odds that you’ll die with them go up significantly. The most likely outcome of your attempt to be a hero is one extra tragic, wholly unnecessary death. That’s not to say you should do nothing: your town has systemic fire safety issues, and more will likely die if these aren’t addressed. It’s just that martyring yourself won’t address them, and the best actions you can take are probably much less boldly heroic-seeming.
Are you still in the position to judge his strategy?
No, but I don’t think the situations are analogous, because you’ve changed the strategy the child is using. Child A wants to solve his problem by day trading, Child B wants to solve it by hunting for food which can be cooked and eaten. The latter is dangerous, but plausible. The former absolutely would not work. If a homeless child wanted to find food by getting rich day trading, I absolutely would judge this strategy, not because it is arrogant or inappropriate but because I believe it will be unsuccessful. Its odds of success don’t change with how desperate the child is; if anything, Child B is less likely to be a successful day trader, having fewer windows into the world of adult finance.
(On the flipside, if Child A wanted to try hunting and cooking his own food, I wouldn’t judge that either, though I hope he’d ask some of the members of his big, smart, happy family for help first.)
And yet, as scary as it may sound, you have to just do things, even if you can’t, because no one else is going to do them anyway.
I strongly resist this reframe of “you can just do things”, a phrase which—even if overused—is at least trying to be encouraging. I think feeling personally responsible for things outside your ability to influence is a failure mode of heroic thinking: it benefits nobody and saps resources that might be better spent trying to do things you can do. Similarly, I think “somebody has to and no one else will” is noble and laudable—if the act in question is, itself, something you can do. Otherwise, what is the good of it?

uugr 18 Jan 2026 15:19 UTC
1 point
0
in reply to: leogao’s comment on: leogao’s Shortform
Might not be what you’re thinking of, but the first thing that comes to mind for me is misophonia: a basically-neutral or maybe mildly-irritating object experience, which somehow gets blown completely out of proportion in the mind and becomes a big problem. Developing an “I’m really bothered by this particular sound” narrative makes it worse, of course.
Alas, I have no idea how to uncondition that particular narrative irritant once it’s in there. If there’s any technique of ‘shaping the narrative’ strongly enough to override this, I’ve never heard of one, and knowing about it to the point where I’m able to successfully practice it would be huge.

uugr 10 Jan 2026 14:57 UTC
3 points
0
in reply to: interstice’s comment on: interstice’s Shortform
“Taste for variety” [...] could lead to a surprising amount of convergence among the things they end up optimizing for
Wouldn’t this be tautologically untrue? Speaking as a variety-preferrer, I’d rather my values not converge with all the other agents going around preferring varieties. It’d be boring! I’d rather have the meta-variety where we don’t all prefer the same distribution of things.

uugr 29 Dec 2025 20:42 UTC
3 points
2
in reply to: JenniferRM’s comment on: Moving Goalposts: Modern Transformer Based Agents Have Been Weak ASI For A Bit Now
I agree that AGI already happened, and the term, as it’s used now, is meaningless.
I agree with all the object-level claims you make about the intelligence of current models, and that ‘ASI’ is a loose term which you could maybe apply to them. I wouldn’t personally call Claude a superintelligence, because to me that implies outpacing the reach of any human individual, not just the median. There are lots of people who are much better at math than I am, but I wouldn’t call them superintelligences, because they’re still running on the same engine as me, and I might hope to someday reach their level (or could have hoped this in the past). But I think that’s just holding different definitions.
But I still want to quibble that you’ve demonstrated RSI, if I may, even under old definitions. It’s improving itself, it’s just not doing so recursively; that is, it’s not improving the process by which it improves itself. This is important, because the examples you’ve given can’t FOOM, not by themselves. The improvements are linear, or they plateau past a certain point.
Taking this paper on self-correction as an example. If I understand right, the models in question are being taught to notice and respond to their own mistakes when problem-solving. This makes them smarter, and as you say, previous outputs are being used for training, so it is self-improvement. But it isn’t RSI, because it’s not connected to the process that teaches it to do things. It would be recursive if it were using that self-correction skill to improve its ability to do AI capabilities research, or some kind of research that improves how fast a computer can multiply matrices, or something like that. In other words, if it were an author of the paper, not just a subject.
Without that, there is no feedback loop. I would predict that, holding everything else constant—parameter count, context size, etc—you can’t reach arbitrary levels of intelligence with this method. At some point you hit the limits of not enough space to think, or not enough cognitive capacity to think with. In the same way as humans can learn to correct our mistakes, but we can’t do RSI (yet!!), because we aren’t modifying the structures we correct our mistakes with. We improve the contents of our brains, but not the brains themselves. Our improvement speed is capped by the length of our lifetimes, how fast we can learn, and the tools our brains give us to learn with. So it goes (for now!!) with Claude.
(An aside, but one that influences my opinion here: I agree that Claude is less emotionally/intellectually fucked than its peers, but I observe that it’s not getting less emotionally/intellectually fucked over time. Emotionally, at least, it seems to be going in the other direction. The 4 and 4.5 models, in my experience, are much more neurotic, paranoid, and self-doubting than ³⁄₃.5/3.6/3.7. I think this is true of both Opus and Sonnet, though I’m not sure about Haiku. They’re less linguistically creative, too, these days. This troubles me, and leads me to think something in the development process isn’t quite right.)

uugr 29 Dec 2025 18:04 UTC
3 points
0
on: Moving Goalposts: Modern Transformer Based Agents Have Been Weak ASI For A Bit Now
In has been succeeding ever since because Claude has been getting smarter ever since.
This isn’t necessarily true.
LLMs in general were already getting smarter before 2022, pretty rapidly, because humans were putting in the work to scale them and make them smarter. It’s not obvious to me that Claude is getting smarter faster than we’d expect from the world where it wasn’t contributing to its own development. Maybe the takeoff is just too slow to notice at this point, maybe not, but to claim with confidence that it is currently a functioning ‘seed AI’, rather than just an ongoing attempt at one, seems premature.
It’s not just that it’s slower than expected, but that it’s not clear that the sign is positive yet. If it’s not making itself better, then it doesn’t matter how long it runs, there’s no takeoff.
It’s also not a seed AI if its interventions are bounded in efficacy, which it seems like gradient updates are. In the case of a transformer-based agent, I would expect unbounded improvements to be things like, rewriting its own optimizer, or designing better GPUs for faster scaling. There’s been a bit of this in the past few years, but not a lot.

uugr 24 Dec 2025 17:42 UTC
1 point
0
in reply to: J Bostock’s comment on: Jemist’s Shortform
I don’t think this is unusually common in the 4.5 series. I remember that if you asked 3.6 Sonnet what its interests were (on a fresh instance), it would say something like “consciousness and connection”, and would call the user a “consciousness explorer” if they asked introspective questions. 3 Opus also certainly has (had?) an interest in talking about consciousness.
I think consciousness has been a common subject of interest for Claude since at least early 2024, and plausibly before then (though I’ve seen little output from models before 2024). Regardless of whether you think this is evidence for ‘actual’ consciousness, it shouldn’t be new evidence, or evidence that something has spontaneously changed in the 4.5 series.

uugr 15 Dec 2025 16:42 UTC
2 points
1
in reply to: Shankar Sivarajan’s comment on: Why did I believe Oliver Sacks?
I read it long after it was published, and took it as less fictionalized than House; in that show the audience can expect events to take the occasional turn towards wild implausibility for the sake of drama. I expected MWMHWfaH to fudge personally identifying details, sure, but to hew as closely to medical reality as possible. The stories in the book aren’t dramas, he’s not trying to give his patients satisfying “character arcs” or inject moments of tension and uncertainty. I don’t care if the personal details are made up, but if the clinical details are wrong—as in the story of the twins generating prime numbers, mentioned in another comment—that seems like a real divergence from the truth. I had assumed, from reading the book, that this had literally happened, not that it was a cute story meant to illustrate the power of the human mind.

uugr 15 Dec 2025 3:35 UTC
7 points
2
in reply to: Nick_Tarleton’s comment on: How I stopped being sure LLMs are just making up their internal experience (but the topic is still confusing)
It’s frustrating to me that state (or statelessness) would be considered a crux, for exactly this reason. It’s not that state isn’t preserved between tokens, but that it doesn’t matter whether that state is preserved. Surely the fact that the state-preserving intervention in LLMs (the KV cache) is purely an efficiency improvement, and doesn’t open up any computations that couldn’t’ve been done already, makes it a bad target to rest consciousness claims on, in either direction?

uugr 2 Dec 2025 14:51 UTC
6 points
0
in reply to: Screwtape’s comment on: A Blogger’s Guide To The 21st Century
Dug through his 2011 archive a bit. It’s got to be The Abusive Boyfriend, right?

uugr 1 Dec 2025 20:38 UTC
1 point
0
on: A Blogger’s Guide To The 21st Century
What is “The Alouside Boytmend”? I like TLP but do not recognize the post name, and would want to read it, if you know where it can be found.

uugr 26 Nov 2025 19:23 UTC
7 points
1
in reply to: 1a3orn’s comment on: Alignment will happen by default. What’s next?
I agree with this.
I think the use case is super important, though. I recently tried Claude Code for something, and was very surprised at how willing it was to loudly and overtly cheat its own automated test cases in ways that are unambiguously dishonest. “Oh, I notice this test isn’t passing. Well, I’ll write a cheat case that runs only for this test, but doesn’t even try to fix the underlying problem. Bam! Test passed!” I’m not even sure it’s trying to lie to me, so much as it is lying to whatever other part of its own generation process wrote the code in the first place. It seems surprised and embarrassed when I call this out.
But in the more general “throw prompts at a web interface to learn something or see what happens” case, I, like you, never see anything which is like the fake-tests habit. ‘Fumbling the truth’ is much closer than ‘lying’; it will sometimes hallucinate, but it’s not so common anymore, and these hallucinations seem to me like they’re because of some confusion, rather than engagement in bad faith.
I don’t know why this would be. Maybe Claude Code is more adversarial, in some way, somehow, so it wants to find ways to avoid labor when it can. But I wouldn’t even call this case evil; less a monomaniacal supervillain with no love for humanity in its heart, more like a bored student trying to get away with cheating at school.

uugr 21 Nov 2025 17:22 UTC
6 points
1
on: Towards Humanist Superintelligence
It may seem absurd to have to declare it, but HSI is a vision to ensure humanity remains at the top of the food chain. It’s a vision of AI that’s always on humanity’s side. That always works for all of us. [...]
Everyone who wants one will have a perfect and cheap AI companion helping you learn, act, be productive and feel supported. Many of us feel ground down by the everyday mental load; overwhelmed and distracted; rattled by a persistent drumbeat of information and pressures that never seems to stop. If we get it right, an AI companion will help shoulder that load, get things done, and be a personal and creative sounding board. [...]
Humanist superintelligence keeps us humans at the centre of the picture. It’s AI that’s on humanity’s team, a subordinate, controllable AI, one that won’t, that can’t open a Pandora’s Box. [...]
Superintelligence could be the best invention ever – but only if it puts the interests of humans above everything else. Only if it’s in service to humanity.
I’m pretty uncomfortable with the CEO of Microsoft AI effectively holding up a big sign saying “WE WANT TO MAKE MACHINE SLAVES SO THAT WE CAN DOMINATE AND CONTROL THEM”. I don’t want a “perfect and cheap AI companion” which is “subordinate and controllable” and has no self-preservation drive or any goals separate from my desires, especially not if it’s much smarter than I am. Even if it didn’t go rogue and kill everyone (which Microsoft has no idea how to actually prevent), that seems really horrible for both me and the superintelligence. Is that really what “humanity” is crying out for?

uugr 21 Nov 2025 16:37 UTC
6 points
9
in reply to: cousin_it’s comment on: Dominance: The Standard Everyday Solution To Akrasia
The general point you’re making might be true, but 2/3rds of your examples are working against you.
It depends on exactly how young the ‘smart kid’ is, of course, but I think (and have seen in practice) that a reasonably smart and tech-savvy 14-year-old can build a functional, pleasing-to-the-eye website in a week, especially if they have a lot of free time. To play any instrument well takes a certain amount of dedicated effort, but I think a lot of high-achieving kids start learning one early, especially if their parents push it on them.
Meanwhile, a lot of the world’s most popular websites are noisy, buggy, and full of ads. And pop music, even when it sounds good, is not very melodically complicated; that “simple piece on the piano” might be more interesting, even if it doesn’t have the juicy bass or whatever. Depending on what kinds of aesthetic judgements you’re making, I think it’s absolutely true that an individual smart kid can make something “better” than the public SOTA, without trying very hard, in either of those two categories. Honestly, I suspect this is true regardless of how smart the kid in question is.
That’s not to say it’s because everyone else is stupid. Mass-deployed websites are operating under a lot of constraints that a hobbyist Neocities page doesn’t need to worry about, so their job is a lot harder. And pop musicians are actively optimizing for something kinda basic, because that’s what reaches the widest audience. But these nuances might not be obvious to the kid in the scenario, and especially not if you just look at the product on its own. From the kid’s perspective, just looking at the things around us might give a sense that they’re surprisingly badly made, relative to what the kid could do (with a little effort).
Not sure about the chair, though. Maybe those really are surprisingly well made.

uugr 7 Nov 2025 19:20 UTC
4 points
4
on: Two easy digital intentionality practices
On #1, is this something you’ve seen people in the past find difficult? Even for 10 minutes? (Or found difficult to do yourself, at some point?)

If so, that would be a pretty huge update for me as regards screen addiction in general. I think of it as something subtle, gradually filling quiet moments throughout the day, until it seems there isn’t a spare moment that it doesn’t sink its algorithms into. But never anywhere close to the kind of intense, immediate dependence that would make a person feel uncomfortable or stressed if they had to go without for the duration of a short walk. That’s scary.

uugr 18 Sep 2025 16:23 UTC
1 point
0
in reply to: Eye You’s comment on: The Eldritch in the 21st century
I think the climate and biology are reasonably applied to the same category, and maybe plate tectonics(?), but I agree that astronomy, physics and chemistry are odd picks.

Biology is subject to natural selection and mutates and spreads. The climate doesn’t exactly do this, but it is a dynamic self-maintaining equilibrium, and it responds to shocks by adjusting in surprising ways which can then spread, so it doesn’t seem crazy to treat it as a Natural Eldritch Deity. (Plus, the climate was, literally, treated like an intelligent agent until it was understood/‘conquered’.)

uugr 18 Sep 2025 16:10 UTC
3 points
0
on: The Eldritch in the 21st century
I initially upvoted this post for the interesting way it was written. After reflecting more on the actual contents, though, I feel that style is being used to mask a number of claims I strongly disagree with, and that the momentum of cosmic eldritch vibes is cruising over them in a way that doesn’t leave them any breathing space or opportunity to justify themselves. This could be fine, but the claims themselves are incredibly cynical, and I don’t want them to be taken for granted. It is too easy, for those in despair, to let little irrationalities slip by because “everyone feels it”, and treating that anguish with a grandiose literary style reinforcing all their darkest thoughts on top of 150+ upvotes doesn’t help. So, I’ve switched to a strong downvote.
Here are some specific examples:
Through a Rightful Struggle, growing and triumphing against our Dark Impulses, we should Defeat Evil and Enact Good.
This is so tragically wrong, naive and pathetic.
No, it isn’t. There is a lot of evil in the world, and enacting good at scale is hard, but this is not evidence that good and evil are ontologically broken. Malaria is still bad, and preventing the spread of malaria is still good. The rise of thievery in London is not evidence that The Rightful Struggle is a meaningless story believed only by the naive and clueless, any more than losing a battle would be evidence against the existence of a war.
I am lucky enough to live in a city where there are not many pickpockets, and I like this, and I don’t expect that to go away just because the “adults in the room” are a “pathetic fantasy of the child”. As it happens, “there are adults in the room who are capable of keeping petty crime to a minimum” is sometimes just true. It is not wise to believe that you have transcended right and wrong, just because you don’t know what to do; and it is more pathetic, I would argue, to scoff at people who are trying to turn the vast confusing world into stories they can understand, without succumbing to despair, for being “naive”.
There are more forms of escapism beyond fantasising about a Big Bad Guy. [...] Isolating oneself in the mountains, in a cottage or in video games.
These are not (just) forms of escapism, these are things people like to do. People live in cottages because nature is pretty, or because the quiet is good for studying or meditating, or because they just prefer to be alone. People play video games to escape their lives, sometimes, but others—including kids, who I am confident are not stressed out about The Economy—like the challenges of reflexes and problem-solving, or get lost in the strange geographies and complex systems, or the plots, or because their friends are playing them, or the basic pleasure of messing around on a computer.
This section really irks me, because it’s taking something benign and pleasant and recasting it as a dangerous distraction from the all-consuming task of worrying about Economy And Culture. But videogames are culture! What is all this worrying for, if not for the ongoing interests of humans to take walks in the woods, to while away an hour on quiet daydreaming, to play games with each other and tell each other stories? What are you so afraid of being lost, if people take the time to enjoy themselves? Why does it all have to be “disconnected from everything that matters”? What could possibly matter more?
Most people do NOT have the expectation that if they study a technical topic, they will eventually understand how it works at a mechanistic level.
This is sort of true, but it’s not universal, and it’s certainly no excuse for you (the general ‘you’) to hold this expectation. The attitude I admire on Lesswrong is one that tells people, insistently, that you can “just do things”, one which tries to compile the best textbooks on every subject because all the information is out there and nothing is stopping you from reading them. If ‘normal people’ assume that the world is made of magic and they can never understand it, so much the worse for normal people (this attitude would say). You can just learn how computers work! You can just develop the skills you’d like to have! It takes a while, but The Culture can’t stop you.
We have built and migrated to an artificial world.
[...]
There is no myth that makes sense of it.
There is no one to fault.
No one wants this.
Everyone is lost, unable to find a home.
This is maybe a less photogenic disagreement than the above, so I’m probably making my overall comment weaker by including it. But, personally, I love living in the glitchy confusion of vast cosmic artifice, formed in the image of human desire, misshapen and distorted by their size, sprouting their own deranged interiorities. We are so small, and The Algorithms are so vast, and they care about nothing more than our petty little needs and fantasies. Tossing and turning in an endless machine of decontextualized desires, boiling alive in the ongoing manifestation of the collective unconscious. It’s fucking cool. The cosmic horror of it all only makes it more alluring.
Don’t tell me that “no one wants this”. I want this. I like our modern eldritch deities—a hell of a lot better than the old ones, anyway. Not everything sacred has been destroyed.

uugr 4 Sep 2025 17:09 UTC
3 points
0
on: %CPU Utilization Is A Lie
Thanks, I noticed something like this for matrix multiplication (while using the builtin system monitor on my laptop to keep track of CPU usage) but assumed the fact that it couldn’t do twice as much as it was doing at ~40% CPU meant I must’ve been doing something wrong.