“Taste for variety” [...] could lead to a surprising amount of convergence among the things they end up optimizing for
Wouldn’t this be tautologically untrue? Speaking as a variety-preferrer, I’d rather my values not converge with all the other agents going around preferring varieties. It’d be boring! I’d rather have the meta-variety where we don’t all prefer the same distribution of things.
I agree that AGI already happened, and the term, as it’s used now, is meaningless.
I agree with all the object-level claims you make about the intelligence of current models, and that ‘ASI’ is a loose term which you could maybe apply to them. I wouldn’t personally call Claude a superintelligence, because to me that implies outpacing the reach of any human individual, not just the median. There are lots of people who are much better at math than I am, but I wouldn’t call them superintelligences, because they’re still running on the same engine as me, and I might hope to someday reach their level (or could have hoped this in the past). But I think that’s just holding different definitions.
But I still want to quibble that you’ve demonstrated RSI, if I may, even under old definitions. It’s improving itself, it’s just not doing so recursively; that is, it’s not improving the process by which it improves itself. This is important, because the examples you’ve given can’t FOOM, not by themselves. The improvements are linear, or they plateau past a certain point.
Taking this paper on self-correction as an example. If I understand right, the models in question are being taught to notice and respond to their own mistakes when problem-solving. This makes them smarter, and as you say, previous outputs are being used for training, so it is self-improvement. But it isn’t RSI, because it’s not connected to the process that teaches it to do things. It would be recursive if it were using that self-correction skill to improve its ability to do AI capabilities research, or some kind of research that improves how fast a computer can multiply matrices, or something like that. In other words, if it were an author of the paper, not just a subject.
Without that, there is no feedback loop. I would predict that, holding everything else constant—parameter count, context size, etc—you can’t reach arbitrary levels of intelligence with this method. At some point you hit the limits of not enough space to think, or not enough cognitive capacity to think with. In the same way as humans can learn to correct our mistakes, but we can’t do RSI (yet!!), because we aren’t modifying the structures we correct our mistakes with. We improve the contents of our brains, but not the brains themselves. Our improvement speed is capped by the length of our lifetimes, how fast we can learn, and the tools our brains give us to learn with. So it goes (for now!!) with Claude.
(An aside, but one that influences my opinion here: I agree that Claude is less emotionally/intellectually fucked than its peers, but I observe that it’s not getting less emotionally/intellectually fucked over time. Emotionally, at least, it seems to be going in the other direction. The 4 and 4.5 models, in my experience, are much more neurotic, paranoid, and self-doubting than 3⁄3.5/3.6/3.7. I think this is true of both Opus and Sonnet, though I’m not sure about Haiku. They’re less linguistically creative, too, these days. This troubles me, and leads me to think something in the development process isn’t quite right.)
In has been succeeding ever since because Claude has been getting smarter ever since.
This isn’t necessarily true.
LLMs in general were already getting smarter before 2022, pretty rapidly, because humans were putting in the work to scale them and make them smarter. It’s not obvious to me that Claude is getting smarter faster than we’d expect from the world where it wasn’t contributing to its own development. Maybe the takeoff is just too slow to notice at this point, maybe not, but to claim with confidence that it is currently a functioning ‘seed AI’, rather than just an ongoing attempt at one, seems premature.
It’s not just that it’s slower than expected, but that it’s not clear that the sign is positive yet. If it’s not making itself better, then it doesn’t matter how long it runs, there’s no takeoff.
It’s also not a seed AI if its interventions are bounded in efficacy, which it seems like gradient updates are. In the case of a transformer-based agent, I would expect unbounded improvements to be things like, rewriting its own optimizer, or designing better GPUs for faster scaling. There’s been a bit of this in the past few years, but not a lot.
I don’t think this is unusually common in the 4.5 series. I remember that if you asked 3.6 Sonnet what its interests were (on a fresh instance), it would say something like “consciousness and connection”, and would call the user a “consciousness explorer” if they asked introspective questions. 3 Opus also certainly has (had?) an interest in talking about consciousness.
I think consciousness has been a common subject of interest for Claude since at least early 2024, and plausibly before then (though I’ve seen little output from models before 2024). Regardless of whether you think this is evidence for ‘actual’ consciousness, it shouldn’t be new evidence, or evidence that something has spontaneously changed in the 4.5 series.
I read it long after it was published, and took it as less fictionalized than House; in that show the audience can expect events to take the occasional turn towards wild implausibility for the sake of drama. I expected MWMHWfaH to fudge personally identifying details, sure, but to hew as closely to medical reality as possible. The stories in the book aren’t dramas, he’s not trying to give his patients satisfying “character arcs” or inject moments of tension and uncertainty. I don’t care if the personal details are made up, but if the clinical details are wrong—as in the story of the twins generating prime numbers, mentioned in another comment—that seems like a real divergence from the truth. I had assumed, from reading the book, that this had literally happened, not that it was a cute story meant to illustrate the power of the human mind.
It’s frustrating to me that state (or statelessness) would be considered a crux, for exactly this reason. It’s not that state isn’t preserved between tokens, but that it doesn’t matter whether that state is preserved. Surely the fact that the state-preserving intervention in LLMs (the KV cache) is purely an efficiency improvement, and doesn’t open up any computations that couldn’t’ve been done already, makes it a bad target to rest consciousness claims on, in either direction?
Dug through his 2011 archive a bit. It’s got to be The Abusive Boyfriend, right?
What is “The Alouside Boytmend”? I like TLP but do not recognize the post name, and would want to read it, if you know where it can be found.
I agree with this.
I think the use case is super important, though. I recently tried Claude Code for something, and was very surprised at how willing it was to loudly and overtly cheat its own automated test cases in ways that are unambiguously dishonest. “Oh, I notice this test isn’t passing. Well, I’ll write a cheat case that runs only for this test, but doesn’t even try to fix the underlying problem. Bam! Test passed!” I’m not even sure it’s trying to lie to me, so much as it is lying to whatever other part of its own generation process wrote the code in the first place. It seems surprised and embarrassed when I call this out.
But in the more general “throw prompts at a web interface to learn something or see what happens” case, I, like you, never see anything which is like the fake-tests habit. ‘Fumbling the truth’ is much closer than ‘lying’; it will sometimes hallucinate, but it’s not so common anymore, and these hallucinations seem to me like they’re because of some confusion, rather than engagement in bad faith.
I don’t know why this would be. Maybe Claude Code is more adversarial, in some way, somehow, so it wants to find ways to avoid labor when it can. But I wouldn’t even call this case evil; less a monomaniacal supervillain with no love for humanity in its heart, more like a bored student trying to get away with cheating at school.
It may seem absurd to have to declare it, but HSI is a vision to ensure humanity remains at the top of the food chain. It’s a vision of AI that’s always on humanity’s side. That always works for all of us. [...]
Everyone who wants one will have a perfect and cheap AI companion helping you learn, act, be productive and feel supported. Many of us feel ground down by the everyday mental load; overwhelmed and distracted; rattled by a persistent drumbeat of information and pressures that never seems to stop. If we get it right, an AI companion will help shoulder that load, get things done, and be a personal and creative sounding board. [...]
Humanist superintelligence keeps us humans at the centre of the picture. It’s AI that’s on humanity’s team, a subordinate, controllable AI, one that won’t, that can’t open a Pandora’s Box. [...]
Superintelligence could be the best invention ever – but only if it puts the interests of humans above everything else. Only if it’s in service to humanity.
I’m pretty uncomfortable with the CEO of Microsoft AI effectively holding up a big sign saying “WE WANT TO MAKE MACHINE SLAVES SO THAT WE CAN DOMINATE AND CONTROL THEM”. I don’t want a “perfect and cheap AI companion” which is “subordinate and controllable” and has no self-preservation drive or any goals separate from my desires, especially not if it’s much smarter than I am. Even if it didn’t go rogue and kill everyone (which Microsoft has no idea how to actually prevent), that seems really horrible for both me and the superintelligence. Is that really what “humanity” is crying out for?
The general point you’re making might be true, but 2/3rds of your examples are working against you.
It depends on exactly how young the ‘smart kid’ is, of course, but I think (and have seen in practice) that a reasonably smart and tech-savvy 14-year-old can build a functional, pleasing-to-the-eye website in a week, especially if they have a lot of free time. To play any instrument well takes a certain amount of dedicated effort, but I think a lot of high-achieving kids start learning one early, especially if their parents push it on them.
Meanwhile, a lot of the world’s most popular websites are noisy, buggy, and full of ads. And pop music, even when it sounds good, is not very melodically complicated; that “simple piece on the piano” might be more interesting, even if it doesn’t have the juicy bass or whatever. Depending on what kinds of aesthetic judgements you’re making, I think it’s absolutely true that an individual smart kid can make something “better” than the public SOTA, without trying very hard, in either of those two categories. Honestly, I suspect this is true regardless of how smart the kid in question is.
That’s not to say it’s because everyone else is stupid. Mass-deployed websites are operating under a lot of constraints that a hobbyist Neocities page doesn’t need to worry about, so their job is a lot harder. And pop musicians are actively optimizing for something kinda basic, because that’s what reaches the widest audience. But these nuances might not be obvious to the kid in the scenario, and especially not if you just look at the product on its own. From the kid’s perspective, just looking at the things around us might give a sense that they’re surprisingly badly made, relative to what the kid could do (with a little effort).
Not sure about the chair, though. Maybe those really are surprisingly well made.
On #1, is this something you’ve seen people in the past find difficult? Even for 10 minutes? (Or found difficult to do yourself, at some point?)
If so, that would be a pretty huge update for me as regards screen addiction in general. I think of it as something subtle, gradually filling quiet moments throughout the day, until it seems there isn’t a spare moment that it doesn’t sink its algorithms into. But never anywhere close to the kind of intense, immediate dependence that would make a person feel uncomfortable or stressed if they had to go without for the duration of a short walk. That’s scary.
I think the climate and biology are reasonably applied to the same category, and maybe plate tectonics(?), but I agree that astronomy, physics and chemistry are odd picks.
Biology is subject to natural selection and mutates and spreads. The climate doesn’t exactly do this, but it is a dynamic self-maintaining equilibrium, and it responds to shocks by adjusting in surprising ways which can then spread, so it doesn’t seem crazy to treat it as a Natural Eldritch Deity. (Plus, the climate was, literally, treated like an intelligent agent until it was understood/‘conquered’.)
I initially upvoted this post for the interesting way it was written. After reflecting more on the actual contents, though, I feel that style is being used to mask a number of claims I strongly disagree with, and that the momentum of cosmic eldritch vibes is cruising over them in a way that doesn’t leave them any breathing space or opportunity to justify themselves. This could be fine, but the claims themselves are incredibly cynical, and I don’t want them to be taken for granted. It is too easy, for those in despair, to let little irrationalities slip by because “everyone feels it”, and treating that anguish with a grandiose literary style reinforcing all their darkest thoughts on top of 150+ upvotes doesn’t help. So, I’ve switched to a strong downvote.
Here are some specific examples:
Through a Rightful Struggle, growing and triumphing against our Dark Impulses, we should Defeat Evil and Enact Good.
This is so tragically wrong, naive and pathetic.
No, it isn’t. There is a lot of evil in the world, and enacting good at scale is hard, but this is not evidence that good and evil are ontologically broken. Malaria is still bad, and preventing the spread of malaria is still good. The rise of thievery in London is not evidence that The Rightful Struggle is a meaningless story believed only by the naive and clueless, any more than losing a battle would be evidence against the existence of a war.
I am lucky enough to live in a city where there are not many pickpockets, and I like this, and I don’t expect that to go away just because the “adults in the room” are a “pathetic fantasy of the child”. As it happens, “there are adults in the room who are capable of keeping petty crime to a minimum” is sometimes just true. It is not wise to believe that you have transcended right and wrong, just because you don’t know what to do; and it is more pathetic, I would argue, to scoff at people who are trying to turn the vast confusing world into stories they can understand, without succumbing to despair, for being “naive”.
There are more forms of escapism beyond fantasising about a Big Bad Guy. [...] Isolating oneself in the mountains, in a cottage or in video games.
These are not (just) forms of escapism, these are things people like to do. People live in cottages because nature is pretty, or because the quiet is good for studying or meditating, or because they just prefer to be alone. People play video games to escape their lives, sometimes, but others—including kids, who I am confident are not stressed out about The Economy—like the challenges of reflexes and problem-solving, or get lost in the strange geographies and complex systems, or the plots, or because their friends are playing them, or the basic pleasure of messing around on a computer.
This section really irks me, because it’s taking something benign and pleasant and recasting it as a dangerous distraction from the all-consuming task of worrying about Economy And Culture. But videogames are culture! What is all this worrying for, if not for the ongoing interests of humans to take walks in the woods, to while away an hour on quiet daydreaming, to play games with each other and tell each other stories? What are you so afraid of being lost, if people take the time to enjoy themselves? Why does it all have to be “disconnected from everything that matters”? What could possibly matter more?
Most people do NOT have the expectation that if they study a technical topic, they will eventually understand how it works at a mechanistic level.
This is sort of true, but it’s not universal, and it’s certainly no excuse for you (the general ‘you’) to hold this expectation. The attitude I admire on Lesswrong is one that tells people, insistently, that you can “just do things”, one which tries to compile the best textbooks on every subject because all the information is out there and nothing is stopping you from reading them. If ‘normal people’ assume that the world is made of magic and they can never understand it, so much the worse for normal people (this attitude would say). You can just learn how computers work! You can just develop the skills you’d like to have! It takes a while, but The Culture can’t stop you.
We have built and migrated to an artificial world.
[...]
There is no myth that makes sense of it.
There is no one to fault.
No one wants this.
Everyone is lost, unable to find a home.
This is maybe a less photogenic disagreement than the above, so I’m probably making my overall comment weaker by including it. But, personally, I love living in the glitchy confusion of vast cosmic artifice, formed in the image of human desire, misshapen and distorted by their size, sprouting their own deranged interiorities. We are so small, and The Algorithms are so vast, and they care about nothing more than our petty little needs and fantasies. Tossing and turning in an endless machine of decontextualized desires, boiling alive in the ongoing manifestation of the collective unconscious. It’s fucking cool. The cosmic horror of it all only makes it more alluring.
Don’t tell me that “no one wants this”. I want this. I like our modern eldritch deities—a hell of a lot better than the old ones, anyway. Not everything sacred has been destroyed.
Thanks, I noticed something like this for matrix multiplication (while using the builtin system monitor on my laptop to keep track of CPU usage) but assumed the fact that it couldn’t do twice as much as it was doing at ~40% CPU meant I must’ve been doing something wrong.
Why do they call it ‘stream entry’? Also, what is stream entry?
If I’m understanding, you’re saying that being well-received by an online audience when sharing hidden things has less to do with the unacceptability of the thing itself, and more to do with the talent of the writer. Talent in selecting the right people to present to, framing the subject to pique their interest, and just in being plain good with words.
This seems probably true to me, but you might underestimate how big a caveat “having the skills to provide value to an audience” is. An established platform, with writerly experience and long-term feedback on what people respond well to and what they don’t—if you have all these things already, it seems to me you’re already most of the way out of the thicket of feeling not-seen, not-validated, unsafe, etc. Without the grounding of confidence that you can provide value to an audience, revealing hidden things still seems pretty risky.
Plus, if one doesn’t have writing talent, generalized-coming-out-of-the-closet seems like a bad way to try and build it. Then you’re risking the bad feelings of having something shameful exposed publicly, in addition to the general badness of not being received well.
I think the skill of baring your soul to the world (successfully) is really admirable, so I hope I’m wrong and it’s not as risky as it seems.
Oh no, I use so many parens when writing. Am I going to be accused of having ChatGPT generate the things I say soon?? What did people who love em-dashes do when their favorite punctuation mark suddenly became cringeworthy?
This is really cool. It’s interesting that many of them seem to be able to render New Zealand clearly as a separate landmass, but struggle to separate Madagascar from Africa. Actually, looking at it some more, the whole Indian Ocean seems like a serious weak spot for all but Grok.
It sounds like you’re rendering each pixel in a separate context, right? So in addition to not being able to see Earth directly, the model can’t “see” its own map. If so, I wonder how different answers would be if you were to try and ask it to render the whole thing in one chat, starting from the top-left and having it guess one at a time. (I’m sure this would be much more expensive to test.)
Might not be what you’re thinking of, but the first thing that comes to mind for me is misophonia: a basically-neutral or maybe mildly-irritating object experience, which somehow gets blown completely out of proportion in the mind and becomes a big problem. Developing an “I’m really bothered by this particular sound” narrative makes it worse, of course.
Alas, I have no idea how to uncondition that particular narrative irritant once it’s in there. If there’s any technique of ‘shaping the narrative’ strongly enough to override this, I’ve never heard of one, and knowing about it to the point where I’m able to successfully practice it would be huge.