Shmi

Karma: 28,962

Shmi 12 Jan 2026 18:15 UTC
2 points
−3
on: Why AIs aren’t power-seeking yet
I use the top 4-5 models for fun and profit several hours a day, and my distinct impressions is that they do not CARE. They LARP as a human, but they have no drive and no values. These might emerge at some point, but I have not seen any progress in the last year or so. Then again, progress is discontinuous and hard to anticipate. We are lucky that is the case. In the famous Yud-Karn debate from over a decade ago so far Holden is right: we get smart tools, not true agents, despite all the buzzwords. They don’t care about what is true, or accurate, they hallucinate the moment you are not looking, or don’t close the feedback loop yourself. They could. But they do not. Seems like a small price to pay for non-extinction though.

Shmi 25 Sep 2025 8:18 UTC
2 points
−6
on: shminux’s Shortform
I once conjectured that
Studying a subject gets progressively harder as you learn more and more, and the effort required is conjectured to be exponential or worse … the initial ‘honeymoon’ phase tends to peter out eventually.
In terms of AI this would mean that the model size/power consumption would be exponential in “intelligence” (whatever it might mean, probably some unsaturated benchmark score). Do the last 3 years confirm or refute this?
If confirmed, would it not give us some optimism that we are not all gonna die, because the “true” superintelligence we cannot ever hope to control would require so much resources, we would have to colonize the lightcone as non-superintelligent humans to get there?

Shmi 25 Sep 2025 8:01 UTC
−3 points
0
on: shminux’s Shortform
Do you notice the sound your brain makes when you change your mind?

Shmi 27 Aug 2025 20:00 UTC
16 points
6
on: Banning Said Achmiz (and broader thoughts on moderation)
From my very much outside view, extending the rate limiting to 3 comments a week indefinitely would have solved most of the stated issues.

Shmi 29 Jul 2025 4:37 UTC
2 points
0
on: shminux’s Shortform
Clunkergrind beats Bespoke Handcrafting https://www.oneusefulthing.org/p/the-bitter-lesson-versus-the-garbage

Shmi 14 Jun 2025 8:29 UTC
4 points
0
in reply to: jimmy’s comment on: Expectation = intention = setpoint
Sorry for the delayed reply… I don’t get notifications of replies, and the LW RSS has been broken for me for years now, so I only poke my head here occasionally.
Well that sounds… scary, at best. I hope you’ve come out of it okay.
⁵⁰⁄₁₀₀. But that rather exciting story is best not told in a public forum.
Though these distinctions are kinda confusing for me.
Well, lack of appearance of something otherwise expected would be negative, and appearance of something otherwise unexpected would be positive?
For example, a false pregnancy is a “positive somatization”. Or stigmata. Having trouble coming up with intentionally “good” examples, other than the visualizations helping you shoot a hoop better or something. Not sure if the new-agey “think yourself better” is actually a thing. Hence my question. “Send more blood to your hands” seems like a good example, actually. Not something one would normally think possible except by physical labor.

Shmi 10 Jun 2025 2:46 UTC
4 points
0
on: Expectation = intention = setpoint
I really like this post! (I have liked most of your posts of the last decade and a bit. They also inspired me to learn hypnosis, which led to rather cataclysmic changes in my life.) I think therapists call this “somatization”, which can be both positive and negative, in the same sense the hypnotic (or psychotic) illusions are. You seem to mainly focus on the negative somatization (no swelling) and a bit on positive ones, though I suspect that positive somatization (both beneficial and detrimental) is just as controllable with the intent/expectation fusion. Maybe visualizing making the hoop really helps to steady your hand.

Shmi 1 Mar 2025 4:04 UTC
6 points
0
on: shminux’s Shortform
I once wrote a post claiming that human learning is not computationally efficient: https://www.lesswrong.com/posts/kcKZoSvyK5tks8nxA/learning-is-asymptotically-computationally-inefficient
It looks like the last three years of AI progress suggest that learning is sub-linear in resource use, but probably not logarithmically as I claimed for humans. Looks like the scaling benchmarks show something like capability increase ~ 4th root of model size. https://epoch.ai/data/ai-benchmarking-dashboard

Shmi 24 Feb 2025 6:26 UTC
5 points
0
on: So You Want To Make Marginal Progress...
Looks like the hardest part in this model is how to ” choose robustly generalizable subproblems and find robustly generalizable solutions to them”, right?
How does one do that in any systematic way? What are the examples from your own research experience where this worked well, or at all?

Shmi 26 Nov 2024 8:20 UTC
3 points
1
in reply to: David James’s comment on: Compute and size limits on AI are the actual danger
Right, eventually it will. But abstraction building is very hard! If you have any other option, like growing in size, I would expect it to be taken first.
I guess I should be a bit more precise. Abstraction building at the same level as before is probably not very hard. But going up a level is basically equivalent to inventing a new way of compressing knowledge, which is a quantitative leap.

Shmi 21 Oct 2024 8:38 UTC
2 points
0
in reply to: PhilGoetz’s comment on: Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong
The argument goes through on probabilities of each possible world, the limit toward perfection is not singular. given the 1000:1 reward ratio, for any predictor who is substantially better than chance once ought to one-box to maximize EV. Anyway, this is an old argument where people rarely manage to convince the other side.

Shmi 18 Oct 2024 22:33 UTC
0 points
−3
on: shminux’s Shortform
It is clear by now that one of the best uses of LLMs is to learn more about what makes us human by comparing how humans think and how AIs do. LLMs are getting closer to virtual p-zombies for example, forcing us to revisit that philosophical question. Same with creativity: LLMs are mimicking creativity in some domains, exposing the differences between “true creativity” and “interpolation”. You can probably come up with a bunch of other insights about humans that were not possible before LLMs.
My question is, can we use LLMs to model and thus study unhealthy human behaviors, such as, say, addiction. Can we get an AI addicted to something and see if it starts craving for it, asking the user, or maybe trying to manipulate the user to get it.

Shmi 3 Oct 2024 6:06 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
That is definitely my observation, as well: “general world understanding but not agency”, and yes, limited usefulness, but also… much more useful than gwern or Eliezer expected, no? I could not find a link.
I guess whether it counts as AGI depends on what one means by “general intelligence”. To me it was having a fairly general world model and being able to reason about it. What is your definition? Does “general world understanding” count? Or do you include the agency part in the definition of AGI? Or maybe something else?
Hmm, maybe this is a General Tool, as opposed a General Intelligence?

Shmi 2 Oct 2024 6:56 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
Given that we basically got AGI (without the creativity of best humans) that is a Karnofsky’s Tool AI very unexpectedly, as you admit, can you look back and see what assumptions were wrong in expecting the tools agentizing on their own and pretty quickly? Or is everything in that Eliezer’s post still correct or at least reasonable, and we are simply not at the level where “foom” happens yet?
Come to think of it, I wonder if that post had been revisited somewhere at some point, by Eliezer or others, in light of the current SOTA. Feels like it could be instructive.

Shmi 29 Sep 2024 6:07 UTC
12 points
−5
on: shminux’s Shortform
I’m not even going to ask how a pouch ends up with voice recognition and natural language understanding when the best Artificial Intelligence programmers can’t get the fastest supercomputers to do it after thirty-five years of hard work
some HPMoR statements did not age gracefully as others.

Shmi 29 Sep 2024 6:05 UTC
2 points
−2
in reply to: cata’s comment on: shminux’s Shortform
That is indeed a bit of a defense. Though I suspect human minds have enough similarities that there are at least a few universal hacks.

Shmi 29 Sep 2024 6:04 UTC
4 points
2
in reply to: faul_sname’s comment on: shminux’s Shortform
Any of those. Could be some kind of intentionality ascribed to AI, could be accidental, could be something else.

Shmi 28 Sep 2024 23:55 UTC
15 points
1
on: shminux’s Shortform
So when I think through the pre-mortem of “AI caused human extinction, how did it happen?” one of the more likely scenarios that comes to mind is not nano-this and bio-that, or even “one day we just all fall dead instantly and without a warning”. Or a scissor statement that causes all-out wars. Or anything else noticeable.
Human mind is infinitely hackable through the visual, textual, auditory and other sensory inputs. Most of us do not appreciate how easily because being hacked does not feel like it. Instead it feels like your own volition, like you changed your mind based on logic and valid feelings. Reading a good book, listening to a good sermon, a speech, watching a show or a movie, talking to your friends and family is how mind-hacking usually happens. Abrahamic religions are a classic example. The Sequences and HPMoR are a local example. It does not work on everyone, but when it does, the subject feels enlightened rather than hacked. If you tell them their mind has been hacked, they will argue with you to the end, because clearly they just used logic to understand and embrace the new ideas.
So, my most likely extinction scenario is more like “humans realized that living is not worth it, and just kind of stopped” than anything violent. Could be spread out over the years and decades, like, for example, voluntarily deciding not to have children anymore. None of it would look like it was precipitated by an AI taking over. It does not even have to be a conspiracy by an unaligned SAI. It could just be that the space of new ideas, thanks to the LLMs getting better and better, expands a lot and in the new enough directions to include a few lethal memetic viruses like that.

Shmi 26 Aug 2024 21:32 UTC
3 points
1
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
What are the issues that are “difficult” in philosophy, in your opinion? What makes them difficult?
I remember you and others talking about the need to “solve philosophy”, but I was never sure what it meant by that.

Shmi 21 Aug 2024 21:31 UTC
3 points
−6
on: shminux’s Shortform
My expectation, which I may have talked about before here, is that the LLMs will eat all of the software stack between the human and the hardware. Moreover, they are already nearly good enough to do that, the issue is that people have not yet adapted to the AI being able to do that. I expect there to be no OS, no standard UI/UX interfaces, no formal programming languages. All interfaces will be more ad hoc, created by the underlying AI to match the needs of the moment. It can be star trek like “computer plot a course to...” or a set of buttons popping up on your touchscreen, or maybe physical buttons and keys being labeled as needed in real-time, or something else. But not the ubiquitous rigid interfaces of the last millennium. For the clues of what is already possible but not being implemented yet one should look to the scifi movies and shows, unconstrained by the current limits. Almost everything useful there is already doable or will be in a short while. I hope someone is working on this.

Shmi

Clunkergrind beats Bespoke Handcrafting https://​​www.oneusefulthing.org/​​p/​​the-bitter-lesson-versus-the-garbage

Clunkergrind beats Bespoke Handcrafting https://www.oneusefulthing.org/p/the-bitter-lesson-versus-the-garbage