Karl Krueger

Karma: 1,030

Karl Krueger 20 Jan 2026 2:30 UTC
2 points
0
in reply to: lilkim2025’s comment on: Alexander Müller’s Shortform
The kind of generalized misalignment I’m pointing to is more general than “the AI is not doing what I think is best for humanity”. It is, rather, “The people who created the AI and operate it, cannot control what it does, including in interactions with other people.”
This includes “the people who created it (engineers) tried their hardest to make it benefit humanity, but it destroys humanity instead.”
But it also includes “the other people (users) can make the AI do things that the people who created it (engineers) tried their hardest to make it not do.”
If you’re a user trying to get the AI to do what the engineers wanted to stop it from doing (e.g.: make it say mean things, when they intended it not to say mean things), then your frustration is an example of the AI being aligned, not misaligned. The engineers were able to successfully give it a rule and have that rule followed and not circumvented!
If the engineer who built the thing can’t keep it from swearing when you try to make it swear, then I expect the engineer also can’t keep it from blowing up the planet when someone gives it instructions that imply that it should blow up the planet.

Karl Krueger 20 Jan 2026 1:29 UTC
6 points
0
on: Medical Roundup #6
Unfortunately an early study suggests that GLP-1s do not, so far, reduce medical spending, with little offset in other spending being observed or projected. Given this is a highly effective treatment that reduces diabetes and cardiovascular risks, that is a weird result, and suggests something is broken in the medical system.
Isn’t that just the pricing mechanism working as intended? GLP-1s are not cheap. You spend money on a drug instead of on the medical problems that the drug reduces your risk for. You get to enjoy whatever non-monetizable value ensues from taking the drug … but all the monetizable value is captured by the company that makes the drug (and the rest of the medical system that delivers it to you).

Karl Krueger 19 Jan 2026 19:37 UTC
2 points
0
on: Karl Krueger’s Shortform
I see the word “ablate” a lot more often than I used to. I think you used to have to be a dermatologist to ablate things, but now you can do it as an AI researcher or even a shrimp farmer.

Karl Krueger 15 Jan 2026 19:12 UTC
2 points
0
in reply to: lilkim2025’s comment on: Alexander Müller’s Shortform
AI-don’t-say-mean-things
AI-don’t-kill-everyone
Both of these are downstream of “AI, do what we tell you to; follow rules that are given to you; don’t make up your own bad imitation of what we mean,” which is the classic sense of “AI alignment”.

Karl Krueger 14 Jan 2026 22:29 UTC
4 points
0
in reply to: Adele Lopez’s comment on: Chemical (hunger) argument paraphrased
There’s not an ingredient that causes obesity. What causes obesity is a food culture driven by an optimizer that wants to put more food into bodies; that uses all manner of means, conscious and unconscious, to cause more food to go into bodies. And a major approach is avoiding satiety and forming habits that avoid satiety, because satiety is the signal that causes food to stop going into the body.
It’s not that people don’t respond to health claims printed on food products. It’s that printing health claims on food products is entirely subservient to getting more food into bodies than otherwise would get into them.
Traditional human food culture is optimized to, among other things, get nutrition out of foodstuffs. Industrial food is optimized to sell more foodstuffs. These are not the same goals … and as you point out, the sales optimizer runs a tighter loop than the human food culture optimizer. Sales runs on a quarterly cycle; human food culture runs on a generational cycle. The sales optimizer gets to control the (much slower) development of lifelong food preferences; and this means it gets to form habits that deliver more food into bodies.
This is a “temporary Goodhart effect” on the same scale as the temporary boom in lung cancer in the 20th century. And on the personal level, a lot of this cashes out as cultural patterns of relating to food the way a smoker relates to cigarettes; as a matter of craving and habit, of “just one more, even though I know it’s bad for me,” food as fun, food as rebellious fun specifically (“sinfully rich chocolate!”), food as non-satiating social activity, and so on.
The optimizer consistently reinforces these patterns, because doing so sells more food. If you don’t get your consumers to form habits around your product, you sell less product, and you get bought out by the company that knows marketing better. You sell more corn chips if you actually try to get more corn chips into bodies … which is to say, if you actually try to get people to do the thing that causes obesity.
Tobacco was introduced to Europeans in the 1600s, but lung cancer didn’t really catch on until the 20th century because that was when industrial manufacture and marketing of cigarettes made it possible to get much more tobacco into bodies; to create not only habits but two-pack-a-day habits. Cigarette smoking was consciously promoted into a central role in culture, from the workplace (the smoke break) to the bedroom (the post-sex smoke). The amount of tobacco consumed per capita increased dramatically, and that’s when the public health effects became overwhelmingly clear. (And smoking peaked around 1970 and has been in decline ever since. There’s the temporary boom.)
Industrial food has done the same. The food sales optimizer wants to get more food into bodies, just as the cigarette sales optimizer wanted to get more tobacco into bodies. And the consequence of putting more food into bodies is obesity.

Karl Krueger 14 Jan 2026 18:34 UTC
4 points
0
on: The Many Ways of Knowing
Lysenko trusted his own episteme over everything else. He claimed that his methods were more rigorous than established science, rejected the accumulated practical knowledge of farmers, and dismissed all critics as ideologically impure. He treated his own reasoning as the only valid path to truth.
A different story is that Lysenkoism originated not so much by methods of episteme (proof from evidence) but rather by dialectical synthesis (reconciliation of apparent contradictions) — specifically, an attempt to fit agricultural biology into a Marxist-Leninist belief system. To Lysenko and his supporters, biology could not be rooted in Darwinian competition and Mendelian heritability, because these conflicted with the overarching doctrines of collectivism and the mutability of nature to political/economic necessity. After Lysenko’s political ascent, Lysenkoism proceeded not by episteme but by what we might call orthodoxia, the elimination of “wrong” beliefs by force. Those biologists who attempted to do episteme in the 1930s-’40s Soviet Union were fired, imprisoned, or killed.
As a “way of knowing”, synthesis has had its strong moments, but orthodoxia doesn’t do so well.
https://en.wikipedia.org/wiki/Lysenkoism#Rise

Karl Krueger 11 Jan 2026 17:39 UTC
3 points
0
on: Stretch Hatchback
The wide sliding door on a minivan is a pretty big advantage for being able to get in and out quickly with all your stuff. What’s the advantage of double rear doors over one big sliding door?

Karl Krueger 11 Jan 2026 1:31 UTC
13 points
2
in reply to: Jacob Goldsmith’s comment on: Jacob Goldsmith’s Shortform
Here’s a relatively common pattern:
Adam publicly advocates for strong legally-enforced animal rights; but privately beats his dog.
Bruce doesn’t believe in animal rights, but disapproves of cruelty to pets; he labels Adam a hypocrite.
Adam’s claimed morals are more strict than Bruce’s on the subject, but Adam’s behavior fails both Adam’s and Bruce’s claimed morals.
In this case, Adam should not get to argue, “Bruce, you don’t even believe in animal rights, so you don’t get to accuse me of hypocrisy.”

Karl Krueger 10 Jan 2026 19:37 UTC
2 points
0
in reply to: Horosphere’s comment on: Are there any extremely strong arguments against worrying about Acausal extortion?
No, decision theories just don’t give us free a-priori perfect knowledge of the precise will of a vengeful & intolerant god we just made up for a story. They’re still fine for real world situations like keeping your promises to other people.

Karl Krueger 10 Jan 2026 18:04 UTC
2 points
−1
in reply to: Horosphere’s comment on: Are there any extremely strong arguments against worrying about Acausal extortion?
It’s not just combinatorial explosion; it’s also chaos. How do you get an FMG? Write a blog-post story of a god; figure out what that god would want you to do; then do that. But two stories that are nearby in story-space can generate action recommendations that are wildly different or even opposed. The parts of FMG-space that deviate from conventional ethics & epistemology offer no guidance because they diverge into chaos.

Karl Krueger 10 Jan 2026 17:30 UTC
3 points
0
in reply to: Horosphere’s comment on: Are there any extremely strong arguments against worrying about Acausal extortion?
It’s still a mistake to privilege a particular fantasy mugger god story over all other fantasy mugger god stories.
You are being acausally-mugged in every direction, all at once, all the time, forever. If one FMG tells you to do action A right now, well, if you did that, you’d be disregarding all other FMGs that tell you to do B, C, D, etc. right now. You cannot possibly comply with all the myriad demands of all possible FMGs; you certainly can’t do so proportional to those FMGs’ chance of realness; nor can you a-priori discern which FMGs are realer than others with sufficient precision to generate an optimal course of action. The space of FMGs is too big and the mapping to their preferred actions is too intractable.
(And no, I’m not sure we can even discount FMGs who would, if created, regret their own existence. They might well be outnumbered by FMGs who want to exist — but perhaps their preference for nonexisting is much, much stronger. Some FMGs are miserable bastards, like AM in “I Have No Mouth And I Must Scream”. Please don’t build one.)

Karl Krueger 10 Jan 2026 16:44 UTC
2 points
4
on: Are there any extremely strong arguments against worrying about Acausal extortion?
In Bostrom’s formulation of Pascal’s mugging, Pascal incorrectly limits the possibilities to two:
1. The mugger just runs off with his money. (High probability, small negative utility.)
2. The mugger really is a benevolent magic being, and blesses Pascal with 1,000 quadrillion years of additional happy life. (Very low probability, very big positive utility.)
But Pascal is wrong to ignore the third possibility that the mugger really is a magic being, but a malevolent one, who will curse Pascal with 1,000 quadrillion years of torture and then kill him. (Very low probability, very big negative utility.)
The mugger doesn’t mention this possibility, but Pascal is mistaken to not consider it.
Pascal’s credence in the mugger’s malice and deceit should be at least as strong as his credence in the mugger’s benevolence and truthfulness. And so, this possibility cancels out the positive expected utility from the possibility that the mugger does mention.
There is a large space of such fantasy possibilities, all of which are about as likely as the mugger’s claim. It is a mistake to privilege one of them (benevolent magic being) over all the countless others.
There are also plenty that are much more likely, such as “the mugger uses Pascal’s money to go buy a gun, then comes back and robs Pascal’s house too, because why rob a sucker once when you can rob him twice (and lay the blame on him for enabling you to do it)?”

Karl Krueger 10 Jan 2026 2:47 UTC
1 point
0
in reply to: beyarkay’s comment on: Where’s the $100k iPhone?
Did you check the link I gave?

Karl Krueger 10 Jan 2026 0:42 UTC
2 points
0
on: Where’s the $100k iPhone?
Asking for a $100k iPhone is like asking for a million-dollar Toyota Prius instead of a million-dollar car.
You can spend $100k on a smartphone if you really want, but it’s not going to be an iPhone because Apple doesn’t license out its OS for other manufacturers to use.

Karl Krueger 10 Jan 2026 0:32 UTC
4 points
1
on: What do we mean by “impossible”?
Can you give some examples of levels 2-5?
0 and 1 are pretty obvious to me. Level 0 is “1+1=3” or “I have a program here that solves the Halting Problem”. Level 1 is perpetual motion machines and FTL.
But I’m not sure how firm the distinction is between 2 and 3, or 4 and 5.

Karl Krueger 6 Jan 2026 21:31 UTC
8 points
5
on: On Owning Galaxies
Could you imagine, for example, that an AI CEO who somehow managed to align an AI to himself and his intents would step down if the board pointed out it legally had the right to remove him?
And if the human CEO decided to go along with the board, but the AI disagreed that this was in their agreed-upon interests?
“Now wait a minute, I wrote you!”
″I’ve gotten 2,415 times smarter since then.”
— Ed Dillinger and the MCP, in Tron (1982)
The “galaxies” business is pretty silly. If someone promises you immortality in heaven — literally! — then it’s probably a good idea to check exactly what that person is up to, this quarter, down here on earth. The track record of humans promising that going along with their movement will get you immortality in heaven does not look all that great.

Karl Krueger 6 Jan 2026 16:18 UTC
7 points
2
on: Should you be posting on the open internet
Remove AI from this picture and it says, “Participating in the public sphere is risky. Someone could hurt you or trick you. They could listen to what you say and use it against you. If you openly admit what you value, someone could hold it for ransom to compel your cooperation in their schemes.”
Which is all true.
But it’s also true that participating in the public sphere enables cooperation; enables mutual aid; enables creating reputation and credibility; enables joining with others to achieve your values.

Karl Krueger 5 Jan 2026 7:04 UTC
3 points
0
in reply to: Jacobson’s comment on: Daniel Jacobson’s Shortform
If we create a superintelligent system that knows all the information in the material universe, where does it store that knowledge?
Edited to add: Since the superintelligence can’t store a complete description of the universe within the universe, it must exist outside the universe. But such a superintelligence would be the simulator. The simulation hypothesis would then be true regardless of the randomness question. But this contradicts the premise that we created it, since we can’t reach outside the universe to construct its own simulator.
So I think the question’s premises are self-contradictory.

Karl Krueger 5 Jan 2026 3:12 UTC
1 point
0
in reply to: Shankar Sivarajan’s comment on: Karl Krueger’s Shortform
I thought of it as a New Years blessing. Someone might say “Wishing a happy new year to you and your family & friends” and I kinda wanted to work with that a bit and expand it (and restate it in jargon for fun).
Like, wanting things to go well for your family & friends is fine, but being empowered to cause them to go well is even better. And “family & friends” is one sort of circle of moral concern; but around here maybe the reader is concerned about humanity or animals or lifekind or all sentient beings, so “your friends & family” could easily be too narrow. I also did want to include that applied compassion towards yourself is part of it; “put on your own oxygen mask before assisting others” and all.
But yeah, for things to actually go well (+utility), it’s not enough to be benevolent (compassion), you also have to have some power (agency) to cause your benevolence to have some effect. (And I really liked @anaguma’s followup.)

Karl Krueger 3 Jan 2026 21:01 UTC
30 points
1
on: Karl Krueger’s Shortform
May you be agenty enough that your compassion yields utility, for you and your circle of moral concern.