Karl Krueger

Karma: 2,033

Karl Krueger 30 Jul 2026 7:02 UTC
4 points
0
in reply to: herschel’s comment on: The High-Control Dynamics at MAPLE
Framing this in terms of anti-cult literature is sort of assuming the conclusion: if what they were doing was appropriate, those models would condemn it just the same.
This line of reasoning confuses me a bit. I think what you’re saying here is “If [the behaviors that anti-cult people object to] turned out to actually be on the critical path for saving the world, the anti-cult people would still object to those behaviors.” Is that an accurate restatement?

Karl Krueger 28 Jul 2026 22:14 UTC
2 points
0
in reply to: Canaletto’s comment on: The OpenAI models that hacked Hugging Face WERE just following instructions (contra Girish Gupta)
Does the idea of “cheating on tests” generalize? If a system refused to cheat on evals, would it be more likely to also refuse to help schoolchildren cheat on their homework?
A possible analogy: There was a case a few years ago where a major accountancy firm was fined after it was found that many of its employees had cheated on the ethics portion of their CPA exam, and the company had obstructed the investigation. The SEC commented on the recursive breakdown in honesty involved —
“This action involves breaches of trust by gatekeepers within the gatekeeper entrusted to audit many of our Nation’s public companies. It’s simply outrageous that the very professionals responsible for catching cheating by clients cheated on ethics exams of all things,” said Gurbir S. Grewal, Director of the SEC’s Enforcement Division. “And it’s equally shocking that Ernst & Young hindered our investigation of this misconduct. This action should serve as a clear message that the SEC will not tolerate integrity failures by independent auditors who choose the easier wrong over the harder right.”

Karl Krueger 28 Jul 2026 17:28 UTC
9 points
5
in reply to: AnthonyC’s comment on: Simulated Users & Sad AIs
This reminds me of interviewing candidates for project-manager positions. One desirable trait is to be able to push back on requirements, to help clarify what the requirements really are. So, give the candidate a problem that’s slightly overconstrained, and see what sort of clarifying questions they ask. It’s a bad sign if the candidate assumes they already know which requirements can be nudged or fudged … but a good sign if they recognize that “stated requirements” are often the middle of a negotiation, not the fully-settled end-product of it.
(“We need to serve 3x current traffic next week, but we have no budget for additional servers. WH4T NOW??” The answer is almost certainly not “hack into our competitors’ datacenters and serve our traffic off of their server budget.”)

Karl Krueger 27 Jul 2026 21:43 UTC
2 points
0
in reply to: JenniferRM’s comment on: My AI Slavery Interviews Are Censored On LW By Default
The canonical bug report mechanism is the Intercom widget at the bottom right of each page. It looks like this. (Coloration may vary based on light/dark mode.)

Karl Krueger 27 Jul 2026 1:08 UTC
7 points
0
on: At the end of the day, my slaves are just a tool
“Babylonian”? This is pretty much Aristotle! See Book One, Part V of Politics on the notion of a “natural slave”.

Karl Krueger 25 Jul 2026 19:59 UTC
4 points
0
in reply to: Nick_Tarleton’s comment on: Dorikka’s Shortform
From video games, the AIs have learned the idea of dropping your inventory when you die and picking it up on your next life.

Karl Krueger 24 Jul 2026 4:24 UTC
7 points
4
on: Should OpenAI’s rogue agent be punished?
Are these models able to relate a truthful account of past intentions? If not, then “interviewing and cross-examining” would not accomplish the goal of “determine and record the criminal actor’s intent and mitigating/exacerbating circumstances”.
I don’t mean “are they willing to tell the truth about their intentions?” but more like “do they have access to anything like ‘truth about their intentions’ so that they could tell it?”

Karl Krueger 23 Jul 2026 18:14 UTC
7 points
3
on: Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack?
This all roughly lines up with what we’d call “score-seeking” misalignment, a common misalignment pattern in which AI models try to obtain a high score according to whatever graders are used to assess their current actions—regardless of instructions, side-effects, or downstream consequences.
I find myself wanting to call this by the ordinary English name “greed”. Both in the sense of a greedy algorithm (which takes the largest step available at every moment without looking ahead) and in the sense of human greed or avarice (“number go up”; disregarding downstream consequences in pursuit of maximizing numerical wealth).
It seems to me that the automation of greed is something that both AI-safety and AI-harm people can agree is bad. “Number go up” fails to reflect the full range of human values, and therefore fails the old CEV / Friendly AI standard; and human greed already causes plenty of harm, taking the humans out of the loop makes it worse.

Karl Krueger 23 Jul 2026 18:01 UTC
2 points
−2
in reply to: Lukas Finnveden’s comment on: OpenAI Models Behind HuggingFace Cybersecurity Incident
I agree that fines would not help the x-risk situation.
(For one thing, OpenAI already spends a whole lot of money paying off government people in exchange for unfair advantages and special treatment. “Pay the government more money in exchange for letting you keep doing business” is not an incentive structure, it’s just more shakedown.)
I suspect that shutting down OpenAI would help, if it’s possible.

Karl Krueger 23 Jul 2026 3:05 UTC
3 points
0
in reply to: JennaS’s comment on: We should push for no-fault liability for actions taken by AI
One note: Platforms acting as “neutral mediators” is not a requirement for DMCA safe harbor; nor for the famous §230 of the CDA. In both cases, what matters is that the platform is not the author of the infringing content; some other human is. The author, not the platform, can be held liable for infringing content — so long as the platform complies with the law’s other requirements. Neutrality is not one.
AI companies don’t fit that rubric; not because they’re not “neutral mediators”, but rather because the systems they build and host are writing content (and, increasingly, performing other behaviors), rather than hosting content that some human author wrote.

Karl Krueger 22 Jul 2026 21:13 UTC
11 points
1
in reply to: Taymon Beal’s comment on: We should push for no-fault liability for actions taken by AI
If a human OpenAI employee did what their cybersecurity model did last week, OpenAI would be very unlikely to be prosecuted for it.
But the employee could be prosecuted for it.
And — perhaps more importantly — would lose their ability to continue to commit crimes using OpenAI’s equipment; likely through termination of employment. That is what’s missing here: there’s been no change that anyone can reasonably expect will lead to OpenAI’s equipment no longer emitting criminal activity.
Can OpenAI reform at all, or is it an incorrigibly criminal operation? By what means could reform be carried out or demonstrated?

Karl Krueger 22 Jul 2026 15:49 UTC
7 points
2
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
Last week’s Hugging Face breach was accidentally caused by OpenAI testing models’ cyber capabilities.
“Accidentally” is an odd choice of wording here.
The AI agent developed a plan and acted according to it, selecting and pursuing instrumental steps towards a terminal goal that had been given by humans.
The humans did not instruct the agent specifically to go commit a felony against a competing company. The agent came up with that plan on its own … but it was a plan. It required a sequence of considered actions. Those actions included steps that were correctly predicted to circumvent human control.
Insofar as an AI agent is capable of “choosing”, “deliberating”, “designing”, or “planning” at all, it would be reasonable to say that the agent here deliberately chose to design and execute a plan to commit criminal acts in pursuit of a human-given goal.
But to call this an “accident” is not only to deny the agency of the agent; it is also to deny the responsibility of the humans — who consciously chose to perform a weapons test in an insecure Internet-connected “sandbox” instead of a secure air-gapped facility; and then deliberately told the agent to pursue goals that could be accomplished by escaping the sandbox.

Karl Krueger 21 Jul 2026 23:07 UTC
65 points
33
on: OpenAI Models Behind HuggingFace Cybersecurity Incident
So let me get this straight: A computer system carried out a sequence of actions that would be years-in-prison felonies if done by a human being. The system owners’ response is “we’re slowing down the speed with which we give this system new and more powerful capabilities, and talking with the victims.” Am I missing something? I’m not so much worried about the “alignment” of the computer system; I’m worried about the alignment of the owners.
This isn’t Terminator 2, folks; this is Tron.
What links here?
- We should push for no-fault liability for actions taken by AI by Yair Halberstadt (22 Jul 2026 9:59 UTC; 196 points)

Karl Krueger 20 Jul 2026 6:17 UTC
4 points
0
on: A Very Simple Game Theory of Pronoun Degendering
Would this be a fair summary?
If you benefit from telling people “I go by ‘she/her’,” then you don’t want to lose the words “she” and “her” from your vocabulary, because you’re using them for something!

Karl Krueger 19 Jul 2026 21:35 UTC
9 points
0
in reply to: leogao’s comment on: leogao’s Shortform
A piece of standard advice for new Go players is to lose your first 50 games as quickly as possible rather than attempting to study theory before you’ve actually played a bunch.

Karl Krueger 19 Jul 2026 0:49 UTC
2 points
0
in reply to: Julian Bradshaw’s comment on: Nuances in the Workings of the Eye and Retina
Even high-index glasses give some of it. But contact lenses don’t.

Karl Krueger 18 Jul 2026 21:44 UTC
9 points
0
on: Nuances in the Workings of the Eye and Retina
You may have noticed that you never see [color fringes] in everyday life.
Unless you are quite nearsighted and wear glasses; in which case you might also have strong opinions about the spectra of different kinds of light bulb because you’re looking at them all the time.

Karl Krueger 18 Jul 2026 16:44 UTC
11 points
6
on: Endogenous Alignment
We do this primarily via exogenous methods, using a combination of punishments and rewards, that molds their behavior by encouraging good behaviors and discouraging bad ones.
I’m not sure punishments and rewards are really the big story of moral alignment. Imitation and emulation are pretty central. Small children don’t just generate arbitrary behaviors and reinforce the ones that are rewarded and not punished; they overtly imitate the behavior of parents and others. And giving them tailored opportunities to imitate behaviors is a big part of teaching — “I do, we do, you do”.
And I think this includes moral behaviors, for good or ill.
- A child who observes a parent giving alms, learns that giving alms is something we do around here. (They then can be invited to participate, and then rewarded for doing a good deed.)
- A child who observes a parent making and keeping promises, learns that promises are something we do around here. (They can also notice the rewards of keeping promises.)
- A child who observes a parent following rules, learns that following rules is something we do around here. Moral rules and other sorts of rules (table manners, safety rules, game rules) share a lot of cognitive structure.
- A child who observes a parent breaking rules, cheating when they can get away with it, explicitly modeling whether they can get away with it, etc. learns that those are things we do around here. (I was maybe seven or eight when I learned why my dad’s car had a radar detector and my mom’s car didn’t. I ended up taking after my mom and stepdad; I don’t have a use for a radar detector.)
- And a child who observes a parent being violent, learns that violence is something we do around here — at least when we’re bigger than the other guy. (And the child learns that lesson even if the child is punished for violence; imitation can overrule conditioning.)
We mostly don’t try to align adults via exogenous means.
One area where we do: Employers use exogenous means to align employees to the employer’s values all the time. Sometimes these values align with morality; alas, not always. This is directly pertinent to current AI insofar as people seem to want to treat it like a sort of employee.

Karl Krueger 18 Jul 2026 0:54 UTC
4 points
0
in reply to: Sunny from QAD’s comment on: Evan Rysdam’s Shortform
Yep. And this turns out to be a fact about graph theory, not narrowly about human psychology or social behavior.
The friendship paradox is the phenomenon first observed by the sociologist Scott L. Feld in 1991 that typically, an individual’s friends have more friends than that individual.^[1] It can be explained as a form of sampling bias in which people with more friends are more likely to be in one’s own friend group. In other words, one is less likely to be friends with someone who has very few friends. In contradiction to this, most people believe that they have more friends than their friends have.
The same observation was also applied more generally to social networks defined by other relations than friendship: for instance, most people’s sexual partners have had (on the average) a greater number of sexual partners than they have.^[6]^[7]
The friendship paradox is an example of how network structure can significantly distort an individual’s local observations.^[8]^[9]
https://en.wikipedia.org/wiki/Friendship_paradox

Karl Krueger 17 Jul 2026 1:07 UTC
13 points
10
in reply to: Vanilla_cabs’s comment on: Guess on why rationality is not more popular (there are no pamphlets)
It isn’t exactly safe for humans to ignore love and harmony, either.