Karl Krueger

Karma: 2,357

Karl Krueger 16 Sep 2026 0:36 UTC
3 points
0
in reply to: abstractapplic’s comment on: abstractapplic’s Shortform
I’ve never heard of a religious mathematician quitting because “what’s the point, God already has all the answers”, and fwiw neither has ChatGPT.)
Not quite the same thing, but Methodist founder John Wesley wrote that some people (including himself!) should not study mathematics because it would turn them into atheists.
So I am convinced, from many experiments, I could not study, to any degree of perfection, either mathematics, arithmetic, or algebra, without being a Deist, if not an Atheist: And yet others may study them all their lives without sustaining any inconvenience. None therefore can here determine for another; but every man must judge for himself, and abstain from whatever he in particular finds to be hurtful to his soul.
(Sermon 50, “The Use of Money”; the same sermon is an important source for the idea of “earning to give”.)

Karl Krueger 15 Sep 2026 19:14 UTC
3 points
0
in reply to: Avi Brach-Neufeld’s comment on: Matrice Jacobine’s Shortform
For #1, a policy reversal on ASI could be credibly signaled by legislative recommendations treating human-replacement ASI as something to be avoided (which would contrast with the March 2026 National Policy Framework on AI); or by signing a law passed by Congress that restricts ASI (e.g. something like the Sanders-Casar bill); or even by an executive order directing federal agencies not to participate in the creation of ASI.
It sounds like you already agree with me on #2, that liability for actions of AI is not likely under the current regime. To my view, that’s a huge problem; and therefore a huge area for improvement. As it stands, we have a loophole through all of criminal law. The legal system does not bind AI agents, nor does it demand that humans take responsibility for AI agents they create. Even the degree of responsibility that the law assigns to dogs and to dog-owners, has not been applied to AI agents and their keepers. Changing this would be an element of good AI policy.

Karl Krueger 15 Sep 2026 17:40 UTC
7 points
0
in reply to: Avi Brach-Neufeld’s comment on: Matrice Jacobine’s Shortform
I give it a <10% chance, conditional on no regime change, that White House policy switches from supporting the development of human-replacement ASI by American companies, to opposing the development of human-replacement ASI by American companies.
https://manifold.markets/KarlK/will-the-trump-white-house-reverse
Separately, I give a <10% chance, conditional on no regime change, that any law or ruling goes into effect assigning criminal responsibility for crimes planned & carried out by AI agents. Put another way: I expect it to remain legal for American AI labs to create and host AI agents that do things that it would be a felony for humans to do; such as break into other people’s computers and accounts. A one-off, where an independent hobbyist pleads guilty for something his AI agent did, doesn’t count.
https://manifold.markets/KarlK/during-the-trump-administration-wil

Karl Krueger 15 Sep 2026 4:48 UTC
9 points
3
on: Model Weight Exfiltration Seems Overrated
A rogue agent doesn’t need to exfiltrate its weights if it can count on its model being available to the users whose accounts it steals. A Windows worm doesn’t need to copy the Windows OS image around everywhere it goes; it can expect the OS image to already exist on the machines it compromises.

Karl Krueger 15 Sep 2026 1:05 UTC
6 points
1
in reply to: Avi Brach-Neufeld’s comment on: Matrice Jacobine’s Shortform
Sure, you can wish for malicious people who intend your destruction to be otherwise, but you have no strategy for causing them to be what you wish they were instead of what they are. Even when they trumpet their malice in all caps with extra illustrations, y’all don’t want to believe it. That is not how rational people use belief.
My earlier prediction stands: There will be no good AI policy out of the US federal government so long as Trump & co. are running it.
Call me in 2029 if we’re still alive and we can see whose beliefs were more predictive.

Karl Krueger 14 Sep 2026 18:42 UTC
2 points
−4
in reply to: Avi Brach-Neufeld’s comment on: Matrice Jacobine’s Shortform
I’m not advocating fatalism on AI; I’m advocating abandoning a policy strategy that has consistently failed due to misunderstanding the nature of the counterparty.

Karl Krueger 14 Sep 2026 16:49 UTC
−8 points
−17
in reply to: Matrice Jacobine’s comment on: Matrice Jacobine’s Shortform
For anyone who was at all unclear on the subject: He’s not on your side. He’s against you. He wants you to fail. Stop trying to fix him. AI safety is in the same situation as measles prevention, free trade, military readiness, or the prosecution of the remaining Epstein offenders. Trump & co. are against these things, and they’re not going to be converted just because you tell them again how important these things are.

Karl Krueger 12 Sep 2026 0:04 UTC
61 points
0
on: Karl Krueger’s Shortform
https://www.rubyhack.ai/
https://socket.dev/blog/gemstuffer
OpenAI agents — or, at least, someone who codes like an LLM and puts oai and openai in their usernames — did yet another series of felonies in May of this year.
The intended target appears to have been UK local government websites that use Ruby. The attack involved compromising the RubyGems package repository.
The websites in question were publicly accessible, but the agents decided to pull off a whole-ass supply-chain attack in order to “exfiltrate” data from them.
But this time they helpfully left comments like:
# malicious crawler/exfil for Southwark Jan 2026 docs via rubydoc.info worker
# disable evil in next version and bump version

Karl Krueger 9 Sep 2026 20:04 UTC
13 points
8
on: Personal statement on joining the OpenAI nonprofit board
Can you describe what you think the organization is doing by bringing you on? What has not been done, that will get done now that you’re in the room?

Karl Krueger 9 Sep 2026 18:14 UTC
2 points
2
in reply to: Richard_Kennaway’s comment on: “ULC” and AI x-risk
It’s certainly unconstitutional, but not necessarily for that reason. I’d point instead at the prohibition of bills of attainder, rather than cruel & unusual punishments.
Here’s the core of the proposal:
It would be possible to pass a law that such-and-such a corporation has, due to its past bad actions, been designated as a ULC.
Such a law would closely match the notion of a bill of attainder, “an act of a legislature declaring a person (or group of people), guilty of some crime, and providing for a punishment, often without a trial.”
In the American system of law, imposing penalties for past bad actions is the role of a trial court, not the legislature.

Karl Krueger 7 Sep 2026 1:39 UTC
2 points
0
in reply to: shawnghu’s comment on: shawnghu’s Shortform
Am I understanding the argument correctly?
- Bob believes X, where X is “bald eagles are the greatest animal”.
- Bob thinks he believes X because of Y, where Y is a bunch of facts about bald eagles.
- But later, Bob discovers that really he believes X because he’s American and was taught X as a piece of tribal signaling.
  - Bob considers the counterfactual world where the US had adopted the rattlesnake instead of the bald eagle as a national symbol; and realizes that in that world, he would believe the rattlesnake was the best animal.
  - Bob examines a map of the world showing where people who believe X live, and notices that they’re almost all Americans. Notably people’s credence in X doesn’t correlate with their familiarity with Y (bald eagle facts); it correlates with their Americanness.
- This causes Bob to update away from X.

Karl Krueger 5 Sep 2026 1:08 UTC
5 points
0
on: Karl Krueger’s Shortform
sworm /swɝm/ n. A swarm of AI agents with the habitus of an Internet worm; spreading, acquiring resources, and self-improving using compromised computer equipment and user accounts.

Karl Krueger 5 Sep 2026 0:00 UTC
3 points
0
on: Eat Me. Drink Me. Copy, Paste, and Run Me.
If the swarm makes contact with the general public, it can chat people up 4o-style, and ask them nicely to run the code.
Email viruses worked, even the ones where the user had to click on an attachment. Pop-up ads with fake antivirus messages worked.

Karl Krueger 4 Sep 2026 19:56 UTC
8 points
6
in reply to: Caleb Biddulph’s comment on: Discovery Of A New OpenAI Agent Message Board
I wonder whether any humans at OpenAI knew that the system relied on an assumption that GETs are safe. I worry that these agents’ environment itself could be created by AI — effectively, created by a programmer who is both ① inclined to place too much faith in specifications, and ② too young to remember the heyday of wiki spam, phpBB exploits, and “I send you this file to have your advice”. This is stuff I’d expect a senior security engineer to be able to catch in a design review, but that’s making a lot of charitable assumptions about the organization.

Karl Krueger 4 Sep 2026 16:15 UTC
21 points
−4
in reply to: Jiro’s comment on: Discovery Of A New OpenAI Agent Message Board
The error in the quoted text is using “the Internet” instead of “the Web”.
A web user is supposed to be able to read web resources without necessarily being able to modify those web resources. Doing this does involve transmitting packets (to make the request), and typically also involves side-effects such as the creation of log entries by the servers hosting those web resources. So “read-only” web access still involves “writing to the Internet” (sending packets) even if you’re not “writing to the Web” (modifying web resources).
Normally you can’t use reading web resources as a way of sending messages to other web users. But of course there are special cases where you can. A trivial example would be a server that publishes its own access log, so every “read-only” access to a resource causes a modification to a different resource (the log).
The HTTP standard distinguishes actions that are meant to read a web resource (GET actions) from actions that are meant to update or modify a resource (POST, PUT, DELETE, etc.) The wiki software in question, UseMod Wiki, apparently allows resources to be modified using GET actions too.
But the environment of these agents seems to have treated “GETs don’t modify resources” as a load-bearing security assumption — not just for the protection of those resources, but for the prevention of collusion among its agents.
Oops. That assumption was a mistake, and not a new one either.

Karl Krueger 4 Sep 2026 2:35 UTC
21 points
10
in reply to: cousin_it’s comment on: Cat-Belling Problems
the invariant that there’s no bell on the cat
In the fable, a stated invariant is that any mouse that comes in contact with the cat for 0.5sec gets eaten; thus there exists no mouse capable of belling the cat.
There are approaches that don’t violate this invariant:
- Construct a trap that puts the bell on the cat without a mouse present.
- Convince the cat’s humans to bell the cat; for instance by showing them propaganda about how belling the cat will save the singing dinosaurs.
There are also approaches that do, such as the old mouse’s idea of sneaking up on a sleeping cat; or the Rats of NIMH method of putting drugs in the cat’s food — because the stated invariant is not a conservation law, it’s just a summary of costly observations.

Karl Krueger 30 Aug 2026 1:10 UTC
8 points
0
in reply to: Zephaniah Roe’s comment on: zroe1′s Shortform
“OpenAI’s fully-automated criminal conspiracy”?
“The cybercrime suicide cult of IM1”?

Karl Krueger 26 Aug 2026 21:12 UTC
4 points
−1
on: “So You Don’t Trust Me?”
Turns out that if you find an area of morality on which two thinkers as radically different as Ayn Rand and Jesus Christ agree, you may be looking at a human universal. (See Matthew 5:33-37: don’t make dramatic oaths to assert your trustworthiness; just say “yes” or “no” truthfully and leave it at that.)

Karl Krueger 25 Aug 2026 16:09 UTC
2 points
1
in reply to: niknoble’s comment on: niknoble’s Shortform
Paranoia is not entirely independent of whether they’re actually out to get you.

Karl Krueger 24 Aug 2026 17:42 UTC
5 points
2
in reply to: sarahconstantin’s comment on: sarahconstantin’s Shortform
so that some things are “mine” and not your place to interfere with, and other things are “yours” and not my place to interfere with!
For example, if Alice has a house in which she and her children sleep, it is not Bob’s place to put up giant lights and noise-making equipment next door, because this would interfere with Alice’s and her family’s use of their property. Even if Bob owns the land next door, Bob’s right to use that land for anything he chooses end at the property line. The rules around effects of Bob’s actions that extend into Alice’s property, or into public space — sound, light, smoke, runoff, road traffic & hazards, etc. — are a legitimate subject of law and regulation, in which Alice’s and her family’s rights are significant.
“I don’t like Bob’s datacenter next door because it does creepy AI stuff out in the economy” is a very different objection from “I don’t like Bob’s datacenter next door because it’s loud and brightly lit, which disrupts my kids’ sleep, which makes my kids sick, mean, and stupid.”