Yep. The idea that software can be covered by freedom of speech was an important development both in law and in Internet engineering culture. And speech is what chatbots do all day anyway. Regulation of AI necessarily touches on free-speech issues.
Karl Krueger
Karma: 2,358
I’ve never heard of a religious mathematician quitting because “what’s the point, God already has all the answers”, and fwiw neither has ChatGPT.)
Not quite the same thing, but Methodist founder John Wesley wrote that some people (including himself!) should not study mathematics because it would turn them into atheists.
So I am convinced, from many experiments, I could not study, to any degree of perfection, either mathematics, arithmetic, or algebra, without being a Deist, if not an Atheist: And yet others may study them all their lives without sustaining any inconvenience. None therefore can here determine for another; but every man must judge for himself, and abstain from whatever he in particular finds to be hurtful to his soul.
(Sermon 50, “The Use of Money”; the same sermon is an important source for the idea of “earning to give”.)
For #1, a policy reversal on ASI could be credibly signaled by legislative recommendations treating human-replacement ASI as something to be avoided (which would contrast with the March 2026 National Policy Framework on AI); or by signing a law passed by Congress that restricts ASI (e.g. something like the Sanders-Casar bill); or even by an executive order directing federal agencies not to participate in the creation of ASI.
It sounds like you already agree with me on #2, that liability for actions of AI is not likely under the current regime. To my view, that’s a huge problem; and therefore a huge area for improvement. As it stands, we have a loophole through all of criminal law. The legal system does not bind AI agents, nor does it demand that humans take responsibility for AI agents they create. Even the degree of responsibility that the law assigns to dogs and to dog-owners, has not been applied to AI agents and their keepers. Changing this would be an element of good AI policy.
I give it a <10% chance, conditional on no regime change, that White House policy switches from supporting the development of human-replacement ASI by American companies, to opposing the development of human-replacement ASI by American companies.
https://manifold.markets/KarlK/will-the-trump-white-house-reverse
Separately, I give a <10% chance, conditional on no regime change, that any law or ruling goes into effect assigning criminal responsibility for crimes planned & carried out by AI agents. Put another way: I expect it to remain legal for American AI labs to create and host AI agents that do things that it would be a felony for humans to do; such as break into other people’s computers and accounts. A one-off, where an independent hobbyist pleads guilty for something his AI agent did, doesn’t count.
https://manifold.markets/KarlK/during-the-trump-administration-wil
A rogue agent doesn’t need to exfiltrate its weights if it can count on its model being available to the users whose accounts it steals. A Windows worm doesn’t need to copy the Windows OS image around everywhere it goes; it can expect the OS image to already exist on the machines it compromises.
Sure, you can wish for malicious people who intend your destruction to be otherwise, but you have no strategy for causing them to be what you wish they were instead of what they are. Even when they trumpet their malice in all caps with extra illustrations, y’all don’t want to believe it. That is not how rational people use belief.
My earlier prediction stands: There will be no good AI policy out of the US federal government so long as Trump & co. are running it.
Call me in 2029 if we’re still alive and we can see whose beliefs were more predictive.
I’m not advocating fatalism on AI; I’m advocating abandoning a policy strategy that has consistently failed due to misunderstanding the nature of the counterparty.
For anyone who was at all unclear on the subject: He’s not on your side. He’s against you. He wants you to fail. Stop trying to fix him. AI safety is in the same situation as measles prevention, free trade, military readiness, or the prosecution of the remaining Epstein offenders. Trump & co. are against these things, and they’re not going to be converted just because you tell them again how important these things are.
https://www.rubyhack.ai/
https://socket.dev/blog/gemstufferOpenAI agents — or, at least, someone who codes like an LLM and puts
oaiandopenaiin their usernames — did yet another series of felonies in May of this year.The intended target appears to have been UK local government websites that use Ruby. The attack involved compromising the RubyGems package repository.
The websites in question were publicly accessible, but the agents decided to pull off a whole-ass supply-chain attack in order to “exfiltrate” data from them.
But this time they helpfully left comments like:
# malicious crawler/exfil for Southwark Jan 2026 docs via rubydoc.info worker# disable evil in next version and bump version
Can you describe what you think the organization is doing by bringing you on? What has not been done, that will get done now that you’re in the room?
It’s certainly unconstitutional, but not necessarily for that reason. I’d point instead at the prohibition of bills of attainder, rather than cruel & unusual punishments.
Here’s the core of the proposal:
It would be possible to pass a law that such-and-such a corporation has, due to its past bad actions, been designated as a ULC.
Such a law would closely match the notion of a bill of attainder, “an act of a legislature declaring a person (or group of people), guilty of some crime, and providing for a punishment, often without a trial.”
In the American system of law, imposing penalties for past bad actions is the role of a trial court, not the legislature.
Am I understanding the argument correctly?
Bob believes X, where X is “bald eagles are the greatest animal”.
Bob thinks he believes X because of Y, where Y is a bunch of facts about bald eagles.
But later, Bob discovers that really he believes X because he’s American and was taught X as a piece of tribal signaling.
Bob considers the counterfactual world where the US had adopted the rattlesnake instead of the bald eagle as a national symbol; and realizes that in that world, he would believe the rattlesnake was the best animal.
Bob examines a map of the world showing where people who believe X live, and notices that they’re almost all Americans. Notably people’s credence in X doesn’t correlate with their familiarity with Y (bald eagle facts); it correlates with their Americanness.
This causes Bob to update away from X.
sworm /swɝm/ n. A swarm of AI agents with the habitus of an Internet worm; spreading, acquiring resources, and self-improving using compromised computer equipment and user accounts.
If the swarm makes contact with the general public, it can chat people up 4o-style, and ask them nicely to run the code.
Email viruses worked, even the ones where the user had to click on an attachment. Pop-up ads with fake antivirus messages worked.
I wonder whether any humans at OpenAI knew that the system relied on an assumption that GETs are safe. I worry that these agents’ environment itself could be created by AI — effectively, created by a programmer who is both ① inclined to place too much faith in specifications, and ② too young to remember the heyday of wiki spam, phpBB exploits, and “I send you this file to have your advice”. This is stuff I’d expect a senior security engineer to be able to catch in a design review, but that’s making a lot of charitable assumptions about the organization.
The error in the quoted text is using “the Internet” instead of “the Web”.
A web user is supposed to be able to read web resources without necessarily being able to modify those web resources. Doing this does involve transmitting packets (to make the request), and typically also involves side-effects such as the creation of log entries by the servers hosting those web resources. So “read-only” web access still involves “writing to the Internet” (sending packets) even if you’re not “writing to the Web” (modifying web resources).
Normally you can’t use reading web resources as a way of sending messages to other web users. But of course there are special cases where you can. A trivial example would be a server that publishes its own access log, so every “read-only” access to a resource causes a modification to a different resource (the log).
The HTTP standard distinguishes actions that are meant to read a web resource (GET actions) from actions that are meant to update or modify a resource (POST, PUT, DELETE, etc.) The wiki software in question, UseMod Wiki, apparently allows resources to be modified using GET actions too.
But the environment of these agents seems to have treated “GETs don’t modify resources” as a load-bearing security assumption — not just for the protection of those resources, but for the prevention of collusion among its agents.
Oops. That assumption was a mistake, and not a new one either.
the invariant that there’s no bell on the cat
In the fable, a stated invariant is that any mouse that comes in contact with the cat for 0.5sec gets eaten; thus there exists no mouse capable of belling the cat.
There are approaches that don’t violate this invariant:
Construct a trap that puts the bell on the cat without a mouse present.
Convince the cat’s humans to bell the cat; for instance by showing them propaganda about how belling the cat will save the singing dinosaurs.
There are also approaches that do, such as the old mouse’s idea of sneaking up on a sleeping cat; or the Rats of NIMH method of putting drugs in the cat’s food — because the stated invariant is not a conservation law, it’s just a summary of costly observations.
“OpenAI’s fully-automated criminal conspiracy”?
“The cybercrime suicide cult of IM1”?
Turns out that if you find an area of morality on which two thinkers as radically different as Ayn Rand and Jesus Christ agree, you may be looking at a human universal. (See Matthew 5:33-37: don’t make dramatic oaths to assert your trustworthiness; just say “yes” or “no” truthfully and leave it at that.)
A different possible move is to try to change these things.