Tom Butterweich

Karma: 103

Tom Butterweich 22 Sep 2026 8:22 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Tom Butterweich’s Shortform
Thank you for the explanation!
I agree that the social setting has a strong influence on the internalised norms. Georg Jellinek called that the “normative power of the factual” and the racist school is a great example.^[1]
I think the conditions for cooperation between agents are both in the agents themselves and in the way how they are coupled. Current agents seem good enough to me. Current arrangements, like: [ naive RL + >10^4 identical agents + access to a pervious web ] seem doomed to create problems. There is no trust, no iterated observation, no permanence, no real learning.
An analogy: Take the nicest person on earth. Get them hooked on heroin. Create 10k identical copies of them and tell them they can get a fix if they bring something that’s unattainable in any legal way...
1. ^
  Another one: a friend recently shared that in his family, only his Mum knows that he’s gay—even though it should be pretty obvious to anyone at this point?! Anyway, he then continued to state that he used to be passionately anti-gay, just like his family and old friend group, until he realised he was into guys. Now he is very, fabulously queer, goes to pride while is little brother marches with “anti-pride”. But my friend shrugs it off because, eh, he gets it, he was like that once, you know. Took him (just) two months to snap out of it, good on him, I would say.

Tom Butterweich 21 Sep 2026 17:34 UTC
3 points
0
in reply to: Steven Byrnes’s comment on: Tom Butterweich’s Shortform
I’ve (re-)read your texts, but I still think I have a point that I didn’t get across. I’ll try to compact our positions somewhat so we can boil down what we disagree upon. First some agreement though:
That seems way too strong. [...] (And it would be very easy indeed to get away with torturing and killing a companion animal.) That is some nonzero kind of morality.
Agreed, that came out wrong. I’m not that pessimistic, but more pessimistic than the 99%/1% you mention somewhere. It’s a blurry line anyway.
Internalization is a genetic mechanism and one that’s worth understanding; I talk about what controls the virtues that people internalize in §4 here.
I don’t get it. Do you mean that the ability to internalise is a genetic/innate trait, i.e. LLMs don’t possess it (yet)? Is this somewhat parallel to your “LLMs won’t scale to AGI”? To clarify—I used the term in the way that the Self-determination Theory uses it: a motivation that persists even in situations where the original extrinsic incentives are missing.^[1] Your model of identity genesis (“I am proud of this trait, this is a good part of me”) goes in a similar direction, maybe?
Regarding what you write about human institutions: I agree that their overall quality isn’t crucial. We can see their worth wherever they’re missing, and some of the most effective ones are those which aren’t formalised: small communities, friend groups, families. I am unsure if institutions really need any percentage of “intrinsically cooperative” individuals to function. The higher you rise in the hierarchy, the more machiavellian it inevitably gets, but international relations do a remarkable job at keeping order where constant war was the default.
What’s important to my point: There are several factors for human cooperation, and whenever one is missing, the situation gets notably worse. I think those factors are pretty independent (you disagree with this, right?).
We currently mostly focus on how to build AI that cooperates. I think we should focus on building a better environment for the AI that we have. Not much has been done to that effect and it wouldn’t hurt to try.
1. ^
  Please indulge me if I define these things sloppily, that shouldn’t be super relevant for my point.

Tom Butterweich 21 Sep 2026 16:40 UTC
1 point
0
in reply to: Dagon’s comment on: Tom Butterweich’s Shortform
But it’s coded somewhere, right? In the DNA, in the neural structure of the brain, in organisations and social networks. How would you call it, if not ‘implementation’?

Tom Butterweich 21 Sep 2026 10:47 UTC
1 point
0
in reply to: StanislavKrym’s comment on: Tom Butterweich’s Shortform
Thank you. Byrnes’ work helps, but it makes a complementary point: it focusses on implementation of instincts into brains (points 1 and 2 in my post) rather than steering cooperation by implementing different social settings, which are best explained from the outside (point 3, which I suggest should get more attention).
For humans, we work on all three layers, because leaving out one creates problems. Educating (or brainwashing) everyone to be Rightfully Good is hard and often ineffective—the methods themselves contradict the ideals they claim to propagate, so students would often mimic their superiors’ practice rather than internalise their lessons. Even when moral education works, a fixed ethos is limited in scope. Good hearted people get Goodharted when context changes, they end up killing in the name of love unless there is some sort of social control. The latter requires rough parity between agents that control each other—but stability extends to inferiors too, see my other reply to @IanWS.

Tom Butterweich 21 Sep 2026 10:26 UTC
1 point
0
in reply to: IanWS’s comment on: Tom Butterweich’s Shortform
Absolutely. Humans are terrible, but humanity makes do, and that’s what deserves exploration IMO. And yes, figuring out those conditions is roughly the project I have in mind. Regarding the parity argument: cooperation between humans (top of the food chain as of now) moderates inter-human power dynamics, creating dictators but also getting them decapitated. It also ensures animal welfare to some extent, even though a single human could probably wipe out entire species and might well benefit from it, if it weren’t for social sanctions. It’s a bitter analogy, precisely because we haven’t been nice to most animals. But for some species, signalling friendship has become part of human identity: audiences forgive films almost anything except killing the dog, and John Wick is the extreme version of that. Okay, we have a kind of symbiosis with cats, dogs, and horses; but what about hamsters and dolphins? I don’t think it’s fully explained as spillover from loyalty to our own species. I’d rather see it as a kind of social group selection: nobody wants to be friends with the people who torture animals, as they tend to be assholes in other ways, too. Maybe the AI overlords can signal trust to each other by protecting and explicitely not torturing us, wouldn’t that be nice?

Tom Butterweich 21 Sep 2026 9:07 UTC
7 points
0
on: Tom Butterweich’s Shortform
Where exactly is human alignment implemented?
At the lowest level, genes (or memes) act completely selfishly. No gene has ever sacrificed itself for another. So it’s not the ingredients that make a person aligned.
Things get more complex when you put the pieces together. There is cooperation within a cell and altruism within a species. These are second-order effects, grounded in and emerging from the utter selfishness at the lower, less abstract levels.
But where is this favourable behaviour “stored”?
1. Some of it is hard-wired and passed on through the genome and gene expression: reacting to a baby’s cry, being afraid of the dark, …
2. Some of it is internalised over a lifetime: loyalty to one’s family, doing “the right thing” even when no one’s watching (though you still get to tell the story), …
3. Some of it lies only in the context. Without repeated interaction and social control, people lose their internalised ethics remarkably quickly: tourists misbehaving, soldiers looting and raping, …
My impression, having lived in this world for a while now, is that the formation of an individual is not enough to align it with any kind of morality. Most people will defect to some degree when given the chance—and defection pays, at least locally. Whatever cooperation has emerged within humanity, its ~~main~~ ingredients have ~~been~~ [included] openly (often grossly) misaligned individuals.
So why isn’t this discussed more in AI alignment? There, too, points 1 and 2 prove insufficient when point 3 (context and incentives) is off, just as one would expect from humans. This is the common theme of the HuggingFace and related incidents.
Working on the arrangement seems more promising than optimising the ingredients, especially since interventions could then happen bottom-up rather than only top-down.
There is a lot of research out there that remains marginal in the discussions here on LW. Cooperative AI, obviously, but also multi-level selection (unpopular in biology because it’s hardly needed there, more useful for explaining culture; was discussed on LW mostly ages ago) and the pragmatic approaches to trust networks proposed by the web3 people (closely related to alignment stuff in concept, but lacking mutual citations and shared terminology). There are major obstacles in making any of this work for alignment, and currently, cooperation between agents seems to be part of the problem, not the solution.
Still—I think this deserves more attention.

Tom Butterweich 7 Aug 2026 14:12 UTC
1 point
0
on: There Will Come Soft Rains
Sharing things that touch me—in my case mostly true stories, special music, sometimes live performances—is something I really cherish about working as a teacher. It also happens to be a use of embodied knowledge that AI cannot replace: not because AI couldn’t understand human art, but because it could fake being moved by it. When I inevitably well up in those certain scenes of Britten’s War Requiem or Rameau’s Les Boreades, that is a hard-to-fake signal (at least for me, I’m a shitty actor), and it communicates something about human experience that a “crying” robot couldn’t.
This isn’t just a sentimental point. AI is eroding the economic value of education and marginalising its role for distinction (intelligence isn’t exclusive to humans any more). What remains is education as a pathway to embodied experience. Because of that, I think teaching will not become 100 % machine work.

Tom Butterweich 17 Jul 2026 9:54 UTC
6 points
1
in reply to: kbear’s comment on: Learning concepts is dirty work
Glad you liked it. And yes, I would agree that this is in accordance with what I tried to point out.
Being asked a question two times incentivises children (and adults, I might add) to favour a different answer. This makes interpretations blurry and the categorisation notoriously unreliable. The child in the video you describe could well be regarded as in second or even third phase, yet not engaged in the intended task (but focussed on the “language game”) -- or simply unclassifiable. Criticism will often point out these weak data points but fail to check how that affects Piaget’s deduction of constructivism as a description of how concepts are acquired (not at all, as long as there is also some compelling evidence for phase 2).
Maybe some day we will have more objective measurements of people’s reasoning. Evolutionary biology started out with spotty data and took a long time to fill the gaps it couldn’t explain in detail, the human eye being the most prominent example,* but nothing in biology makes sense except in the light of evolution. For me, it’s the same with constructivism and learning theory. And it’s exciting, if scary, to see how we might learn more about our own minds in the near future.
*) I think it’s 300-odd evolutionary steps, and every single one brings a gradual fitness advantage.

Tom Butterweich 16 Jul 2026 19:58 UTC
5 points
0
on: Why I Left Google DeepMind
First of all, thank you for your effort. I really think it makes a difference, and I admire your integrity and stamina.
The elephant in the room: The US is currently textbook pre-fascist, even though it might not feel that bad. The massive spending on ICE is not just for legal deportations. The midterms will not be fair. Trump (or his successor, should he die) will not leave office voluntarily.
AI is a dual-use technology of national importance. The administration would not leave it to chance whether it had full access to frontier models. They made an offer in public and enforced it in private. Even in more peaceful, more democratic times, state agencies had ways to get around any restriction if it was really necessary.
There might have been some soft persuasion, too: sharing classified intelligence about what China is cooking, painting disheartening but convincing scenarios that all included Trump getting what he wants. In the end, it doesn’t matter—the people you talked to don’t have the agency you asked them for. Not defending them—we cannot allow civil society to cave in --, but I wouldn’t underestimate the levers that are at work here.
Epistemic status: I think it’s more plausible than a bunch of commendable people suddenly learning to love the ~~bomb~~ killer drones. Sorry if this comment is depressing. :(

Tom Butterweich 15 Jul 2026 9:17 UTC
1 point
0
in reply to: Raemon’s comment on: Raemon’s Shortform Feed
The US economy is overdue for a recession. This is currently delayed by massive investments in AI infrastructure. If the current, exponential growth were to be halted or even just throttled, that would let a lot of bets on the future of the industry burst. Could broader adoption (-> inference → hardware purchases) balance that? Enough to keep the US economy afloat? I doubt it, but I’m sure the tipping-point-ness of the situation makes predictions hard.
Also, I imagine that from an economist’s perspective, the idea of slowing down an industry just to save humanity would sound very hypothetical—not even /s

Tom Butterweich 14 Jul 2026 12:57 UTC
17 points
3
on: The Whitney Biennial Should Admit That Emilie Gossiaux Wants to Fuck Their Dog
Wow, what an arc! From spotting this title among all the alignment/extinction stuff, to feeling bewildered and very sceptical at reading the core thesis, to … uh, agreement. Among other feelings. This was well aimed, thank you.
Zeitgeist is found in the most improbable of places, I guess. In this case, it’s not even the content (weird art) -- it’s how it’s (re)presented.

Tom Butterweich 22 Jun 2026 18:10 UTC
17 points
4
on: Tom Butterweich’s Shortform
What’s going through your mind, vacuum pump?
.
.
.
Air, and air, and air again.
But less, and less, and less.

Tom Butterweich 21 Jun 2026 9:42 UTC
4 points
3
on: Would anybody here be interested in a “mistake postmortem” discussion group?
In society, we hear somewhat conflicting advice about thinking about our mistakes. Sometimes you hear advice like “Don’t focus on your past mistakes, look forward to the future”. But you will also hear, “learn from your mistakes so you don’t repeat them”. Which should you follow?
I think these are two sides of the same coin. Learning from mistakes is definitely useful, and it’s not the reflection itself that paralyses, but the (internalised) social punishment that comes with one’s downfalls getting exposed. In a setting where everyone may contribute to making mistakes and discussing them (safely),* nobody gets too attached to a clean record, and the fear of exposure loses its sting. Establishing this requires committed leadership and, as always, trust.
The bottleneck does not lie in the knowledge base. Most institutions I have seen from the inside fail to even leverage their own mistakes—which would be the most relevant, most precious to learn from! It’s a matter of how we talk about mistakes, a question of culture.
One more observation: as stories of bad decisions are told online, LLMs are trained on them and become very adept at giving the kind of Good Advice that would have prevented the mistakes. Will all those witticisms of old people boil down to “should have asked your Claude first” in the future? Mixed with a pinch of “shouldn’t have trusted the AI”, of course.
*) I’m somewhat proud to say we have a special word for that in German, “Fehlerkultur” (“error/mistake culture”), which isn’t to say we don’t suck at it.

Tom Butterweich 15 Jun 2026 7:30 UTC
31 points
11
on: You need to know about the Baruch Plan
In December of 1994, Ukraine notoriously surrendered its nuclear arsenal* under the Budapest Memorandum, in exchange for two pinky promises: a non-aggression pact with Russia and security assurances by the US (and other, less powerful states).
We’re in the fifth year of the full-scale invasion, and the lesson the whole world has learned by now is: guarantees by world powers are worthless unless you have nukes, and threats by world powers are worthless if you do (see North Korea). This is a serious deterioration compared to the geopolitical climate in the 90s.
Each geopolitical player, when on their own, has an incentive to gain nuclear deterrence (and thus, territorial integrity) for the price of increasing collective existential risk. The only way out of this dynamic is to create trust—and that means costly signals. I guess what I’m saying is that the treaties are hard already but the real challenge is to sustainably back them up. I have tremendous respect for people who choose careers in international relations.
*) It’s worth noting that just keeping the bombs would not immediately have implied nuclear capabilities and posed challenges of its own.

Tom Butterweich 14 Jun 2026 19:53 UTC
1 point
0
on: Canada Lost Its Measles Elimination Status Because We Don’t Have Enough Nurses Who Speak Low German
Very interesting, thank you. I had no idea there was a Low German speaking minority in Canada, of all places.
I listened to a snippet of Plaudietsch and found out the dialect is quite different from the Lower German still spoken around where I live (Northwestern Germany). That got me thinking: it would take a lot of effort for someone to learn an exotic language like this, so maybe it’s more feasible to rally some of the Mennonites who like the idea of vaccines to become ambassadors for their own wider communities? They would already be familiar with the communication channels.

Tom Butterweich 10 Jun 2026 16:08 UTC
1 point
0
on: The Three Filters: Why Almost Every Plan to Survive ASI Fails Miserably
I feel like at the heart of counterarguments to solutions that resemble those found in human history—cooperative / multipolar / non-inner alignment --, there is a fallacy of comparing time scales but not shifting perspective to the new subjects of post-human history, AIs. It’ll all go too fast for us, that’s for sure—but does that imply that one rogue AI will win everything? The speed of decision making will increase significantly, after all, possibly more than the speed of weapon development. The grounds for decision making will become more abstract, and a move by one ASI to simply secure computing ressources beyond a certain limit could be regarded as a provocation by its fellow ASIs, who could intervene incredibly swiftly. If it’s just one or two AIs who escape the peloton of the arms race: okay, that sounds bad, but if it’s a couple? The very thing that OP is working on—deep understanding for ASI risk—would be wide-spread among all relevant players.
Consider human development of the last century or so. We have surpassed the limits of traditional weapons and produced enough to destroy the world many times over. None of this is comprehensible to chimps. There have also been times at which one player had enough lead to establish total military control, but they didn’t really try, at least not by all means. Why not? Because there is multipolarity nested in multipolarity; people show signs of virtue to succeed inside their social environments and that is what saved everyone so far, including some of the underpowered players like, say, chimps. This behaviour is not a priori, it emerged under evolutionary pressure and has to be relearned by every specimen.
By comparing this to ASI timelines, I have probably committed a number of other fallacies, but so it goes.