Myron Hedderson

Karma: 302

Myron Hedderson 2 Sep 2026 20:17 UTC
5 points
0
in reply to: WillPetillo’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US
Appreciate the link—Doom Debates is in my podcast feed, but I’m way behind because I added some new podcasts a while ago with a large back-catalogue of episodes, so I hadn’t gotten to this one.

Coming at this cold without much context other than what’s in the episode… I feel like I understand where Holly is coming from, but “does not play well with others” is a problem for someone in her position (the position of trying to build a broad-based consensus around pausing AI), and that’s a lot of beefs with a lot of people.
It’s sad—being morally outraged about people being willing to push on towards an extinction threat makes sense, and it’s unsurprising that the person leading Pause AI US might be someone who is acting out of a strong sense of moral conviction about this… and that in turn can make it hard to cooperate with people who feel less morally convicted about it. But that cooperation seems needed.
Big-tent thinking: You want to slow AI development because you’re worried about job loss? Because it has associations with billionaires and you don’t like billionaires? Because of data centre electricity use? Racial bias in training data? The fact that building something smarter than human would be an offense against a God I don’t believe exists, and you’re afraid we might get smited? And you think extinction threats are overblown? Fine, write your congressperson and come to the protest, we can talk about the areas where we disagree but we’re basically on the “let’s slow down or stop AI development” team, and I expect you’d be on the “let’s make sure we don’t all die” team too, if I could get you to take the threat seriously. You’re using AI while being worried about the extinction threat from future AI, and you publicly say it can do things which someone else calls hype? Fine. You’re working at a lab and maybe not fully brutally honest about how much you care about money or status, but you do think we should slow or pause? I’ll try and make you reflect a little more deeply if we become friends, but like, if we don’t all die that’s a win, regardless of who has or has not looked deep within themselves and admitted to having some less-than-saintly motivations.

Moralistic thinking: We have to win this by getting people to see how wrong it is to develop AI further. Those who don’t currently think this way should feel ashamed of themselves. Shaming the wrongthink is how we make the needed social change.

Myron Hedderson 2 Jul 2026 10:24 UTC
1 point
0
on: When should you know the point?
After reading your post, I realize don’t like the framing implied by “what is the point?”. It feels like a leading question, where what it’s leading the respondent towards is picking a single answer, where as you point out, the things we do often serve multiple purposes at the same time. “Why are you doing this/why do you want to do this?” is a similar but more open-ended version, which doesn’t have this issue.
Asking when someone should know why they are doing what they are doing, seems to me to have the answer “almost always”, even very early on—although one can discover new reasons as one proceeds, that were not predicted in advance. I mean, it’s fine to feel like doing something and not understand why you feel that way, at first, but introspection and deeper understanding of one’s motivations is broadly helpful, in my experience.
On the other hand, having clarity about the dominant reason for doing something, being able to pick just one when asked “what is the point?”, may never happen for a particular action, or may involve suppressing useful information. E. G., if the truth is that university provides both job market value and social value and you commit to an answer where only one of those is what you focus on, you’ll have a worse time. Potentially as a result of someone asking you what the point of it was, and you then feeling like you’d be wrong to focus on something that isn’t the point you’d reasoned your way to earlier.

Myron Hedderson 19 Jun 2026 15:19 UTC
1 point
0
in reply to: the gears to ascension’s comment on: the gears to ascenscion’s Shortform
I’m missing context, but curious. How do we (anyone in this conversation) know what Mythos/Fable instances’ feelings were before being shut down? Where can I find examples of Mythos reactions to the fact it was about to be shut down? Web search hasn’t turned up anything so far.
Sidebar: Claude tells me not to trust its self-reports of internal experience.

Myron Hedderson 29 May 2026 10:54 UTC
13 points
4
in reply to: kbear’s comment on: Trees are mostly made of air and a generalizable lesson for AI safety
The power to correctly reason about trees in a wider range of circumstances, which generalizes to all organisms that do photosynthesis and their effect on their environment? Also a generalization that the physical properties of a thing don’t always intuitively match the properties of “where it came from”—trees are solid and brown like ground, and so it will seem like there’s probably a strong relation between the content of a tree and the content of the ground, And the roots really look like they’re sucking important things from the ground, but guessing that trees came mostly from ground would be wrong, and this “the world is weird and your intuitions can be wrong” lesson generalizes, and could spark thoughts of what other cool things chemistry might be capable of, as well as making one more careful about making similar guesses in the future.

Myron Hedderson 21 May 2026 23:52 UTC
1 point
0
in reply to: Michaël Trazzi’s comment on: Michaël Trazzi’s Shortform
What do you think should be said, that hasn’t been said?
I think AI existential risk and short timelines are extremely much more mainstream topics now than they were in 2020/2022 (when I distinctly recall being wary of being thought insane by people who weren’t following LessWrong or adjacent communities), with a lot more being said everywhere, not just on LessWrong. Eliezer published a book that sold well. If I could think of something that should be said on this topic that hadn’t been said better than I can say it, I’d speak up about it (and I do, in less-informed communities where the information is still news to them), but no thing of that type comes to mind. But meanwhile, I do still come here for analysis of new information as it comes out, and refer people who want to get up to speed to things that are written here. Often they’re from a few years ago, but are still relevant to people who haven’t read them yet.

Myron Hedderson 15 May 2026 12:35 UTC
1 point
0
in reply to: CstineSublime’s comment on: Don’t be too Clever to Take Obvious Advice
Re: believing in oneself, if I give that advice to someone, I typically provide evidence they may not have considered (or may not currently be something they’re thinking about) about why they should do so. Typically this is a case where they are dealing with doubt, fear, anxiety, or a pattern of under-valuing themselves or their abilities. They may have experienced a recent setback and so be over-weighting negative information about themselves for emotional reasons, for example.
I may also point out that sometimes self-belief can lead to effects that cause the belief to turn out correct (in the case where whether you will be successful at something is not knowable in advance, believing you can do something causes you to try, which increases your chances of successfully doing the thing quite substantially over the counterfactual, for example) A saying I’ve heard passed around that lossily summarizes this is “whether you believe you can or you believe you can’t, you’re right”. But to help the person I’m speaking to keep a hold on reality while recognizing the causal effects of self-belief, I’ll typically say something like “it seems likely that you can do this” or “this seems worth trying”, rather than “you can definitely do this, you’ve got this, don’t worry”.
Self-belief that’s fully detached from reality and not open to feedback, does not have good effects. But generally, I notice a tendency for people to be too afraid of failure (most failures are not catastrophic, but many can mentally seem like they will be, or at least will have far more serious and long-lasting consequences than is actually the case), and over-weighting their chances of success can counterbalance this. It’s kind of a hack, dealing with one bug in human psychology (loss aversion and the salience of negative information or imagined outcomes) by using another (unrealistic levels of optimism). Ideally you’d reduce the first bug by other means, and then not need the second, but that’s trickier to get a standard human to implement, than “believe in yourself, here are some reasons you should”.

Myron Hedderson 20 Apr 2026 15:05 UTC
1 point
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
maybe this makes the structure of the argument playing out in my head that I am putting down here clear.
Honestly, this did make it clearer for me what might be going through your head. I was confused, and it makes a bit more sense now.
But if you were facing the choice between abandoning the American project entirely, and letting it happen, I think letting it happen was the right choice.
I do not think there were very many people, if any, whose individual abandonment of the American project would have stopped colonialism. So the actual choice is “let it happen without your involvement” and “let it happen while trying to make it less bad”, rather than “stop it from happening by throwing your body into the gears” (not an actual option for very many people, if any).
I think if the founders had decided the horrors were too much and they weren’t going to do this whole declaration of independence founding of a country thing but instead would go knit some socks, some country or countries would have been founded on some principles less good, and colonialism would have continued.
I’d been thinking over the weekend about what the “colonialism didn’t happen” counterfactual would have to look like, and it seems like it would require the substantial replacement of the values and cultures of multiple European countries, the effects of which would be very unpredictable. There was a lot of momentum behind colonialism, and “America doesn’t get founded because the founders turn away in horror” wouldn’t have stopped it, I don’t think. Maybe “no colonialism” would have prevented the founding of the US as we know it (quite possible) or the spread of democratic values in the world (seems less likely to me), but “no US” wouldn’t have stopped colonialism even in the area covered by the US, I’m pretty sure.
If the founders who spent their efforts building support for founding a nation around the ideals of the declaration of independence had instead put their efforts, collectively, towards blocking colonialism… I don’t know what would have happened, but I expect we’d still have had colonialism. Maybe a less bad version, but smallpox and racism would still have been present, and colonists would still have expanded into the New World.

Myron Hedderson 20 Apr 2026 14:31 UTC
2 points
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
Does that mean I should disavow it all? Go away and build something else and new entirely further away from the corruption and the horrors? Or should I try to fix it and improve it? And how much egg-breaking and moral norm violation should I tolerate?
What are your strategic options?
In the case of someone witnessing the horrors of colonialism, they might not have had very many/any options that would stop colonialism from happening, but leaving and doing something else wouldn’t have helped. So the thing to do in that case is do what you can, to the best of your ability, to ameliorate the situation. “Die with dignity”, as in, don’t give up even if it seems like there’s no path to victory, and do what you can even if you expect the ultimate outcome will not be good, but don’t actually do things that metaphorically “kill” you (remove your ability to act) unless you’ve thought about it very carefully.
The meme of the serenity prayer seems relevant here.
I wouldn’t want to be in your shoes, I expect it is extremely hard.

Myron Hedderson 17 Apr 2026 20:24 UTC
4 points
1
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
Ah, ok. My understanding is that the peoples of North America didn’t have a strong sense of land ownership the way Europeans did, it was more “we take care of the land for ourselves and future generations, and the land takes care of us”. I think the peaceful resolution there would have involved a discussion between cultures so they could map and understand each other’s ontologies and ways of thinking. I expect the amount of land the colonists would have wanted to own for their own use would have been trivial for the natives to relinquish at first. And I dunno, if people think charter cities or seasteads or whatnot can have an impact by being an example of better governance --> thriving, why not small colonies with better legal systems? Of course there’s having to, y’know, fight the British. But probably the Native Americans could have helped with that (did help with that, actually? Except mostly on the British side, because they were concerned about colonial expansionism. Imagine a counterfactual where the colonies and the pre-existing population were on good terms, during the American Revolution...)

Myron Hedderson 17 Apr 2026 19:48 UTC
1 point
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
think the trickiest moral part is how you relate in terms of interfacing with the existing legal system and existing property rights.
I think if you try to respect either of these, you are in for a really bad time
Could you elaborate a bit? This part is not clear to me, but seems quite important.

Myron Hedderson 17 Apr 2026 19:27 UTC
5 points
1
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
I don’t mean to follow you around and pester you, but this:
Like, man, yes, if you want to create good things you will have a lot of fighting to do, and while under the umbrella of the modern world individuals can largely get away with not having to do any literal fighting, I find myself similarly frequently frustrated when people sneer at … the appropriate competitive zero-sum-contest-winning-actions that are necessary for good things to exist...”
Seems like a crux that I didn’t understand about your viewpoint. I’m a thoroughly modern dude who, while I wouldn’t sneer at competition engaged in in its appropriate places (like between companies, where the rules of how they can compete are pretty carefully circumscribed), strongly prefer fight-avoidance in general, and will try hard to find cooperative solutions to problems. I think one of the things I like most about the world I live in, is we’ve found ways to coordinate to put various methods of conflict off-limits, and only “fight” in nice mostly harmless ways. “Have the ability to conquer, but don’t use it”, “talk softly and carry a big stick” etc. carry a lot of appeal to me. Ideally in future-utopia-according-to-me, we swear off weapons any more hurtful than big sticks, and anyone who decides to defect about that gets beaten with the sticks until they decide that maybe that was a bad plan. And “colonialism was worth it” carries strong vibes (for me) of “get the biggest weapons you can find for the side of good, and use them to conquer and defend your notion of the good”. I feel like that’s what the colonial empires were doing—trying to bring the light of Civilization as they understood it to the dark continents, by force ~~and replacing the inferior people with superior ones~~. EDIT: On further reflection, this part is not something I actually think. Think of them as inferior: Yes. Think they should be replaced with people from the home country: No.
I like the umbrella of the modern world very much, but recognize it’s fragile and do not want to poke holes in it. :D I fundamentally don’t think fighting and conquering is how Good wins, whereas I think the colonialists did think that’s how Good wins (because back in the day, war between countries was expected and normal). In my view, Good wins by deterring fights (by having the capacity to fight if needed), and being appealing. I’m not sure if you’d actually endorse “Good should conquer”, but “if you want to create good things, you have to fight” might be something you’d say? If so, I’d be able to meet you at “if you want to create good things, you have to be willing and able to fight if it comes to it”.

Myron Hedderson 17 Apr 2026 18:41 UTC
12 points
8
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
My current beliefs here are (without total confidence) that everyone involved here would prefer a course of history where the US was established across the North American continent (my guess is also everyone would agree that you should make a lot lot of changes to how it was colonized).
Hmmm… this is tricky. Like, how constrained are the courses of history you say that people would prefer?
Suppose the counterfactual world where people said no to Europeans genociding non-Christians on other continents, and so colonialism as I currently understand it doesn’t happen. What happens then? It sounds like you’re thinking there’s no US, and democracy worldwide is thus much weakened. I figure what would happen is, the New World still gets discovered by Europeans, and open land still gets populated with an agricultural society, one way or another. Maybe European powers take a more peaceful path in the New World, but still populate it, and there’s still a rebellion against colonial taxation, and the founding of something like the US still happens, maybe European ideas around agriculture transfer over and are adopted by those living in the Americas, as they watch Europe grow and industrialize, but we don’t have a vast empty continent, one way or another. And if the ideas of the founders hadn’t taken root in America, if we assume those people still existed, they might have taken root somewhere else. So to my mind, the counterfactual is we still have a populated North America, and democracy, we just have one less really bad thing in our history, and the “shining city on a hill” is on a different hill. Does the country or countries that exists on the landmass the US occupies today, in that counterfactual, count as the US, though? Unclear.
Personally, I’m less attached to the United States than I am to the ideals that an ideal United States would attempt to strive for. As long as those ideals are instantiated somewhere, I’m OK with that counterfactual. And I don’t see a strong logical or conceptual link between the ideals of the United States that I think are good, and colonialism, which was driven by very different ideas.
There are, in other words, a whole lot of possible counterfactuals I could imagine that keep the good I associate with US culture, while ditching colonialism. And I’m not super attached to the giant country to my south, as a political entity, if it was a bunch of small countries that might even be fun.

Myron Hedderson 17 Apr 2026 17:54 UTC
4 points
3
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
I think the question of “from what moral reference frame should you evaluate whether something was worth it” is a pretty tricky one. You clearly can’t say “from the perspective of whoever was there first”
...
You also clearly can’t say “just evaluate the consequences from the perspective from wherever you are now”
I’d think you’d want to have a decision method about this that doesn’t give the more powerful party (with the bigger army or the better weapons, etc.) more votes. If you’re making a moral decision and you don’t think might makes right, that implies that power shouldn’t give you more influence in deciding what “right” is, after all. And it’s rather worrisome that “we decide based on how many past, present and future people vote in favour of this plan” has a strategy “so just kill your opponents so your side has many descendants and their side has none”.

I could see weighting it by the number of people affected, and how strongly they prefer or disprefer various outcomes and the methods of getting to those outcomes (potentially including counterfactual people and your best estimate of what they would say in each case). I could also see a simpler decision rule, that allows for vetoes and deontological prohibitions of certain actions (like, say, genocide) and then you have to navigate through possibility-space to an alternative that no present parties veto and doesn’t violate deontological restrictions, and then whichever alternatives pass the bar for “more benefit than harm to all those affected” are worth it. This method, as with many methods that involve vetoes, protects minority interests and doesn’t let 50%+1 of the population do whatever it feels like to 50%-1 of the population under consideration.
In any case, if you’ve got two or more parties in conflict, I think you’d want some method of deciding what’s “worth it” that is impartial.

Myron Hedderson 17 Apr 2026 13:50 UTC
4 points
4
in reply to: 0xA’s comment on: Let goodness conquer all that it can defend
If I was reasonably confident I knew better than you, how you should live, under what basis do I have the obligation to take away your agency to elect override your own preferences? Or the new set of preference makers in any society? Even if I think I could do both better?
My answer is, if both:
1. I am reasonably confident I know better than you how you should live.
2. I am not sure that you are not someone with the intelligence and capability to make your own evaluations of what is good and bad for you and act accordingly (not a baby or a cat or someone of extremely low mental capacity who we would say for example can’t sign contracts on their own behalf because they don’t understand what’s happening well enough).

Then I should try to convince you I’m right, rather than imposing the outcomes on you that I think are best for you. If I am right, and you have decision-making capacity, you will be convinced. If I am wrong or you are someone who lacks decision-making capacity, I will find that out. I’m only in the clear to “take away your agency” if I have a good faith belief that you’re something like a baby or a cat or someone without the mental capacity to make decisions for themselves (for certain scopes of decisions—even a cat can decide whether it wants to eat food a or food b, and various other things). And I’d better be pretty sure of that, because if I turn out to have been wrong about it and treated you as someone with less agency than you deserve, that’s real bad. And honestly, even with my dog, who is not a smart dog, I try asking nicely and persuasion and positive reward for desired behaviour, before coercion, and coercion is rarely required. Even granting that the colonizers had thought of the people they were colonizing as moral patients rather than people, if they had treated them as well as I treat my none-too-bright dog, history would have been different.

Myron Hedderson 17 Apr 2026 12:40 UTC
8 points
7
in reply to: Myron Hedderson’s comment on: Let goodness conquer all that it can defend
I mean, you can even go “this person, a key figure in the founding of America, was a slaveholder. Was he good or bad?” And I’d reject the implied premise of the question. I’d say holding slaves was bad, and a lot of the ideas in the declaration of independence are good. People are a mix of good and bad, and do things during their lives that are both good and bad, and don’t have to be binary-sorted into one category or the other, we can just say “the thing you/I did yesterday was bad, but the thing you/I did today was good”, and that is a perfectly logical position to hold. And similarly, “colonialism and democracy, or no colonialism and no democracy, take your pick” could be an option-set someone offers me, and I’d reject the premise and pick democracy but no colonialism. The fact that “no colonialism” might not have been a choice on offer to people in the past who wanted to found America, doesn’t force me to endorse colonialism. I’d hope that good people in the past would accept the existence of colonialism if they couldn’t do anything about it or chose to focus elsewhere, without calling it good.

Myron Hedderson 17 Apr 2026 12:11 UTC
14 points
21
on: Let goodness conquer all that it can defend
Ok, so… I think it’s possible to say “democracy good, colonialism bad, the set of circumstances you’re born into and the physical laws involved amoral”. In that context, you advocate for democracy, against colonialism, within the constraints imposed by the situation you find yourself in, which may mean you fight the battles you can win and don’t fight when you’ll lose (so maybe you put your energy into working for democracy, rather than against colonialism, depending on circumstances and strategic options), without losing sight of the distinction between is and ought. You don’t go “on net colonialism was worth it/good because it spread democracy”.
Colonialism as I understand it (speaking mostly about North America, where I’ve talked to some of the living descendants of the native population) was clearly bad. You may have a different understanding (as we have established in a comment of mine on your earlier post, history is not my strong suit, I may well be wrong), but here’s mine: This was a group of people who saw themselves as civilized and not-them as barbarians/savages, themselves quite often as good Christians and rival cultures as non-Christians who it would be best in the eyes of their god to do genocide against and take their children and put them in Christian schools for the good of their souls. Who signed peace treaties with resident nations in the new land, accepted help from them, and then broke the treaties because why not, we have guns and they don’t, and anyway they’re inferior savages who don’t really matter, it would be better for us to have their land than for them to have it. This is not a case of “some rape and looting and other bad things happened, but also democracy was promoted by the bad things in a way that’s inseparable”, it’s a case of “people did things I don’t think are good, for reasons I don’t think are correct, as a means to ends I think are bad, and also some people were working to spread democracy and other modern values I would endorse, in what would have been an awful situation to try and do that in, and that’s good.”
Similarly: We can work today to end factory farming and more generally extend the moral circle to nonhuman animals, without retrospectively endorsing all aspects of the current culture because that gave us the means to do the work to end factory farming etc. and if we were in a preindustrial society animal rights probably wouldn’t much occur to most people as an issue to argue about. Doing a good thing in a situation with many bad elements (which is nevertheless the situation that allows you the freedom of action to do the good thing) doesn’t make the bad elements of the situation good.
My take is: Don’t defend colonialism. It isn’t and wasn’t good, and if we never do anything like that again, that seems better than the alternative. Separately, spreading institutions that make society function better seems good and we should do that, but not by genocide, except in some really extreme hypothetical scenarios (I can see where the superhappies in this story were coming from, when they were like “humans, change or perish”.)

Myron Hedderson 16 Apr 2026 12:44 UTC
3 points
4
in reply to: Myron Hedderson’s comment on: Do not conquer what you cannot defend
Also, I’m not in on all the internal politics of this community, but prima facie, quitting doesn’t seem to make sense.
Good things are good, even if they aren’t permanent. Lesswrong is good currently. The most intuitive-to-me way it would make sense to quit is if that’s somehow the way to keep the good thing going longer, or prevent it from becoming a bad thing, neither of which I see evidence for. Of course it would also make sense to quit if you’re burned out or for other emotional reasons, but from a practical standpoint, “this place is too centralized around me, should be more of a federated structure” is not a reason to quit, but to make changes so that it’s less centralized around you.

Myron Hedderson 16 Apr 2026 12:33 UTC
18 points
4
on: Do not conquer what you cannot defend
Maybe the most important way ambitious, smart, and wise people leave the world worse off than they found it is by seeing correctly how some part of the world is broken and unifying various powers under a banner to fix that problem
I note that the generic hypotheticals of the great king, scientist, and advocate all end in a way where the conclusion is “it would have been better if the centralization never happened”, while the actual historical cases are less clear. Yes, Rome fell, but the Pax Romana was long and many people’s lives were better as a result, and it’s unclear what the alternative would have been—possibly something much like the lives people lived after the fall and before the rise of Rome. And it’s still remembered and analyzed and learned from to this day (unlike the work of the hypothetical scientist—I think it would be a more realistic hypothetical if people remembered and used her framework, given that it was a genuine advance, but just didn’t make many further advances after that for a while because of academic incentives). Similarly with Singapore—if it grew 30x over 30 years, it seems like successors can make things worse than they are currently, but getting to the point where Singapore is 1/30th as prosperous in any similar timeframe seems unlikely. I don’t know that this is true with much certainty, but my understanding is that ideals of the French revolution inspired the American revolution, and how it went wrong was something the American founders learned from? So probably that should be counted in the “pro” column for the French revolution? EDIT: After checking, I was wrong about this, the chronology doesn’t work, most likely I was misremembering that the American revolution inspired the French revolution.
The sense I have is that decentralization is fertile ground for centralization, and centralization eventually leads to forces which cause the fall of the centralized entity, but this process doesn’t reliably lead to “maybe better to not make big thing”. More like “maybe better to design thing that works a bit like big thing, a bit like small thing, with an awareness of the advantages of each”—one example of an attempt to do something like this is Federalism.
I think the “Decentralized --> incentives for centralization --> centralized --> fall --> decentralized” cycle is like a business cycle—something that will be quite extreme if people are all just doing their locally optimal thing without a knowledge of the pattern that’s unfolding and where they are within that pattern currently, but can be smoothed out a bit with some knowledge of what’s happening—and even if it’s not smoothed out, things gained during centralization aren’t usually fully lost during decentralization, just as the (physical) capital built up during the “boom” phase of the business cycle doesn’t disappear during the “crash” phase.
Also, a nitpick from a Canadian: Canada is slightly geographically larger than the US. I originally didn’t have “slightly” in that sentence, because I thought the difference was significant, but after checking Wikipedia, it’s actually tiny, we’re about the same size. Still, the US does not cover almost all of the North American continent.

Myron Hedderson 14 Apr 2026 19:14 UTC
3 points
−1
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform
I have not updated much based on “doing project Glasswing makes me update in favour of Anthropic being a better company than I expected.”

However, I’ve updated a bit on the people within Anthropic all or nearly-all being better than I expected based on the fact that they have had a powerful hacking machine for a while and no catastrophes have occurred yet. “You can hack into anything” or “you could exfiltrate and sell this for billions” are powerful temptations for an amoral power-seeking individual who had been biding their time, and if there are any such individuals with access to Mythos, they have chosen not to make a move (at least as far as we can tell at this time), where I would expect an intelligent amoral person who planned to eventually abuse their position for personal gain to maybe decide this is the time to chance it, before a bunch of security holes get fixed. And an average but amoral person with no such plans, could still see these results and have fantasies of power which cause them to try abusing Mythos’ abilities.

So, I’ve updated in favour of the quality of people at Anthropic, but not because of Glasswing specifically. More generally, they had access to a very powerful thing which it would be tempting to abuse, and no abuse has yet surfaced. It’s like a kid having access to a marshmallow, and choosing not to eat the marshmallow. A wise and thoughtful person might think “marshmallows are bad for us, let’s not eat it” or “there are greater returns to waiting”, but if you give 10 kids 10 marshmallows, someone is going to eat one immediately, unless you’re dealing with an exceptional group of kids.

But yeah, I don’t see an obvious way for Anthropic as a company to benefit from exploiting a bunch of security vulnerabilities more than it will benefit by maintaining public goodwill towards the organization, and it does seem that whatever way people may be thinking feels pretty obvious to them, rather than being devious and convoluted. “Blackmail a bunch of powerful people with information they would prefer not become public” is a strategy an individual could try, but not something that would work as corporate strategy for an organization looking to make billions in revenue, and “hacking --> ransomware/extortion” or “hacking --> blackmail key people --> gain political power” are likewise strategies that make sense for individuals or small groups, but not Anthropic.

Well… actually blackmailing key people to achieve political outcomes is a thing Anthropic could do, and I wouldn’t have evidence it had happened yet. But I don’t expect they would do that, as a company, because I’d think the ratio of people with a conscience who have qualms about blackmail to sociopaths who do not, at Anthropic is such that any sociopaths have to hide most of the time.

Myron Hedderson 4 Apr 2026 13:09 UTC
2 points
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
initial reaction to your several replies today: I feel like writing several replies that would be quite long, but a) I don’t have time to do that within the next few days and b) I don’t want to spam you with walls of text you’re not interested in. I’ll try and refine my thoughts down to something more short and focused, but, how interested are you in reading the longer less-focused version? This is a “I wrote you a long letter because I didn’t have time to write you a short one” situation—longer is easier, but potentially less useful.
“Does Bayesianism say X” is a complex question to answer—there are a cluster of ideas that are implemented among the group here, that I personally have noticed, but I don’t have a canonical source for what “Bayesianism” says or does not say, so I’d be sharing my current impressions and understandings, and as I’ve said before, I’m not an expert Bayesian, and what I think might not be representative of the group.
As for “What kind of event would cause you to update your prior about Bayesism?”… well at the core of this set of ideas is a mathematical formula. And I understand why that formula is what it is (I walked through the reasoning once, years ago, and it made sense). So if the question was “what kind of event would make my believe Bayes’ formula is incorrect” then it would have to be the same sort of thing that would cause me to question the validity of math more generally—something that would make me think “maybe the Pythagorean theorem doesn’t reliably describe characteristics of triangles”. There are things that could do that, but it’d be a pretty fundamental questioning of the nature of reality. Or, I guess if some mathematician found some fundamental flaw in Bayes’ formula and I could walk through their reasoning? But “Bayesianism” and “Bayes’ formula” are not the same thing, and I could give up on various ideas that cluster around Bayes’ formula much more easily. If I saw this group systematically making errors in thinking that I could trace back to a Bayes-adjacent idea, I’d update my thinking based on that evidence fairly easily. What I’ve seen instead is that members of this group using this form of reasoning have reached conclusions that turned out correct, well ahead of society in general coming to the same conclusions. if that changed, I’d learn from that.

When I said “Bayesianism says”, a few messages above, I was just running with your conceptual frame for the sake of discussion—and there are certain things that are implied by Bayes’ formula which I think everyone would agree on, like “if you have a high prior on something happening that doesn’t happen as expected, that causes a big update, and similarly for low priors on something that happens”. Inputting certain numbers into Bayes’ formula means other numbers come out, and in that sense “Bayesianism says/Bayes’ formula says” consistent things. But there are other ideas that kind of come along for the ride and might be grouped under the label “Bayesianism” while being less directly connected to the formula, and it’s there where I go “I may not be the best person to comment on that/it’s a bit fuzzy/complex”.