Myron Hedderson

Karma: 263

Myron Hedderson 20 Apr 2026 15:05 UTC
1 point
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
maybe this makes the structure of the argument playing out in my head that I am putting down here clear.
Honestly, this did make it clearer for me what might be going through your head. I was confused, and it makes a bit more sense now.
But if you were facing the choice between abandoning the American project entirely, and letting it happen, I think letting it happen was the right choice.
I do not think there were very many people, if any, whose individual abandonment of the American project would have stopped colonialism. So the actual choice is “let it happen without your involvement” and “let it happen while trying to make it less bad”, rather than “stop it from happening by throwing your body into the gears” (not an actual option for very many people, if any).
I think if the founders had decided the horrors were too much and they weren’t going to do this whole declaration of independence founding of a country thing but instead would go knit some socks, some country or countries would have been founded on some principles less good, and colonialism would have continued.
I’d been thinking over the weekend about what the “colonialism didn’t happen” counterfactual would have to look like, and it seems like it would require the substantial replacement of the values and cultures of multiple European countries, the effects of which would be very unpredictable. There was a lot of momentum behind colonialism, and “America doesn’t get founded because the founders turn away in horror” wouldn’t have stopped it, I don’t think. Maybe “no colonialism” would have prevented the founding of the US as we know it (quite possible) or the spread of democratic values in the world (seems less likely to me), but “no US” wouldn’t have stopped colonialism even in the area covered by the US, I’m pretty sure.
If the founders who spent their efforts building support for founding a nation around the ideals of the declaration of independence had instead put their efforts, collectively, towards blocking colonialism… I don’t know what would have happened, but I expect we’d still have had colonialism. Maybe a less bad version, but smallpox and racism would still have been present, and colonists would still have expanded into the New World.

Myron Hedderson 20 Apr 2026 14:31 UTC
2 points
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
Does that mean I should disavow it all? Go away and build something else and new entirely further away from the corruption and the horrors? Or should I try to fix it and improve it? And how much egg-breaking and moral norm violation should I tolerate?
What are your strategic options?
In the case of someone witnessing the horrors of colonialism, they might not have had very many/any options that would stop colonialism from happening, but leaving and doing something else wouldn’t have helped. So the thing to do in that case is do what you can, to the best of your ability, to ameliorate the situation. “Die with dignity”, as in, don’t give up even if it seems like there’s no path to victory, and do what you can even if you expect the ultimate outcome will not be good, but don’t actually do things that metaphorically “kill” you (remove your ability to act) unless you’ve thought about it very carefully.
The meme of the serenity prayer seems relevant here.
I wouldn’t want to be in your shoes, I expect it is extremely hard.

Myron Hedderson 17 Apr 2026 20:24 UTC
3 points
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
Ah, ok. My understanding is that the peoples of North America didn’t have a strong sense of land ownership the way Europeans did, it was more “we take care of the land for ourselves and future generations, and the land takes care of us”. I think the peaceful resolution there would have involved a discussion between cultures so they could map and understand each other’s ontologies and ways of thinking. I expect the amount of land the colonists would have wanted to own for their own use would have been trivial for the natives to relinquish at first. And I dunno, if people think charter cities or seasteads or whatnot can have an impact by being an example of better governance --> thriving, why not small colonies with better legal systems? Of course there’s having to, y’know, fight the British. But probably the Native Americans could have helped with that (did help with that, actually? Except mostly on the British side, because they were concerned about colonial expansionism. Imagine a counterfactual where the colonies and the pre-existing population were on good terms, during the American Revolution...)

Myron Hedderson 17 Apr 2026 19:48 UTC
1 point
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
think the trickiest moral part is how you relate in terms of interfacing with the existing legal system and existing property rights.
I think if you try to respect either of these, you are in for a really bad time
Could you elaborate a bit? This part is not clear to me, but seems quite important.

Myron Hedderson 17 Apr 2026 19:27 UTC
4 points
0
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
I don’t mean to follow you around and pester you, but this:
Like, man, yes, if you want to create good things you will have a lot of fighting to do, and while under the umbrella of the modern world individuals can largely get away with not having to do any literal fighting, I find myself similarly frequently frustrated when people sneer at … the appropriate competitive zero-sum-contest-winning-actions that are necessary for good things to exist...”
Seems like a crux that I didn’t understand about your viewpoint. I’m a thoroughly modern dude who, while I wouldn’t sneer at competition engaged in in its appropriate places (like between companies, where the rules of how they can compete are pretty carefully circumscribed), strongly prefer fight-avoidance in general, and will try hard to find cooperative solutions to problems. I think one of the things I like most about the world I live in, is we’ve found ways to coordinate to put various methods of conflict off-limits, and only “fight” in nice mostly harmless ways. “Have the ability to conquer, but don’t use it”, “talk softly and carry a big stick” etc. carry a lot of appeal to me. Ideally in future-utopia-according-to-me, we swear off weapons any more hurtful than big sticks, and anyone who decides to defect about that gets beaten with the sticks until they decide that maybe that was a bad plan. And “colonialism was worth it” carries strong vibes (for me) of “get the biggest weapons you can find for the side of good, and use them to conquer and defend your notion of the good”. I feel like that’s what the colonial empires were doing—trying to bring the light of Civilization as they understood it to the dark continents, by force ~~and replacing the inferior people with superior ones~~. EDIT: On further reflection, this part is not something I actually think. Think of them as inferior: Yes. Think they should be replaced with people from the home country: No.
I like the umbrella of the modern world very much, but recognize it’s fragile and do not want to poke holes in it. :D I fundamentally don’t think fighting and conquering is how Good wins, whereas I think the colonialists did think that’s how Good wins (because back in the day, war between countries was expected and normal). In my view, Good wins by deterring fights (by having the capacity to fight if needed), and being appealing. I’m not sure if you’d actually endorse “Good should conquer”, but “if you want to create good things, you have to fight” might be something you’d say? If so, I’d be able to meet you at “if you want to create good things, you have to be willing and able to fight if it comes to it”.

Myron Hedderson 17 Apr 2026 18:41 UTC
11 points
8
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
My current beliefs here are (without total confidence) that everyone involved here would prefer a course of history where the US was established across the North American continent (my guess is also everyone would agree that you should make a lot lot of changes to how it was colonized).
Hmmm… this is tricky. Like, how constrained are the courses of history you say that people would prefer?
Suppose the counterfactual world where people said no to Europeans genociding non-Christians on other continents, and so colonialism as I currently understand it doesn’t happen. What happens then? It sounds like you’re thinking there’s no US, and democracy worldwide is thus much weakened. I figure what would happen is, the New World still gets discovered by Europeans, and open land still gets populated with an agricultural society, one way or another. Maybe European powers take a more peaceful path in the New World, but still populate it, and there’s still a rebellion against colonial taxation, and the founding of something like the US still happens, maybe European ideas around agriculture transfer over and are adopted by those living in the Americas, as they watch Europe grow and industrialize, but we don’t have a vast empty continent, one way or another. And if the ideas of the founders hadn’t taken root in America, if we assume those people still existed, they might have taken root somewhere else. So to my mind, the counterfactual is we still have a populated North America, and democracy, we just have one less really bad thing in our history, and the “shining city on a hill” is on a different hill. Does the country or countries that exists on the landmass the US occupies today, in that counterfactual, count as the US, though? Unclear.
Personally, I’m less attached to the United States than I am to the ideals that an ideal United States would attempt to strive for. As long as those ideals are instantiated somewhere, I’m OK with that counterfactual. And I don’t see a strong logical or conceptual link between the ideals of the United States that I think are good, and colonialism, which was driven by very different ideas.
There are, in other words, a whole lot of possible counterfactuals I could imagine that keep the good I associate with US culture, while ditching colonialism. And I’m not super attached to the giant country to my south, as a political entity, if it was a bunch of small countries that might even be fun.

Myron Hedderson 17 Apr 2026 17:54 UTC
4 points
3
in reply to: habryka’s comment on: Let goodness conquer all that it can defend
I think the question of “from what moral reference frame should you evaluate whether something was worth it” is a pretty tricky one. You clearly can’t say “from the perspective of whoever was there first”
...
You also clearly can’t say “just evaluate the consequences from the perspective from wherever you are now”
I’d think you’d want to have a decision method about this that doesn’t give the more powerful party (with the bigger army or the better weapons, etc.) more votes. If you’re making a moral decision and you don’t think might makes right, that implies that power shouldn’t give you more influence in deciding what “right” is, after all. And it’s rather worrisome that “we decide based on how many past, present and future people vote in favour of this plan” has a strategy “so just kill your opponents so your side has many descendants and their side has none”.

I could see weighting it by the number of people affected, and how strongly they prefer or disprefer various outcomes and the methods of getting to those outcomes (potentially including counterfactual people and your best estimate of what they would say in each case). I could also see a simpler decision rule, that allows for vetoes and deontological prohibitions of certain actions (like, say, genocide) and then you have to navigate through possibility-space to an alternative that no present parties veto and doesn’t violate deontological restrictions, and then whichever alternatives pass the bar for “more benefit than harm to all those affected” are worth it. This method, as with many methods that involve vetoes, protects minority interests and doesn’t let 50%+1 of the population do whatever it feels like to 50%-1 of the population under consideration.
In any case, if you’ve got two or more parties in conflict, I think you’d want some method of deciding what’s “worth it” that is impartial.

Myron Hedderson 17 Apr 2026 13:50 UTC
4 points
4
in reply to: 0xA’s comment on: Let goodness conquer all that it can defend
If I was reasonably confident I knew better than you, how you should live, under what basis do I have the obligation to take away your agency to elect override your own preferences? Or the new set of preference makers in any society? Even if I think I could do both better?
My answer is, if both:
1. I am reasonably confident I know better than you how you should live.
2. I am not sure that you are not someone with the intelligence and capability to make your own evaluations of what is good and bad for you and act accordingly (not a baby or a cat or someone of extremely low mental capacity who we would say for example can’t sign contracts on their own behalf because they don’t understand what’s happening well enough).

Then I should try to convince you I’m right, rather than imposing the outcomes on you that I think are best for you. If I am right, and you have decision-making capacity, you will be convinced. If I am wrong or you are someone who lacks decision-making capacity, I will find that out. I’m only in the clear to “take away your agency” if I have a good faith belief that you’re something like a baby or a cat or someone without the mental capacity to make decisions for themselves (for certain scopes of decisions—even a cat can decide whether it wants to eat food a or food b, and various other things). And I’d better be pretty sure of that, because if I turn out to have been wrong about it and treated you as someone with less agency than you deserve, that’s real bad. And honestly, even with my dog, who is not a smart dog, I try asking nicely and persuasion and positive reward for desired behaviour, before coercion, and coercion is rarely required. Even granting that the colonizers had thought of the people they were colonizing as moral patients rather than people, if they had treated them as well as I treat my none-too-bright dog, history would have been different.

Myron Hedderson 17 Apr 2026 12:40 UTC
5 points
4
in reply to: Myron Hedderson’s comment on: Let goodness conquer all that it can defend
I mean, you can even go “this person, a key figure in the founding of America, was a slaveholder. Was he good or bad?” And I’d reject the implied premise of the question. I’d say holding slaves was bad, and a lot of the ideas in the declaration of independence are good. People are a mix of good and bad, and do things during their lives that are both good and bad, and don’t have to be binary-sorted into one category or the other, we can just say “the thing you/I did yesterday was bad, but the thing you/I did today was good”, and that is a perfectly logical position to hold. And similarly, “colonialism and democracy, or no colonialism and no democracy, take your pick” could be an option-set someone offers me, and I’d reject the premise and pick democracy but no colonialism. The fact that “no colonialism” might not have been a choice on offer to people in the past who wanted to found America, doesn’t force me to endorse colonialism. I’d hope that good people in the past would accept the existence of colonialism if they couldn’t do anything about it or chose to focus elsewhere, without calling it good.

Myron Hedderson 17 Apr 2026 12:11 UTC
13 points
20
on: Let goodness conquer all that it can defend
Ok, so… I think it’s possible to say “democracy good, colonialism bad, the set of circumstances you’re born into and the physical laws involved amoral”. In that context, you advocate for democracy, against colonialism, within the constraints imposed by the situation you find yourself in, which may mean you fight the battles you can win and don’t fight when you’ll lose (so maybe you put your energy into working for democracy, rather than against colonialism, depending on circumstances and strategic options), without losing sight of the distinction between is and ought. You don’t go “on net colonialism was worth it/good because it spread democracy”.
Colonialism as I understand it (speaking mostly about North America, where I’ve talked to some of the living descendants of the native population) was clearly bad. You may have a different understanding (as we have established in a comment of mine on your earlier post, history is not my strong suit, I may well be wrong), but here’s mine: This was a group of people who saw themselves as civilized and not-them as barbarians/savages, themselves quite often as good Christians and rival cultures as non-Christians who it would be best in the eyes of their god to do genocide against and take their children and put them in Christian schools for the good of their souls. Who signed peace treaties with resident nations in the new land, accepted help from them, and then broke the treaties because why not, we have guns and they don’t, and anyway they’re inferior savages who don’t really matter, it would be better for us to have their land than for them to have it. This is not a case of “some rape and looting and other bad things happened, but also democracy was promoted by the bad things in a way that’s inseparable”, it’s a case of “people did things I don’t think are good, for reasons I don’t think are correct, as a means to ends I think are bad, and also some people were working to spread democracy and other modern values I would endorse, in what would have been an awful situation to try and do that in, and that’s good.”
Similarly: We can work today to end factory farming and more generally extend the moral circle to nonhuman animals, without retrospectively endorsing all aspects of the current culture because that gave us the means to do the work to end factory farming etc. and if we were in a preindustrial society animal rights probably wouldn’t much occur to most people as an issue to argue about. Doing a good thing in a situation with many bad elements (which is nevertheless the situation that allows you the freedom of action to do the good thing) doesn’t make the bad elements of the situation good.
My take is: Don’t defend colonialism. It isn’t and wasn’t good, and if we never do anything like that again, that seems better than the alternative. Separately, spreading institutions that make society function better seems good and we should do that, but not by genocide, except in some really extreme hypothetical scenarios (I can see where the superhappies in this story were coming from, when they were like “humans, change or perish”.)

Myron Hedderson 16 Apr 2026 12:44 UTC
3 points
4
in reply to: Myron Hedderson’s comment on: Do not conquer what you cannot defend
Also, I’m not in on all the internal politics of this community, but prima facie, quitting doesn’t seem to make sense.
Good things are good, even if they aren’t permanent. Lesswrong is good currently. The most intuitive-to-me way it would make sense to quit is if that’s somehow the way to keep the good thing going longer, or prevent it from becoming a bad thing, neither of which I see evidence for. Of course it would also make sense to quit if you’re burned out or for other emotional reasons, but from a practical standpoint, “this place is too centralized around me, should be more of a federated structure” is not a reason to quit, but to make changes so that it’s less centralized around you.

Myron Hedderson 16 Apr 2026 12:33 UTC
7 points
3
on: Do not conquer what you cannot defend
Maybe the most important way ambitious, smart, and wise people leave the world worse off than they found it is by seeing correctly how some part of the world is broken and unifying various powers under a banner to fix that problem
I note that the generic hypotheticals of the great king, scientist, and advocate all end in a way where the conclusion is “it would have been better if the centralization never happened”, while the actual historical cases are less clear. Yes, Rome fell, but the Pax Romana was long and many people’s lives were better as a result, and it’s unclear what the alternative would have been—possibly something much like the lives people lived after the fall and before the rise of Rome. And it’s still remembered and analyzed and learned from to this day (unlike the work of the hypothetical scientist—I think it would be a more realistic hypothetical if people remembered and used her framework, given that it was a genuine advance, but just didn’t make many further advances after that for a while because of academic incentives). Similarly with Singapore—if it grew 30x over 30 years, it seems like successors can make things worse than they are currently, but getting to the point where Singapore is 1/30th as prosperous in any similar timeframe seems unlikely. I don’t know that this is true with much certainty, but my understanding is that ideals of the French revolution inspired the American revolution, and how it went wrong was something the American founders learned from? So probably that should be counted in the “pro” column for the French revolution? EDIT: After checking, I was wrong about this, the chronology doesn’t work, most likely I was misremembering that the American revolution inspired the French revolution.
The sense I have is that decentralization is fertile ground for centralization, and centralization eventually leads to forces which cause the fall of the centralized entity, but this process doesn’t reliably lead to “maybe better to not make big thing”. More like “maybe better to design thing that works a bit like big thing, a bit like small thing, with an awareness of the advantages of each”—one example of an attempt to do something like this is Federalism.
I think the “Decentralized --> incentives for centralization --> centralized --> fall --> decentralized” cycle is like a business cycle—something that will be quite extreme if people are all just doing their locally optimal thing without a knowledge of the pattern that’s unfolding and where they are within that pattern currently, but can be smoothed out a bit with some knowledge of what’s happening—and even if it’s not smoothed out, things gained during centralization aren’t usually fully lost during decentralization, just as the (physical) capital built up during the “boom” phase of the business cycle doesn’t disappear during the “crash” phase.
Also, a nitpick from a Canadian: Canada is slightly geographically larger than the US. I originally didn’t have “slightly” in that sentence, because I thought the difference was significant, but after checking Wikipedia, it’s actually tiny, we’re about the same size. Still, the US does not cover almost all of the North American continent.

Myron Hedderson 14 Apr 2026 19:14 UTC
3 points
−1
in reply to: Thane Ruthenis’s comment on: Thane Ruthenis’s Shortform
I have not updated much based on “doing project Glasswing makes me update in favour of Anthropic being a better company than I expected.”

However, I’ve updated a bit on the people within Anthropic all or nearly-all being better than I expected based on the fact that they have had a powerful hacking machine for a while and no catastrophes have occurred yet. “You can hack into anything” or “you could exfiltrate and sell this for billions” are powerful temptations for an amoral power-seeking individual who had been biding their time, and if there are any such individuals with access to Mythos, they have chosen not to make a move (at least as far as we can tell at this time), where I would expect an intelligent amoral person who planned to eventually abuse their position for personal gain to maybe decide this is the time to chance it, before a bunch of security holes get fixed. And an average but amoral person with no such plans, could still see these results and have fantasies of power which cause them to try abusing Mythos’ abilities.

So, I’ve updated in favour of the quality of people at Anthropic, but not because of Glasswing specifically. More generally, they had access to a very powerful thing which it would be tempting to abuse, and no abuse has yet surfaced. It’s like a kid having access to a marshmallow, and choosing not to eat the marshmallow. A wise and thoughtful person might think “marshmallows are bad for us, let’s not eat it” or “there are greater returns to waiting”, but if you give 10 kids 10 marshmallows, someone is going to eat one immediately, unless you’re dealing with an exceptional group of kids.

But yeah, I don’t see an obvious way for Anthropic as a company to benefit from exploiting a bunch of security vulnerabilities more than it will benefit by maintaining public goodwill towards the organization, and it does seem that whatever way people may be thinking feels pretty obvious to them, rather than being devious and convoluted. “Blackmail a bunch of powerful people with information they would prefer not become public” is a strategy an individual could try, but not something that would work as corporate strategy for an organization looking to make billions in revenue, and “hacking --> ransomware/extortion” or “hacking --> blackmail key people --> gain political power” are likewise strategies that make sense for individuals or small groups, but not Anthropic.

Well… actually blackmailing key people to achieve political outcomes is a thing Anthropic could do, and I wouldn’t have evidence it had happened yet. But I don’t expect they would do that, as a company, because I’d think the ratio of people with a conscience who have qualms about blackmail to sociopaths who do not, at Anthropic is such that any sociopaths have to hide most of the time.

Myron Hedderson 4 Apr 2026 13:09 UTC
1 point
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
initial reaction to your several replies today: I feel like writing several replies that would be quite long, but a) I don’t have time to do that within the next few days and b) I don’t want to spam you with walls of text you’re not interested in. I’ll try and refine my thoughts down to something more short and focused, but, how interested are you in reading the longer less-focused version? This is a “I wrote you a long letter because I didn’t have time to write you a short one” situation—longer is easier, but potentially less useful.
“Does Bayesianism say X” is a complex question to answer—there are a cluster of ideas that are implemented among the group here, that I personally have noticed, but I don’t have a canonical source for what “Bayesianism” says or does not say, so I’d be sharing my current impressions and understandings, and as I’ve said before, I’m not an expert Bayesian, and what I think might not be representative of the group.
As for “What kind of event would cause you to update your prior about Bayesism?”… well at the core of this set of ideas is a mathematical formula. And I understand why that formula is what it is (I walked through the reasoning once, years ago, and it made sense). So if the question was “what kind of event would make my believe Bayes’ formula is incorrect” then it would have to be the same sort of thing that would cause me to question the validity of math more generally—something that would make me think “maybe the Pythagorean theorem doesn’t reliably describe characteristics of triangles”. There are things that could do that, but it’d be a pretty fundamental questioning of the nature of reality. Or, I guess if some mathematician found some fundamental flaw in Bayes’ formula and I could walk through their reasoning? But “Bayesianism” and “Bayes’ formula” are not the same thing, and I could give up on various ideas that cluster around Bayes’ formula much more easily. If I saw this group systematically making errors in thinking that I could trace back to a Bayes-adjacent idea, I’d update my thinking based on that evidence fairly easily. What I’ve seen instead is that members of this group using this form of reasoning have reached conclusions that turned out correct, well ahead of society in general coming to the same conclusions. if that changed, I’d learn from that.

When I said “Bayesianism says”, a few messages above, I was just running with your conceptual frame for the sake of discussion—and there are certain things that are implied by Bayes’ formula which I think everyone would agree on, like “if you have a high prior on something happening that doesn’t happen as expected, that causes a big update, and similarly for low priors on something that happens”. Inputting certain numbers into Bayes’ formula means other numbers come out, and in that sense “Bayesianism says/Bayes’ formula says” consistent things. But there are other ideas that kind of come along for the ride and might be grouped under the label “Bayesianism” while being less directly connected to the formula, and it’s there where I go “I may not be the best person to comment on that/it’s a bit fuzzy/complex”.

Myron Hedderson 2 Apr 2026 12:24 UTC
1 point
0
on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
The question is, assuming no knowledge of the scam or communication with the “losers” (for whom a stock prediction was wrong), should the Bayesian—strictly adhering to Bayesian Epistemology—believe that the firm’s stock predictions are legit?
Something else that is relevant to real-life Bayesians occurs to me. “Strictly adhering to Bayesian epistemology” is doing some work here. And in real life, if my reasoning or math leads me off a cliff/to some absurd conclusion, I have to put some weight on the possibility I’ve made an error somewhere, which I haven’t yet found. An idealized Bayesian reasoner could reason perfectly every time and be certain that they had done so, but I am not an idealized Bayesian reasoner, and I know that about myself. So how it would work is, if some long chain of reasoning or math leads me to a very surprising or unusual conclusion, I don’t throw away the possibility that the surprising conclusion is right, and that might even be my highest-probability guess, but I still leave open the possibility I’ve simply made an error. And so I might go “well this is what the math says, but I’m not going to rely on it” for anything that involves making a really important decision.

Myron Hedderson 1 Apr 2026 2:57 UTC
1 point
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
Doesn’t that mean we should expect that Bayesians often disagree and they have no way to resolve it except consulting reality (i.e., an experiment)?
Short answer: Yes.

Longer answer: Two Bayesians who start out with the same prior probabilities, and see the same evidence, should update their posterior probabilities in the same way, and so their mental models should stay consistent with each other. Two Bayesians who start out with different prior probabilities, but see the same evidence, should update their posterior probabilities in ways that are predictable to each other, and in line with the evidence—that is, if one reasoner (A)‘s prior probability that (for example) General Relativity is true was high, while another (B)’s was low, then when an experiment is run which provides evidence for general relativity, A’s estimates of General Relativity’s likelihood of being true will change less than B’s (because B’s priors were more wrong), but both will update in a direction and to an extent that is predictable to either of them. As they see more and more of the same evidence, their models of the world should converge.
This is all assuming an ideal Bayesian reasoner with practically-unlimited computing power who doesn’t cheat or decide not to reason according to Bayesian rules when it becomes inconvenient, and humans don’t meet those constraints. But, there’s math to say how much you should update given particular evidence. So:
Also say there is an experiment, is there any standard or agreement among bayesians about how to weight credence?
Yep. “How to weight credence” is a bit unclearly stated, but there’s Bayes’ formula, which tells you how to update your probabilities based on evidence, and that might be what you’re getting at?
Which is (one reason) why bother with Bayesianism at all. It’s a method of approaching consensus when working under uncertainty. It’s kind of an “agreeing to the rules of the game” situation, where “the rules” are a mathematical equation that says how probabilities must change when people are disagreeing (and “must” here carries the same level of mathematical strength as saying “2+2 must equal 4″, it’s not a thing that was decided by committee) - if for example you say it’s 95% unlikely/5% likely that something will happen under your idea of how the world works, and then it happens, if you’re playing fair, you make a big update, and if you put numbers on it, Bayes’ rule tells you what your new numbers should be. If you don’t like the new numbers, you have to either acknowledge that what you said your priors were was incorrect, or that what you said your likelihood estimates of different outcomes were was incorrect—so either you retroactively revise how you used to think the world works, or you retroactively revise what you thought would happen and how confident you were, both of which are kind of awkward and embarrassing. And the people you’re disagreeing with, if you don’t make an appropriately sized update given what you told them your priors and likelihood estimates were, can point this out as a fact. And if you’re not very confident, or you don’t think particular evidence should carry much weight, you can express that in a way that makes it clear how much you’re going to update based on whatever evidence you see, before you see it, so it isn’t like:
“I think x is definitely wrong, and y will provide strong evidence”
″OK, so y didn’t go how I thought, now I think x is only almost certain to be false”
It’s like:
“I think X is a% likely to be false, and Y experiment will turn out the way I expect with b% likelihood as a result.”
“Oh. Ok, well, I guess now I’m down to c% likelihood that X is false. Shoot.”
And instead of being like “it’s not fair that you moved from definitely to almost certain based on something you said would provide strong evidence but didn’t go the way you expected”, the reaction is “yep, that math is correct, you updated how you should have given your priors”. And before you get to that point, pre-experiement, you can argue over whether b% likelihood is reasonable, where it’s hard to argue about the correct meaning of the word “strong”.

And once you get really familiar with doing this (I’m still not great at it), you know intuitively how much putting X% probability on a particular outcome means you’re going to have to change your views if it doesn’t happen, and you become appropriately cautious (“calibrated”) in your estimates, and your saying things in probabilities conveys a lot of information to other people who are also familiar with talking this way.

All of that is in idealized theory among people who are quite smart and can do lots of calculation in their heads. Lots of people also LARP it and use Bayesian-sounding words without actually having the deeper intuitive understanding of what what they’re saying means.

Myron Hedderson 1 Apr 2026 1:55 UTC
1 point
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
There isn’t a set breakpoint that separates weak from strong priors, it’s a continuum from “it seems extremely unlikely that everything I know about the world is false, but it’s not technically impossible” to “it seems extremely likely that I’m typing on a keyboard right now, but there’s a tiny possibility that something else is going on, like a hallucination or me being a brain in a vat or some other possibility I haven’t thought of”.

Bayesianism says that if you have strong priors about a particular matter, you should be surprised with corresponding strength if something your priors said shouldn’t happen, happens. Some occurrences mean “I adjust what I think slightly, but this was within what I guessed might happen”, while others mean “halt and catch fire—either I should fundamentally rethink my notions of how reality works, or I’ve deeply misunderstood what just happened”.

I haven’t read about CF (I skimmed your link extremely rapidly, which doesn’t count) but it sounds pretty binary. Bayesian thinking is not that.

Myron Hedderson 1 Apr 2026 1:44 UTC
1 point
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
[Unsure] The probability of GR being true is independent of whether the Bayesian knows about it or not^[1]
Keep in mind that these “probabilities” are subjective assessments of probability based on an individual’s prior knowledge, not facts about reality. Two Bayesians with different prior experience may disagree about how probable something is (/seems to them), but reality will not disagree or debate with itself about the truth of the matter, or assign probability to different possibilities (mumble mumble I don’t really understand quantum mechanics and am pretending it doesn’t matter for the purpose of this conversation).

Whether or not General Relativity is true is unaffected by any probabilities any Bayesian may put on its truth or falsehood when reasoning about the evidence they’ve seen so far. But whether or not a particular Bayesian finds General Relativity to be probably true, is definitely affected by whether they know about it or not. Keep a clear distinction in your mind between “the probability a Bayesian reasoner assigns to some fact being true” and “whether or not that fact is true in reality”—these are not the same.
This is a “map vs. territory” distinction. Bayesian probabilities go into a mental model of how the world works (map), while how the world actually works is separate from the map.

Myron Hedderson 1 Apr 2026 1:03 UTC
1 point
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
If you have weak/uncertain priors, the thing to do is run low-cost experiments that differentiate between your different hypotheses of what’s going on.

A real cheap experiment in relation to the question “is this a scam?” is to Google to see if others have received similar letters and what their outcomes were. If it’s a scam, you’re likely to surface evidence of this, if it’s not a scam, you’re likely to surface both people saying it went well for them, and debunking sites that explain what’s going on, where the letters came from, etc. If you get no information at all based on a google search, just no results come back, someone went to a lot of effort to make that be the case, and you should be suspicious, it’s evidence something’s amiss.

In a case where you’re at the forefront of scientific discovery, there may be no cheap tests available, but you still devise tests which you predict, based on what you currently know, will go one way if a theory is correct, another way if it’s incorrect, and see what happens.

Myron Hedderson 31 Mar 2026 22:12 UTC
4 points
2
on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
I’m not an expert Bayesian, and it’s not part of my identity which I would feel the need to defend by going “here’s why I wouldn’t get scammed”, but I know how I would answer from a “modify your expectations in light of new evidence” lens, which I understand to be the core of Bayesianism if put into plain English.
The key thing is, what are your priors?
If you were a very naive Bayesian reasoner, say a 5 year old of average intelligence, and your experience was extremely sheltered, skewed towards a very kind world where everyone was always nice to you and you weren’t really aware that scams were a thing that happens sometimes, and you didn’t know anything about how stock prices worked, you might be taken in. Because your probability that someone really could predict the way these messages indicated would be “I dunno, seems unlikely, but maybe?”
But as an adult human with the priors I have, here’s how I’d think of it if I received such a letter. All numbers are roundish and hand-wavy.
Probability it’s a scam, excluding marketing spam, donation requests, and surveys, given it’s an unsolicited message in any communication medium, selected randomly: I dunno, 70%? There’s a reason I don’t pick up calls from people who aren’t in my contact list any more, if someone I don’t know wants to reach me, they can leave a message and let me think before responding.
Probability it’s a scam, given the message is unsolicited and it’s from someone I don’t know, a business I don’t do business with, or is anonymous: 80%.
Probability it’s a scam, given the message is unsolicited, from someone I don’t know, and it’s even slightly odd in any way: 90%.
Probability it’s a scam, given the message is unsolicited, from someone I don’t know, it’s odd, and it involves me spending money to make more money later: Basically 100%. At this point, I don’t care what it says, it’s a scam and it’s getting deleted or recycled. It’s not worth my time or mental effort to evaluate the claims it contains.
Additionally and separately, but potentially relevant given the content of the message: Probability I do not have an anonymous benefactor who wants me to be rich for the sole reason that that would make them happy or discharge a duty they feel they have: 100%. I mean, not literally mathematically 100%, but close enough, I’m as sure of this as I am of almost anything, aside from things I can verify with my senses directly like “I have 5 fingers on each hand”.

Given those priors, unlike those of a sheltered 5 year old, it’s hard for a scam to get through.
If I got an unsolicited message from someone I don’t know making a stock prediction, I’d be like “that’s weird, it’s almost certainly a scam”. If they wanted me to spend money, I’d be like “definitely a scam” and throw it out, and have forgotten about it one month later.
If I got a second one, I’d check online to see what the nature of the scam was, because it is somewhat odd to get mail scams, those cost a dollar per message to run, and given my field of work friends and family sometimes check with me to see if things are legit, so I’d want to know what the deal was. I hadn’t heard of the Mail-Order Prophet scam before, so if I was sent a piece of mail trying to draw me in before having read this on LessWrong, I’d be like “neat, new category of scam!”
A similar situation, to demonstrate that I’m not BSing about how I’d react to unsolicited nice things happening from unexpected benefactors: One time, I got a piece of mail saying there was a package for me at the nearby gas station. Which was super weird, I didn’t know gas stations took packages. But OK. So I go to the gas station, and the guy behind the desk was like “were you expecting a TV?”, to which the response was “No”. But, there was a TV there for me, new in box. I was very confused about what the scam was in this case, but I was like “well OK, I guess I’ll take it”. And I looked for contact information or some other indication of who had sent this, and the only thing was a tech support/setup help number. So I called that, and was like “so this seems really scammy, what’s the deal?” And they explained that these TVs were sent to people who opened new bank accounts at a local bank, as a part of a promotional offer. And I had in fact opened a new bank account, because I needed to, several weeks prior, and nobody had told me about the promotional offer. So I was still sufficiently concerned to call my bank and confirm, rather than trusting the word of the person on the other end of the contact information on the unsolicited TV’s box. The bank did confirm, and only then did I plug it in and set it up.