MichaelStJules

Karma: 648

MichaelStJules 28 Jul 2026 1:44 UTC
2 points
0
in reply to: Vasco Grilo’s comment on: antimonyanthony’s Shortform
Consequentialism should guarantee the best (expected) outcomes for the consequentialist. That’s the whole point. If Hammond’s version doesn’t do that, then it seems like a bad version. That being said, his kind of consequentialist can work, you just need to be willing to assign higher value to actually paying in town, say by caring about keeping promises even if you’d predict with your current values you’d rather break them later.
For me the upshot of the thought experiment is that I am basically certain to stay in the desert. I will never be able to commit 100 % to paying in town. The simple fact that I would have to think about commiting is a sign that the chance of not paying would be above 0. So a driver who only drove me to town if they were certain I would pay would certainly leave me in the desert. So, regardless of how much I commit, I think I would be certain to end up in the desert.
Let’s assume the driver will drive you if they predict you’re at least 80% likely to pay. Maybe you can’t fully guarantee a ride, but you can increase the probability by at least temporarily modifying how you make decisions, e.g. by caring more about keeping promises.
Anyway, here are my takeaways from Parfit’s hitchiker:
1. If sticking to your generally preferred decision rule leads to worse outcomes in some cases, you should consider modifying your rule, e.g. by caring about keeping promises for their own sake.
2. If you’re okay accepting that your rule will sometimes lead to predictably worse outcomes than another rule, then this at least partly undermines all dominance based arguments, including money pumps.
1 and 2 undermine Elga’s arguments against unsharp credences.

MichaelStJules 27 Jul 2026 5:15 UTC
2 points
0
in reply to: Vasco Grilo’s comment on: antimonyanthony’s Shortform
Fair that my proposal violates Hammond’s conception of consequentialism, and that conception is at least a very plausible-at-first-glance conception of consequentialism: only the consequences matter, not the structure of the decision tree.
However, my proposal and argument seem compatible with a broader conception of consequentialism, and indeed your “what matters are the consequences of my actions given the current state of the world regardless of how I arrived at the current state”. We accept bet A (rule out rejecting bet A) and rule out the non-dominated option [reject A, accept B] for entirely consequentialist reasons: to prevent ourselves from rejecting both bets, which would be dominated in terms of consequences by accepting both bets. The ends, i.e. avoiding consequence-dominated sequences, justify the means, i.e. ruling out both a dominated option and an undominated option together. In the alternative where we just pick one of the 4 options directly in one step, ruling out [reject A, accept B] doesn’t help us avoid rejecting both bets, the dominated option.
If the driver is able to predict what I will do in town with perfect accuracy, and they will only drive me to town if I pay, then I will pay in town. Not paying would go against the perfect accuracy of the driver, and therefore the thought experiment would not be coherent, as the driver cannot have and not have perfect accuracy? As far as I understand, if I was driven to town, paying would be the only option I would consider.
Yes, you can’t not pay if you got a ride, assuming perfect accuracy. But this doesn’t solve the problem (guarantee a ride to town, which is presumably better). Before the driver decides whether or not to drive you, they predict that if they drove you, you paying would be worse to you once you got into town and feasible, so you wouldn’t pay. So, they don’t drive you, and you end up in the worst case.
That is unless unless you change how you think about the utilities of each option, give up consequentialism or otherwise find a way to commit to preventing yourself from not paying in town. And then all of these kinds of solutions are available to those with unsharp credences in the two bet problem by Elga.

MichaelStJules 23 Jul 2026 17:42 UTC
2 points
0
in reply to: Vasco Grilo’s comment on: antimonyanthony’s Shortform
I just have a strong intuition that what matters are the consequences of my actions given the current state of the world regardless of how I arrived at the current state.
This seems completely consistent with the backward induction argument I made in the second half of my comment here.
I think my thoughts are compatible with the premises.
Conditional on being in town, you wouldn’t pay, because this is lower EV. It doesn’t matter how you got there (by your own claim). By consistency in subtrees (which you claim), this means outside of town, ahead of time, you won’t pay even if you get to town (unless you can force yourself to override your later local choice). So you will be predicted not to pay, and you will be left stranded, which is worse than getting into town and paying.
If it helps, build the decision tree.
Elga’s paper supports my belief in sharp expected values (EVs), including EVs of distributions describing probabilities.
Are you saying you believed in sharp credences primarily because of Elga’s arguments?

MichaelStJules 22 Jul 2026 18:32 UTC
2 points
0
in reply to: Vasco Grilo’s comment on: antimonyanthony’s Shortform
Terminal nodes have consequences in a given domain [unrestricted domain]. Behaviour is required to be consistent in subtrees [dynamic consistency]. Consequentialist behaviour, by definition, reveals a consequence choice function independent of the structure of the decision tree [consequentialism].
Why do you believe behaviour should be consistent in subtrees? Why can’t just the fact that the trees are different be enough to permit (but not necessarily require) different behaviour? I think this premise is basically assuming away our responses, so it’s doing a lot of work.
There could be specific differences we could point to to justify different behaviour, like the argument I made here, which does not apply to each bet on its own.
EDIT: Actually, I’m not sure how the premises relate to my argument, since IIUC, Hammond’s model is that outcomes are in the terminal nodes, and those differ between the isolated bets and the two bets in sequence.
And what do the premises together imply for Parfit’s hitchhiker and this decision problem with St Petersburg lotteries? Are those acceptable conclusions to you?
I wrote “a complete [sharp]” in my last comment. I do not think completeness alone implies sharp probabilities. One can collapse unsharp probabilities to sharp ones using something like the midpoint rule discussed in Adam Elga’s paper. However, assuming one rejects rules like this, completeness leads to sharp probabilities?
The space of potential rules could be huge, so that could be assuming quite a lot.
Completeness is one of Savage’s axioms, which together imply sharp probabilities. Completeness probably does a lot of the work, but other parts may be important, too. (I’m not that familiar with Savage’s axioms or theorem, though, so I wouldn’t be able to explain much more with reading more myself.)
I did not know about Hammond (1988) before linkposting Adam’s paper. I found it thanks to this comment from Simon on my linkpost. I would still have endorsed the premises if I had read about them before publishing the post.
Why did you believe in sharp credences when you linkposted Elga’s paper? Do you no longer endorse those reasons as important for your beliefs?
I do not feel like I have been making arguments. My recollection is that I have mostly been quoting Claude, and asking questions.

I lightly endorsed Adam’s claim that “Sharp does not entail Uniqueness” before linkposting the paper. I did not think much about it before linkposting. A comment from Jim Buhler prompted me to think a bit more about it, and come to the view that “I endorse sharpness and uniqueness. As far as I can tell, the issues of unsharp probabilities would apply in the same way to non-unique probabilities”. There is more context about my views about uniqueness in my reply to Jim.
So when you’ve quoted Claude to us (including the summary post of Elga’s paper itself), say the answer “makes sense” and/or ask for our reactions, you didn’t endorse any of what it says?

MichaelStJules 22 Jul 2026 14:46 UTC
2 points
0
in reply to: Vasco Grilo’s comment on: antimonyanthony’s Shortform
Could you answer my other questions, too?
Can you spell out these 3 premises explicitly? I’m not sure what you’re referring to.
And did you know about and endorse these 3 premises when you posted Elga’s paper on the EA Forum?

MichaelStJules 22 Jul 2026 13:08 UTC
5 points
3
in reply to: Vasco Grilo’s comment on: antimonyanthony’s Shortform
I don’t understand. Anthony already addressed Claude’s 4th point above:
Following (3): There are positive epistemic motivations for imprecise credences. So yes, Elga is asking the impreciser to override what they consider locally (epistemically) rational.
More explicitly:
So “adopt sharp credences” means “have a state that doesn’t need forcing” — which is the opposite of forcing.
For someone who finds imprecise credences locally (epistemically) rational, “have a state that doesn’t need forcing” requires forcing themself into that state, and overturning the verdicts that would be implied (or allowed) by imprecise credences.
I fear what’s going on here and in the EA Forum thread is that you’ve picked a position and you’re prompting Claude in a way that will defend the position even if the arguments are weak, applied unfairly or ignore important context, and then pick and present any remaining argument that seems plausible to you. But Claude can generally quite easily make plausible-sounding arguments that are actually bad, and it can keep doing so ad nauseum.^[1] This can mean being more exhaustive of the space of arguments and making sure each one is addressed, but it’s also exhausting and frustrating to deal with, because it seems like it will never end and it’s indistinguishable from motivated reasoning.
So, I think answering these would help:
1. What exactly is the position you’re defending? Has it changed throughout these discussions (in this thread or on the EA Forun post)?
2. Are there any cases in these discussions where you think an argument you made or endorsed turned out wrong, or a counterargument was correct? And if so, how has that affected your credences in the position you’ve been defending?
3. After all of these discussions, in your own words, what are your reasons for believing in sharp credences (or whatever position you’re defending)?
1. ^
  And this is why I recommend you filter through a round of exhaustive critique in another session, including checking if the arguments are being applied unfairly or have been already addressed, and for counterarguments. Better yet to do this with an LLM from another family.

MichaelStJules 27 Apr 2026 18:46 UTC
50 points
18
in reply to: habryka’s comment on: Forecasting is Way Overrated, and We Should Stop Funding It
What decisions do you think this has affected and what would you estimate the differences in outcomes to be as a result? Or, say, the most important impacts?

MichaelStJules 16 Sep 2025 14:39 UTC
3 points
0
on: Shutdownable Agents through POST-Agency
Cool direction and results!
Can we prevent such an agent from having a preference to create agents that do resist shutdown?
EDIT: And if they’re going to create agents anyway, actually make sure those agents don’t resist shutdown, too, rather than, say, being indifferent about whether those other agents resist shutdown.

MichaelStJules 24 Aug 2025 1:09 UTC
3 points
0
in reply to: Lukas Finnveden’s comment on: Winning isn’t enough
Would some version of this still work if you have imprecise credences about the signs (and magnitudes) of considerations you’ll come up with, rather than 50-50 (or some other ratio)?
Even if not 50-50 but precise, we could adjust the donation amounts to match the probabilities and maintain the expected amount donated at $1000.
But if the probabilities are imprecise, I don’t think we can (precisely) maintain the expected donation amounts. We could pick donation amounts such that $1000, <$1000 and >$1000 are all possible expected donation amounts, in our set of credences (representor).

MichaelStJules 11 Jan 2025 19:52 UTC
1 point
0
in reply to: Pablo’s comment on: Actualism, asymmetry and extinction
FWIW, users can at least highlight text in a post to disagree with.
What links here?
- Pablo's comment on Actualism, asymmetry and extinction by MichaelStJules (11 Jan 2025 23:03 UTC; 4 points)

MichaelStJules 9 Jan 2025 15:00 UTC
10 points
0
in reply to: Noosphere89’s comment on: Quick general thoughts on suffering and consciousness
Interesting! Graziano’s Attention Schema Theory is also basically the same: he proposes consciousness to be found in our models of our own attention, and that these models evolved to help control attention. To be clear, though, it’s not the mere fact of modelling or controlling attention, but that attention is modelled in a way that makes it seem mysterious or unphysical, and that’s what explains our intuitions about phenomenal consciousness.^[1]
Graziano thinks mammals, birds and reptiles are conscious, is 50-50 on octopuses and very skeptical of fish and arthropods.
1. ^
  Graziano, 2021:
  In the attention schema theory (AST), having an automatically constructed self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. The reason why such a self-model evolved in the brains of complex animals is that it serves the useful role of modeling, and thus helping to control, the powerful and subtle process of attention, by which the brain seizes on and deeply processes information.
  Suppose the machine has a much richer model of attention. Somehow, attention is depicted by the model as a Moray eel darting around the world. Maybe the machine already had need for a depiction of Moray eels, and it coapted that model for monitoring its own attention. Now we plug in the speech engine. Does the machine claim to have consciousness? No. It claims to have an external Moray eel.
  Suppose the machine has no attention, and no attention schema either. But it does have a self-model, and the self-model richly depicts a subtle, powerful, nonphysical essence, with all the properties we humans attribute to consciousness. Now we plug in the speech engine. Does the machine claim to have consciousness? Yes. The machine knows only what it knows. It is constrained by its own internal information.
  AST does not posit that having an attention schema makes one conscious. Instead, first, having an automatic self-model that depicts you as containing consciousness makes you intuitively believe that you have consciousness. Second, the reason why such a self-model evolved in the brains of complex animals, is that it serves the useful role of modeling attention.

Some implications of radical empathy

MichaelStJules7 Jan 2025 16:10 UTC

3 points

0 comments7 min readLW link

Actualism, asymmetry and extinction

MichaelStJules7 Jan 2025 16:02 UTC

8 points

4 comments9 min readLW link

Really radical empathy

MichaelStJules6 Jan 2025 17:46 UTC

19 points

0 comments10 min readLW link

Better difference-making views

MichaelStJules21 Dec 2024 18:27 UTC

9 points

0 comments14 min readLW link

MichaelStJules 20 Dec 2024 22:35 UTC
2 points
0
in reply to: transhumanist_atom_understander’s comment on: Iron deficiencies are very bad and you should treat them
Also oysters and mussels can have a decent amount of presumably heme iron, and they seem unlikely to be significantly sentient, and either way, your effects on wild arthropods are more important in your diet choices. I’m vegan except for bivalves.

MichaelStJules 14 Oct 2024 6:56 UTC
3 points
0
in reply to: Nathan Helm-Burger’s comment on: LLMs are likely not conscious
Since consciousness seems useful for all these different species, in a convergent-evolution pattern even across very different brain architectures (mammals vs birds), then I believe we should expect it to be useful in our homonid-simulator-trained model. If so, we should be able to measure this difference to a next-token-predictor trained on an equivalent number of tokens of a dataset of, for instance, math problems.
What do you mean by difference here? Increase in performance due to consciousness? Or differences in functions?
I’m not sure we could measure this difference. It seems very likely to me that consciousness evolved before, say, language and complex agency. But complex language and complex agency might not require consciousness, and may capture all of the benefits that would be captured by consciousness, so consciousness wouldn’t result in greater performance.
However, it could be that
1. humans do not consistently have complex language and complex agency, and humans with agency are fallible as agents, so consciousness in most humans is still useful to us as a species (or to our genes),
2. building complex language and complex agency on top of consciousness is the locally cheapest way to build them, so consciousness would still be useful to us, or
3. we reached a local maximum in terms of genetic fitness, or evolutionary pressures are too weak on us now, and it’s not really possible to evolve away consciousness while preserving complex language and complex agency. So consciousness isn’t useful to us, but can’t be practically gotten rid of without loss in fitness.
Some other possibilities:
1. The adaptive value of consciousness is really just to give us certain motivations, e.g. finding our internal processing mysterious, nonphysical or interesting makes it seem special to us, and this makes us
  1. value sensations for their own sake, so seek sensations and engage in sensory play, which may help us learn more about ourselves or the world (according to Nicholas Humphrey, as discussed here, here and here),
  2. value our lives more and work harder to prevent early death, and/or
  3. develop spiritual or moral beliefs and adaptive associated practices,
2. Consciousness is just the illusion of the phenomenality of what’s introspectively accessible to us. Furthermore, we might incorrectly believe in its phenomenality just because of the fact that much of the processing we have introspective access to is wired in and its causes are not introspectively accessible, but instead cognitively impenetrable. The full illusion could be a special case of humans incorrectly using supernatural explanations for unexplained but interesting and subjectively important or profound phenomena.
What links here?
- Michael St Jules 🔸's comment on Why I Think All The Species Of Significantly Debated Consciousness Are Conscious And Suffer Intensely by Bentham's Bulldog (EA Forum; 20 Nov 2024 18:09 UTC; 8 points)
- Michael St Jules 🔸's comment on Why I Think All The Species Of Significantly Debated Consciousness Are Conscious And Suffer Intensely by Bentham's Bulldog (EA Forum; 21 Nov 2024 1:04 UTC; 2 points)

MichaelStJules 16 Aug 2024 5:37 UTC
1 point
0
in reply to: Dagon’s comment on: Utilitarianism and the replaceability of desires and attachments
Sorry for the late response.

If people change their own preferences by repetition and practice, then they usually have a preference to do that. So it can be in their own best interests, for preferences they already have.

I could have a preference to change your preferences, and that could matter in the same way, but I don’t think I should say it’s in your best interests (at least not for the thought experiment in this post). It could be in my best interests, or for whatever other goal I have (possibly altruistic).

In my view, identity preservation is vague and degreed, a matter of how much you inherit from your past “self”, specifically how much of your memories and other dispositions.