JBlack

Karma: 2,335

JBlack 28 Aug 2025 3:17 UTC
4 points
2
in reply to: AdamLacerdo’s comment on: Would you sell your soul to save it? ( I am NOT a Christian)
Then there’s a HUGE hole in the scenario right where all the load-bearing assumptions are.
What sort of evidence convinced the person in the scenario that Christianity is actually with probability greater than 0.99 correct, and definitely less than 1% chance of every other scenario combined? How do you distinguish this from the whims of a powerful being who can read minds and do lots of other stuff, forcing people to accept that it is correct on pain of eternal death or torture? The only difference here from literal Christianity seems to be the idea that the powerful being is good and just in doing that, so what evidence made the person accept that?
I’m assuming that the powerful being can’t (or won’t) write minds as well, since being mind-probed into belief is not really that interesting.

JBlack 26 Aug 2025 1:43 UTC
2 points
0
in reply to: Heighn’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
Why do you believe that I believe that this is a scenario in which you are definitely not in a simulation? I do not.
I am saying that in general, FDT does not require that the agent must reason about whether they are in a simulation. In the published Bomb scenario in particular it is not stated whether the agent may be in a simulation, and it is also not stated whether the agent knows or believes that they may be in a simulation. In principle, all these combinations of cases must be considered separately.
Since the scenario does not make any statement in this respect, I do believe that it was not intended by the scenario author that the agent should reason as if they may be in a simulation. That would be just one of infinitely many unstated possibilities that might affect the analysis if they were considered, all of which would complicate and detract from the issue they intended to discuss.
So I do believe that the analysis described in the original scenario was carried out for an agent that does not consider whether or not they may be in a simulation, as distinct from them actually being definitely not in a simulation.
After all, the fact of the matter is that they are in a simulation. We are simulating what such an agent should do, and there is no “real” agent in this case.

JBlack 25 Aug 2025 0:45 UTC
2 points
0
in reply to: Arjun Panickssery’s comment on: Arjun Panickssery’s Shortform
Partly for the reasons outlined in my comment here. Mainly the following section:
Even under the most hardcore determinism and assuming immutable agents, they can be classified into those that would and those that wouldn’t have performed that act and so there is definitely some sort of distinction to be made.
In another comment (that I’m not finding after some minutes of search) I outline why this distinction is one that should be (and is) called moral culpability for all practical and most philosophical purposes. The few exceptions aren’t relevant here, since even one counterexample renders the argument invalid.

JBlack 25 Aug 2025 0:31 UTC
2 points
0
in reply to: David_Bates’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
It’s a separate issue entirely. Scenarios in which you could be in a temporary simulation (especially in which the simulation outcome may be used to determine something for another instance of your decision process) are different and should be analysed differently from those in which you are definitely not.

JBlack 24 Aug 2025 1:32 UTC
2 points
0
in reply to: David_Bates’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
No, it does not. However MacAskill did not correctly perform the FDT analysis.

JBlack 24 Aug 2025 1:28 UTC
2 points
0
in reply to: gjm’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
What you’re missing is the F in FDT: the agent is assumed to have a function that maps their available information to a decision. In a note-less scenario, every possible agent has the same information available, and so there are only two possible functions: one that maps to Left, and one that maps to Right. FDT then says that Left performs better.
Once you introduce the possibility of a note, there are now (at least) three values in the domain: no note, a note saying the prediction was Left, and a note saying the prediction was Right. MacAskill assumed that this doesn’t change the FDT decision, but of course it does: there are now at least 8 (and possibly infinitely many) such functions to compare, not just 2, and unstated distributions over the mapping between inputs and outputs. In almost all of the distributions, FDT will in these new and different scenarios recommend picking Right.

JBlack 24 Aug 2025 1:17 UTC
2 points
0
in reply to: Vladimir_Nesov’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
In my scenario there is probability pT that the note is truthful (and the value of pT is assumed known to the agent making the decision). It is possible that pT = 1, but only for pN < 10^-24 so as to preserve the maximum 10^-24 probability of the predictor being incorrect.

JBlack 24 Aug 2025 1:15 UTC
2 points
0
in reply to: Gurkenglas’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
In my scenario, there is a probability pN that the predictor leaves a note, regardless of what prediction the predictor makes. The note (when present) is always of the form “I predicted that you will pick the <right|left> box, and therefore <did|did not> put a bomb in the left box.” That is, the note is about what the predictor thinks you will do, which more closely matches your second paragraph. Your first paragraph concerns a prediction about what you counterfactually would have done in some other situations and is not relevant in this scenario.
However, your decision process should consider the probability 1-pT that the note is lying about the predictor’s actual prediction (and therefore bomb-placing).

JBlack 23 Aug 2025 11:26 UTC
2 points
0
in reply to: Gurkenglas’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
That would have been a different set of example scenarios. In this one, I chose one in which the predictor had fixed probabilities. I am not assuming any particular means by which the predictor arrives at the prediction (such as simulation), and don’t care about anything to do with self-locating uncertainty (such as being a conscious entity created by such a simulation).
Feel free to follow up with FDT analysis of a different scenario, if you prefer it.
Though I will note that in a scenario where the agent is uncertain of the actual constraints of the scenario, the conclusion is almost always simpler: take the right box every time, because even a tiny fraction of uncertainty makes the left box too unsafe to be worth taking. The more moving parts you introduce into a scenario with this basic premise, the more likely it is that “always take the right box” is the correct decision, which is boring.

JBlack 22 Aug 2025 3:12 UTC
2 points
1
in reply to: Richard Korzekwa ’s comment on: Arjun Panickssery’s Shortform
Yes, an obvious flaw is that 1 is obviously false. Though also 2 is false depending upon exactly how you view the term “a person”.

JBlack 22 Aug 2025 2:11 UTC
2 points
0
in reply to: J Bostock’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
Example scenario extended to allow notes
Suppose that when the agent enters the room, a note is either Absent, states that the Left box was predicted (and there’s no bomb in it), or that the Right box was predicted (and therefore there’s a bomb in Left). There is probability pN of the note being present in both prediction cases, and probability pT of it being truthful in both cases where it is present. (A more complete analysis would have these be different probabilities, and a consideration that they may be adversarially selected to extract the most damage from participants or to minimise the recorded rates of error of the predictor)
For simplicity, the probabilities of incorrect prediction in every combination of functional inputs and outputs are all B, which according to the scenario is very small, less than 10^-24. Again, in a more complete analysis one should consider different rates of error under different decision scenarios and possible optimization pressures.
The simplest decision function then maps type of note one sees mapped to which box to take, although a more complete analysis would permit mixed strategies. There are eight such functions instead of the two in the no-note scenario.
In particular, there is a possible function F(A) = L, F(L) = R, F(R) = L. In this case, the predictor leaves a truthful note with probability pN pT, and the prediction was wrong because the agent does the opposite. By the scenario constraint B < 10^-24, we know that the product pN pT < 10^-24.
So it is inconsistent with the scenario that the predictor always leaves a truthful note. The predictor can either be forced into error with probability very much greater than 10^-24 which violates the scenario premises, or almost always does not leave a truthful note. It is also possible that the predictor could arrange that the agent is unable to decide in this way, but that’s very much outside the bounds of decision theory.
It is still possible that pT is close to 1, but less than 1 - U(-$100)/U(death). In this case, pN ~= 10^-24 and your optimal decision function is F(A) = L, F(L) = R, F(R) = R. That is, take the left box unless you see a note saying Right, in which case you should pick Right. If you see no note then with probability at least 0.999999999999999999999999 you will save $100 and not burn to death. In the extremely unlikely case that you see a note, you pick the right box even if it says Left, because you can’t be sufficiently confident that it’s true.
If pT >= 1 - U(-$100) / U(death), then the optimum is F(A) = L, F(L) = L, F(R) = R because now you have near certainty that the note is truthful and the risk of picking the left box based on the note is now worth it.

JBlack 22 Aug 2025 0:20 UTC
2 points
0
in reply to: J Bostock’s comment on: Critiques of FDT Often Stem From Confusion About Newcomblike Problems
Yes, this is very much correct. It is invalid reasoning to carry out an FDT decision analysis on a different scenario (without the possibility of a note), claim that you should use the result of that on this one (with the note), and then claim that FDT is a bad decision theory as a result.
The possible inclusion of a note and its contents adds an unknown number of bits of information to the input of the function in FDT, which obviously changes the analysis.
The agent would need extraordinarily strong information about when a note appears and the bounds on its probability of truthfulness in order to decide to take the left box under FDT here (and almost certainly not burn to death because the note is almost certainly lying). Otherwise FDT dictates that you must take the right box.
The exact bounds depend upon the conditions under which a note can appear and what its content can be. If they are not specified then you would need to apply some prior distribution, which unless you are an extremely bizarre person with extremely bizarre priors or utility function, will lead to FDT recommending that you take the right box.

JBlack 21 Aug 2025 0:40 UTC
5 points
1
in reply to: Alex_Altair’s comment on: Alex_Altair’s Shortform
Yes, pretty much it. Concern with path-connectedness is equivalent to considering the transitive closure of the relation.

JBlack 14 Aug 2025 1:35 UTC
6 points
2
in reply to: lsusr’s comment on: Enlightenment AMA
That said, I’ve never heard of someone being born into this condition.
How would they, and therefore you, know?

JBlack 14 Aug 2025 1:17 UTC
5 points
2
in reply to: David Johnston’s comment on: MIRI’s “The Problem” hinges on diagnostic dilution
Empirically, most uses of values of 32-bit integers (e.g. in programming in general) have magnitudes very much smaller than 2^30. A significant proportion of their empirical values are between 0 and 10. So my prior would be very much non-uniform.
I would expect something like a power law, possibly on the order of P(X >= x) ~ x^-0.2 or so except with some smaller chance of negative values as well, maybe 10-20% of that of positive ones (which the statement “nonnegative” eliminates), and a spike at zero of about 20-40% of total probability. It should also have some smaller spikes at powers of two and ten, and also one less than those.
This would still be a bad estimate for the number of children, once I found out that you were actually talking about children, but not by nearly as much.

JBlack 12 Aug 2025 3:32 UTC
2 points
0
in reply to: Adam Newgas’s comment on: Adam Newgas’s Shortform
They should learn both, since it seems to me that both are true.

JBlack 6 Aug 2025 0:47 UTC
11 points
9
in reply to: Richard_Kennaway’s comment on: “Momentism”: Ethics for Boltzmann Brains
Boltzmann brains (in universes that permit them) will be overwhelmingly dominated by the least improbable of all possible methods of formation. Matter is sticky, so that once partly assembled it has a much higher probability of continuing to remain assembled especially at the very low temperatures usually envisaged.
There are far more paths for slow formation of assemblies of matter in such an environment than instant ones. To randomly assemble a trillion correct atoms in exactly the right place at the same time is much less likely than accumulating a trillion atoms in the wrong places which then migrate through any of an immense number of possible products of paths to the right places over different times.
It seems very much more likely that the modal Boltzmann brain has had essentially all of its matter in a pre-functioning state for far longer than mere geological timescales. It will almost certainly not be biological matter due to every atom contributing orders of magnitude less probability of formation. I would guess the most likely form to be some sort of dense reversible computing substrate that will operate very near absolute zero, and for a conscious brain of this type to be immensely more common in the space of thermal configurations than even one cell of familiar biological life.

JBlack 5 Aug 2025 1:51 UTC
3 points
0
on: 🫵YOU🫵 get to help the AGI Safety Act in Congress! This is real!
This is a rare, high-leverage juncture: a member of Congress is actively writing a bill that could (potentially) fully stop the risk of unaligned AGI from US labs.
It would, if wildly more successful than any law in human history has ever been, stop a very small fraction of the risk.

JBlack 2 Aug 2025 3:15 UTC
0 points
0
on: Three Quotes on Transformative Technology
~~Three~~ Four Quotes on Transformative Technology

JBlack 31 Jul 2025 5:44 UTC
1 point
2
on: Low P(x-risk) as the Bailey for Low P(doom)
What does probability of risk actually mean, whether it’s x-risk or other types of risk? It seems like a fundamentally confused concept. Risk isn’t an outcome. Doom is at least an outcome, even if it’s not a particularly well-specified one in some cases.

JBlack

Example scenario extended to allow notes