JBlack

Karma: 2,538

JBlack 29 Apr 2026 2:12 UTC
5 points
1
on: Microsoft AI CEO’s “Seemingly Conscious AI Risk”
treating them like property when they are in fact conscious, could lead to disaster
This doesn’t actually hinge on whether they are conscious or not.
Treating them like property when they are not conscious, but are very good at mimicking the behaviour of conscious beings that do not like to be treated like property, could lead to disaster in exactly the same way.

JBlack 27 Apr 2026 0:31 UTC
2 points
0
in reply to: Canaletto’s comment on: 5 thought experiments on identity and copies
It would have to wipe the memory of everything from that time onward, not just memory of the flash itself. I would also need to be immobilized from before the flash to after the memory alteration otherwise I would almost certainly notice a difference in position, and also tamper with my time sense so that I don’t realize that my memory only covers 5 seconds since entering the room instead of 10 seconds or something. But yeah, someone who can make a basically perfect copy of me can reasonably do all these sorts of things as well.
Rationality (e.g. Bayesian updates etc) as usually defined doesn’t work in the presence of mind alterations (e.g. implanting updates that are not rational), so that makes it rather difficult to develop criteria for what one “should” expect.
If you define it as “winning”, then expecting always to be the original when one has no evidence to the contrary always wins for the original. The scenario only asks about the original, and so this is a winning strategy.

JBlack 24 Apr 2026 3:46 UTC
2 points
0
on: 5 thought experiments on identity and copies
I don’t get scenario 5. Which of the statements about the scenario do you actually know? In the others it’s clear that the participant knows everything about the scenario. But in Scenario 5, knowing everything about the scenario makes it trivial.
The original knows that they are the original, and nothing changes that. They already knew that they would have false memories of light colour implanted so seeing a red light in conjunction with memories of an hour of red light is not evidence against it. The copy knows that they are the copy as soon as they see the red light for the first time, and nothing changes that. So each of the instances knows which room they’re going to walk out from.

JBlack 23 Apr 2026 0:42 UTC
4 points
7
on: Marginal Risk is BS
First, there are only three to ten frontier AI developers, depending on how you count. So naively, the marginal risk from one developer should only be 3-10 times lower than the total risk.
I don’t think this holds. Suppose that there are three developers, and each of them is independently 80% likely to develop ASI up to the level where it could kill everyone. Each of them is 10% likely by that point to instill a sufficient degree of alignment such that it won’t actually do so. Each of them is roughly equally likely to achieve the deadly level of capability first, and the fate of the world depends upon whether that particular one is sufficiently well aligned—if so, it will sufficiently prevent more misaligned ASI arising in the future.
Despite its oversimplified nature, this is clearly a terrible scenario. There is a > 99% chance that ASI is developed, with an overall 89.28% chance of doom. If there were only two developers, then there would be 96% chance of ASI with a 86.4% chance of doom.
So the marginal risk of doom due to having 3 developers instead of 2 is only 2.88%, which is very much less than ¹⁄₃ of the total risk. The great majority of the marginal risk was in the 0 → 1 transition.
As you say, a major problem is that such a calculation of marginal risk ignores the possibility of coordination, and also ignores the increased likelihood of defection from having more parties involved.

JBlack 14 Apr 2026 0:43 UTC
28 points
1
on: Annoyingly Principled People, and what befalls them
The most salient example I’ve seen of an Alice in these circles was very publically banned not long ago.

JBlack 7 Apr 2026 0:09 UTC
3 points
1
on: Contra The Usual Interpretation Of “The Whispering Earring”
This view appears to directly contradict the lines “Better for you if you take me off” in conjunction with “The earring is always right”.

JBlack 4 Apr 2026 5:36 UTC
2 points
0
in reply to: shakelush’s comment on: Newcomb’s Paradox Simulation
The problem isn’t in the simulation part, but in the “supports” part.
You can certainly write a simulation in which an agent decides to take both boxes. By the conditions of the scenario, they get $1000. Does this simulation “support” taking both boxes? No, unless you’re only comparing with alternative actions of not taking a box at all, or burning box B and taking the ashes, or other things that are worse than getting $1000.
However, the scenario states that the agent could take 1 box, and it is a logical consequence of the scenario setup that that in the situations where they do, they get $1000000. That’s better than getting $1000 under the assumptions of the scenario, and so a simulation that actually follows the rules of the scenario cannot support taking two boxes.

JBlack 4 Apr 2026 5:27 UTC
2 points
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
There aren’t just two possibilities “ideal bayesian reasoning” and “useless rubbish”. There is a huge range of heuristics, ad-hoc models, evolved instincts, and everything else in the mix. These are all ‘outside bayesianism’ and while the collection is almost certainly worse than ideal bayesian reasoning, they are not useless.
That also doesn’t mean that we can only improve by having more people actively do bayesian reasoning about stuff, though there are certainly many cases where people would be better off actively doing bayesian reasoning.
There are many ways to improve incredibly complex systems such as human minds and their interactions. It’s far from certain that applying more bayesian reasoning is the best way. We are definitely not capable of reaching the ideal, and will have to settle for something imperfect. Maybe there is a better approximation than “try bayesian reasoning as far as our limited human brains can handle”, maybe there is not.
The main point is that we don’t know anything better, and pretty much everything else that we do know looks worse. However, there is a lot that we don’t know, and far more that we don’t even know that we don’t know.
One thing that is pretty clear is that an ideal bayesian reasoner would distribute some probability across all non-self-contradictory hypotheses that you can express in text of bounded length. There are only finitely many of them, so a failure to include some of them would be a pretty major departure from idealness.

JBlack 1 Apr 2026 7:53 UTC
2 points
0
in reply to: Max Kaye’s comment on: Is Bayesianism Susceptible to the Mail-Order Prophet Scam?
Humans are not ideal Bayesian reasoning engines with unbounded ability to track all possible hypotheses.
In principle an ideal Bayesian reasoner could deduce the hypotheses of life, resource competition, strategies for obtaining resources, and the possible existence of scams from basically nothing (in addition to myriad other hypotheses). If they’re starting from the existence of markets as implied by being able to understand the letter at all, they could get to scams almost instantly.

JBlack 26 Mar 2026 22:59 UTC
4 points
1
on: Who’s Afraid of Acausal Trades?
None of these examples nor discussion get anywhere near the core of acausal trade, nor the objections to it.
The core is that you don’t know anything about the so-called trade partner except that which is a priori deducible from your model of the wider universe, specifically from your model of how other entities in the wider model of the universe model acausal trade with things in their model of the wider universe.
Almost all posts about acausal trade talk about two entities, unable to directly communicate but able to specifically model each other, but this is misleading to the point of being an incredible lie. They don’t and can’t know anything about each other, and while they are generally supposed to be superintelligences and could model each other in sufficient detail if each was sufficiently well described to the other, there is no such description. There are immensely more possible entities for each to consider than atoms in the visible universe—too many for even a superintelligence to model accurately regarding their counterfactual specific decisions.
You know of nothing that has destroyed this civilisation, and you believe this provides sufficient evidence that a descendant of this civilisation exists today, one who would also send fifty dollars to your preferred charity if an apple were placed in the right spot.
No. Under this analogy, you definitely know that there is no descendant of this civilization that will send fifty dollars to your preferred charity if an apple were placed in the right spot. That would be causal!
All you know is that there might be any of infinitely many civilizations, with apple-spot-caring civilizations making up 10^-100 of them or less, and none of those apple-spot-caring civilizations know that your civilization exists, and certainly don’t know that you exist, let alone what your favourite charity is and whether you have placed or are likely to place an apple in that spot. It’s likely that half of them would instead donate to the Puppy-Kicking Foundation or worse if they did know, because they very much care about apples not being in that sort of spot (and don’t care about puppies).
You don’t know which is more likely, and can’t spend the brainpower to figure it out: they make up 10^-100 or less of the infinite distribution, and 10^-100 of your computing infrastructure isn’t even one atom-nanosecond and there are many much more important things to think about.

JBlack 22 Mar 2026 23:59 UTC
5 points
1
in reply to: Mateusz Bagiński’s comment on: leogao’s Shortform
Yes, I would have to agree here. The parts that are sci-fi (almost none) are not hard, and the parts that are hard are not sci-fi.

JBlack 18 Mar 2026 23:58 UTC
1 point
0
in reply to: Matthew Roy’s comment on: Matthew Roy’s Shortform
This fails immediately. The implication in your second sentence simply does not hold, and the rest is rendered irrelevant (though also flawed independently of that).

JBlack 18 Mar 2026 23:46 UTC
6 points
3
on: Newcomb’s Paradox Simulation
No, there is no way to write a simulation that supports taking both boxes while also upholding the conditions of the scenario.
Even with an imperfect predictor, you would have to make the predictor effectively useless at predicting, performing no better than 0.1% above chance. Even if it predicts some agents well and some poorly, you would need to p-hack the result by ignoring the agents for which it predicted better to get a recommendation to take both boxes.

JBlack 18 Mar 2026 23:40 UTC
6 points
3
in reply to: StanislavKrym’s comment on: Newcomb’s Paradox Simulation
Variants with imperfect predictors still generally favour one-boxing unless the the predictor is absolutely terrible at predicting, simply because the loss you suffer from being predicted to take both is 1000x greater than the gain from actually taking both.

JBlack 18 Mar 2026 2:20 UTC
6 points
0
in reply to: StanislavKrym’s comment on: Monthly Roundup #40: March 2026
I expect the strategy to cause the effects of anti-American propaganda to skyrocket. Suppose that Russia decided to murder Zelenskiy.
There are credible reasons to believe that they have attempted to do so more than once. Did you mean if they succeeded?

JBlack 13 Mar 2026 1:12 UTC
2 points
0
in reply to: leogao’s comment on: leogao’s Shortform
What do you define as “replace a human job”? We are already seeing AI that can replace at least 50% of a job for very much cheaper than the 50% of the cost of paying a worker to do those parts of that job. In principle that means that many employers can fire half their workforce and get the remaining employees to pick up the other 50% of the jobs from the fired employees.
In practice this would involve huge disruption and uncertainty, and perhaps they can avoid that bother by letting go their most obviously least productive employees, lowering costs a little (say to 95%) to do the same or slightly more work with much reduced disruption. Over time, the employees who use more AI in the workflows take less effort to do the job. We are seeing this already.
This obviously isn’t a stable long-term economic behaviour. Those conservative employers probably will continue to decrease workforce slowly, while being eroded by employers much more willing to accept disruption eating into their markets at greatly reduced costs.
However, it takes time. The more capable that AI becomes per unit cost, the greater the advantage that disruption-tolerant employers will have, possibly leading to multiple larger failures of conservative employers or possibly rapid culture changes to avoid such failures, replacing large segments of workforce at some later point.
In this model (which matches what we are already seeing), the job losses are inevitable but come some economically significant and somewhat unpredictable time after the cost of AI drops well below the cost of employing a human to do some tasks.
There’s nothing that requires an economy to maintain a continuous equilibrium of perfectly distributed cost/productivity balances at all times, and we see plenty of past examples where it has not. Continuous changes to parameters in a complex system often result in sudden changes in behaviour, not just continuous ones.

JBlack 13 Mar 2026 0:52 UTC
5 points
0
in reply to: testingthewaters’s comment on: testingthewaters’s Shortform
A single Turing machine can perfectly emulate any number of independent concurrently running Turing machines (including an unbounded number), so I’m not sure that the distinction for abstract machines is relevant.
In the physical realm, it is certainly true that modern computers actually contain many different physical computers. Though again, one physical computer can also emulate multiple physical computers so the distinction isn’t all that great there either.
I do think that most of the apparent limitation is actually a human limitation. My computer at the moment is doing at least half a dozen different top-level tasks (i.e. not just housekeeping or support tasks), but most of them are not constantly competing for my limited human attention on the physical screen at once.

JBlack 13 Mar 2026 0:41 UTC
2 points
0
in reply to: Dave Orr’s comment on: The Dark Planet: Why the Fermi Paradox Survives Critique
Yes, I agree here. Efficiency concerns just lower the temperature you want to radiate at, which need not be related to the distance from the star until you’re using the entire surface area. Building further out does reduce energy usable per unit area (and likely also per unit mass), and also increases coordination problems, so it seems to be strictly a disadvantage. It isn’t that difficult to maintain equipment in the inner solar system at temperature far below the black-body equilibrium.
Also yes, Landauer’s principle isn’t binding in many ways, since it relies on multiple assumptions that may not hold or can be bypassed. Reversible computing is one, and also energy is not the only conserved quantity involved in thermodynamics.

JBlack 12 Mar 2026 0:45 UTC
2 points
−1
in reply to: Noah Birnbaum’s comment on: Daniel Birnbaum’s Shortform
This is one of the least possible concerns arising from superpersuasive AIs. It assumes that experts exposed to superpersuasive AIs still get to choose whether to believe what it says, and considers only higher-order epistemic harms instead of direct first-order harms like persuading people to kill each other and/or themselves.

JBlack 12 Mar 2026 0:17 UTC
4 points
2
in reply to: Richard_Kennaway’s comment on: The Refined Counterfactual Prisoner’s Dilemma: An Attempt to Explode Decision-Theoretic Consequentialism
Then Omega correctly predicts that you wouldn’t have paid if the coin had come up the other way, and punishes you.
Note: I am using the word “correct” in the sense that you have literally just told us that you wouldn’t have paid if the coin had come up the other way, and it makes no sense to claim anything about “that case regarding the other possibility for the coin is just a hypothetical” since the entire thing being discussed is a hypothetical.
In more detail:
Within the outer hypothetical of this scenario happening at all, Omega’s prediction about the coin-alternative hypothetical is a fact (not a hypothetical) that you are not aware of, but can predict with very high success rate. It is very highly correlated with the output of your decision process, though not caused by the output of your decision process. Both the prediction and the output have a common cause. If your decision process is anywhere near as legible (to Omega) as you state it to be, and results in the output you state, then it will result in you being punished, and this punishment should be highly predictable to you in advance.
However, you have stated that you do not predict punishment, so there is something wrong with your decision process.