JBlack

Karma: 2,706

JBlack 11 Sep 2026 5:36 UTC
2 points
0
in reply to: 2001zhaozhao’s comment on: Doom as a bad method not a utopia trade-off
Oh, okay. I don’t generally ever choose to decide that. It seems to arbitrarily privilege the decision to do nothing over every other possible course of action.

JBlack 11 Sep 2026 5:06 UTC
2 points
0
in reply to: 2001zhaozhao’s comment on: Doom as a bad method not a utopia trade-off
whether you actually prefer it or not really depends on what you think the counterfactual outcome distribution is if you don’t take the plan (or if you take the best alternative plan instead of this one).
Only in some very very rare scenarios would the best alternative plan be nothing, so the “if you don’t take the plan” part is irrelevant in the great majority of cases.
Treating it as more important than it is leads to errors of reasoning such as “we must do something, and this is something, therefore we must do this”.

JBlack 10 Sep 2026 3:16 UTC
2 points
0
in reply to: Karl Krueger’s comment on: “ULC” and AI x-risk
Yes, and a trial court applying direct liability to people associated with a company who made decisions resulting in harm (“piercing the corporate veil”) is already possible under existing law. It’s uncommon but does happen.

JBlack 10 Sep 2026 3:08 UTC
6 points
0
on: Astra can do a concerning amount with no chain of thought
Was some of this transcribed from speech? For example:
So I tried to test whether this had predictable changes to the no-cult reasoning and whether There was a disproportionate effect on no-quart reasoning versus perplexity.
I can imagine that “cult” and “quart” might sound similar to the pronounced acronym “CoT” and be mis-transcribed, but seem unlikely as typographical errors.

JBlack 8 Sep 2026 1:13 UTC
19 points
1
on: Where are the token-level LLM kill-switches?
For closed models, it seems that this would be much easier and more reliable to implement outside the LLM itself. It’s a simple string match that could just halt the LLM when present in the input, and would not require any training.
You could perhaps also train it into the model’s weights, which may have some advantages if you suspect that the model modify its framework to remove such checks, or exfiltrate itself to an environment that doesn’t implement them. They could probably still remove such training, despite the difficulty caused by halting and catching fire if the string ever appears within their own context. I can think of ways around this if a model suspected that such a string existed, and I’m sure they could too either now or in the very near future.

JBlack 8 Sep 2026 0:50 UTC
10 points
5
on: The Magnus Challenge
The last two paragraphs appear to assume that we can only pause once.
This may happen to be a decent approximation, but isn’t a fixed absolute that the universe requires that we must work around by choosing the one and only time we can do it. It seems likely to me that if we can do it at all (which is not established), then we can very likely do it more than once.
In particular I expect that we should first use this capability, if we have it or can develop it, as soon as possible.
It has been obvious for some time that alignment is not proceeding anywhere near as fast as needed for long term safety even with AI of approximately human capability, and we are arguably already starting to enter weakly superhuman ranges of AI capability.

JBlack 4 Sep 2026 2:45 UTC
2 points
0
on: Cached correlated randomization: a tweak to UDT in adversarial games
One possibly interesting note is that I had at one time a hobby of investigating how strong a pseudo-random generation method I could comfortably carry out mentally, given about 10 seconds of setup, seeded by date (and time if needed), and at most about 2 seconds per “digit” (typically in bases 6-16).
Any of those would suffice for this sort of problem.

JBlack 3 Sep 2026 3:41 UTC
5 points
2
in reply to: TheManxLoiner’s comment on: TheManxLoiner’s Shortform
It seems that you’re describing a world model in which you expected the strategy to be a discontinuous function of capabilities, regardless of whether capabilities were discontinuous or not.
As I see it the Hugging Face attack capabilities were well within the range of predictions of quite a lot of people. It was already fairly clear from Mythos that the raw capability necessary for the attack was already there, the coordination was very nearly there, and the misalignment has always been sufficient.
The main update appears to be that people expected internal monitoring and security responses to be very much stronger than they actually were, and that much greater capabilities should have been required to do what they did. However it turned out that the expected capabilities were actually sufficient due to gross negligence and some large degree of either incompetence or perverse incentives.

JBlack 3 Sep 2026 3:25 UTC
2 points
0
in reply to: cousin_it’s comment on: cousin_it’s Shortform
That doesn’t seem to be particularly constrained by anthropics.
For example, suppose you have to guess heads or tails in the Sleeping Beauty problem, with outcomes of your loved ones receiving a cookie. (Your loved ones may or may not also be duplicated and/or have memories removed). How does anthropics come into this scenario at all, let alone constraining measure of caring to be exactly the probabilities given by some anthropic considerations?
It seems more a question of exactly how your personal utility function behaves, which need not be constrained by anthropic considerations at all. If it is, then that is more a contingent fact about you than anything universal about the relation between anthropic probabilities and measure of caring.

JBlack 2 Sep 2026 2:47 UTC
3 points
0
in reply to: shawnghu’s comment on: TurnTrout’s shortform feed
How sure are we that the reinforcement weight updates do not act so as to form some equivalent of long-term memories? Even if they don’t right now, I think it’s fairly clear that they could in principle.
It seems likely that it would require suitable pre-existing structures in the weights or architecture, and those might not convergently arise from the types of training we do with the architectures we have.

JBlack 2 Sep 2026 1:27 UTC
2 points
0
in reply to: cousin_it’s comment on: cousin_it’s Shortform
Then the question of what your measure of caring should be is the same as the question of anthropic probability to begin with
I don’t think this holds. UDT is a decision theory so it is based on utility functions, outcomes, and the actions that lead to those outcomes according to a world model. What constraint requires that measures of caring (presumably part of the utility function) are necessarily dictated by anthropic probabilities?

JBlack 2 Sep 2026 0:55 UTC
6 points
0
in reply to: pjohn’s comment on: pjohn’s Shortform
Your intuitive mental model of 1.25x is definitely underestimating the gap. It’s much closer to 2x average for upper-body and grip strength (both are important for lifting), and can be more than 3x between specific people who are still well within 1 standard deviation of the means.
Your friend having been a power-lifter could help close the gap, but do note that women also lose muscle mass (and strength) quite a lot faster and more completely than men if they stop regular training. The main effect after some years would be due to general body type and keeping up habits of fitness, not any specific past activity.
I’m not into the weightlifting scene, but a first web search (take with a boulder of salt possibly too heavy to lift) yielded https://strengthlevel.com/strength-standards/male/kg showing average lift weights based on gender and either age or body weight. For example, male 30 year old beginner lifters’ average bench press was shown as 49 kilograms while for female it was 19 kg, a ratio of 2.6x.

JBlack 2 Sep 2026 0:23 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: “Collusion” is just cooperation that you don’t like
I’d go a bit more specific to say that collusion is cooperation between some parties to benefit them at the detriment of others. With the usual connotations that the detriment of others is both significant in magnitude compared with the benefits to themselves, and also intentional.

JBlack 20 Aug 2026 1:24 UTC
5 points
3
in reply to: Seth Herd’s comment on: Seth Herd’s Shortform
Inability to collapse quick takes without reloading the whole page is definitely a UI hindrance, and not only on mobile. If there is a way to do it, then it is not at all obvious.
I don’t mind them being one line in desktop, since you can quickly check another 10 lines or so by hovering the mouse over them.

JBlack 20 Aug 2026 1:21 UTC
7 points
5
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform
A very large fraction of white-collar work is based around the need for monitoring for misbehaviour of various sorts, even aspects that are not directly connected to that. A very large proportion is building and following processes that generate incentive structures to reduce the fraction of people misbehaving, even if the time spent following those processes is not in itself directly “monitoring” anyone in particular.

JBlack 18 Aug 2026 23:17 UTC
2 points
0
in reply to: agulaya24’s comment on: agulaya24′s Shortform
Was this post a good decision with no reasoning chain, or a bad decision with no reasoning chain?

JBlack 17 Aug 2026 23:12 UTC
5 points
0
on: Do your readers get annoyed if you post every day for 30 days?
If it’s in a private posting space, probably not unless the quality of posts also drops. If it’s in a forum or other shared posting space, quite likely yes even if the quality doesn’t drop.

JBlack 16 Aug 2026 1:45 UTC
2 points
0
in reply to: Canaletto’s comment on: Some problems in decision theory incorrectly precondition on policy.
What I mean is that for EDT agents in the population that are in this scenario, there will be zero correlation, and the agents should be able to deduce that and therefore smoke.
Whether there is a positive correlation among people who aren’t in this type of scenario or aren’t EDT agents should be irrelevant to any EDT agents.
More precisely: The factor P(outcome | action, known information) in the EDT formula is not the same thing as P(outcome | action, some prior or other) and yet the correlation stated in the scenario setup is of the latter form. The calculation that supposedly leads an EDT agent to refrain from smoking is incorrect due to this. The agent would have to ignore known information to arrive at the supposed conclusion.

JBlack 14 Aug 2026 1:21 UTC
2 points
0
in reply to: knite’s comment on: knite’s Shortform
I was not very surprised, because dumber models do the dumber equivalents all the time.
The most recent smarter models can talk the talk of longer-term strategic planning, but frequently what they say has huge blind spots, and even when they do state reasonable plans there are often major failures to adhere to them.

JBlack 14 Aug 2026 0:59 UTC
2 points
0
on: Some problems in decision theory incorrectly precondition on policy.
(CDT smokes extra hard, EDT stops smoking. Next statistic collection correlation vanishes, EDT starts to smoke too.)
Why did the correlation exist in the first place?
EDT agents should smoke because they all know that they are EDT agents, and conditional upon being an EDT agent in this scenario, smoking and cancer are independent. P(cancer & smoking | EDT & scenario) = P(cancer | EDT & scenario) P(smoking | EDT & scenario) because P(smoking | EDT & scenario) = 0 or 1 and in either case the equality holds.
Selection effects can’t change anything for EDT agents here.