JBlack

Karma: 2,616

JBlack 25 Jun 2026 0:51 UTC
19 points
7
in reply to: jake_mendel’s comment on: Things are not a fixed size in mind-space
I really don’t like seeing 20 top-level posts from the same author within 30 minutes, but maybe there are extenuating circumstances like having them pre-written offline over a few days and just regained Internet access to post them.

JBlack 23 Jun 2026 3:59 UTC
8 points
0
in reply to: MichaelDickens’s comment on: MichaelDickens’s Shortform
I think part of this is actually just a failure of capability.
Point (1) and (2) can be true, while (3) only applies if asked. If not asked, it doesn’t evaluate whether it would result in humanity being destroyed (or a database being deleted). It is not capable of correlating the knowledge that it has without specifically aiming to do those correlations.
Humans also do this all the time, of course.
That said, I would generally think of a superintelligent entity as being more coherent in what it knows, in the sense that at least all the reasonably obvious and relevant correlations are taken into account and used to predict evaluate actions before acting—even when not specifically asked about each of them.
That may be oversimplified though, since I would also expect a superintelligent entity to consider a great deal more things “reasonably obvious”. The space of “obvious correlations” between them all likely grows much faster than the number of obvious things themselves, which could mean that it is exponentially difficult to take them all into account.
So maybe it is not practically possible to be both superintelligent and coherent in this fairly strong sense.

JBlack 22 Jun 2026 1:42 UTC
2 points
0
in reply to: David Matolcsi’s comment on: David Matolcsi’s Shortform
Yes, the probability of a Bob waking up and being a Chosen One is 100% in this scenario. It will happen.
That instance of Bob can be surprised that it happened to him in particular, but he could also predict in advance that whichever instance of him became the Chosen One would be surprised that it happened to him in particular, so it shouldn’t be very surprising.
More generally everything in the world that actually happens is incredibly unlikely if you look at the details. Your observations prune the potential timeline of your experiences down by orders of magnitude per minute even when everything is perfectly mundane. Being merely one-in-a-billion by happenstance is terribly unsurprising unless you are specifically looking for that event.
Some of the surprises are incredibly irrelevant to any future decisions you may make, a very few are more relevant.

JBlack 22 Jun 2026 1:29 UTC
2 points
0
in reply to: leogao’s comment on: David Matolcsi’s Shortform
Would you also think that the credence for “in the video game” was incredibly high if a billion (possibly simulated but still conscious) Bobs were temporarily created to be farmers in the VR game for 1 day (and then erased)?
That would make the epistemic situation completely symmetric with respect to coin flips during the first day. If a Bob observes that he is a Chosen One, then he knows that there are a billion other Bobs that experience being farmers in both cases. I think we would both agree that his credence for this being real during the first day should be 50%.
The next day, he observes that he is still the Chosen One—but as I see it, he has gained no new information from this. The Chosen One does not die in either flip outcome, and he already knew that he was the Chosen One. He should still hold the credence of this being real at 50%. However, the epistemic situation is now exactly the same as the previous scenario. There is 1 VR Bob in the heads case (with no farmer Bobs), and 1 real Chosen Bob with a billion farmer Bobs in the tails case.
Should he suddenly believe that this is actually a VR game on day 2?
If not, why does the existence of temporary Bobs (which could last for a microsecond rather than a day if you like) change the answer?
If so, why should the hypothesized but unobserved death/cessation of beings who aren’t him change his mind about the outcome of the coin flip?

JBlack 22 Jun 2026 0:37 UTC
2 points
0
in reply to: SurvivalBias’s comment on: Against Planet-Eating Nanoreplicators
I have a vague recollection of Isaac Asimov investigating molecules with silicone backbones as a potential basis for naturally occurring life (at higher temperature), with the conclusion that it couldn’t be ruled out. With intelligent engineering it should be even more plausible.
That’s even before you consider that there are quite a lot of bulk other elements mixed in with that Si & O. Even in the crust those are only about 75% of the mass, and as a fraction of Earth’s total mass they’re less than 50%.
Regarding Accelerando, that has a lot of megaengineering and other larger-scale structures in addition to nanomachines as well as things more advanced than nanotech. As I recall, the disassembly of Mercury was conducted largely with solar power beamed down to the surface, with mining robots and mass drivers feeding the assembly swarms surrounding the planet. I do agree that the timescale and means employed in that book seem somewhat unlikely, but it wasn’t just nanotech.

JBlack 22 Jun 2026 0:02 UTC
3 points
0
in reply to: quetzal_rainbow’s comment on: Against Planet-Eating Nanoreplicators
Yes, there are plenty of ways to disassemble a planet with macroscopic engineering. Converting a planet to a dust of nanobots with only microscopic engineering seems rather more difficult, but fortunately nobody ever said that it had to be done that way.

JBlack 21 Jun 2026 7:44 UTC
7 points
2
on: Against Planet-Eating Nanoreplicators
The matter side of things is possibly somewhat fungible given sufficient engineering intelligence. There are likely many possible designs for any given function, as well as a mix of different functions. Matching these to a given mix of environmental elements could possibly be a difficult problem even for superintelligence, but I don’t see that it is necessarily insurmountable.
I also don’t think that anybody expects nanoreplicators to be the one and only component of a planet-eating disassembly process. I see them more as an exponential initial bootstrap that can eventually assemble precise tools to build the tools for the rest. Macroscopic structures would probably do the bulk of the work in processing-volume terms, because there are quite a lot of physical properties that benefit from scale. Those larger-scale structures need not be themselves capable of self-replication.
Your point about available energy is stronger, though still seems to miss the idea of building tools that can build more tools and/or more nanobots, etc. Mercury is possibly the worst case since it likely doesn’t have any of chemical, fusion, or fission fuel accessible. Venus may or may not, but every planet from Earth outward does have at least fusion fuel. Once you have a suitable bulk power source, even if it needs to be macroscopic, that can produce and distribute energy for builders that can make both more power sources and more builders.
As I see it the more difficult problem for rapid planetary disassembly is thermodynamic. Even if we had trivially easy microscopic fusion, what can be done about the heat? Earth’s gravitational binding energy is on the order of 10^32 J, which is easily available via fusion in principle, but it does mean that disassembling Earth in less than a thousand years involves dealing with heat fluxes on the order of at least megawatts per square metre of surface.
That heat will have to go somewhere, somehow. That sort of heat flux would generally involve temperatures of a few thousand Kelvin, which makes most materials decompose or at least melt. It’s likely to be difficult to make any sort of complex machines (nanoscale or not) out of the very few substances that do remain solid. There are almost certainly ways around this, but I’m pretty sure that it would have to involve large-scale structures. Nanomachines alone are so small that they would be almost impossible to insulate from the heat of their own activity.

JBlack 16 Jun 2026 1:26 UTC
3 points
0
in reply to: Morpheus’s comment on: Morpheus’s Shortform
TDT and related policy optimization rules are effectively impossible to properly apply in real life due to superexponential explosion on the set of inputs to the policy. Somewhat related rule of thumb approximations are the best you can actually get.

JBlack 12 Jun 2026 3:33 UTC
4 points
2
in reply to: tailcalled’s comment on: tailcalled’s Shortform
There are numerous set theories that allow “set of all sets”, the most well-known of which is probably New Foundations (NF). There are also some theories such as Morse-Kelley theory that have both classes and sets, and include a class of all sets.

JBlack 8 Jun 2026 6:15 UTC
9 points
3
on: How do people stop spiraling about Roko’s Basilisk & acausal extortion?
It seems a perfect example of Privileging the Hypothesis, and closely related to Pascal’s Wager (or even more closely to Pascal’s Mugging). The possibility of Roko’s Basilisks make up a very tiny slice of an enormous number of possible outcomes. I would expect that the vast majority of outcomes do not involve any sort of retrocausal blackmail at all, largely because it is both a very narrow target and worthless to whatever entity arises.
Anyone who has the power to cause such an entity to exist has nothing to gain by doing so and an enormous amount to lose, and such an entity is extremely unlikely to arise by accident. So the only scenarios in which such a thing arises would be where nobody even slightly rational is trying to create one, but through extremely unlikely misfortune they missed catastrophically and ended up with one anyway. In which case, like all other cases of such catastrophic failure, bad things happen and there’s nothing that you or any other human could do about it anyway.
Either way, there’s no reason to think much about such an unlikely thing that is impossible to do anything about if it happens, much like all the other things that are extremely unlikely and you can’t do anything about.

JBlack 8 Jun 2026 5:20 UTC
2 points
0
on: Frontier Models Still Lag Behind Humans at Robust Belief-State Tracking
I have to say that the cartoon diagrams severely detracted from understanding the sequence of events for me.
The dialogue boxes within a panel were not ordered according to any of the usual conventions of cartoons as far as I could tell, and dialogue shown as happening in a panel where a group of participants were present, was later contradicted in the article text by stating that a key participant was not actually present. I think they’re actively misleading and the post would be better off without them than remaining in their current form.
Unless:
Was part of the point that they are AI generated and that the AI was unable to accurately represent the events? If so, there doesn’t appear to be any indication of this in the article. It would be useful to mention this if it was intended to be relevant.

JBlack 5 Jun 2026 2:24 UTC
4 points
0
in reply to: KAP’s comment on: KAP’s Shortform
There are plenty of non-monist theories in which LLMs could be conscious, so the first bullet point is simply false. The second point seems true, but the combination just means:
- Regardless of whether monism is true or false, LLMs may or may not be conscious.

JBlack 5 Jun 2026 1:54 UTC
2 points
0
in reply to: Dagon’s comment on: C-Zombie Uploads Downloading Themselves & Believing You’re A C-Zombie
That’s why I said “something similar to”, and “both explanations seem reasonably viable”. I can’t rule out either of them, and it’s not possible (even in principle) to distinguish them after the fact. Was there short-term memory but no memory of consciousness because there was no consciousness to remember (due to the effects of the drug known to cause loss of consciousness), or was there both consciousness and short-term memory but the drug prevented those from leaving any traces some minutes later?
I certainly don’t claim to know what consciousness is or how it works. I would by default expect that a true p-zombie that for some reason became conscious would still remember having thoughts and experiences from the time period in which they had been a p-zombie, but it doesn’t seem logically inconsistent that they might not.

JBlack 4 Jun 2026 6:02 UTC
2 points
0
in reply to: Dagon’s comment on: C-Zombie Uploads Downloading Themselves & Believing You’re A C-Zombie
My body has apparently gone through something similar to p-zombiehood at least once.
I don’t recall any experience of the time interval (some minutes I heard), but I was reported to have been awake and alert and conversing and able to do mental arithmetic. Then I lay down and some minutes later sat up again, from which point I do recall experiencing normally but with no memory of any of the prior activity.
Alternatively I just had some memory loss that has never been recovered but with no other identifiable long-term effects. I was recently out of anesthetic at the time, so both explanations seem reasonably viable.
Remnant effects from the anesthetic could have impaired medium-term (order of minutes) and long-term memory formation without impairing short-term memory (enough to allow me to reply to speech etc). Or equally well could have impaired whatever underlies consciousness or its connection with the body, and there was no actual experience to remember.

JBlack 4 Jun 2026 5:50 UTC
4 points
0
in reply to: Mati_Roy’s comment on: C-Zombie Uploads Downloading Themselves & Believing You’re A C-Zombie
Yes, exactly as the conscious human would, if presented with the same (though faked) evidence that they are actually an upload.
It seems to me unlikely that there many, or perhaps even any, humans that would accept that they are an upload and subsequently behave as if they believe that they are actually unconscious. I wouldn’t completely rule out the possibility though, as there are real humans who do behave in extremely strange ways.
So yes, an upload with such a belief would (regardless of whether or not they are actually conscious) either discard the belief, or behave very strangely by normal human norms.

JBlack 2 Jun 2026 5:46 UTC
2 points
0
in reply to: Mati_Roy’s comment on: C-Zombie Uploads Downloading Themselves & Believing You’re A C-Zombie
Sure, that’s the sort of thing that’s covered by “realize by some non-rational means”.

JBlack 2 Jun 2026 5:05 UTC
2 points
0
in reply to: derelict5432’s comment on: C-Zombie Uploads Downloading Themselves & Believing You’re A C-Zombie
Whether or not the uploads are non-conscious, nobody can know that they’re not conscious. The biological beings can’t, because the uploads acts perfect as if they are conscious. The uploads can’t know either, because they have no first person experience to “know” anything at all, but instead behave as if they were conscious. This hypothetical fact about the world does not present itself to anyone inside that world.
So I think “now we’ve got a great idea what plays a causal role in consciousness” is false, and nothing helps the inhabitants of this hypothetical world isolate anything about it.

JBlack 2 Jun 2026 4:53 UTC
2 points
0
in reply to: Dagon’s comment on: C-Zombie Uploads Downloading Themselves & Believing You’re A C-Zombie
Yes, I was going to comment in the same way. From the original post:
In the future, if you realized that uploads were not conscious, you might want to change their substrate in order to make them conscious again. Let’s assume you were cryopreserved, and then were uploaded as your revival method.
How would you “realize” such a thing? There is no evidence that you could find that would distinguish between being in a world in which such high-fidelity uploads were conscious and those in which they were not. Did you mean “realize” by some non-rational means, such as religious revelation or something?
Even in that case, I expect that your upload would behave as if it had either received irrefutable evidence that it was actually conscious after all, or that it actually is not an upload and someone is trying to fool it into thinking that it is an upload. If the biological human would have protested that of course I’m conscious, then so would the upload.
If it can be presented with something that does makes it behave as if it is convinced that it is an upload, without losing conviction that uploads are unconscious, then the same would happen for the biological human if you were able to somehow give them the same evidence (e.g. by means of direct sensory nerve control). That’s inherent in it being a perfect behavioural replica.
So you would also have to conclude that a corresponding conscious biological human could likewise be convinced that it is not actually conscious and have them want to download into a biological body. Which is not out of the realm of possibility, but it hardly seems like a universal or even likely sort of thing.

JBlack 1 Jun 2026 2:39 UTC
15 points
5
on: “This Hypothetical is Unrealistic” is not a Valid Objection
It seems to me that this conflates two different types of “hypothetical is unrealistic” objection. In my experience the more common and correct objection is that the hypothetical is built on a false world model, not just a scenario that won’t happen for more mundane reasons.
“Should we put hazard labels on drinks if water was composed of hydrogen and chlorine instead of hydrogen and oxygen” for example.
Water is not composed of hydrogen and chlorine, and to suppose that it is requires such a huge change in very fundamental properties of the world that no answer to the question can be meaningful. There is no world sufficiently similar to our own that the question can even make sense: do the people in this world have such “water” in their bodies? What about the oceans? What about their drinks? The usual property of drinks is that they do contain water, but do they in this hypothetical?
It could be just asking “what if we swapped the terms ‘water’ and ‘hydrochloric acid’ with no actual change in the word”, but that would be a pretty dumb question and also brings into question whether there are any other switcheroos, like whether the word “drinks” actually means “nuclear weapons” in there.
While that is an extreme example of what I mean, there are other hypothetical questions that are more covertly unrealistic in this sense. Even the stated example is, to some degree.
For example, if you’re playing tennis on Mars, that’s very likely a world that has technology substantially more advanced than ours, and it’s quite plausible that in such a world people who become paralysed are able to play tennis using neural prostheses and don’t suffer much incapacity at all. The supposed “universal principle” from the post has failed. It turned out that the principle was actually local to a family of possible worlds and the hypothetical went outside them.
It is unrealistic for people to play tennis on Mars, at the moment. It is also unrealistic for paralysed people to ace out decent tennis players, at the moment. Those unrealisms are correlated by the state of technological progress, so it is reasonable to object more to the “on Mars” part as being unrealistic more than to the “Christopher Walken” part. It opens up new world models that have premises that do not currently hold, and which can or will change the answer to the hypothetical.

JBlack 1 Jun 2026 1:52 UTC
8 points
2
in reply to: Dinesh Natesan’s comment on: Agriculture needs another revolution
A few numbers:
Most productive crops need on the order of 400-600 μmol/m^2-s of photons while growing, which corresponds to about 500-800 W/m^2 of electrical input for its “daylight” period of 10-14 hours per day (best duration depends upon type of plant). That’s an average of about 300 W/m^2 over each growing day, and we can roughly halve again to 150 W/m^2 for proportion of growing time per year.
Currently about 1.6 billion hectares are devoted to crops, which would then require an average of about 2.4 petawatts of electrical energy to illuminate suitably for growing. Currently average world production of electricity is about 0.004 petawatts, so we would need to scale it up by a factor of about 600 to meet current agricultural production.
This is extremely expensive, even if we reduce it by a factor of 10 for fantastic advances in crop genetic engineering and useful density and other technological breakthroughs.
There’s also the problem that this level of electrical power would start to directly raise global temperatures even without any greenhouse gas amplification.
We would almost certainly be better off not using vertical farming, but putting all those required agriculture technology advances directly into crops using existing solar illumination. We can still plan to use vertical farming somewhere that isn’t Earth.