Dweomite

Karma: 1,676

Dweomite 5 Aug 2026 21:10 UTC
19 points
11
on: Generalized atheism rules out “inaccurate simulation”-ism.
I don’t think this affects your central point, but I’d expect a fairly large class of simulations to have exactly one miracle in them, and I don’t think your argument weighs against this.
For example, someone with a question like “what if technology X had been discovered 100 years earlier?” might run a simulation with exactly one miraculous intervention to cause technology X to be discovered at the time of the simulator’s choosing. Or “what if WW2 had ended differently?” could lead to a simulation with exactly one miraculous intervention to change the outcome of the war, etc.
Similarly, simulations about alternate rules (“what if humans had psychic powers?”) might choose to start when things are getting interesting (e.g. when humans evolve, or when civilization kicks off) rather than starting with the big bang, and might fill in the history before that point from some other data source (e.g. basement reality), and then the simulation will have exactly one miraculous discontinuity in its history (at the moment the simulation started). h/t to (major spoilers)
the novel Ra by qntm
(It has occurred to me that “that’s when the simulation started” is a possible explanation for (at most one of) various historical discontinuities like the year the singularity was canceled, though I don’t give it much credence.)
This doesn’t change your take-away that you shouldn’t expect miracles in the future.

Dweomite 15 Jul 2026 15:29 UTC
11 points
0
in reply to: Bistrofan’s comment on: Surprising facts about the slave trade
When you are counting up the total value being produced in a free labor market, I believe you are implicitly including the part of that value that is captured by the laborers. That is, if I pay you $10 to produce a good worth $20 to me and so two people each end up $10 richer, you are counting that as $20 worth of production. Under this accounting, I agree it would be quite surprising if slavery were efficient.
But surely no slavery advocate was accounting that way? The efficiency seems more doubtful if you only care about a fixed group of people that doesn’t include the people whose slavery/freedom/nonexistence is being decided. From their perspective, isn’t this like arguing that a farmer should spend $20 to produce $10 worth of extra crops, on the grounds that that’s a net gain if you include the $20 of value received by the plants?

Dweomite 10 Jul 2026 20:43 UTC
3 points
0
on: A Mechanistic Explanation of Prompt Injection (and why you should study roles)
I think your description of how LLMs receive all their input in a single stream does a good job of explaining the issue but overstates the difference compared to a human.
Your examples of how humans are different focus on our internal thoughts and memories. I agree that these are privileged at a hardware level, but I think they’re the exception. Instructions from your boss, your client, your child, and a stranger all arrive via the same channels, and you still need to treat them differently. And there are effective attacks in the wild that rely on confusion about where a message comes from; e.g. phishing.
It is often the case for humans that important communication uses some hard-to-fake markers to verify authenticity...and also often the case (especially online) that humans mostly ignore those markers and rely on other indicators that are easier to use, but also easier to fake. For example, you might recognize that you are on your bank’s website mostly by the logo and overall layout rather than by its URL and security certificate, even though the former identifiers are much easier to fake.
In face-to-face interactions, I think humans rely mostly on appearance and how a voice sounds, and that humans have specialized submodules to recognize those on a subconscious level. This may make it feel like they’re coming through different input channels, but that isn’t true; it is difficult but possible to fake both of those well enough to fool someone.
For security purposes, I suggest that we should think of a human’s internal thoughts as being analogous to the LLM’s activations, rather than its chain-of-thought, and the chain-of-thought as being more like a scratchpad or journal. In this light, I think the LLM problem of identifying roles starts to look very much like the human problem of identifying the source of a communication. I do think humans are currently much better at this problem than LLMs are, but also that essentially all the vulnerabilities you identified in LLMs are analogous to known attacks against humans.
I’d guess the reason that tricking a human by faking their notepad is less effective than spoofing a LLM’s CoT is about 50% because humans are better at paying attention to context (e.g. noticing that the text is in your web browser vs your notepad app) and about 50% because humans keep more of their memory internally (e.g. I have enough memory of what I wrote to myself to become puzzled if a note doesn’t feel familiar). But if someone snuck into my home and edited the to-do list I keep on my computer to add an entry about fixing a false-but-plausible bug in the software I’m working on, it might be possible to fool me.

Dweomite 10 Jul 2026 0:45 UTC
3 points
0
in reply to: Jasmine C.’s comment on: A Mechanistic Explanation of Prompt Injection (and why you should study roles)
even when models notice spoofed prefills, the attack can still participate in computation.
Also true for humans; attacks that the human successfully recognizes as an attack can still affect the human’s subsequent reasoning. For example, the anchoring effect works even when the anchor is blatantly wrong; blatant advertising tricks can still work against humans who know how they work; etc.

Dweomite 6 Jul 2026 7:46 UTC
3 points
0
in reply to: HedonicEscalator’s comment on: The LLM shoggoth meme is weirder than you think
This isn’t especially relevant, but I’ve started to occasionally notice that some snippet of song lyrics unintentionally could be interpreted as being about rationalsphere ideas if taken completely out of context, and your comment reminded me of one such:
But can you make the difference?
Are endings set in stone?
A band of wayward strangers
can’t stand up to gods alone
So what could you become, then—
But not lose who you are?
And will you stop before
you go too far?
Make your move and change it all
Forevermore
(From Make Your Move by Aviators; there’s 2 slightly different versions of these lyrics starting at 1:09 and 2:59)

Dweomite 27 May 2026 21:51 UTC
10 points
1
in reply to: aphyer’s comment on: Bad Problems Don’t Stop Being Bad Because Somebody’s Wrong About Fault Analysis
As I read the OP, I thought to myself: If I were to steelman the people the post is complaining about, I would guess that they are interpreting the complaint about the problem as an implied proposal for how to address the problem, and they are reacting to the perceived-implied-proposal.
It seems like you’re thinking along similar lines, but are about 3 assumptions further down that road:
- Not only are you guessing that they might have thought this, you’re pretty sure this is what they were thinking
- Not only are you sure they thought this, you’re sure they were justified in thinking this
- Not only were they justified, but all of this is so totally obvious that you’re not even going to bother articulating it, and instead treat it as an unspoken presumption of your snarky comeback
It seems reasonable to me that some people are confused by your comment.

Dweomite 24 May 2026 8:07 UTC
2 points
0
in reply to: Shubhorup Biswas’s comment on: My hobby: running deranged surveys
Alternate steelman—they’re worried that you’re intending to misleadingly quote their answer in a different context, and have rigged the question to get the quote you want.

Dweomite 21 May 2026 22:59 UTC
2 points
0
in reply to: Mateusz Bagiński’s comment on: My hobby: running deranged surveys
My primary guess is that the 6% who like RSI but not ASI are answering based on vibes rather than coherent models, and ASI currently has worse vibes.
Though I could imagine some people thinking that RSI will stop before “superintelligence”, and other people thinking that orthogonality is wrong and RSI would continue beyond some window of “dangerous superintelligence” into “godlike benevolence”. I personally consider both of those possibilities to be so staggeringly implausible that they’re basically just wishful thinking, but I also think that more than 6% of people are engaging in some amount of wishful thinking about AI.

Dweomite 21 May 2026 22:39 UTC
2 points
0
in reply to: Gavin Runeblade’s comment on: My hobby: running deranged surveys
Could you clarify what you meant by this combination of remarks?
I looked at all of those questions and thought, “two years down the road that’s hell. No thank you.”
...
Let me say, for myself, I absolutely want such an outcome.
My personal take is that I can imagine there are people who really would be happier in a scenario where they’ll literally starve to death if they aren’t productive enough, and I guess it would be good if those people could experience the scenario where they thrive, but if there are also people who do fine with a permanent vacation then it doesn’t seem ethical to forcibly keep everyone in the “work or starve” scenario. If resources aren’t actually scarce, then that’s basically slavery, and arguably worse.

Dweomite 21 May 2026 22:21 UTC
2 points
0
in reply to: oligo’s comment on: My hobby: running deranged surveys
For some reason, I didn’t think of this when I read the results, but immediately thought of it when I read the actual question’s wording (even though the question doesn’t mention this).
Framing effects are scarily powerful.

Dweomite 17 May 2026 23:52 UTC
0 points
−2
in reply to: programjames’s comment on: A relatively brief explanation of Boltzmann Brains
How is that different from saying that you do not have an Earth unless you can point to it? If we require that you augment the recipe with some spacetime coordinates that tell you where in the resulting universe to look, why are those coordinates any longer for a BB than for an Earth?

Dweomite 17 May 2026 23:21 UTC
−2 points
0
in reply to: programjames’s comment on: A relatively brief explanation of Boltzmann Brains
Any given configuration of the universe either eventually will produce a BB, or won’t.
Because BBs are much smaller than Earth, the number of configurations that eventually lead to BBs should be much larger than the number that eventually lead to Earths, and therefore the number of variables you need to control to ensure you eventually get a BB should (in expectation) be much smaller than the number of variables you need to control to ensure you eventually get an Earth. BBs are a larger target in possibility space, and therefore (in expectation) easier to hit.

Dweomite 17 May 2026 20:59 UTC
2 points
0
in reply to: DanArmak’s comment on: A relatively brief explanation of Boltzmann Brains
1. Doesn’t this argument prove too much? Why doesn’t similar reasoning rule out other mentally-impaired states like drunkenness, dreams, and brain injuries? In fact, why doesn’t it rule out being a flawed human being, rather than an ideal reasoner?
2. Even if all your memories are false, couldn’t you still reason validly about abstract concepts (like math) and hypothetical scenarios? This seems to show that BBs can do at least some valid reasoning.
3. Even if you might be in some state S that would totally prevent you from reasoning validly, if you conjectured you were in some state T that would permit you to reason validly, and your reasoning based on that conjecture leads to a contradiction, wouldn’t that allow you to rule out state T without ruling out state S? This seems to show that you can do at least some useful reasoning without completely ruling out that you are in an impaired state.

Dweomite 17 May 2026 20:43 UTC
2 points
2
in reply to: programjames’s comment on: A relatively brief explanation of Boltzmann Brains
I’m confused about why you think BBs should be complexity-penalized for the difficulty of specifying which of all hypothetically-possible BBs we’re talking about, but you don’t think Earth should be complexity-penalized for the difficulty of specifying which of all hypothetically-possible Earths we’re talking about.
I get that you can (we think) specify a recipe that eventually produces Earth, rather than explicitly specifying every detail of the current Earth. But presumably you could also specify a recipe that eventually produces a BB. And since the final result you are aiming with for the BB is much simpler than the final result you are aiming for with Earth, I would intuitively expect the recipe will also be simpler (in expectation).

Dweomite 17 May 2026 20:33 UTC
8 points
2
in reply to: David Matolcsi’s comment on: A relatively brief explanation of Boltzmann Brains
What makes you think the explanation for why you won the lottery won’t help you make useful predictions about what follow-up actions will fulfill your values? For example, if the explanation were something like “the lottery was rigged, and there’s about to be a criminal investigation targeting you”, that seems pretty relevant to your follow-up plans.
Explanations of previously-mysterious phenomena are often useful in ways that are hard to foresee before you know the explanation.
If you think that understanding “normal” things is typically useful, why single out this one specific thing to be incurious about?

Dweomite 15 May 2026 0:18 UTC
3 points
0
on: Designing AI factual claims for “easy verification”
This seems like a neat idea, but I’d like to flag that this strategy only seems applicable when the fact you are looking for already exists explicitly and at an appropriate explainer level. I’m not sure you can do anything equivalent if you want the LLM to explain, synthesize, summarize, or do original reasoning.

Dweomite 5 May 2026 21:47 UTC
34 points
5
in reply to: Matthew Barnett’s comment on: Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI
I would describe a critical try as one where the act of trying is likely to prevent further attempts. Launching an ASI is a critical try because the ASI itself could likely stop you from launching more ASIs later on (e.g. by killing you).
If it’s possible to send out missions to intercept the asteroid before it arrives, then it seems to me that the asteroid is better understood as a time limit than as a critical try. You could set the parameters of the asteroid scenario in such a way that you have time for exactly one try, but you could also set the parameters so that you have time to send up a mission to deflect the asteroid, observe its results, and then make a second try before the asteroid arrives. You could also set the parameters such that you have time for zero tries! The key consideration is how fast you can work vs how much time you have.
Contrariwise, if you assume that you are stopping the asteroid with a shield that is close to the earth, such that no matter how fast you build the shield you have to wait for the asteroid to arrive before you can see how well it works, then I’d call that a critical try, because the part of the plan where you wait for the asteroid to arrive severely depletes a critical resource (time) and makes that resource unavailable for later attempts. (Note similarity to the Maginot Line.)
By similar reasoning, I’d say your #2 (global warming) is also more of a time limit, but your #3 (creating a new type of human that potentially kills you) is a critical try (though compared to launching an ASI, it’s more likely to get a middle-ground outcome).

Dweomite 22 Apr 2026 7:30 UTC
2 points
0
in reply to: polytope’s comment on: Annoyingly Principled People, and what befalls them
Yes, that is the sort of example I meant. Though of course this particular example does not prove that the game of Catan, in particular, has situations like this.
Based on his other reply, I expect James would want to point out that there is an equivalent equilibrium where player A, instead of saying “button N is blue”, says “either button N is blue or no button is”, which produces the same outcome without technically lying.
I’m coming to think that there should be some other distinction we can draw that rhymes with the truthful/lying distinction but that talks about consequences instead of semantics, and therefore can’t be dodged by relabeling the signals. Still thinking about it.

Dweomite 22 Apr 2026 4:20 UTC
2 points
0
in reply to: programjames’s comment on: Annoyingly Principled People, and what befalls them
In principle, IF the norms are more important to you than to everyone else combined, then there should be some amount you can pay them that is higher than how much they care about the norms but lower than how much you care about them.
(In practice, finding that amount may be hard, and treating it too much like a transaction may have friendship-corroding effects.)

Dweomite 22 Apr 2026 4:15 UTC
2 points
0
in reply to: programjames’s comment on: Annoyingly Principled People, and what befalls them
Your final sentence clarified some things for me:
- In that sentence, you are not arguing that the lie is no better than silence, you are arguing that it is no better than some truthful message. (This is technically still a falsification of my previously-stated interpretation of your claim.)
- This argument is based on the assumption that the other players already know all circumstances under which you would transmit this message, so there’s no harm in admitting them.
I now realize that if all players have perfect knowledge of the exact conditions under which you would transmit some message, then the actual informational payload of every message is that those conditions are true. (Even with a randomized strategy, you can just interpret the RNG output as part of the conditions.) You might as well literally say “message #27”. Classifying the message itself as truth or lie becomes academic, because no one is expected or intended to pay attention to its face-value claim, and in fact there’s no reason for it to make a face-value claim at all. (Under this very strong assumption.)
So if we’re going to assume players have perfect knowledge of each others’ strategies (including what messages they send under what circumstances), I no longer think it makes sense to distinguish “true” and “false” messages.
I note that “common knowledge that all players are perfectly rational” does not (I think) logically entail perfect knowledge of everyone’s strategy, since a game can have more than one Nash equilibrium. So technically neither of us stated “perfect knowledge of everyone’s strategy” as an assumption in the first place, though I admit I sort of hand-waved towards it when I talked about what players would infer from your failure to say something.
I still think that if we don’t assume “perfect knowledge of everyone’s strategy” then lying is potentially beneficial.
Given that clarification, I’m not sure if your numbered chain of reasoning is a crux for either of us, but for the record I found that chain extremely confusing to read, I think step 3 is invalid, and your final paragraph (after the numbered list) was the only part of the comment I found helpful.
In step 3, you seem to be trying to treat the groups and as if they were each a single player so that you can apply the conclusions from two-player games, but I don’t think that’s valid. The two-player result was based on an implicit assumption that transmitting a message from Beth to Adam cannot have any effect on the game except through Adam’s reaction, but that’s not true here because isn’t a unified agent, so transmitting can change the game (by affecting other members of ) even if refuses to react to it. does not get a veto on changing the game, like Adam does. Chris does not need to be listening in order for Adam and Beth to strike a mutually-beneficial deal at his expense.
So the inference that as a group cannot be profiting is invalid.
(Also note that your claim proves too much: If this were accepted, you haven’t proven that false messages are useless, you’ve proven that all messages are useless.)