Steffan Spalthos

Karma: 1

Steffan Spalthos 6 Jun 2026 2:17 UTC
1 point
0
on: Empowerment, corrigibility, etc. are simple abstractions (of a messed-up ontology)
On the topic of distinguishing between Counsel and Manipulation, without going directly through whether the resulting beliefs are accurate or not (Because that’s adequately covered and I think the observation that you can manipulate through selective truths somewhat defeats it- given someone who’s already wrong or who has underspecified preferences, you can probably manipulate their actions while only making their beliefs more correct.), I propose: That one(and maybe the) distinction between Counsel and Manipulation is whether the outcome of the action depends on the person being spoken to.
That is: If the AI wants and manipulates for Pink Ponies, then it will tailor its arguments to make sure, no matter who it is speaking to, they will conclude the right answer is pink ponies. We do not need to know who the AI is talking to; maybe they hate horses. Maybe they’re obsessed with manly colors. Regardless, we know the outcome will end up being pink ponies. The AI may need to simulate the target in detail to manipulate them, but the AI can predict where the conversation will go without doing the simulation.
If the AI is counselling, then it will not do this. It might tell the target true facts, which are relevant to the target’s preferences and choices, allowing them to make better decisions; or it still might extrapolate the target’s values and then tell the target selective facts to guide them to that outcome^[1]. But regardless, you will need to know what the target is actually like to figure out what is going to happen, and changing the target’s identity will result in corresponding (and probably proportional) changes to the outcome.
This does mean the distinction between counselling and manipulation applies to the AI’s decision-making, and not to any particular series of words or actions, but that’s just what… that word means. If I advise you to play a slot machine I claim is bugged to give out extra jackpots, I’m counselling you if I believe that’s a good idea and manipulating you if I know you’ll still run out of money before the jackpot, even if the machine is the same; the distinction being made is indeed, my thought process.
(To formalize this you’d need to define ‘depends on the person being spoken to’ better, I imagine. an AI that lets the target implicitly-pick between pink ponies and magenta mermaids by demonstrated-receptiveness-in-conversation then manipulates them in the opposite direction from there is depending on the person a little, but much less than actual counsel and it still involves a lot of manipulation. I’m not totally sure whether or not it’s desirable to specify ‘and depends on the intermediate states of conversation’, because as intelligent beings one would hope we’d sometimes converge on the same answers given enough helpful advice.)
1. ^
  I think in this case what the AI is doing is somewhere between counselling and manipulation, where the outcome depends on the target, but only on the extrapolated-values from the start of the process, and so you don’t need to know most of the person or any of the intermediate states to know the conclusion, even though you need to know a little.

Steffan Spalthos 23 May 2026 0:57 UTC
1 point
0
in reply to: TsviBT’s comment on: A relatively brief explanation of Boltzmann Brains
In a sense, yes- of course you can construct something without any given characteristic of a mind, such as an inert rock or a pair of dice.
That said, I’d argue the presence of something like a simplicity prior- not necessarily something that fits the formal definition of a prior, but some sort of tendency for simple beliefs over complicated ones- is a necessity of having beliefs of any kind at all.
For instance- you might have no prior belief, but whenever you generate a belief of any kind, assign it 50% probability. This isn’t a prior(it’s more like a change in how you update from your prior) and you end up with very stupid beliefs; but the ‘generate a belief of any kind’ step necessarily encodes a simplicity tendency, weakly in the sense that more complicated beliefs broadly require more of a prompt to pick out in the first place, and also strongly in the sense that your beliefs cannot be of infinite length, and any finite belief encodes a simplicity tendency over the far-more-numerous versions that are twice as complex. To generate even a single belief requires something at least resembling a simplicity prior.
You can dispense with having beliefs at all as well, but by that point we’re essentially dispensing with being a mind at all,^[1] and that means you can no longer have a self-ratifying state of a mind for lack of a mind;^[2] and if you have to unwind to a rock to avoid having a simplicity-tendency then that’s about as far from it being optional as is possible.^[3]
1. ^
  I think for the purposes of this discussion many types of thoughts can be thought of as either being a type of belief or implying a belief, in that they follow the same rules of requiring some sort of preference for simplicity… either directly or just because they’re produced through a sequential process that takes finite time to reach a stopping point.
2. ^
  and also because ‘this state of mind is more correct’ is a belief.
3. ^
  ...I think I may have somewhat lost the thread here. The original argument here is something like ‘perhaps the reason the world seems orderly and not blotzmany, is just that we’re sneaking in that assumption by using reasoning that assumes an orderly world’, right? I guess this is somewhat true, in the sense that anthropically we can’t perceive something no mind perceives and you need a certain amount of order for a mind.^[4] But it doesn’t seem to be true in a way that would work to avoid blotzman brains, and if we’re sneaking in that assumption anywhere I think it’s located inside our definition of ‘mind’ rather than how we reason about that definition.
4. ^
  (I actually expect- Insofar as we can draw any conclusions from this kind of subjective observation at all, which is not obviously the case- a stronger version of this where, ie, my subjective observations don’t draw from the pool of insect observations or even those of any other person, and so it’s not surprising that I don’t find myself being a fly even if there are a lot of flies, because anyone who finds themself a fly is a fly and not me, but that’s besides the point because blotzman brains can still match that criteria.)

Steffan Spalthos 22 May 2026 14:14 UTC
1 point
0
in reply to: TsviBT’s comment on: A relatively brief explanation of Boltzmann Brains
On the topic of unwinding to some state of mind besides the simplicity prior- are the alternatives not uncomputable as a class?
That is: There is a finite number of theories below a certain level of complexity, so the simplicity prior allocates a finitely-small-amount-of its expectation to a finite number of outcomes. For the opposite of the simplicity prior, a complexity prior- where you prefer a theory more the more complicated it is- any theory you consider has an infinite number of even more complicated counterparts, because you can always make the theory more complicated by adding another detail, such that you replace ‘gravity’ with ‘gravity, except on tuesdays’ with ‘gravity except on tuesdays, except for next tuesday where there will be gravity after all’ and so on forever. As such, it’s impossible to compute the complexity prior.^[1]
It appears this should apply to any prior that doesn’t contain the simplicity prior- Even without weighing towards complexity, if you don’t weigh towards simplicity, every question has an infinite number of equally good answers, like in the gravity answer above.
Priors which contain the simplicity prior have a similar problem, but one that’s closer to leaning on the simplicity prior itself rather than just the implementability of the prior- that is, you can implement ‘the simplicity prior, but theories with an even number of bits/words/etc are more likely’, but, with there being an infinite number of possible simplicity-prior-variations like this, making a decision between these possible priors, without using the simplicity prior, is itself uncomputable.^[2]
A dynamic being impossible to implement- not just due to limited computational power but because there is no right answer within the dynamic, seems like a good reason not to implement it, that doesn’t itself depend on the simplicity prior.
1. ^
  If I were trying to compute the complexity prior in finite time, I’d want to aggregate entire infinite classes of outcomes into one larger claim that shares all their details to save time, IE, ‘all theories wherein gravity works most of the time’. However, in doing so I would just be turning the complexity prior back into the simplicity prior, because the weight of a hypothesis-class is proportional to the fraction of all possible hypotheses which fall under it, which is proportional to its simplicity!
2. ^
  and also, if you picked a prior at random from this set, it would probably tell you to adopt the simplicity prior instead, because, ie, ‘simplicity but even-bitted theories are x2 likely’ adds more than 2 bits of complexity over the simplicity prior. Though I’m not sure whether it works out to ‘slightly over 50% odds of the simplicity prior’^[3] or ‘approximately 100% odds of the simplicity prior’, it depends on how we abstract this infinity; the former if we figure each specific simplicity-prior-variant is equally likely to favor or not favor itself and that most of them do this by a huge amount that more than cancels out their complexity and so you get to the simplicity prior whenever a prior doesn’t favor itself, the latter if complexity tends to grow faster than self-favoring.
3. ^
  or slightly over ¹⁄₃ odds of the simplicity prior, if a prior that doesn’t favor itself has a 50% chance to not favor the simplicity prior either and end up on a new random prior that repeats the calculation. Or ~0% odds if each prior is so complex and specific that the odds of any prior coming up simplicity are minimal… or maybe that goes back to 50 / 100% because the odds of a prior not favoring itself the most go up by the same amount… It’s complicated but probably not relevant to the conclusion.

Steffan Spalthos 26 Aug 2019 21:28 UTC
2 points
0
on: Moloch’s Toolbox (1/2)
cecie: You can’t buy the sovereignty on the land. Even if you had a lot of money, any country poor enough and desperate enough to consider your offer might just steal your stuff after you moved in.
Negotiating the right to bring in weapons to defend yourself in this kind of scenario would be even more unthinkable, and would spark international outrage that could prevent you from trading with other countries.
To be clear, it’s not that there’s a global dictator who prevents new countries from popping up; but every potentially useful part of every land is under some system’s control, and all of those systems would refuse you the chance to set up your own alternative system, for very similar reasons.
visitor: So there’s no way for your planet to try different ways of doing things, anywhere. You literally cannot run experiments about things like this.
Technically speaking, there is a large chunk of unclaimed land in Antarctica which nobody has sovereignty of. Antarctica isn’t the best place to be, but it’s at least Possible to colonise by this stage. Not easily, but possible.