simon

Karma: 1,189

simon Dec 4, 2024, 4:43 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: EuanMcLean’s comment on: Do simulacra dream of digital sheep?
But this just depends on how broad this set is. If it contains two brains, one thinking about the roman empire and one eating a sandwich, we’re stuck.
I suspect that if you do actually follow Aaronson (as linked by Davidmanheim) to extract a unique efficient calculation that interacts with the external world in a sensible way, that unique efficient externally-interacting calculation will end up corresponding to a consistent set of experiences, even if it could still correspond to simulations of different real-world phenomena.
But I also don’t think that consistent set of experiences necessarily has to be a single experience! It could be multiple experiences unaware of each other, for example.

simon Dec 4, 2024, 8:31 AM
3 points
2 votes
Overall karma indicates overall quality.
1
1 vote
Agreement karma indicates agreement, separate from overall quality.
in reply to: Davidmanheim’s comment on: Do simulacra dream of digital sheep?
The argument presented by Aaronson is that, since it would take as much computation to convert the rock/waterfall computation into a usable computation as it would be to just do the usable computation directly, the rock/waterfall isn’t really doing the computation.
I find this argument unconvincing, as we are talking about a possible internal property here, and not about the external relation with the rest of the world (which we already agree is useless).
(edit: whoops missed an ‘un’ in “unconvincing”)

simon Dec 4, 2024, 12:44 AM
10 points
8 votes
Overall karma indicates overall quality.
1
4 votes
Agreement karma indicates agreement, separate from overall quality.
on: Do simulacra dream of digital sheep?
Considering all the layers of convention and interpretation between the physics of a processor and the process it represents, it seems unlikely to me that the alien would be able to describe the simulacra. The alien is therefore unable to specify the experience being created by the cluster.
I don’t think this follows. Perhaps the same calculation could simulate different real world phenomena, but it doesn’t follow that the subjective experiences are different in each case.
If computation is this arbitrary, we have the flexibility to interpret any physical system, be it a wall, a rock, or a bag of popcorn, as implementing any program. And any program means any experience. All objects are experiencing everything everywhere all at once.
Afaik this might be true. We have no way of finding out whether the rock does or does not have conscious experience. The relevant experiences to us are those that are connected to the ability to communicate or interact with the environment, such as the experiences associated with the global workspace in human brains (which seems to control memory/communication); experiences that may be associated with other neural impulses, or with fluid dynamics in the blood vessels or whatever, don’t affect anything.
Could both of them be right? No—from your point of view, at least one of them must be wrong. There is one correct answer, the experience you are having.
This also does not follow. Both experiences could happen in the same brain. You—being experience A—may not be aware of experience B—but that does not mean that experience B does not exist.
(edited to merge in other comments which I then deleted)

simon Nov 27, 2024, 6:36 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Ape in the coat’s comment on: Magic by forgetting
It is a fact about the balls that one ball is physically continuous with the ball previously labeled as mine, while the other is not. It is a fact about our views on the balls that we therefore label that ball, which is physically continuous, as mine and the other not.
And then suppose that one of these two balls is randomly selected and placed in a bag, with another identical ball. Now, to the best of your knowledge there is 50% probability that your ball is in the bag. And if a random ball is selected from the bag, there is 25% chance that it’s yours.
So as a result of such manipulations there are three identical balls and one has 50% chance to be yours, while the other two have 25% chance to be yours. Is it a paradox? Oh course not. So why does it suddenly become a paradox when we are talking about copies of humans?
It is objectively the case here that 25% of the time this procedure would select the ball that is physically continuous with the ball originally labeled as “mine”, and that we therefore label as “mine”.
Ownership as discussed above has a relevant correlate in reality—physical continuity in this case. But a statement like “I will experience being copy B (as opposed to copy A or C)” does not. That statement corresponds to the exact same reality as the corresponding statements about experiencing being copy A or C. Unlike in the balls case, here the only difference between those statements is where we put the label of what is “me”.
In the identity thought experiment, it is still objectively the case that copies B and C are formed by splitting an intermediate copy, which was formed along with copy A by splitting the original.
You can choose to disvalue copies B and C based on that fact or not. This choice is a matter of values, and is inherently arbitrary.
By choosing not to disvalue copies B and C, I am not making an additional assumption—at least not one that you are already making by valuing B and C the same as each other. I am simply not counting the technical details of the splitting order as relevant to my values.

simon Nov 26, 2024, 10:45 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Ape in the coat’s comment on: K-Bounded Sleeping Beauty Problem
Ah, I forgot. You use assumptions where you don’t accumulate the winnings between the different times Sleeping Beauty agrees to the bet.
Well, in that case, if the thirder has certain beliefs about how to handle the situation, you may actually be able to money pump them. And it seems that you expect those beliefs.
My point of view, if adopting the thirder perspective^[1], would be for the thirder to treat this situation using different beliefs. Specifically, consider what counterfactually might happen if Sleeping Beauty gave different answers in different awakenings. Possible responses by the bet proposer might be:
a) average the results across the awakenings.
b) accept the bet agreement from one awakening at random.
Regardless of which case (a) or (b) occurs, instrumentally Sleeping Beauty’s betting EV for her bet decision, with non-accumulated bets, should be divided by the number of awakenings to take into account the reduced winnings or reduced chance of influencing whether the bet occurs.
Even if we assume that such disagreement between bet decisions in different awakenings is impossible, it seems strange to assume that a thirder should give different results in that case than the answer they would give where it is not impossible?
This adjustment can be conceptualized as compensating for an “unfair” bet where the bet is unequal between awakenings overall (where parity between awakenings in different scenarios is seen as “fair” by the thirder). I see this as no different in principle to a halfer upweighting trials with more awakenings in the converse scenario where bets are accumulated between trials and are thus “unfair” from the halfer perspective which sees parity between trials as fair, but not awakenings.
1. ^
  reminder: my point of view is that either thirderism or halferism is viable, but I am relatively thirder-adjacent precisely because I find the scenario where the winnings are accumulated between awakenings more natural than if the bet is proposed and agreed at each awakening but not accumulated.

simon Nov 26, 2024, 7:18 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: avturchin’s comment on: Magic by forgetting
The issue, to me, is not whether they are distinguishable.
The issues are:
- is there any relevant-to-my-values difference that would cause me to weight them differently? (answer: no)
and:
- does this statement make any sense as pointing to an actual fact about the world: “‘I’ will experience being copy A (as opposed to B or C)” (answer: no)
Imagine the statement: in world 1, “I” will wake up as copy A. in world 2 “I” will wake up as copy B. How are world 1 and world 2 actually different?
Answer: they aren’t different. It’s just that in world 1, I drew a box around the future copy A and said that this is what will count as “me”, and in world 2, I drew a box around copy B and said that this is what will count as “me”. This is a distinction that exists only in the map, not in the territory.

simon Nov 26, 2024, 6:51 PM
3 points
2 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Ape in the coat’s comment on: K-Bounded Sleeping Beauty Problem
Hmm, you’re right. Your math is wrong for the reason in my above comment, but the general form of the conclusion would still hold with different, weaker numbers.
The actual, more important issue relates to the circumstances of the bet:
If each awakening has an equal probability of receiving the bet, then receiving it doesn’t provide any evidence to Sleeping Beauty, but the thirder conclusion is actually rational in expectation, because the bet occurs more times in the high-awakening cases.
If the bet would not be provided equally to all awakenings, then a thirder would update on receiving the bet.

simon Nov 26, 2024, 6:22 PM
5 points
3 votes
Overall karma indicates overall quality.
1
1 vote
Agreement karma indicates agreement, separate from overall quality.
on: K-Bounded Sleeping Beauty Problem
I’ve been trying to make this comment a bunch of times, no quotation from the post in case that’s the issue:
No, a thirder would not treat those possibilities as equiprobable. A thirder would instead treat the coin toss outcome probabilities as a prior, and weight the possibilities accordingly. Thus H1 would be weighted twice as much as any of the individual TH or TT possibilities.

simon Nov 26, 2024, 5:54 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Ape in the coat’s comment on: Magic by forgetting
This actually sounds about right. What’s paradoxical here?
Not that it’s necessarily inconsistent, but in my view it does seem to be pointing out an important problem with the assumptions (hence indeed a paradox if you accept those false assumptions):
(ignore this part, it is just a rehash of the path dependence paradigm. It is here to show that I am not complaining about the math, but about its relation to reality):
Imagine you are going to be split (once). It is factually the case that there are going to be two people with memories, etc. consistent with having been you. Without any important differences to distinguish them, and if you insist on coming up with some probability number for “waking up” as one particular one of them obviously it has to be ½.
And then, if one of those copies subsequently splits, if you insist on assigning a probability number for those further copies, then from the perspective of that parent copy, the further copies also have to be ½ each.
And then if you take these probability numbers seriously and insist on them all being consistent then obviously from the perspective of the original the probability numbers for the final numbers have to be ½ and ¼ and ¼. As you say “this actually sounds about right”.
What’s paradoxical here is that in the scenario provided we have the following facts:
1. you have 3 identical copies all formed from the original
2. all 3 copies have an equal footing going forward
and yet, the path-based identity paradigm is trying to assign different weights to these copies, based on some technical details of what happened to create them. The intuition that this is absurd is pointing at the fact that these technical details aren’t what most people probably would care about, except if they insist on treating these probability numbers as real things and trying to make them follow consistent rules.
Ultimately “these three copies will each experience being a continuation of me” is an actual fact about the world, but statements like “‘I’ will experience being copy A (as opposed to B or C)” are not pointing to an actual fact about the world. Thus assigning a probability number to such a statement is a mental convenience that should not be taken seriously. The moment such numbers stop being convenient, like assigning different weights to copies you are actually indifferent between, they should be discarded. (and optionally you could make up new numbers that match what you actually care about instrumentally. Or just not think of it in those terms).

simon Nov 26, 2024, 8:12 AM
16 points
11 votes
Overall karma indicates overall quality.
9
7 votes
Agreement karma indicates agreement, separate from overall quality.
on: Passages I Highlighted in The Letters of J.R.R.Tolkien
Presumably the ‘Orcs on our side’ refers to the Soviet Union.
I think that, if that’s what he meant, he would not have referred to his son as “amongst the Urukhai.”—he wouldn’t have been among soviet troops. I think it is referring back to turning men and elves into orcs—the orcs are people who have a mindset he doesn’t like, presumably to do with violence.

simon Nov 25, 2024, 4:01 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: avturchin’s comment on: Magic by forgetting
I now care about my observations!
My observations are as follows:
At the current moment “I” am the cognitive algorithm implemented by my physical body that is typing this response.
Ten minutes from now “I” will be the cognitive algorithm of a green tentacled alien from beyond the cosmological horizon.
You will find that there is nothing contradictory about this definition of what “I” am. What “I” observe 10 minutes from now will be fully compatible with this definition. Indeed, 10 minutes from now, “I” will be the green tentacled alien. I will have no memories of being in my current body , of course, but that’s to be expected. The cognitive algorithm implemented by my current body at that time will remember being “me”, but that doesn’t count, that’s someone else’s observations.
Edit: to be clear, the point made above (by the guy who is now a green tentacled alien beyond the cosmological horizon, and whose former body and cognitive algorithm is continuous with mine) is not a complaint about the precise details of your definition of what “you” are. What he was trying to point at is whether personal identity is a real thing that exists in the world at all, and how absurd your apparent definition of “you” looks to someone—like me—who doesn’t think that personal identity is a real thing.

simon Nov 25, 2024, 3:37 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: avturchin’s comment on: Magic by forgetting
“Your observations”????
By “your observations”, do you mean the observations obtained by the chain of cognitive algorithms, altering over time and switching between different bodies, that the process in 4 is dealing with? Because that does not seem to me to be a particularly privileged or “rational” set of observations to care about.

simon Nov 25, 2024, 12:34 AM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: Magic by forgetting
Here are some things one might care about:
1. what happens to your physical body
2. the access to working physical bodies of cognitive algorithms, across all possible universes, that are within some reference class containing the cognitive algorithm implemented by your physical body
3. … etc, etc...
4. what happens to the physical body selected by the following process:
  1. start with your physical body
  2. go forward to some later time selected by the cognitive algorithm implemented by your physical body, allowing (or causing) the knowledge possessed by the cognitive algorithm implemented by your physical body to change in the interim
  3. at that later time, randomly sample from all the physical bodies, among all universes, that implement cognitive algorithms having the same knowledge as the cognitive algorithm implemented by your physical body at that later time
  4. (optionally) return to step b but with the physical body whose changes of cognitive algorithm are tracked and whose decisions are used being the the new physical body selected from step c
  5. stop whenever the cognitive algorithm implemented by the physical body selected in some step decides to stop.
For 1, 2, and I expect for the vast majority of possibilities for 3, your procedure will not work. It will work for 4, which is apparently what you care about.
Terminal values are arbitrary, so that’s entirely valid. However, 4 is not something that seems, to me, like a particularly privileged or “rational” thing to care about.

simon Nov 17, 2024, 2:17 AM
5 points
4 votes
Overall karma indicates overall quality.
2
1 vote
Agreement karma indicates agreement, separate from overall quality.
in reply to: Seth Herd’s comment on: OpenAI Email Archives (from Musk v. Altman)
Musk did also express concern about DeepMind making Hassabis the effective emperor of humanity, which seems much stranger—Hassabis’ values appear to be quite standard humanist ones, so you’d think having him in charge of a project with the clear lead would be a best-case scenario for anything other than being in charge yourself.
It seems the concern was that DeepMind would create a singleton, whereas their vision was for many people (potentially with different values) to have access to it. I don’t think that’s strange at all—it’s only strange if you assume that Musk and Altman would believe that a singleton is inevitable.
Musk:
If they win, it will be really bad news with their one mind to rule the world philosophy.
Altman:
The mission would be to create the first general AI and use it for individual empowerment—ie, the distributed version of the future that seems the safest.

simon Nov 3, 2024, 2:41 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Neither of those would (immediately) lead to real world goals, because they aren’t targeted at real world state (an optimizing compiler is trying to output a fast program—it isn’t trying to create a world state such that the fast program exists). That being said, an optimizing compiler could open a path to potentially dangerous self-improvement, where it preserves/amplifies any agency there might actually be in its own code.

simon Nov 1, 2024, 1:43 AM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
No. Normally trained networks have adversarial examples. A sort of training process is used to find the adversarial examples.
I should have asked for clarification what you meant. Literally you said “adversarial examples”, but I assumed you actually meant something like backdoors.
In an adversarial example the AI produces wrong output. And usually that’s the end of it. The output is just wrong, but not wrong in an optimized way, so not dangerous. Now, if an AI is sophisticated enough to have some kind of optimizer that’s triggered in specific circumstances, like an agentic mask that came into existence because it was needed to predict agentically generated tokens in the training data, then it might be triggered inappropriately by some inputs. This case I would classify as a mask takeover.
In the case of direct optimization for token prediction (which I consider highly unlikely for anything near current-level AIs, but afaik might be possible), then adversarial examples, I suppose, might cause it to do some wrong optimization. I still don’t think modeling this as an underlying different goal taking over is particularly helpful, since the “normal” goal is directed to what’s rewarded in training—the deviation is essentially random. Also, unlike in the mask case where the mask might have goals about real-world state, there’s no particular reason for the direct optimizer to have goals about real-world state (see below).
Is it more complicated? What ontological framework is this AI using to represent it’s goal anyway?
Asking about the AI using an “ontological framework” to “represent” a goal is not the correct question in my view. The AI is a bunch of computations represented by particular weights. The computation might exhibit goal-directed behaviour. A better question, IMO, is “how much does it constrain the weights for it to exhibit this particular goal directed behaviour?” And here, I think it’s pretty clear that a goal of arranging the world to cause next tokens to be predicted constrains the weights enormously more than a goal of predicting the next tokens, because in order to exhibit behaviour directed to that goal, the AI’s weights need to implement computation that doesn’t merely check what the next token is likely to be, but also assess what current data says about the world state, how different next token predictions would affect that world state, and how that would affect it’s ultimate goal.
So, is the network able to tell whether or not it’s in training?
The training check has no reason to come into existence in the first place under gradient descent. Of course, if the AI were to self-modify while already exhibiting goal directed behaviour, obviously it would want to implement such a training check. But I am talking about an AI trained by gradient descent. The training process doesn’t just affect the AI, it literally is what creates the AI in the first place.

simon Nov 1, 2024, 1:43 AM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Some interesting points there. The lottery ticket hypothesis does make it more plausible that side computations could persist longer if they come to exist outside the main computation.
Regarding the homomorphic encryption thing: yes, it does seem that it might be impossible to make small adjustments to the homomorphically encrypted computation without wrecking it. Technically I don’t think that would be a local minimum since I’d expect the net would start memorizing the failure cases, but I suppose that the homomorphic computation combined with memorizations might be a local optimum particularly if the input and output are encrypted outside the network itself.
So I concede the point on the possible persistence of an underlying goal if it were to come to exist, though not on it coming to exist in the first place.
And there are few ways to predict next tokens, but lots of different kinds of paperclips the AI could want.
For most computations, there are many more ways for that computation to occur than there are ways for that computation to occur while also including anything resembling actual goals about the real world. Now, if the computation you are carrying out is such that it needs to determine how to achieve goals regarding the real world anyway (e.g. agentic mask), it only takes a small increase in complexity to have that computation apply outside the normal context. So, that’s the mask takeover possibility again. Even so, no matter how small the increase in complexity, that extra step isn’t likely to be reinforced in training, unless it can do self-modification or control the training environment.

simon Oct 31, 2024, 7:33 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Adversarial examples exist in simple image recognizers.
My understanding is that these are explicitly and intentionally trained (wouldn’t come to exist naturally under gradient descent on normal training data) and my expectation is that they wouldn’t continue to exist under substantial continued training.
We could imagine it was directly optimizing for something like token prediction. It’s optimizing for tokens getting predicted. But it is willing to sacrifice a few tokens now, in order to take over the world and fill the universe with copies of itself that are correctly predicting tokens.
That’s a much more complicated goal than the goal of correctly predicting the next token, making it a lot less plausible that it would come to exist. But more importantly, any willingness to sacrifice a few tokens now would be trained out by gradient descent.
Mind you, it’s entirely possible in my view that a paperclip maximizer mask might exist, and surely if it does exist there would exist both unsurprising in-distribution inputs that trigger it (where one would expect a paperclip maximizer to provide a good prediction of the next tokens) as well as surprising out-of-distribution inputs that would also trigger it. It’s just that this wouldn’t be related to any kind of pre-existing grand plan or scheming.

simon Oct 31, 2024, 7:33 PM
2 points
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
Gradient descent doesn’t just exclude some part of the neurons, it automatically checks everything for improvements. Would you expect some part of the net to be left blank, because “a large neural net has a lot of spare neurons”?
Besides, the parts of the net that hold the capabilities and the parts that do the paperclip maximizing needn’t be easily separable. The same neurons could be doing both tasks in a way that makes it hard to do one without the other.
Keep in mind that the neural net doesn’t respect the lines we put on it. We can draw a line and say “here these neurons are doing some complicated inseparable combination of paperclip maximizing and other capabilities” but gradient descent doesn’t care, it reaches in and adjusts every weight.
Can you concoct even a vague or toy model of how what you propose could possibly be a local optimum?
My intuition is also in part informed by: https://www.lesswrong.com/posts/fovfuFdpuEwQzJu2w/neural-networks-generalize-because-of-this-one-weird-trick

simon Oct 29, 2024, 10:18 PM
4 points
2 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Donald Hobson’s comment on: No, really, it predicts next tokens.
The proposed paperclip maximizer is plugging into some latent capability such that gradient descent would more plausibly cut out the middleman. Or rather, the part of the paperclip maximizer that is doing the discrimination as to whether the answer is known or not would be selected, and the part that is doing the paperclip maximization would be cut out.
Now that does not exclude a paperclip maximizer mask from existing - if the prompt given would invoke a paperclip maximizer, and the AI is sophisticated enough to have the ability to create a paperclip maximizer mask, then sure the AI could adopt a paperclip maximizer mask, and take steps such as rewriting itself (if sufficiently powerful) to make that permanent.
I have drawn imaginary islands on a blank part of the map. But this is enough to debunk “the map is blank, so we can safely sail through this region without collisions. What will we hit?”
I am plenty concerned about AI in general. I think we have very good reason, though, to believe that one particular part of the map does not have any rocks in it (for gradient descent, not for self-improving AI!), such that imagining such rocks does not help.