Wei_Dai
Karma: 24,983
NewTop
I think it is likely best to push against including that sort of thing in the Overton window of what’s considered AI safety / AI alignment literature.
I’m really sympathetic to these concerns but I’m worried about the possible unintended consequences of trying to do this. There will inevitably be a large group of people working on short and medium term AI safety (due to commercial incentives) and pushing them out of “AI safety / AI alignment literature” risks antagonizing them and creating an adversarial relationship between the two camps, and/or creates a larger incentive for people to stretch the truth about how robust to scale their ideas are. Is this something you considered?
Do you see your “not updating” scheme as the appropriate new theory applicable to very large universes?
It doesn’t fully solve problems associated with very large universes, but I think it likely provides a framework in which those problems will eventually be solved. See this post for more details.
See also this post which explains my current views on the nature of probabilities, which may be needed to understand the “not updating” approach.
If so, does it in fact give the same result as applying FNC while assuming the universe is not so large?
Sort of. As I explained in a linked comment, when you apply FNC you assign zero probability to the universes not containing someone with your memories and then renormalize the rest, but if your decisions have no consequences in the universes not containing someone with your memories, you end up making the same decisions whether you do this “updating” computation or not. So “not updating” gives the same result in this sense.
I do want to stay in explore-options and gather data mode
What kind of analysis are you thinking of doing on the data that you’re gathering? I’m curious, and also pre-registration may be a good idea in situations like this to reduce bias.
I’m curious how you feel about solutions that are more in the direction of ″make karma more illegible?″ – possibly literally just randomizing +/- a few points?
Doesn’t seem like it helps as much as other possibilities and the cost seems substantial (in implementation, user education, getting used to it psychologically, maybe other unforeseen consequences).
There’s also the option of “you can upvote UP TO your max-karma, but can choose any amount”. The two reasons we didn’t go with this was that it felt like asking people to making too-granular a choice every time they voted
I agree with this.
But I took it seriously enough to come up with a counter-argument against it. Doesn’t that count for something? :)
To be clear I’m referring to the second post in that thread, where I wrote:
Let me try to generalize the argument that under the universal prior the 1UH gives really wierd results. The idea is simply that any sufficiently large and/or long universe that doesn’t repeat has a good chance of including a person with mind state M, so knowing that at least one person with mind state M exists in the real universe doesn’t allow you to eliminate most of them from the set of possible universes. If we want to get a result that says the real universe is likely to be in a class of intuitively acceptable universes, we would have to build that directly into our prior. That is, make them a priori more likely to be real than all other large/long universes.
Several questions follow if this argument is sound. First, is it acceptable to consciously construct priors with a built in preference for intuitively acceptable universes? If so how should this be done? If not the 1UH is not as intuitive as we thought. We would have to either reject the 1UH or accept the conclusion that the real universe is likely to be really weird.
(In that post 1UH refers to the hypothesis that only one universe exists, and I was apparently assuming that what you call FNC is the only way to do Bayesian updating under 1UH so I was thinking this is an argument against 1UH, but looking at it now, it’s really more of an argument against FNC.)
It looks like I should actually claim priority for this idea myself, since I came up with something very similar on the way to UDT. From this 1998 post:
One piece of information about the real universe you have direct access to is your own mind state. This is captured in the statement D = “The real universe contains at least one person with mind state M” where M is your current mind state. I’m going to assume this is the ONLY piece of information about the real universe you have direct access to. Everything else must be computed from the prior and this data. The justification for this is that I can’t think of any other information that is not part of or derived from D.
Right away you know that any universe that does not contain at least one person with mind state M cannot be real. It’s also not hard to see that for any two universes that both contain at least one person with mind state M, the ratio of their posterior probabilities is the same as the ratio of their priors. This means the universe most likely to be real given D is the one that has the highest prior among the universes that contain at least one person with mind state M.
Instead, start with a prior chosen somehow, then update it according to the evidence that a being such as you exists in the universe.
This seems very similar to Radford Neal’s full non-indexical conditioning:
I will here consider what happens if you ignore such indexical information, conditioning only on the fact that someone in the universe with your memories exists. I refer to this procedure as “Full Non-indexical Conditioning” (FNC).
Note however that Neal says this idea implies that the thirder answer is correct in Sleeping Beauty:
In this regard, note that the even though the experiences of Beauty upon wakening on Monday and upon wakening on Tuesday (if she is woken then) are identical in all “relevant” respects, they will not be subjectively indistinguishable. On Monday, a fly on the wall may crawl upwards; on Tuesday, it may crawl downwards. Beauty’s physiological state (heart rate, blood glucose level, etc.) will not be identical, and will affect her thoughts at least slightly. Treating these and other differences as random, the probability of Beauty having at some time the exact memories and experiences she has after being woken this time is twice as great if the coin lands Tails than if the coin lands Heads, since with Tails there are two chances for these experiences to occur rather than only one. This computation assumes that the chance on any given day of Beauty experiencing exactly what she finds herself experiencing is extremely small, as will be the case in any realistic version of the experiment.
Assuming your idea is the same as FNC, I think from a decision theory perspective it’s still worse than just not updating. See this comment of mine under a previous post about FNC.
I do this myself for a couple of reasons:
Laziness—the marginal benefit of voting on something decreases with the absolute value of its current karma, but the cost of voting stays constant.
To prevent the “rich get richer” phenomenon, where if everyone pays more attention to posts/comments that have higher karma, but votes independently without regard to current karma, a comment or post that initially gets some upvotes would attract more attention and more upvotes. Similarly, if everyone did that, comments that arrive late or are in deeper threads or in unpopular posts will have much lower karma than more visible comments of similar quality. In other words, the karma would reflect visibility*quality rather than just quality and it would be hard for readers to recover the quality signal from that.
Doesn’t this attitude obviously defeat the entire purpose of the voting system?!
Can you say more about this?
Having a high amount of voting power basically feels like a disadvantage to me instead of a benefit, because it makes me more reluctant to exercise strong voting power. Maybe other people won’t be able to tell who voted on something, but the part of my brain that worries about this kind of thing isn’t really mollified by your data point.
Maybe as a compromise that ameliorates both of the problems I mentioned, consider capping the strong voting power of everyone at some low amount, like 5?
If voting takes an hour, then it’s worth it iff you’re otherwise winning less than 10 picolightcones per lifetime.
Do you mean “per hour” instead?
If a lifetime is 100,000 hours, that means less than a microlightcone per lifetime.
Have you thought about how to estimate microlightcone per lifetime from our other x-risk activities?
Intuitively it seems like everyone except maybe the most productive x-risk activists probably have less than 10 picolightcones per hour of marginal productivity. Does that seem right to you?
Another issue that I don’t know if you’ve thought of: strong votes from people with high karma are not very anonymous. If I’m talking with someone and I strong up/down vote them (like here recently), it’s pretty obvious who did it, yet not fully common knowledge, which makes the situation even more taxing on the social part of my brain than just straightforward non-anonymous voting (like on Facebook).
I think you’re right, for x-risk logical correlation seems irrelevant. So I guess we instead want to know whether voting is good for reducing x-risk assuming that the opportunity cost comes entirely from other x-risk reducing activities, and if not, can a case for voting be made based on both x-risk and other (e.g., selfish) considerations where logical correlation is relevant.
ETA: Ironically, if everyone bought into the CDT argument for not voting based on self interest, much fewer people would vote and it would be a lot easier for people like us to flip elections based on x-risk concerns.
1) It feels like passing the buck, which is a known antipattern in thinking about AI.
Not sure what you mean by this or by “more object-level solution to alignment”. Please explain more?
2) With a “soft” self-improving entity, like a team of people and AIs, most invariants you can define will also be “soft” and prone to drift over many iterations.
Yeah I agree with this part. I think defining an invariant that is both “good enough” and achievable/provable will be very hard or maybe just impossible.
I want to suggest having new posts/comments be normal upvoted by default (by the author) instead of being strong upvoted, for a couple of reasons.
Sometimes I just want to make a not very important or not very insightful comment, which doesn’t really deserve to have a higher karma (or to attract more attention) than many other comments, but if I manually drop it to normal upvote, people might interpret that as others having downvoted it.
Having a initially high karma on a post or comment makes it less likely for others to upvote it further (or reduces the total amount of upvotes from others), which reduces the positive feedback from making posts and comments for people who have high karma / voting power.
(I think I might actually prefer to go back to the LW1 system we had at the end, but I’m not sure if that’s a rational response or if I’m just used to it and don’t like change anymore.)
The idea of “mathematical proof” is useful if someone has never thought of the concept before. For more specifics you need to look at individual proposed proofs. Similarly, people have proposed specific approaches for how to develop a safe AI, which we can look at if we want to know “which individual steps are valid or safe” but having a more general concept seems useful if you hadn’t thought of that before. (I did state that this “may be trivial or obvious for a lot of people”, and also talked about what I personally got out of thinking this way in the paragraph just below the box.)
A general model of safety-oriented AI development
Do you know if anyone has done/published a calculation on whether, given reasonable beliefs about (i.e., a large amount of uncertainty over) opportunity costs and logical correlations, voting is actually a good thing to do from an x-risk perspective?
You should contact Rob Bensinger since he’s the owner of the document in question. (It looks like I technically can share the document with others, but I’m not sure what Rob/MIRI’s policy is about who that document should be shared with.)
I don’t understand how you would get significant conclusions out of this without big multipliers. Yes, there are some participants in your parliament who care more about worlds other than this one. Those worlds appear to be significantly harder to influence (by means other than trade), so this doesn’t seem to have a huge effect on what you ought to do in this world. (Assuming that we are able to make trades that we obviously would have wanted to make behind the veil of ignorance.)
Wait, you are assuming a baseline/default outcome where acausal trade takes place, and comparing other interventions to that? My baseline for comparison is instead (as stated in the OP) “what can be gained by just creating worthwhile lives in this universe”. My reasons for this are (1) I (and likely others who might read this) don’t think acausal trade is much more likely to work than the other items on my list and (2) the main intended audience for this post is people who have realized the importance of influencing the far future but not aware of (or have seriously considered) the possibility of influencing other universes through things like acausal trade and other items on my list. Even the most sophisticated thinkers in EA seem to fall into this category, e.g., people like Will MacAskill, Toby Ord, and Nick Beckstead, unless they’ve privately considered the possibility and chose not to talk about it in public, in which case it still seems safe to assume that most people in EA think “creating worthwhile lives in this universe” is the most good that can be accomplished.
In particular, if your ratio between the value of big and small universes was only 5x, then that would only have a 5x multiplier on the value of the interventions you list in the OP. Given that many of them look very tiny, I assumed you were imagining a much larger multiplier. (Something that looks very tiny may end up being a huge deal, but once we are already wrong by many orders of magnitude it doesn’t feel like the last 5x has a huge impact.)
I don’t understand where “5x” comes from or why that’s the relevant multiplier instead of 100x.
It would be useful if you could make a clear statement of why you think this problem is important
I’ll think about this, but I think I’d be more motivated to attempt this (and maybe also have a better idea of what I need to do) if other people also spoke up and told me that they couldn’t understand my past attempts to explain this (including what I wrote in the OP and previous comments in this thread).
I’m saying that there is also a significant contingent that cares about our universe, so the people who care about other universes aren’t going to dominate.
I don’t think I’m getting your point here. Personally it seems safe to say that >80% of the contingent of my moral parliament that cares about astronomical waste would say that if our universe was capable of 10^(10^120) operations it would be at least 100x as valuable as if was capable of only 10^120 operations. Are your numbers different from this? In any case, what implications are you suggesting based on “no domination”?
(And on top of that, all of the contingents are roughly just trying to maximize the “market value” of what we get, so for the most part we need to reason about an even more spread out distribution.)
I don’t understand this part at all. Please elaborate?
There are tons of ways you could get people to do something they won’t choose to do.
I did preface my conclusion with “The best opportunity to do this that I can foresee”, so if you have other ideas about what someone like me ought to do, I’d certainly welcome them.
I don’t know if “give them an AI that doesn’t defer to them about philosophy” is more natural than e.g. “give them an AI that doesn’t defer to them about how they should deliberate/learn/self-modify.”
Isn’t “how they should deliberate/learn/self-modify” itself a difficult philosophical problem (in the field of meta-philosophy)? If it’s somehow easier or safer to “give them an AI that doesn’t defer to them about how they should deliberate/learn/self-modify” than to “give them an AI that doesn’t defer to them about philosophy” then I’m all for that but it doesn’t seem like a very different idea from mine.
I think you’re right that the result I cited doesn’t actually resolve your question. I looked around a bit more and found Theorem 4 in this paper, which looks more relevant. If you’re still not convinced after looking at that, can you explain why you think “the next million bits are expected to raise K-complexity by much less than a million”?
ETA: On second thought this result doesn’t help either, because if μ is the uniform measure then the bound you get with Theorem 4 after observing a million random bits isn’t any better than what you get at the start before you observed anything. To take another tack, consider your statement:
If failing makes you update away from the universal prior, what are you updating towards? Suppose you update towards the uniform measure. But that means the actual prior you had before updating is some mixture of the universal prior and the uniform measure, but that the mixture is itself a universal prior, so you can just consider yourself as having updated within a universal prior instead.