This post contains my answer.
Wei_Dai
Karma: 31,067
Can you give some specific examples of me having security mindset, and why they count as having security mindset? I’m actually not entirely sure what it is or that I have it, and would be hard pressed to come up with such examples myself. (I’m pretty sure I have what Eliezer calls “ordinary paranoia” at least, but am confused/skeptical about “deep security”.)
Combining hash functions is actually trickier than it looks, and some people are doing research in this area and deploying solutions. See https://crypto.stackexchange.com/a/328 and https://tahoe-lafs.org/trac/tahoe-lafs/wiki/OneHundredYearCryptography. It does seem that if cryptography people had more of a security mindset (that are not being defeated) then there would be more research and deployment of this already.
Maybe this could be done by something else besides the core LW team, but I’d like to have an Android app for LW and EA Forum, that would give me periodic notifications for my inbox, posts/authors/threads I subscribe to, karma changes, maybe new high-karma posts and comments, so I don’t have to constantly refresh many different pages on LW/GW/EAF to keep up with what’s going on. Having to do that is really a pain when you’re trying to use forum participation as a research strategy .
I’ve been thinking about writing this app myself, but thought I’d ask first to see if anyone else wants to do it.
I don’t see anything wrong with what you’re saying, but if you did that you’d end up not being an indexically selfish person anymore. You’d be selfish in a different, perhaps alien or counterintuitive way. So you might be reluctant to make that kind of commitment until you’ve thought about it for a much longer time, and UDT isn’t compatible with your values in the meantime. Also, without futuristic self-modification technologies, you are probably not able to make such a commitment truly binding even if you wanted to and you tried.
I suggest a “I already replied” indicator for messages in my inbox. Also, flags I can set on certain comments/messages to indicate that I should reply to them with high priority. (These could basically work the same way as in a typical email client.) This could be integrated with the bookmark feature that I suggested earlier.
Also, is it just me or does development happen rather slowly on LW? I’ve done some web development myself and it seems like on a codebase that I’m familiar with, it would take a few weeks at most to implement some of the feature suggestions that have been sitting here for many months.
From the perspective of an organism trying to propagate its genes, sex is like a trade: I’ll put half of your DNA in my offspring if you put half of my DNA in yours. I still pass one copy of my genes onto the next generation per unit of investment in children, so it’s a fair deal. And it doesn’t impact the average fitness of my kids very much, since on average my partner’s genes will be about as good as mine.
Wait, you seem to be assuming that both parents invest equally in the offspring, but in most (vast majority?) of species, one sex invests more than the other. In some species the male makes virtually no investment at all, and what you say here clearly do not apply to those species.
That’s true, but they could say, “Well, given that no binding commitment was in fact made, and given my indexically selfish values, it’s rational for me to choose Right.” And I’m not sure how to reply to that, unless we can show that such indexically selfish values are wrong somehow.
So if anyone has a safety idea to which utility uncertainty feels central
These two posts looked at some possibilities for using utility uncertainty but they didn’t seem that promising and I don’t know if anyone is still going in these directions:
Sorry, I think you misunderstood my question about combining human imitations with more general oracles/predictors. What I meant is that you could use general oracles/predictors to build models of the world, which the human imitators could then query or use to test out potential actions. This perhaps lets you overcome the problem of human imitators having worse world models than ~AIXI and narrows the capability gap between them.
I’m pretty sure N = 7 billion is enough.
Why? What are those 7 billion HSIFAUH doing?
In another comment you said “If I’m understanding correctly, the concern is that the imitator learns how humans plan before learning what humans want, so then it plans like a human toward the achievement of some inhuman goal. I don’t think this causes an existential catastrophe.” But if there are 7 billion HSIFAUH which are collectively capable of taking over the world, how is not a potential existential catastrophe if they have inhuman values?
Or maybe the right way to look at it is whether N = 10 could finance a rapidly exponentially growing N.
How? And why would it grow fast enough to get to a large enough N before someone deploys ~AIXI?
It should be possible to weaken the online version and get some of this speedup.
What do you have in mind here?
I don’t know how to do this. But it’s the same stuff the reckless team is doing to make standard RL powerful.
You do have to solve some safety problems that the reckless team doesn’t though, don’t you? What do you think the main safety problems are?
I furthermore, I think that should be a universal policy on LessWrong, though maybe this is just an idiosyncratic neurosis of mine.
If it’s not just you, it’s at least pretty rare. I’ve seen the mods “helpfully” edit posts several times (without asking first) and this is the first time I’ve seen anyone complain about it.
It seems important to acknowledge that there’s a version of the Bomb argument that actually works, at least if we want to apply UDT to humans as opposed to AIs, and this may be part of what’s driving Will’s intuitions. (I’ll use “UDT” here because that’s what I’m more familiar with, but presumably everything transfers to FDT.)
First there’s an ambiguity in Bomb as written, namely what does my simulation see? Does it see a bomb in Left, or no bomb? Suppose the setup is that the simulation sees no bomb in Left. In that case since obviously I should take Left when there’s no bomb in it (and that’s what my simulation would do), if I am seeing a bomb in Left it must mean I’m in the 1 in a trillion trillion situation where the predictor made a mistake, therefore I should (intuitively) take Right. UDT also says I should take Right so there’s no problem here.
Now suppose the simulation is set up to see a bomb in Left. In that case, when I see a bomb in Left, I don’t know if I’m a simulation or a real person. If I was selfish in an indexical way, I would think something like “If I’m a simulation then it doesn’t matter what I choose. The simulation will end as soon as I make a choice so my choice is inconsequential. But if I’m a real person, choosing Left will cause me to be burned. So I should choose Right.” The thing is, UDT is incompatible with this kind of selfish values, because UDT takes a utility function that is defined over possible histories of the world and not possible centered histories of the world (i.e., histories with an additional pointer that says this is “me”). UDT essentially forces an agent to be altruistic to its copies, and therefore is unable to give the intuitively correct answer in this case.
If we’re doing decision theory for humans, then the incompatibility with this kind of selfish values would be a problem because humans plausibly do have this kind of selfish values as part of our complex values and whatever decision theory we use perhaps should be able to handle it. However if we’re building an AI, it doesn’t seem to make sense to let it have selfish values (i.e., have a utility function over centered histories as opposed to uncentered histories), so UDT seems fine (at least as far as this issue is concerned) for thinking about how AIs should ideally make decisions.
Even though alignment may take more years/decades to solve in this scenario, it’s a much safer environment to do so.
It seems safer, but I’m not sure about “much safer”. You now have an extremely powerful AI that takes human commands, lots of people and governments would want to get their hands on it, and geopolitics is highly destabilized due to your unilateral actions. What are your next steps to ensure continued safety?
Although, these hypotheticals are unlikely (their purpose is pedagogical). It’s likely due to my ignorance, but I am unaware of any pivotal acts attached to anyone’s research agenda.
I think the examples in that Arbital post are actually intended to be realistic examples (i.e., something that MIRI or at least Eliezer would consider doing if they managed to build a safe and powerful task AGI). If you have reason to think otherwise, please explain.
What we want the training process to produce is a mesa-optimizer that tries to minimize the actual distance between its output and the training label (i.e., the actual loss).
Hmm, actually this still doesn’t fully address TurnTrout’s (Alex Turner’s) concern, because this mesa-optimizer could try to minimize the actual distance between its output and the training label by changing the training label (what that means depends on how the training label is defined within its utility function). To do that it would have to break out of the box that it’s in, which may not be possible, but this is still a system that is “looking for ways to hurt you.” It seems that what we really want is a mesa-optimizer that tries to minimize the actual loss while pretending that it has no causal influence on the training label (even if it actually does because there’s a way to break out of its box).
This seems like a harder inner alignment problem than I thought, because we have to make the training process converge upon a rather unnatural kind of agent. Is this still a feasible inner alignment problem to solve, and if not is there another way to get around this problem?
As the paper acknowledges, this introduces several risks, and so it calls for deep engagement with AI safety researchers (but sadly it does not propose ideas on how to mitigate the risks).
As far as I can tell, IA-GA doesn’t fit into any of the current AI safety success stories, and it seems hard to imagine what kind of success story it might fit into. I’m curious if anyone is more optimistic about this.
Thanks for the references. I think I should also credit you with being the first to use “success story” the way I’m using it here, in connection with AI safety, which gave me the idea to write this post.
It’s not the same as your Interim Quality-of-Life Improver, but it’s got similar aspects.
The main difference seems to be that you don’t explicitly mention strong global coordination to stop unaligned AI from arising. Is that something you also had in mind? (I seem to recall someone talking about that in connection with this kind of scenario.)
It’s also related to the concept of a “Great Deliberation” where we stabilize the world and then figure out what we want to do. (I don’t have a reference for that though.)
There’s also Will MacAskill and Toby Ord’s “the Long Reflection” (which may be the same thing that you’re thinking of), which as far as I know isn’t written up in detail anywhere yet. However I’m told that both of their upcoming books will have some discussions of it.
I vaguely agree with this concern but would like a clearer understanding of it. Can you think of a specific example of how this problem can happen?
When I use the term “RL agent,” I always mean an agent trained via RL.
I think the problem with this usage is that “RL agent” originally meant something like “an agent designed to solve a RL problem” where “RL problem” is something like “a class of problems with the central example being MDP”. I think it’s just not a well-defined term at this point, and if you Google it, you get plenty of results that say things like “the goal of our RL agent is to maximize the expected cumulative reward”, or “AIXI is a reinforcement learning agent”. I guess this is fine for AI capabilities work but really confusing for AI safety work.
So, consider switching to “RL-trained agent” for greater clarity (unless someone has a better suggestion)? ETA: Maybe “reinforcement trained agent”?
I’m actually thinking more about comments than PMs. There are comments (either addressed to me or just general comments) that I want to make sure to eventually read/digest/respond to, and right now there’s no good way to keep track of that.
Not sure what you mean by “this thread”, but I still want to see the backlinks feature, and this.