Wei_Dai
Karma: 26,594
NewTop
Thanks, I think I understand now, and made some observations about EDT+SSA at the old thread. At this point I’d say this quote from the OP is clearly wrong:
So, we could say that CDT+SIA = EDT+SSA = UDT1.0; or, CDT=EDT=UDT for short.
In fact UDT1.0 > EDT+SSA > CDT+SIA, because CDT+SIA is not even able to coordinate agents making the same observation, while EDT+SSA can do that but not coordinate agents making different observations, and UDT1.0 can (probably) coordinate agents making different observations (but seemingly at least some of them require UDT1.1 to coordinate).
I noticed that the sum inside is not actually an expected utility, because the SSA probabilities do not add up to 1 when there is more than one possible observation. The issue is that conditional on making an observation, the probabilities for the trajectories not containing that observation become 0, but the other probabilities are not renormalized. So this seems to be part way between “real” EDT and UDT (which does not set those probabilities to 0 and of course also does not renormalize).
This zeroing of probabilities of trajectories not containing the current observation (and renormalizing, if one was to do that) seems at best useless busywork, and at worst prevents coordination between agents making different observations. In this formulation of EDT, such coordination is ruled out in another way, namely by specifying that conditional on o→a, the agent is still sure the rest of π is unchanged (i.e., copies of itself receiving other observations keep following π). If we remove the zeroing/renormalizing and say that the agent ought to have more realistic beliefs conditional on o→a, I think we end up with something close to UDT1.0 (modulo differences in the environment model from the original UDT).
(Oh, I ignored the splitting up of probabilities of trajectories into SSA probabilities and then adding them back up again, which may have some intuitive appeal but ends up being just a null operation. Does anyone see a significance to that part?)
𝔼[X|Φ(𝔼¹, 𝔼²)] = 𝔼[𝔼¹[X]|Φ(𝔼¹, 𝔼²)]
What if X is something that only 𝔼 knows, like a random number that the agent that 𝔼 represents just thought up? Then there would be no way for this equality to hold? Maybe it should be something like this instead?
𝔼[X|𝔼¹[X]] = 𝔼[X|𝔼¹[X], Φ(𝔼¹, 𝔼²)]
I’m not sure this is right either, because if 𝔼 is smart enough maybe it can extract some information about X from Φ(𝔼¹, 𝔼²) that’s not in 𝔼¹[X]. Edit: Never mind, that’s fine because it just means that 𝔼¹ has to be smart enough that 𝔼¹[X] takes into account everything that 𝔼 might be able to extract from Φ(𝔼¹, 𝔼²).
The author suggests that just slowing down research into risky technologies in other countries would be worthwhile:
The lack of acceleration of science following the high skill immigration shock to the US is not necessarily bad news: it may also imply that future shocks won’t accelerate risky technologies, that research funding is a more fundamental constraint, or that other sectors of the economy are better at absorbing high skill immigrants.
Further emigration likely decelerated progress for potentially risky technologies in the former USSR, which is a net reduction of risk: there is less incentive for the US government to engage in an arms race if there is no one to race.
The result there is sometimes abbreviated as UDT=CDT+SIA, although UDT⊂CDT+SIA is more accurate, because the optimal UDT policies are a subset of the policies which CDT+SIA can follow. This is because UDT has self-coordination power which CDT+SIA lacks.
I feel like this understates the way in which CDT+SIA is philosophically/intuitively crazy/implausible. Consider a variant of AMD where U(A)=1, U(B)=0, U(C)=2. Obviously one should select CONT with probability 1 in order to reach C, but “EXIT with probability 1” seems to be another CDT+SIA solution. The CDT+SIA reasoning there (translated from math to English) is: Suppose my policy is “EXIT with probability 1″. Then I’m at X with probability 1. Should I deviate from this policy? If I do CONT instead, I’m still at X with probability 1 and the copy of me at Y will still do EXIT with probability 1 so I’ll end up at B for sure with utility 0, therefore I should not deviate. Isn’t this just obviously crazy (assuming I didn’t misunderstand something)?
we could say that UDT1.0 = CDT+SIA
But UDT1.0 already gives a unique and correct solution to the problem above.
Caspar Oesterheld commented on that post with an analogous EDT+SSA result.
I tried to understand Caspar’s EDT+SSA but was unable to figure it out. Can someone show how to apply it to an example like the AMD to help illustrate it?
One more, because one of my posts presented two open problems, and I only listed one of them above:
15. Our current theoretical foundations for rationality all assume a fully specified utility function (or the equivalent), or at least a probability distribution on utility functions (to express moral/value uncertainty). But to the extent that humans can be considered to have a utility function at all, it’s may best be viewed as a partial function that returns “unknown” for most of the input domain. Our current decision theories can’t handle this because they would end up trying to add “unknown” to a numerical value during expected utility computation. Forcing humans to come up with an utility function or even a probability distribution on utility functions in order to use decision theory seems highly unsafe so we need an alternative.
Also, I’m confused about the practical case.
For example, suppose that C formulates a plan to “trick” A[C]. Then the subjective universality condition implies that we don’t expect C to succeed.
What does “expect” mean here? Probability > .5? Also, can you walk through an example of how C might “trick” A[C] and how subjective dominance implies that we don’t expect C to succeed?
We’d like to say that the impoverished perspective is still “good enough” for us to feel safe, despite not being good enough to capture literally everything we know. But now we risk begging the question: how do we evaluate whether the impoverished perspective is good enough? I think this is probably OK, but it’s definitely subtle.
I don’t know how to make sense of this. If an impoverished perspective expects C not to be able to “trick” A, it seems kind of obvious that’s not good enough for me to think the same? You must have reason to think otherwise but the inferential distance is too far for me to figure out what it is.
I went through all my LW posts and gathered the ones that either presented or reminded me of some problem in human rationality.
1. As we become more rational, how do we translate/transfer our old values embodied in the less rational subsystems?
2. How to figure out one’s comparative advantage?
3. Meta-ethics. It’s hard to be rational if you don’t know where your values are supposed to come from.
4. Normative ethics. How much weight to put on altruism? Population ethics. Hedonic vs preference utilitarianism. Moral circle. Etc. It’s hard to be rational if you don’t know what your values are.
5. Which mental subsystem has one’s real values, or how to weigh them.
6. How to handle moral uncertainty? For example should we discount total utilitarianism because we would have made a deal to for total utilitarianism to give up control in this universe?
7. If we apply UDT to humans, what does it actually say in various real-life situations like voting or contributing to x-risk reduction?
8. Does Aumann Agreement apply to humans, and if so how?
9. Meta-philosophy. It’s hard to be rational if one doesn’t know how to solve philosophical problems related to rationality.
10. It’s not clear how selfishness works in UDT, which might be a problem if that’s the right decision theory for humans.
11. Bargaining, politics, building alliances, fair division, we still don’t know how to apply game theory to a lot of messy real-world problems, especially those involving more than a few people.
12. Reality fluid vs. caring measure. Subjective anticipation. Anthropics in general.
13. What is the nature of rationality, and more generally normativity?
14. What is the right way to handle logical uncertainty, and how does that interact with decision theory, bargaining, and other problems?
Comparing the rate of problems opened vs problems closed, we have so far to go....
Strategic High Skill Immigration seems to be a very high quality and relevant post that has been overlooked by most people here, perhaps due to its length and rather obscure title. (Before I strongly upvoted it, it had just 12 points and 4 voters, presumably including the author.) If a moderator sees this, please consider curating it so more people will read it. And for everyone else, I suggest reading at least part 1. It talks about why global coordination on AI safety and other x-risks is hard, and suggests a creative solution for making it a little easier. Part 2 is much longer and goes into a lot of arguments, counterarguments, and counter-counterarguments, and can perhaps be skipped unless you have an interest in this area.
So “simpler” in that sentence should be replaced by “simple enough”? In other words, it’s not the case that A is better-informed than every computation C that is simpler than A, right? Also, can you give a sense of how much simpler is simple enough?
This post defines the concept of “ascription universality,” which tries to capture the property that a question-answering system A is better-informed than any particular simpler computation C.
I’m getting lost right away here. In the case of “large enough teams of humans can carry out arbitrarily complicated reasoning”, what is A and C? Presumably A is the whole team, but I have no idea what C is.
That said, I disagree with Wei that this is relatively crisp: taken literally, the definition is vacuous because all behavior maximizes some expected utility.
I think I meant more that an AGI’s internal cognition resembles that of an expected utility maximizer. But even that isn’t quite right since it would cover AIs that only care about abstract worlds or short time horizons or don’t have general intelligence. So yeah, I definitely oversimplified there.
Maybe we mean that it is long-term goal-directed, but at least I don’t know how to cash that out.
What’s wrong with cashing that out as trying to direct/optimize the future according to some (maybe partial) preference ordering (and using a wide range of competencies, to cover “general”)? You said “In fact, I don’t want to assume that the agent even has a preference ordering” but I’m not sure why. Can you expand on that?
I found an old comment by Morendil that provides a useful (but possibly non-exhaustive) taxonomy of questions:
“challenge” questions like “how confident are you about this”—they are really intended to prompt the askee to ask the question of themselves
“clarification” questions—like “what did you mean by X”
“genuinely curious” questions—the first to seem so was “How does any particular agent go about convincing me that it’s Omega?”
“mocking rhetorical” questions—like “Don’t they know (...) this thing called ‘social networking’? ”
“hypothetical” questions—like “what would I think of an amateur who argues with me in the area of my competence?”
“question and answer” pairs—like “does that mean they are wasting their time there? Of course not...”—obviously rhetorical
“what if” questions—like “And what if you wanted to grow?”
“agreement seeking” questions—like “Agreed?”
“information-providing” questions—like “Anyone heard of Marblar?” (with a link, so we know the asker has heard of it)
This makes me think that none of the current answers have provided a full explanation of what questions are yet.
Most of these sections seem to only contain arguments that AGI won’t come earlier than CAIS, but not that it would come later than CAIS. In other words, they don’t argue against the likelihood that under CAIS someone can easily build an AGI by connecting existing AI services together in a straightforward way. The only section I can find among the ones you listed that tries to argue in this direction is Section 13, but even it mostly just argues that AGI isn’t simpler than CAIS, and not that it’s more complex, except for this paragraph in the summary, Section 13.5:
To summarize, in each of the areas outlined above, the classic AGI model both obscures and increases complexity: In order for general learning and capabilities to fit a classic AGI model, they must not only exist, but must be integrated into a single, autonomous, self-modifying agent. Further, achieving this kind of integration would increase, not reduce, the challenges of aligning AI behaviors with human goals: These challenges become more difficult when the goals of a single agent must motivate all (and only) useful tasks.
So putting alignment aside (I’m assuming that someone would be willing to build an unaligned AGI if it’s easy enough), the only argument Eric gives for greater complexity of AGI vs CAIS is “must be integrated into a single, autonomous, self-modifying agent”, but why should this integration add a non-negligible amount of complexity? Why can’t someone just take a plan maker, connect it to a plan executer, and connect that to the Internet to access other services as needed? (I think your argument that strategic planning may be one of the last AIS to arrive is plausible, but it doesn’t seem to be an argument that Eric himself makes.) Where is the additional complexity coming from?
I’m curious about something that maybe you can answer, since you seem to have a strong interest in nuclear strategy and arms control: why does Russia have so many more nuclear weapons than China? The answer I’ve been able to find online are:
Russia wants to have the option of doing a first strike (like the US) instead of just a second strike (like China). (But why does Russia feel a need for this and China doesn’t?)
The Cold War arms race caused the Soviet Union to have many more nuclear weapons than China, which Russia inherited. (But why doesn’t Russia unilaterally reduce its nuclear arsenal to China’s size?)
Let me know if you have any thoughts on this, or can point me to any papers or articles.
if Alice sees Bob make good remarks etc., she’s more interested in ‘running a draft by him’ next time, or to respond positively if Bob asks her to look something over
This dynamic contributes to anxiety for me to comment in Google Docs, and makes it less fun than public commenting (apparently the opposite of many other people). I feel like if I fail to make a good contribution, or worse, make a dumb comment, I won’t be invited to future drafts and will end up missing a lot of good arguments, or entire articles because many drafts don’t get published until much later or ever. This is not a theoretical fear because I’ve “gotten in” to a couple of important and still unpublished drafts only by accidentally finding out about them and requesting invites.
Another thing that contributes to the anxiety is feeling a need to make a good first impression to people who are in the discussion who I’ve never talked to before because they don’t participate on public forums.
However, I worry that the fuzziness of the usual concept of AGI has now been replaced by a fuzzy notion of “service” which makes sense in our current context, but may not in the context of much more powerful AI technology.
It seems to me that “AGI” is actually relatively crisp compared to “service”: it’s something that approximates an expected utility maximizer, which seems like a pretty small and relatively compact cluster in thing-space. “Service” seems to cover a lot more varied ground, from early simple things like image classifiers to later strategic planners, natural language advice givers, AI researchers, etc., with the later things shading into AGI in a way makes it hard to distinguish between them.
it definitely seems worth investigating ways to make modular and bounded AIs more competitive, and CAIS more likely.
A major problem in predicting CAIS safety is to understand the order in which various services are likely to arise, in particular whether risk-reducing services are likely to come before risk-increasing services. This seems to require a lot of work in delineating various kinds of services and how they depend on each other as well as on algorithmic advancements, conceptual insights, computing power, etc. (instead of treating them as largely interchangeable or thinking that safety-relevant services will be there when we need them). Since this analysis seems very hard to do much ahead of time, I think we’ll have to put very wide error bars on any predictions of whether CAIS would be safe or unsafe, until very late in the game. (This seems like a natural perspective for thinking about CAIS safety, which appears to be missing from Eric’s report.)
Having said that, my feeling is that many risk-reducing services (especially ones that can address human safety problems) seem to require high-level general reasoning abilities, whereas many risk-increasing services can just be technical problem solvers or other kinds of narrow intelligences or optimizers, so the latter is likely to arrive earlier than the former, and as a result CAIS is actually quite unsafe, and hard to make safe, whereas AGI is by default highly unsafe, but with appropriate advances in safety research can perhaps be made safe. So I disagree with the proposal to push for CAIS, at least until we can better understand the strategic landscape. See also this comment where I made some related points.
I’m not sure I understand the question. Inverse reinforcement learning, preference learning (eg. deep RL from human preferences) and inverse reward design are some existing examples of narrow value learning.
Thanks for the existing examples, which are helpful, but I guess what I was trying to ask was, is there a mathematical theory of instrumental value learning, that we can expect practical algorithms to better approximate over time, which would let us predict what future algorithms might look like or be able to do?
You have to put active work to make sure the value learner continues to do what you want.
“You” meaning the user? Does the user need to know when they need to provide the AI with more training data? (For example, if there was a massive devaluation of the US dollar, they need to predict that the AI might sell all their other possessions for dollars, and actively provide the AI with more training data before that happens?) Or can we expect the AI to know when it should ask the user for more training data? If the latter, what can we expect the AI to do in the meantime (e.g., if the user is asleep and it can’t ask)?
How would this kind of narrow value learning work in a mathematical or algorithmic sense? For example, one question I have is, since instrumental goals and values can be invalidated by environmental changes (e.g., I’d stop valuing US dollars if I couldn’t buy things with them anymore), how does the value learner know when that has happened? Are there any papers I can read or tutorials I can watch to learn about this? Or feel free to give an explanation here if it’s simple enough.
I think I expressed myself badly there. What I mean is that it seems a sensible default to not trust an impoverished perspective relative to oneself, and you haven’t stated a reason why we should trust the impoverished perspective. This seems to be at least a big chunk of the formalization of universality that you haven’t sketched out yet.