“inclusion map” refers to the map , not the coproduct . The map is a coprojection (these are sometimes called “inclusions”, see https://ncatlab.org/nlab/show/coproduct).
A simple example in sets: We have two sets , , and their disjoint union . Then the inclusion map is the map that maps (as an element of ) to (as an element of ).
What is an environmental subagent? An agent on a remote datacenter that the builders of the orginal agent don’t know about?
Another thing that is not so clear to me in this description: Does the first agent consider the alignment problem of the environmental subagent? It sounds like the environmental subagents cares about paperclip-shaped molecules, but is this a thing the first agent would be ok with?
This does not sound very encouraging from the perspective of AI Notkilleveryoneism. When the announcement of the foundation model task force talks about safety, I cannot find hints that they mean existential safety. Rather, it seems about safety for commercial purposes.
A lot of the money might go into building a foundation model. At least they should also announce that they will not share weights and details on how to build it, if they are serious about existential safety.
This might create an AI safety race to the top as a solution to the tragedy of the commons
This seems to be the opposite of that. The announcement talks a lot about establishing UK as a world leader, e.g. “establish the UK as a world leader in foundation models”.
There is an additional problem where one of the two key principles for their estimates is
Avoid extreme confidence
If this principle leads you to picking probability estimates that have some distance to 1 (eg by picking at most 0.95).
If you build a fully conjunctive model, and you are not that great at extreme probabilities, then you will have a strong bias towards low overall estimates. And you can make your probability estimates even lower by introducing more (conjunctive) factors.
Nitpick: The title the authors picked (“Current and Near-Term AI as a Potential Existential Risk Factor”) seems to better represent the content of the article than the title you picked for this LW post (“The Existential Risks of Current and Near-Term AI”).
Reading the title I was expecting an argument that extinction could come extremely soon (eg by chaining GPT-4 instances together in some novel and clever way). The authors of the article talk about something very different imo.
From just reading your excerpt (and not the whole paper), it is hard to determine how much alignment washing is going on here.
what is aligned chain-of-thought? What would unaligned chain-of-thought look like?
what exactly means alignment in the context of solving math problems?
But maybe these worries can be answered from reading the full paper...
I think overall this is a well-written blogpost. His previous blogpost already indicated that he took the arguments seriously, so this is not too much of a surprise. That previous blogpost was discussed and partially criticized on Lesswrong. As for the current blogpost, I also find it noteworthy that active LW user David Scott Krueger is in the acknowledgements.
This blogpost might even be a good introduction for AI xrisk for some people.
I hope he engages further with the issues. For example, I feel like inner misalignment is still sort of missing from the arguments.
I googled “Zeitgeist Addendum” and it does not seem to be a thing that would be useful for AGI risk.
is a followup movie of a 9/11 conspiracy movie
has some naive economic ideas (like abolishing money would fix a lot of issues)
the venus project appears to not be very successful
Do you claim the movie had any great positive impact or presented any new, true, and important ideas?
There is also another linkpost for the same blogpost: https://www.lesswrong.com/posts/EP92JhDm8kqtfATk8/yoshua-bengio-argues-for-tool-ai-and-to-ban-executive-ai
There is also some commentary here: https://www.lesswrong.com/posts/kGrwufqxfsyuaMREy/annotated-reply-to-bengio-s-ai-scientists-safe-and-useful-ai
Overall this is still encouraging. It seems to take serious that
value alignment is hard
executive-AI should be banned
banning executive-AI would be hard
alignment research and AI safety is worthwhile.
I feel like there are enough shared assumptions that collaboration or dialogue with AI notkilleveryoneists could be very useful.
That said, I wish there were more details about his Scientist AI idea:
How exactly will the Scientist AI be used?
Should we expect the Scientist AI to have situational awareness?
Would the Scientist AI be allowed to write large scale software projects that are likely to get executed after a brief reviewing of the code by a human?
Are there concerns about Mesa-optimization?
Also it is not clear to me whether the safety is supposed to come from:
the AI cannot really take actions in the world (and even when there is a superhuman AI that wants to do large-scale harms, it will not succeed, because it cannot take actions that achieve these goals)
the AI has no intrinsic motivation for large-scale harm (while its output bits could in principle create large-scale harm, such a string of bits is unlikely because there is no drive towards these string of bits).
a combination of these two.
Potentially relevant: Yoshua Bengio got funding from OpenPhil in 2017:
There is this documentary: https://en.wikipedia.org/wiki/Do_You_Trust_This_Computer%3F Probably not quite what you want. Maybe the existing videos of Robert Miles (on Mesa-Optimization and other things) would be better than a full documentary.
Maybe something like this can be extracted from stampy.ai (I am not that familiar with stampy fyi, its aims seem to be broader than what you want.)
But why? And which user?
I fear that making karma more like a currency is not good for the culture on LW.
I think money would be preferable to karma bounties in most situations. An alternative for bounties could be a transfer of Mana on Manifold: Mana is already (kind of) a currency.
Certain kinds of “thinking ahead” is difficult to do within 1 forward pass. Not impossible, and GPT-4 likely does a lot of thinking ahead within 1 forward pass.
If you have lots of training data on a game, you often can do well without thinking ahead much. But for a novel game, you have to mentally simulate a lot of options how the game could continue. For example, in Connect4, if you consider all your moves and all possible responses, these are 49 possible game states you need to consider. But with experience in this game, you learn to only consider a few of these 49 options.
Maybe this is a reason why GPT-4 is not so good when playing mostly novel strategy games.
I think the conjecture is also false in the case that utility functions map from Oω to [0,1].
Let us consider the case of A={a1,a2} and O={o1,o2}. We use U1(o)=1−2−k, where k is the largest integer such that o starts with ok1 (and U1(oω1)=1). As for U2, we use U2(o)=1−3−k, where k is the largest integer such that o starts with ok1 (and U2(oω1)=1). Both U1 and U2 are computable, but they are not locally equivalent. Under reasonable assumptions on the Solomonoff prior, the policy π that always picks action a1 is the optimal policy for both U1 and U2 (see proof below).
Note that since the policy is computable and very simple, g0(π)=∞ is not true, and we have g0(π)=O(1) instead. I suspect that the issues are still present even with an additional g0(π)=∞ condition, but finding a concrete example with an uncomputable policy is challenging.
proof: Suppose that U1 and U2 are locally equivalent. Let V be an open neighborhood of the point x=oω1 and α>0, β∈R be such that U1(y)=αU2(y)+β for all y∈V.
Since x∈V, we have 1=U1(x)=αU2(x)+β=α+β. Because V is an open neighborhood of oω1, there is an integer N such that on1oω2∈V for all n≥N. For such n≥N, we have 1−2−n=U1(on1oω2)=αU2(on1oω2)+β=α(1−3−n)+β=α+β−α3−n=1−α3−n. This implies α=(2/3)−n. However, this is not possible for all n≥N. Thus, our assumption that U1 and U2 are locally equivalent was wrong.
Assumptions about the solomonoff prior: For all n, the sequence of actions that produces the sequence of on1 with the highest probability is an−11 (recall that we start with observations in this setting). With this assumption, it can be seen that the policy that always picks action a1 is among the best policies for both U1 and U2.
I think this is actually a natural behaviour for a reasonable Solomonoff prior: It is natural to expect that o1a1o1 is more likely than o1a2o1. It is natural to expect that the sequence of actions that leads to o1 over o2 has low complexity. Always picking a1 is low complexity.
It is possible to construct an artificial UTM that ensures that “always take a1” is the best policy for U1, U2: An UTM can be constructed such that the corresponding Solomonoff prior assigns 3⁄4 probability to the program/environment “start with o_1. after action a_i, output o_i”. The rest of the probability mass gets distributed according to some other more natural UTM.
Then, for U1, in each situation with history on1 the optimal policy has to pick a1 (the actions outside of this history have no impact on the utility): With 3⁄4 probability it will get utility of at least 1−2−(n+1). And with 1/4 probability at least 1−2−n. Whereas, for the choice of a2, with probability 3/4 it will have utility of 1−2−n, and with probability 1/4 it can get at most 1. We calculate (1−2−(n+1))3/4+(1−2−n)1/4=1−52−(n+3)1−3⋅2−(n+2)=(1−2−n)3/4+1/4, ie. taking action a1 is the better choice.
Similarly, for U2, the optimal policy has to pick a1 too in each situation with history on1. Here, the calculation looks like (1−3−(n+1))3/4+(1−3−n)1/4=1−3−n/21−⋅3−n+1/4=(1−3−n)3/4+1/4.