There will be plenty of functions $R_i$ that have fewer bits in their encoding than the real function $R^d$ used by the demonstrator.

I don’t think this is a problem. There will be plenty of them, but when they’re wrong they’ll get removed from the posterior.

A policy outputs a distribution over $\{0,1\}\times A$, and equations 3 and 4 define what this distribution is for the imitator. If it outputs (0, a), that means $q_t=0$ and and $a_t=1$ and if it outputs (1, a), that means $q_t=1$ and $a_t=a$. When I say

The 0 on the l.h.s. means the imitator is picking the action itself instead of deferring to the demonstrator,

that’s just describing the difference between equations 3 and 4. Look at equation 4 to see that when $q_t=1$, the distribution over the action is equal to that of the demonstrator. So we describe the behavior that follows $q_t=1$ as “deferring to the demonstrator”. If we look at the distribution over the action when $q_t=0$, it’s something else, so we say the imitator is “picking its own action”.

The 0 on the l.h.s. means the imitator is picking the action $a$ itself instead of deferring to the demonstrator or picking one of the other actions???

The 0 means the imitator is picking the action, and the $a$ means it’s not picking another action that’s not $a$.

Most complexity measures give roughly similar values for the (relative) complexity of most objects

I’ll write mostly about this statement, as I think it’s the crux of our disagreement.

The statement may be true as long as we hold the meaning of “objects” constant as we vary the complexity measure.

However, if we translate objects from one mathematical space to another (say by discretizing, or adding/​removing a metric structure), we can’t simply say the complexity measures for space A on the original A-objects inevitably agree with those space B on the translated B-objects. Whether this is true depends on our choice of translation.

(This is clear in the trivial cases of bad translation where we, say, map every A-object onto the same B-object. Now, obviously, no one would consider this a correct or adequate way to associate A-objects with B-objects. But the example shows that the claim about complexity measures will only hold if our translation is “good enough” in some sense. If we don’t have any idea what “good enough” means, something is missing from the story.)

In the problem at hand, the worrying part of the translation from real to boolean inputs is the loss of metric structure. (More precisely, the hand-waviness about what metric structure survives the translation, if any.) If there’s no metric, this destroys the information needed by complexity measures that care about how easy it is to reconstruct an object “close to” the specified one.

Basic information theory doesn’t require a metric, only a measure. There’s no sense of “getting an output approximately right,” only of “getting the exactly right output with high probability.” If you care about being approximately right according to some metric, this leads you to rate-distortion theory.

Both of these domains—information theory without a metric, and with one—define notions of incompressibility/​complexity, but they’re different. Consider two distributions on R:

1. The standard normal,

2. The standard normal, but you chop it into a trillion pieces on the x axis, and translate the pieces to arbitrary locations in R

According to basic information theory, these are equally simple/​compressible. (They have the same differential entropy, or the same K-L divergence from a uniform distribution if you want to be pedantic.)

But in rate-distortion theory, (1) is way more simple/​compressible than (2). If you’re coding (2) over a noisy channel, you have to distinguish really hard between (say) a piece that stayed in place at [0, 0.1] and another piece that got translated to [1e8, 1e8 + 0.1]. Whereas if you’re coding a standard normal, with its light tails, a 1e8-magnitude mistake is effectively impossible.

If you do all your analysis in the metric-less space, hoping it will cleanly pass over to the metric space at the end, you have no way of distinguishing these two possibilities. When you remove the metric, they’re identical. So you have limited power to predict what the rate-distortion theory notion of complexity is going to say, once you put the metric back in.

Curated.

This post laid out some important arguments pretty clearly.

I think that “think up good strategies for achieving [reward of the type I defined earlier]” is likely to be much, much more complex (making it much more difficult to achieve with a local search process) than an arbitrary goal X for most sorts of rewards that we would actually be happy with AIs achieving.

[edited to delete and replace an earlier question] Question about the paper: under equation (2) on page 4 I am reading:

The 0 on the l.h.s. means the imitator is picking the action itself instead of deferring to the demonstrator,

This confused me initially to no end, and still confuses me. Should this be:

The 0 on the l.h.s. means the imitator is picking the action $a$ itself instead of deferring to the demonstrator or picking one of the other actions???

This would seem to be more consistent with the definitions that follow, and it would seem to make more sense overall.

[long comment, my bigger agenda here is to get to a state where discussions on this forum start using much more math and subcase analysis when they talk about analysing and solving inner alignment problems.]

If we set $\alpha$ small enough, we can make it arbitrarily like[ly] that ${\pi }^d$ never leaves the set of top policies.

OK, I think you are kind of saved here, in the case of my counter-example, because of the special nature of your equation (2) in the paper that computes the set of policies. To make this more specific:

Say I refine the counter-example by adding the prior $w\left({\pi }_i\right)=1/|\Pi |$, so each candidate is considered equally likely from the start. Say that $|\Pi |={10}^{10}$.

The key thing to note in my example is that, before the time-synchronized treacherous turn happens for all policies ${\pi }_i\ne {\pi }^d$, all policies ${\pi }_i$ are equally compatible with events so far, so the posterior $w\left({\pi }_i|h_{<t}\right)$ is the same for all of them.

When I use $\alpha ={10}^{-4}$ with these all-the-same-posterior policies, then (if I am reading equation (2) correctly) the set ${\Pi }_{h<t}^{\alpha }$ of top policies will have size $N={10}^4$. Ties are broken arbitrarily, so the real ${\pi }^d$ is unlikely to be among these top $N={10}^4<<{10}^{10}$ candidates. In that case, the agent will perform the treacherous turn and kill the operator. Correct me if I am wrong here.

However.

Case of $\alpha <1/|\Pi |$: much better than I thought!

However, and this is where things get interesting, if we use $\alpha ={10}^{11}$, so $\alpha <1/|\Pi |$, then equation (2) will compute ${\Pi }_{h<t}^{\alpha }=\Pi$, the top set of policies is all policies. The correct policy is therefore still in the top set, and we get no treacherous turn.

So equation (2) is doing a lot of heavy lifting in making the approach work! This also seems to be a major difference with the $\beta$ based approach in your earlier paper. Not sure if you made this change specifically to deal with examples like my counter-example above, or if you had a more general reason -- the paper does not say.

Pretty interesting! So I would now agree with your statement that your approach provably solves the inner alignment problem, at least for $\alpha <1/|\Pi |$ and for certain values of ‘the inner alignment problem’. I can also imagine several setups where $\alpha <1/|\Pi |$ would still lead to reasonably fast learning (=reasonably fast drop in frequency of demonstrator requests).

This is what progress looks like. There are certain boundary conditions here that might be unrealistic: $\Pi$ needs to be a finite set and the real policy needs to be in there, but at least we have some well-defined boundary conditions we can now look at, where we can try to understand and interpret the math further.

Case of $\alpha >1/|\Pi |$: the challenge of designing a prior

In the case of using $\alpha ={10}^{-4}$ or $\alpha ={10}^{-9}$, because you want the system to learn faster, you are instead looking at a setup where, if you want to solve the inner alignment problem, you will have to shape the prior values $w\left({\pi }_i\right)$ in such a way that $w^d$ gets a higher prior than the all the treacherous policies.

My counter-example above shows that if you are using the 1/​(number of bits in the program that computes ${\pi }_i$) as the prior, you will not get what you need. There will be plenty of functions $R_i$ that have fewer bits in their encoding than the real function $R^d$ used by the demonstrator.

Earlier in this comment section, there is a whole sub-thread with speculation on the number of bits needed too encode benign vs. treacherous policies, but for me that discussion does not yet arrive at any place where I would get convinced that the problem of assigning higher priors to benign vs. treacherous policies has been made tractable yet. (Vanessa has identified some additional moving parts however.)

There is of course a tradition in the AI safety community where this is made ‘tractable’ by the device of polling many AI researchers to ask them whether they think that bits(benign policy)<bits(treacherous policy) for future ML systems, and then graphing the results, but this is not what I would call a tractable solution.

What I would call tractable is a solution like the one, for a much simpler case, in section 10.2.4 of my paper Counterfactual Planning in AGI Systems. I show there that random exploration can be used to make sure that bits(agent environment model which includes unwanted self-knowledge about agent compute core internals) $\gg$ bits(agent environment model that lacks this unwanted self-knowledge), no matter what the encoding. Extending this to the bits(benign policy) case would be nice, but I can’t immediately see a route here.

My answer to the above hypothetical bits(benign policy)<bits(treacherous policy) poll is that we cannot expect this to be true any possible encoding of policies (see counter-example above), but it might be true for some encodings. Figuring out where deep neural net encodings fit on the spectrum would be worthwhile.

Also. my answer to bits(benign policy)<bits(treacherous policy) would depend on whether the benign policy is supposed to be about making paperclips in the same way humans do, or about maximizing human values over the lifetime of the universe in ways that humans will not be able to figure out themselves.

For the paperclip making imitation policy, I am somewhat more optimistic about tractability than in the more general case.

Well, I’ve got five tentative answers to Question One in this post. Roughly, they are: Souped-up AlphaStar, Souped-up GPT, Evolution Lite, Engineering Simulation, and Emulation Lite. Five different research programs basically. It sounds like what you are talking about is sufficiently different from these five, and also sufficiently promising/​powerful/​‘fun’, that it would be a worthy addition to the list basically. So, to flesh it out, maybe you could say something like “Here are some examples of meta-learning/​NAS/​AIGA in practice today. Here’s a sketch of what you could do if you scaled all this up +12 OOMs. Here’s some argument for why this would be really powerful.”

What I’m suggesting is that volume in high-dimensions can concentrate on the boundary.

Yes. I imagine this is why overtraining doesn’t make a huge difference.

Falsifiable Hypothesis: Compare SGD with overtaining to the random sampling algorithm. You will see that functions that are unlikely to be generated by random sampling will be more likely under SGD with overtraining. Moreover, functions that are more likely with random sampling will be become less likely under SGD with overtraining.

See e.g., page 47 in the main paper.

(Maybe you weren’t disagreeing with Zach and were just saying the same thing a different way?)

I’m honestly not sure, I just wasn’t really sure what he meant when he said that the Bayesian and the Kolmogorov complexity stuff were “distractions from the main point”.

This feels similar to:

Saying that MLK was a “criminal” is one way of saying that MLK thought and acted as though he had a moral responsibility to break unjust laws and to take direct action.

(This is an exaggeration but I think it is directionally correct. Certainly when I read the title “neural networks are fundamentally Bayesian” I was thinking of something very different.)

Haha. That’s obviously not what we’re trying to do here, but I do see what you mean. I originally wanted to express these ideas in more geometric language, rather than probability-theoretic language, but in the end we decided to go for more probability-theoretic language anyway.

I agree that this arguably could be mildly misleading. For example, the correspondence between SGD and Bayesian sampling only really holds for some initialisation distributions. If you deterministically initialise your neural network to the origin (i.e., all zero weights) then SGD will most certainly not behave like Bayesian sampling with the initialisation distribution as its prior. Then again, the probability-theoretic formulation might make other things more intuitive.

I agree with your summary. I’m mainly just clarifying what my view is of the strength and overall role of the Algorithmic Information Theory arguments, since you said you found them unconvincing.

I do however disagree that those arguments can be applied to “literally any machine learning algorithm”, although they certainly do apply to a much larger class of ML algorithms than just neural networks. However, I also don’t think this is necessarily a bad thing. The picture that the AIT arguments give makes it reasonably unsurprising that you would get the double-descent phenomenon as you increase the size of a model (at small sizes VC-dimensionality mechanisms dominate, but at larger sizes the overparameterisation starts to induce a simplicity bias, which eventually starts to dominate). Since you get double descent in the model size for both neural networks and eg random forests, you should expect there to be some mechanism in common between them (even if the details of course differ from case to case).

But if you are expecting a 100% guarantee that the uncertainty metrics will detect every possible bad situation

I’m more thinking of how we could automate the navigating of these situations. The detection will be part of this process, and it’s not a Boolean yes/​no, but a matter of degree.

Welcome in the (for now) small family of people funded by Beth! Your research looks pretty cool, and I’m quite excited when seeing how different it is from mine. So Beth is funding quite a wide range of researchers, which is what makes most sense to me. :)

IIUC, you’re saying something like: suppose trained-$A$ computes the source code of the complex core of $B$ and then runs it. But then, define $B^{\prime }$ as: compute the source code of the complex core of $B$ (in the same way $A$ does it) and use it to implement $B$. $B^{\prime }$ is equivalent to $B$ and has about the same complexity as trained-$A$.

Or, from a slightly different angle: if “think up good strategies for achieving $X$” is powerful enough to come up with $M$, then “think up good strategies for achieving [reward of the type I defined earlier]” is powerful enough to come up with $B$.

A’s structure can just be “think up good strategies for achieving X, then do those,” with no explicit subroutine that you can find anywhere in A’s weights that you can copy over to B.

Hmm, sorry, I’m not following. What exactly do you mean by “inference-time” and “derived”? By assumption, when you run $A$ on some sequence it effectively simulates $M$ which runs the complex core of $B$. So, $A$ trained on that sequence effectively contains the complex core of $B$ as a subroutine.

A is sufficiently powerful to select M which contains the complex part of B. It seems rather implausible that an algorithm of the same power cannot select B.

A’s weights do not contain the complex part of B—deception is an inference-time phenomenon. It’s very possible for complex instrumental goals to be derived from a simple structure such that a search process is capable of finding that simple structure that yields those complex instrumental goals without being able to find a model with those complex instrumental goals hard-coded as terminal goals.

Sure, but then I think B is likely to be significantly more complex and harder for a local search process to find than A.

$A$ is sufficiently powerful to select $M$ which contains the complex part of $B$. It seems rather implausible that an algorithm of the same power cannot select $B$.

if we look at actually successful current architectures, like CNNs or transformers, they’re designed to work well on specific types of data and relationships that are common in our world—but not necessarily at all just in a general simplicity prior.

CNNs are specific in some way, but in a relatively weak way (exploiting hierarchical geometric structure). Transformers are known to be Turing-complete, so I’m not sure they are specific at all (ofc the fact you can express any program doesn’t mean you can effectively learn any program, and moreover the latter is false on computational complexity grounds, but it still seems to point to some rather simple and large class of learnable hypotheses). Moreover, even if our world has some specific property that is important for learning, this only means we may need to enforce this property in our prior. For example, if your prior is an ANN with random weights then it’s plausible that it reflects the exact inductive biases of the given architecture.

Sure, but in what way?
Also I’d be happy to do a quick video chat if that would help (PM me).

B can be whatever algorithm M uses to form its beliefs + confidence threshold (it’s strategy stealing.)

Sure, but then I think B is likely to be significantly more complex and harder for a local search process to find than A.

Why? If you give evolution enough time, and the fitness criterion is good (as you apparently agreed earlier), then eventually it will find B.

I definitely don’t think this, unless you have a very strong (and likely unachievable imo) definition of “good.”

Second, why? Suppose that your starting algorithm is good at finding results when a lot of data is available, but is also very data inefficient (like deep learning seems to be). Then, by providing it with a lot of synthetic data, you leverage its strength to find a new algorithm which is data efficient. Unless you believe deep learning is already maximally data efficient (which seems very dubious to me)?

I guess I’m skeptical that you can do all that much in the fully generic setting of just trying to predict a simplicity prior. For example, if we look at actually successful current architectures, like CNNs or transformers, they’re designed to work well on specific types of data and relationships that are common in our world—but not necessarily at all just in a general simplicity prior.