I am a PhD student in computer science at the University of Waterloo.
My current research is related to Kolmogorov complexity. Sometimes I build robots, professionally or otherwise.
See my personal website colewyeth.com for an overview of my interests.
When I say that regularization is an approximation to Solomonoff induction, I don’t mean its a perfect or uniform approximation. The usual justification for regularization is to decrease overfitting, that is, to improve generalization performance, by choosing the simplest model that fits the data. Intuitively, I believe Kolmogorov complexity is the right formalization for simplicity, and in that case regularization should incentivize models with lower Kolmogorov complexity.
As you pointed out, there may be many cases in which a function with relatively low Kolmogorov complexity will still be ruled out by standard ML regularization techniques.
In the case of linear models, L1 regularization probably does tend to decrease Kolmogorov complexity by forcing some weights to 0 - specifying less weights should take a shorter description. The picture may be more complicated regarding L2 regularization, though it is still the case that it decreases the weights, and the liminf of Kolmogorov complexity does (very very very slowly) increase to infinity, so this will asymptotically decrease Kolmogorov complexity of the weights (if not the model). However I think the heuristic argument here is very weak and the reason that L2 regularization works in linear models is because a Gaussian prior on weights is reasonable—an explanation in terms of Kolmogorov complexity is in that sense unnecessary.
(The example in your footnote may be slightly misleading in the linear case because no choice of features allows any individual weight to control the frequency of a sawtooth function, but does demonstrate that some highly jagged functions have low Kolmogorov complexity as you said).
You seem to be focusing on deep learning when you say ML, in which case L2 regularization is usually referred to as weight decay (or Tychonoff regularization). In this case, I have read the argument (don’t remember where) that regularized weights tend to put sigmoids in the linear regime, in which case DL approximately collapses to a linear transformation, which I would expect to have low Kolmogorov complexity. However, you may be interested in this paper https://arxiv.org/abs/1805.08522 which claims that DL actually generalizes because the parameter-function map is biased toward functions with low Kolmogorov complexity, and NOT because of weight decay or other regularization techniques (it’s not clear to me whether the authors actually argue that weight decay doesn’t lead to a simplicity bias, in which case I would be at least partially wrong, or only that weight decay is not necessary for a simplicity bias). Still, this paper does suggest that the view I expressed is the conventional one in the ML community (so is hopefully at least reasonable).
As for your second point, I should note that I discuss two uses of the word regularity—one refers to explicitly regularizing a learned model, the other to regularities in the world (or the data). Ideally our models will pick up on regularities in the data! But I am pointing out that “regularities in the data” should not be used to “explain away” unexpected generalization performance, and should instead make us curious about which precise regularities are being exploited, and where they come from.
This is funny but realistically the human who physically swapped out the device driver for the virtual person would probably just swap the old one back. Generally speaking, digital objects that produce value are backed up carefully and not too fragile. At later stages of self improvement, dumb robots could be used for “screwdriver” tasks like this.
I believe you’re saying that if foom is more than a few years away, it becomes easy to solve the alignment problem before then. I certainly think it becomes easier.
But the view that “foom more than a few years away → the alignment problem is easy” is not the one I expressed, which contained among other highly tentative assertions: “the alignment problem is hard → foom more than a few years away”, and the two are opposed in the sense that they have different truth values when alignment is hard. The distinction here is that the chances we will solve the alignment problem depend on the time to takeoff, and are not equivalent to the difficulty of the alignment problem.
So, you mentioned a causal link between time to foom and chances of solving alignment, which I agree on, but I am also asserting a “causal link” between difficulty of the alignment problem and time to foom (though the counterfactuals here may not be as well defined).
As for how you could possibly believe foom is not going to happen any year now: My opinion depends on precisely what you mean by foom and by “any year now” but I think I outlined scenarios where it takes ~25 years and ~60 years. Do you have a reason to think both of those are unlikely? It seems to me that hard takeoff within ~5 years relies on the assumptions I mentioned about recursive algorithmic improvement taking place near human level, and seems plausible, but I am not confident it will happen. How surprised will you be if foom doesn’t happen within 5 years?
I do expect the next 10 years to be pretty strange, but under the assumptions of the ~60 year scenario the status quo may not be completely upset that soon.
Not really. To be clear, I am criticizing the argument Eliezer tends to make. There can be flaws in that argument and we can still be doomed. I am saying his stated confidence is too high because even if alignment is as hard as he thinks, A.I. itself may be harder than he thinks, and this would give us more time to take alignment seriously.
In the second scenario I outlined (say, scenario B) where gains to intelligence feed back into hardware improvements but not drastic software improvements, multiple tries may be possible. On the whole I think that this is not very plausible (1/3 at most), and the other two scenarios look like they only give us one try.
How would you take the average over an infinite number of UTM’s? You would first need to choose a distribution on them.
It has become slightly more plausible that Melanie Mitchell could come around.
As noted by interstice, this has been defined for infinite and finite strings (instead of functions). I believe Li and Vitanyi use the notation KM.
Another good example is the Goedel machine
As has been observed by other commenters, the argument fails to take into account runtime limitations—in the real world programs can self improve by finding (provably) faster programs that (provably) perform the same inferences that they do, which most people would consider self improvement. However the argument may be onto something: it is indeed true that a program p cannot output a program q with K(q) > K(p) by more than a constant (there is a short program which simulates any input program and then runs its output). Here K(p) is the length of the shortest program with the same behavior as p—in this case we seem to require p to both output another program q and learn to predict a sequence. It is also true that a high level of Kolmogorov complexity is required to eventually predict all sequences up to a high level of complexity: https://arxiv.org/pdf/cs/0606070.pdf.
The real world implications of this argument are probably lessened by the fact that predictors are embedded, and can improve their own hardware or even negotiate with other agents for provably superior software.
I have discussed this problem with Professor Hutter, and though I wouldn’t claim to be able to predict how he would respond to this dialogue, I think his viewpoint is that the anvil problem will not matter in practice. In rough summary of his response: an agent will form a self model by observing itself taking actions through its own camera. When you write something on a piece of paper, you can read what you are writing, and see your own hand holding the pen. Though AIXI may not compress its own action bits, it will compress the results it observes of its actions, and will form a model of its hardware (except perhaps the part that produces and stores those action bits).
I roughly agree with the case made here because I expect interpretability research to be much, much harder than others seem to appreciate. This is a consequence of strong intuitions from working on circuit complexity. Figuring out the behavior of a general circuit sounds like it’s in a very hard complexity class - even writing down the truth table for a circuit takes exponential time in the number of inputs! I would be surprised if coming up with a human interpretable explanation of sub circuits is easy; there are some reasons to believe that SGD will usually produce simple circuits so some success in the average case is possible (see recent work of Ard Louis), but it would be pretty shocking if the full problem had a solution fast enough to run on the huge transformer circuits we are dealing with.
I outlined this position (and pointed out that there is some hope of at least understanding some individual circuits and learning about intelligence) here: https://www.lesswrong.com/posts/RTmFpgEvDdZMLsFev/mechanistic-interpretability-is-being-pursued-for-the-wrong
(Not my best writing though)
I suggest the paper I mentioned on sequential extensions of causal and evidential decision theory. Sequential policy evidential decision theory is definitely a policy theory. But sequential action evidential decision theory is a decision theory making slightly weaker assumptions than CDT. So it’s not clear where the general category EDT should go; I think I’ll update the post to be more precise about that.
I am skeptical of going beyond Decision Theory, though I think to some extent it is necessary for actually building an intelligent machine. Agent Theory seems to exist mostly to address that problem, but I believe that the proper way to make any headway on Agent Theory is likely to start with the much easier to analyze Decision Theory and then make corrections.
It’s interesting to consider whether there is another level beyond Agent Theory. I am not sure if I would consider Society Theory to be a generalization of the same type, I would have to consider that.
You might like my recent post which also argues that FDT (or at least the sequential version of EDT, I don’t feel that FDT is well specified enough for me to talk about it) shouldn’t be considered a decision theory.
This is a useful post that helped me finally understand the importance of inner alignment. However, as the author is careful to note by discussing “vanilla” RL, there are settings in which the model directly sees the rewards, such as history-based RL, which is the framework chosen for Professor Hutter’s development of AIXI. In my mind this is the “true” RL setting in the sense that it most closely reflects the problem humans face in the real world, and it is what I personally usually mean when I think of the RL framework, so I believe this is worth clarifying. Thanks!