Hmm, I think I get it. Correct me if I’m wrong.
Your paper is about an agent which can perform well in any possible universe. (That’s the “for all ν in ℳ”). That includes universes where the laws of physics suddenly change tomorrow. But in real life, I know that the laws of physics are not going to change tomorrow. Thus, I can get optimal results without doing the kind of exhaustive exploration that your paper is talking about. Agree or disagree?
Hmm, interesting. I think human cloning is an imperfect analogy because the only real reason to do it is to impress your friends, so if everyone coordinates on being scornful towards the first person to do human cloning (rather than being impressed), then there’s no more personal benefit to cheating. By contrast, with an AGI, there would be the hope that you’ll actually solve the safety problems, and then get tons of money and power and respect.
Biological weapons is maybe a better example, but not an especially encouraging one: as many as 8 countries may have secret bio-weapons programs, including North Korea. Maybe one could make an argument that there’s a taboo against using bio-weapons, as opposed to merely stockpiling them? Likewise, the taboo against using nuclear weapons was not successfully turned into a taboo against countries starting new nuclear weapons programs. Maybe it’s hard to get riled up against someone doing something that is not purposely aggressive? I don’t know. I can’t think of a great historical analogy.
There’s also the issue that there’s not too many actors who have any reason to start a bio-weapons programs, and the ability to do so without getting shut down. Really just secret military labs. Whereas in the worst case, many orders of magnitude more people would be willing and able to start doing illegal AGI experiments without the authorities realizing it.
This question is probably stupid, and also kinda generic (it applies to many other papers besides this one), but forgive me for asking it anyway.
So, I’m trying to think through how this kind of result generalizes beyond MDPs. In my own life, I don’t go wandering around an environment looking for piles of cash that got randomly left on the sidewalk. My rewards aren’t random. Instead, I have goals (or more generally, self-knowledge of what I find rewarding), and I have abstract knowledge constraining the ways in which those goals will or won’t happen.
Yes, I do still have to do exploration—try new foods, meet new people, ponder new ideas, etc.—but because of my general prior knowledge about the world, this exploration kinda feels different than the kind of exploration that they talk about in MDPs. It’s not really rolling the dice, I generally have a pretty good idea of what to expect, even if it’s still a bit uncertain along some axes.
So, how do you think about the generalizability of these kinds of MDP results?
(I like the paper, by the way!)
I wrote that based on their newsletter: “A significant portion of MIRI’s current work is in Haskell”.
Not a philosopher, but common-sensically, I understand utilitarianism as saying that actions that create more good for more people are progressively more praiseworthy. It’s something else to label the one very best possible action as “moral / permitted” and label every other action as “immoral / forbidden”. That seems like a weird and counterproductive way to talk about things. Do utilitarians actually do that?
Here are three areas where I think I have a different perspective:
I think we should be careful to quantify how compute-efficient, how compressible, etc. I agree that vision relies on lots of heuristics. Vision is not possible in 100 lines of python code with no ML! But are we talking about megabytes of heuristics, or gigabytes, or terabytes? I don’t know.
It also seems to me that we can plausibly declare vision, audio, etc. to be “compute-efficient enough”—that explicit, high-level reasoning is the only bottleneck, and that the other 6 boxes are “solved problems”, or close enough anyway.
I think we shouldn’t confuse two types of “simple”: A “simple core learning algorithm” can still learn arbitrarily complicated heuristics from data.
I don’t think we currently know of an algorithm which would be an AGI if only we had enough compute. Yes we have search algorithms, but I don’t think we have data structures that can encode the kind of arbitrary abstract understanding of the world that is needed, and I don’t think we have a way to build and sort through those data structures. (But I do think there are ideas out there that are getting a lot closer.) Supposedly, the human brain does about as much calculation as a supercomputer. But we can’t take a supercomputer and have it be a remote-work software engineer, doing all the things that software engineers do, like answering emails and debugging code. (Equivalently, we can’t take a normal computer and have it be a software engineer running in slow-motion.)
Therefore, I think we have to say that we’re not putting our computer cycles to the maximally-efficient use in creating intelligence.
The line of research I would most expect to lead to AGI soon would be algorithms explicitly trying to emulate (at a high level) the algorithms of human intelligence. While these tend to be remarkably sample efficient (just as humans can learn a new word from a single exposure), they are not particularly easy computations to do today (they’re vaguely related to probabilistic programming), and they are also substantially different from ResNets and the other most popular ML approaches. Are they “compute efficient”? I wouldn’t know how to answer that, because I think you can only fairly compare it to other algorithms “doing AGI”, and I don’t think there are any other such algorithms today. I would expect a pretty simple core learning algorithm, that is (at first) painfully slow and inefficient to run, and which creates a not-at-all-simple mess of heuristics as it learns about the world.
For example, one fast-takeoff-ish scenario I might imagine would be that someone gets AGI-level algorithms just barely working inefficiently on small toy problems, and then gets a 1000× speedup by translating the algorithms to CUDA-C++ / FPGA / whatever (or opening a collaboration with google, or finding a better implementation, or whatever), and the result takes people by surprise—maybe even including the programmers, and certainly people outside the group.
Just trying to think this through … at the risk of proving I haven’t carefully read all your posts … :-)
I program my AI to invent a better solar cell. So it starts by reading a materials science textbook. OK, now it knows materials science … it didn’t before … Is that a disallowed AU increase? (As the saying goes, “knowledge is power”...?)
Cool. We’re probably on the same page then.
Sure. Looking forward to that. My current intuition is: Humans have a built-in reward system based on (mumble mumble) dopamine, but the existence of that system doesn’t make it easy for us to understand dopamine, or reward functions in general, or anything like that, nor does it make it easy for us to formulate and pursue goals related to those things. It takes quite a bit of education and beautifully-illustrated blog posts to get us to that point :-D
I have previously criticized value learning for needing to locate the human within some kind of prespecified ontology (this criticism is not new). By taking only the agent itself as primitive, perhaps we could get around this (we don’t need any fancy engineering or arbitrary choices to figure out AUs/optimal value from the agent’s perspective).
Wouldn’t you need to locate the abstract concept of AU within the AI’s ontology? Is that easier? Or sorry if I’m misunderstanding.
Very clever! Yes I agree with you that there is a parameter setting for modern DRL architectures for an agent that has an “instinct” to walk over to the nearest computer, and write and execute code that turns on a real-deal superintelligent AGI. Or for a program that manually steps through the execution steps of an AGI Turing machine. I guess I interpreted the question to say that that kind of thing doesn’t count. :-P
I know at least Alan Turing, IJ Good, Marvin Minsky, and Norbert Wiener discussed AI risks, but I can’t offer quotes and references right at this moment. And I’m not sure how much they dove into it, versus talking about it in one speech or whatever.
ETA: I have the impression that AI risk is one of the major themes of Norbert Wiener’s book The Human Use of Human Beings. I haven’t read it myself, so I’m not really sure. For Alan Turing it may have just been the one speech excerpted by ofer below.
Jumping out on a limb—and I might change my mind next week—but I would say “no”, if using current popular mainstream DRL techniques, because these lack (1) foresight (i.e., running a generative model to predict the result of different possible courses of action, and choosing on the basis of the results), and (2) analysis-by-synthesis (processing inputs by continually running searches through a space of generative models to find the model that best matches that input). I think humans do both, and without both (among other requirements), I picture systems as sorta more like “operating on instinct” rather than “intelligent”.
So (in my mind), your question would be “can we get ‘robustly human+ capabilities’ from a system operating on instinct?” and the answer is “Obviously yes when restricted to any finite set of tasks in any finite set of situations”, e.g. AlphaStar. With enough parameters, the set of tasks and situations could get awfully high, and maybe that counts as “robustly human+”—just as a large enough Giant Lookup Table might count as “robustly human+”. But my hunch is that systems with foresight and analysis-by-synthesis will be “robustly human+” earlier than any systems that operate on instinct.
I have yet to see something which strikes me as a compelling argument in its favor.
At some level, any system that does foresight-based planning has to treat its own actions as part of its world-model. In that context, the idea of “Treat my own actions as random variables—just like everything else in the world—and condition on good things happening in the future” is either exactly equivalent to Monte Carlo Tree Search, or awfully close, I think.
But whereas AlphaGo does MCTS in a simplistic, one-timestep-at-a-time manner, the vision here is to bring to bear the full power of the world-modeling framework—hierarchy, analogizing, compositionality, abstraction, etc. etc.—and apply all those tools to the task of MCTS action planning. Thus things like imitating conspecifics, and taking ideas from the world (“I see that wall is not fixed in place; so maybe I can move it!”), and building hierarchical probabilistic plans over arbitrary time-scales (“Maybe I can squeeze through that hole and see what’s on the other side”), and interleaving multiple plans, and on and on—all these things and more arise organically in this kind of framework. (...if the predictive-world-modeling tools are good enough, of course.)
Maybe there are other ways to formulate action selection that has all those features, but I don’t know them.
(Or I guess the simpler “compelling argument in its favor” is “It appears to be very effective in practice”.)
instrumental occam seems almost like a bug rather than a feature
Hmm. On further reflection, I don’t think the brain really uses Occam’s razor at all, either for action-selection or world-modeling, unless you define “complexity” in some way that makes it tautological. (As Charlie Steiner writes, you have to search through the infinite space of possibilities in some order, with some prior, and it’s both tempting and vacuous to say that the earlier choices are “simpler in the context of this algorithm” or whatever).
I think a more typical dynamic of brain algorithms is what Dileep George calls “memorize—generalize”, i.e. memorize a little snippet of pattern / idea (at some level of abstraction), and then see if the same pattern works on other contexts, or when glued together with other patterns, or when analogized to a different area.
I don’t currently think brain world-modeling algorithms have anything analogous to regularization.
One way of solving the problem of action selection is to treat our own actions as random variables, then build a probabilistic model, and condition that probability distribution on things going well for us in the future. See, for example, Planning By Probabilistic Inference, human brains (I would argue), upside-down RL (kinda, I think?), etc.
In this formulation, instrumental Occam’s razor is just a special case of epistemic Occam’s razor, it seems to me.
I don’t understand your comment...
+1, if someone gives me a gadget I’ve never seen before, and I try to figure out what it is and how to use it, that process feels like I’m building a generative model by mainly gluing together a bunch of pre-existing pieces, not like I’m doing gradient descent. And yeah, it’s occurred to me that this (the human brain approach) is a little bit like evolution. :)
Thanks!