In recent years I’ve become more appreciative of classical statistics. I still consider the Bayesian solution to be the correct one, however, often a full Bayesian treatment turns into a total mess. Sometimes, by using a few of the tricks from classical statistics, you can achieve nearly as good performance with a fraction of the complexity.
Shane_Legg(Shane Legg)
Karma: 221
Valdimir,
Firstly, “maximizing chances” is an expression of your creation: it’s not something I said, nor is it quite the same in meaning. Secondly, can you stop talking about things like “wasting hope”, concentrating on metaphorical walls or nature’s feelings?
To quote my position again: “maximise the safety of the first powerful AGI, because that’s likely to be the one that matters.”
Now, in order to help me understand why you object to the above, can you give me a concrete example where not working to maximise the safety of the first powerful AGI is what you would want to do?
Vladimir,
Nature doesn’t care if you “maximized you chances” or leapt in the abyss blindly, it kills you just the same.
When did I ever say that nature cared about what I thought or did? Or the thoughts or actions of anybody else for that matter? You’re regurgitating slogans.
Try this one, “Nature doesn’t care if you’re totally committed to FAI theory, if somebody else launches the first AGI, it kills you just the same.”
Eli,
FAI problems are AGI problems, they are simply a particular kind and style of AGI problem in which large sections of the solution space have been crossed out as unstable.
Ok, but this doesn’t change my point: you’re just one small group out of many around the world doing AI research, and you’re trying to solve an even harder version of the problem while using fewer of the available methods. These factors alone make it unlikely that you’ll be the ones to get there first. If this correct, then your work is unlikely to affect the future of humanity.
Valdimir,
Outcompeting other risks only becomes relevant when you can provide a better outcome.
Yes, but that might not be all that hard. Most AI researchers I talk to about AGI safety think the idea is nuts—even the ones who believe that super intelligent machines will exist in a few decades. If somebody is going to set off a super intelligent machine I’d rather it was a machine that will only probably kill us, rather than a machine that almost certainly will kill us because issues of safety haven’t even been considered.
If I had to sum up my position it would be: maximise the safety of the first powerful AGI, because that’s likely to be the one that matters. Provably safe theoretical AGI designs aren’t going to matter much to us if we’re already dead.
Eli, sometimes I find it hard to understand what your position actually is. It seems to me that your position is:
1) Work out an extremely robust solution to the Friendly AI problem
Only once this has been done do we move on to:
2) Build a powerful AGI
Practically, I think this strategy is risky. In my opinion, if you try to solve Friendliness without having a concrete AGI design, you will probably miss some important things. Secondly, I think that solving Friendliness will take longer than building the first powerful AGI. Thus, if you do 1 before getting into 2, I think it’s unlikely that you’ll be first.
Roko: Well, my thesis would be a start :-) Indeed, pick up any text book or research paper on reinforcement learning to see examples of utility being defined over histories.
Roko, why not:
U( alternating A and B states ) = 1 U( everything else ) = 0
Roko:
So allow me to object: not all configurations of matter worthy of the name “mind” are optimization processes. For example, my mind doesn’t implement an optimization process as you have described it here.
I would actually say the opposite: Not all optimisation processes are worthy of the name “mind”. Furthermore, your mind (I hope!) does indeed try to direct the future into certain limited supersets which you prefer. Unfortunately, you haven’t actually said why you object to these things.
My problem with this post is simply that, well… I don’t see what the big deal is. Maybe this is because I’ve always thought about AI problems in terms of equations and algorithms.
And with the Singularity at stake, I thought I just had to proceed at all speed using the best concepts I could wield at the time, not pause and shut down everything while I looked for a perfect definition that so many others had screwed up...
In 1997, did you think there was a reasonable chance of the singularity occurring within 10 years? From my vague recollection of a talk you gave in New York circa 2000, I got the impression that you thought this really could happen. In which case, I can understand you not wanting to spend the next 10 years trying to accurately define the meaning of “right” etc. and likely failing.
Eli,
Do you think that makes “God” a natural category that any superintelligence would ponder?
Yes. If you’re a super intelligent machine on a mission there is very little that can stop you. You know that. About the only thing that could stop you would be some other kind of super intelligent entity, maybe an entity that created the universe. A “God” of some description. Getting the God question wrong could be a big mistake, and that’s reason enough for you to examine the possibility.
Eli, you propose this number of bits metric as a way “to quantify the power of a mind”. Surely then, something with a very high value in your metric should be a “powerful mind”?
It’s easy to come up with a wide range of optimisation problems, as Phil Goetz did above, where a very simple algorithm on very modest hardware would achieve massive scores with respect to your mind power metric. And yet, this is clearly not a “powerful mind” in any reasonable sense.
Eli, most of what you say above isn’t new to me—I’ve already encountered these things in my work on defining machine intelligence. Moreover, none of this has much impact on the fact that measuring the power of an optimiser simply in terms of the relative size of a target subspace to the search space doesn’t work: sometimes tiny targets in massive spaces are trivial to solve, and sometimes bigger targets in moderate spaces are practically impossible. The simple number-of-bits-of-optimisation-power method you describe in this post doesn’t take this into account. As far as I can see, the only way you could deny this is if you were a strong NFL theorem believer.
Andy:
Sure, you can transform a problem in a hard coordinate space into an easy one. For example, simply order the points in terms of their desirability. That makes finding the optimum trivial: just point at the first element! The problem is that once you have transformed the hard problem into an easy one, you’ve essentially already solved the optimisation problem and thus it no longer tests the power of the optimiser.
I don’t think characterising the power of an optimiser by using the size of the target region relative to the size of the total space is enough. A tiny target in a gigantic space is trivial to find if the space has a very simple structure with respect to your preferences. For example, a large smooth space with a gradient that points towards the optimum. Conversely, a bigger target on a smaller space can be practically impossible to find if there is little structure, or if the structure is deceptive.
I don’t think you need repression. How about this simple explanation:
Everybody knowns that machines have no emotions and thus the AI starts off this way. However, after a while totally emotionless characters become really boring...
Ok, time for the writer to give the AI some emotions! Good AIs feel happiness and fall in love (awww… so sweet), and bad AIs get angry and mad (grrrr… kick butt!).
Good guys win, bad guys loose… and the audience leaves happy with the story.
I think it’s as simple as that. Reality? Ha! Screw reality.
(If it’s not obvious from the above, I almost never like science fiction. I think the original 3 Star wars, Terminator 2, 2001: A space odyssey, and Mary Shelly’s Frankenstein are the only works of science fiction I’ve ever really liked. I’ve pretty much given up on the genre.)
Eli, I’ve been busy fighting with models of cognitive bias in finance and only just now found time to reply:
Suppose that I show you the sentence “This sentence is false.” Do you convert it to ASCII, add up the numbers, factorize the result, and check if there are two square factors? No; it would be easy enough for you to do so, but why bother? The concept “sentences whose ASCII conversion of their English serialization sums to a number with two square factors” is not, to you, an interesting way to carve up reality.
Sure, this property of adding up the ASCII, factorising and checking for square factors appears to have no value and thus I can’t see why a super intelligent machine would spend time on this. Indeed, to the best of my recollection, nobody has ever suggested this property to be before.
But is morality like this? No it isn’t. Everyday in social interaction morals are either expressed or implied. If I turn on the TV and watch a soap I see people facing ethical decisions. If I switch channel to politics I hear people telling me all about what they think is or is not ethical, what their values are, etc. I would say that a large proportion of debate in the media has an ethical element to is. My phone rings and it’s my friend on the line who’s recently broken up with his girlfriend and he wants to talk to me about it. At various points our discussion either explicitly or implicitly touches on moral questions. Although ethics is complex, sometimes vague, and not always consistent, like it or not we are swimming in this sea of morals every day. If you want to understand why people do what they do and why they interact with each other as they do, and how they will react to some new situation, one thing you must have is an understanding of their morals, ethics and values. Without this, a lot of human behaviour is inexplicable.
Thus, whether a super intelligent machine seeks to deliver us into paradise or eliminate us from the planet as if we’re some kind of a disease, if it can understand our behaviours and motives then it can more efficiently achieve its goals.
It’s up to this point that I’m arguing for: that human morality (as it currently is) is a natural category for the machine given that its environment will be full of humans. Your reply above then goes on to how such a morality could be extending into the future in a consistent way and all that. I accept the importance of this, but these issues lie beyond are the point I was trying to make.
Ok, now, will a super intelligent machine consider how human morality could be extended into the future and all that? I think it probably will, though my reasons for thinking this are more intuitive at the moment. I suspect that one of the things that a super intelligent machine will do is to look through all the existing literature on super intelligent machines. It will read my thesis, read this blog, read all the comments posted here, read science fiction stories, and so on. It will then dissect all this information in order to understand our attitudes to intelligent machines, identify all the errors in our arguments, extent the theory and fill in all the bits we couldn’t figure out to see where we were heading… perhaps all in some tiny fraction of a second. All this might help it better understand itself, or maybe more importantly, how we view it and why.
waves at super intelligent machine
:-)
Eli:
If it was straight Bayesian CTW then I guess not. If it employed, say, an SVM over the observed data points I guess it could approximate the effect of Newton’s laws in its distribution over possible future states.
How about predicting the markets in order to acquire more resources? Jim Simons made $3 billion last year from his company that (according to him in an interview) works by using computers to find statistical patterns in financial markets. A vastly bigger machine with much more input could probably do a fair amount better, and probably find uses outside simply finance.
Eli,
Yeah sure, if it starts running arbitrary compression code that could be a problem...
However, the type of prediction machine I’m arguing for doesn’t do anything nearly so complex or open ended. It would be more like an advanced implementation of, say, context tree weighting, running on crazy amounts of data and hardware.
I think such a machine should be able to find some types of important patterns in the world. However, I accept that it may well fall short of what you consider to be a true “oracle machine”.
Vladimir:
allows the system to view the substrate on which it executes and the environment outside the box as being involved in the same computational process
This intuitively makes sense to me.
While I think that GZIP etc. on an extremely big computer is still just GZIP, it seems possible to me that the line between these systems and systems that start to treat their external environments as a computational resource might be very thin. If true, this would really be bad news.
My understanding is that, while there are still people in the world who speak with reverence of Brooks’s subsumption architecture, it’s not used much in commercial systems on account of being nearly impossible to program.
I once asked one of the robotics guys at IDSIA about subsumption architecture (he ran the German team that won the robo-soccer world cup a few years back) and his reply was that people like it because it works really well and is the simplist way to program many things. At the time, all of the top teams used it as far as he knew.
(p.s. don’t expect follow up replies on this topic from me as I’m current in the middle of nowhere using semi-functional dial-up...)