A couple of things come to mind, but I’ve only been studying the surrounding material for around eight months so I can’t guarantee a wholly accurate overview of this. Also, even if accurate, I can’t guarantee that you’ll take to my explanation.
Anyway, the first thing is that brain form computing probably isn’t a necessary or likely approach to artificial general intelligence (AGI) unless the first AGI is an upload. There doesn’t seem to be good reason to build an AGI in a manner similar to a human brain and in fact, doing so seems like a terrible idea. The issues with opacity of the code would be nightmarish (I can’t just look at a massive network of trained neural networks and point to the problem when the code doesn’t do what I thought it would).
The second is that consciousness is not necessarily even related to the issue of AGI, the AGI certainly doesn’t need any code that tries to mimick human thought. As far as I can tell, all it really needs (and really this might be putting more constraints than are necessary) is code that allows it to adapt to general environments (transferability) that have nice computable approximations it can build by using the data it gets through it’s sensory modalities (these can be anything from something familiar, like a pair of cameras, or something less so like a geiger counter or some kind of direct feed from thousands of sources at once).
Also, a utility function that encodes certain input patterns with certain utilities, some [black box] statistical hierarchical feature extraction [/black box] so it can sort out useful/important features in its environment that it can exploit. Researchers in the areas of machine learning and reinforcement learning are working on all of this sort of stuff, it’s fairly mainstream.
As far as computing power—the computing power of the human brain is definitely measurable so we can do a pretty straightforward analysis of how much more is possible. As far as raw computing power, I think we’re actually getting quite close to the level of the human brain, but I can’t seem to find a nice source for this. There are also interesting “neuromorphic” technologies geared to stepping up the massively parallel processing (many things being processed at once) and scale down hardware size by a pretty nice factor (I can’t recall if it was 10 or 100), such as the SyNAPSE project. In addition, with things like cloud/distributed computing, I don’t think that getting enough computing power together is likely to be much of an issue.
Bootstrapping is a metaphor referring to the ability of a process to proceed on its own. So a bootstrapping AI is one that is able to self-improve along a stable gradient until it reaches superintelligence. As far as “how does it know what bits to change”, I’m going to interpret that as “How does it know how to improve itself”. That’s tough :) . We have to program it to improve automatically by using the utility function as a guide. In limited domains, this is easy and has already been done. It’s called reinforcement learning. The machine reads off its environment after taking an action an updates its “policy” (the function it uses to pick its actions) after getting feedback (positive or negative or no utility).
The tricky part is having a machine that can self-improve not just by reinforcement in a single domain, but in general, both by learning and by adjusting its own code to be more efficient, all while keeping its utility function intact—so it doesn’t start behaving dangerously.
As far as SIAI, I would say that Friendliness is the driving factor. Not because they’re concerned about friendliness, but because (as far as I know) they’re the first group to be seriously concerned with friendliness and one of the only groups (the other two being headed by Nick Bostrom and having ties to SIAI) concerned with Friendly AI.
Of course the issue is that we’re concerned that developing a generally intelligent machine is probable, and if it happens to be able to self improve to a sufficient level it will be incredibly dangerous if no one put in some serious, serious effort into thinking about how it could go wrong and solving all of the problems necessary to safeguard against that. If you think about it, the more powerful the AGI is, the more needs to be considered. An AGI that has access to massive computing power, can self improve and can get as much information (from the internet and other sources) as it wants, could easily be a global threat. This is, effectively, because the utility function has to take into account everything the machine can affect in order to guarantee we avoid catastrophe. An AGI that can affect things at a global scale needs to take everyone into consideration, otherwise it might, say, drain all electricity from the Eastern seaboard (including hospitals and emergency facilities) in order to solve a math problem. It won’t “know” not to do that, unless it’s programed to (by properly defining its utility function to make it take those things into consideration). Otherwise it will just do everything it can to solve the math problem and pay no attention to anything else. This is why keeping the utility function intact is extremely important. Since only a few groups, SIAI, Oxford’s FHI and the Oxford Martin Programme on the Impacts of Future Technologies, seem to be working on this, and it’s an incredibly difficult problem, I would much rather have SIAI develop the first AGI than anywhere else I can think of.
Hopefully that helps without getting too mired in details :)
The second is that consciousness is not necessarily even related to the issue of AGI, the AGI certainly doesn’t need any code that tries to mimick human thought. As far as I can tell, all it really needs (and really this might be putting more constraints than are necessary) is code that allows it to adapt to general environments (transferability) that have nice computable approximations it can build by using the data it gets through it’s sensory modalities (these can be anything from something familiar, like a pair of cameras, or something less so like a geiger counter or some kind of direct feed from thousands of sources at once).
Also, a utility function that encodes certain input patterns with certain utilities, some [black box] statistical hierarchical feature extraction [/black box] so it can sort out useful/important features in its environment that it can exploit. Researchers in the areas of machine learning and reinforcement learning are working on all of this sort of stuff, it’s fairly mainstream.
I am not entirely sure I understood what was meant by those two paragraphs. Is a rough approximation of what you’re saying “an AI doesn’t need to be conscious, an AI needs code that will allow it to adapt to new environments and understand data coming in from its sensory modules, along with a utility function that will tell it what to do”?
Yeah, I’d say that’s a fair approximation. The AI needs a way to compress lots of input data into a hierarchy of functional categories. It needs a way to recognize a cluster of information as, say, a hammer. It also needs to recognize similarities between a hammer and a stick or a crow bar or even a chair leg, in order to queue up various policies for using that hammer (if you’ve read Hofstadter, think of analogies) - very roughly, the utility function guides what it “wants” done, the statistical inference guides how it does it (how it figures out what actions will accomplish its goals). That seems to be more or less what we need for a machine to do quite a bit.
If you’re just looking to build any AGI, he hard part of those two seems to be getting a nice, working method for extracting statistical features from its environment in real time. The (significantly) harder of the two for a Friendly AI is getting the utility function right.
An AGI that has access to massive computing power, can self improve and can get as much information (from the internet and other sources) as it wants, could easily be a global threat.
Interestingly, hypothetical UFAI (value drift) risk is something like other existential risks in its counterintuitive impact, but more so, in that (compared to some other risks) there are many steps where you can fail, that don’t appear dangerous beforehand (because nothing like that ever happened), but that might also fail to appear dangerous after-the-fact, and therefore as properties of imagined scenarios where they’re allowed to happen. The grave implications aren’t easy to spot. Assuming soft takeoff, a prototype AGI escapes to the Internet—would that be seen as a big deal if it didn’t get enough computational power to become too disruptive? In 10 years it grown up to become a major player, and in 50 years it controls the whole future…
Even without assuming intelligence explosion or other extraordinary effects, the danger of any misstep is absolute, and yet arguments against these assumptions are taken as arguments against the risk.
A couple of things come to mind, but I’ve only been studying the surrounding material for around eight months so I can’t guarantee a wholly accurate overview of this. Also, even if accurate, I can’t guarantee that you’ll take to my explanation.
Anyway, the first thing is that brain form computing probably isn’t a necessary or likely approach to artificial general intelligence (AGI) unless the first AGI is an upload. There doesn’t seem to be good reason to build an AGI in a manner similar to a human brain and in fact, doing so seems like a terrible idea. The issues with opacity of the code would be nightmarish (I can’t just look at a massive network of trained neural networks and point to the problem when the code doesn’t do what I thought it would).
The second is that consciousness is not necessarily even related to the issue of AGI, the AGI certainly doesn’t need any code that tries to mimick human thought. As far as I can tell, all it really needs (and really this might be putting more constraints than are necessary) is code that allows it to adapt to general environments (transferability) that have nice computable approximations it can build by using the data it gets through it’s sensory modalities (these can be anything from something familiar, like a pair of cameras, or something less so like a geiger counter or some kind of direct feed from thousands of sources at once).
Also, a utility function that encodes certain input patterns with certain utilities, some [black box] statistical hierarchical feature extraction [/black box] so it can sort out useful/important features in its environment that it can exploit. Researchers in the areas of machine learning and reinforcement learning are working on all of this sort of stuff, it’s fairly mainstream.
As far as computing power—the computing power of the human brain is definitely measurable so we can do a pretty straightforward analysis of how much more is possible. As far as raw computing power, I think we’re actually getting quite close to the level of the human brain, but I can’t seem to find a nice source for this. There are also interesting “neuromorphic” technologies geared to stepping up the massively parallel processing (many things being processed at once) and scale down hardware size by a pretty nice factor (I can’t recall if it was 10 or 100), such as the SyNAPSE project. In addition, with things like cloud/distributed computing, I don’t think that getting enough computing power together is likely to be much of an issue.
Bootstrapping is a metaphor referring to the ability of a process to proceed on its own. So a bootstrapping AI is one that is able to self-improve along a stable gradient until it reaches superintelligence. As far as “how does it know what bits to change”, I’m going to interpret that as “How does it know how to improve itself”. That’s tough :) . We have to program it to improve automatically by using the utility function as a guide. In limited domains, this is easy and has already been done. It’s called reinforcement learning. The machine reads off its environment after taking an action an updates its “policy” (the function it uses to pick its actions) after getting feedback (positive or negative or no utility).
The tricky part is having a machine that can self-improve not just by reinforcement in a single domain, but in general, both by learning and by adjusting its own code to be more efficient, all while keeping its utility function intact—so it doesn’t start behaving dangerously.
As far as SIAI, I would say that Friendliness is the driving factor. Not because they’re concerned about friendliness, but because (as far as I know) they’re the first group to be seriously concerned with friendliness and one of the only groups (the other two being headed by Nick Bostrom and having ties to SIAI) concerned with Friendly AI.
Of course the issue is that we’re concerned that developing a generally intelligent machine is probable, and if it happens to be able to self improve to a sufficient level it will be incredibly dangerous if no one put in some serious, serious effort into thinking about how it could go wrong and solving all of the problems necessary to safeguard against that. If you think about it, the more powerful the AGI is, the more needs to be considered. An AGI that has access to massive computing power, can self improve and can get as much information (from the internet and other sources) as it wants, could easily be a global threat. This is, effectively, because the utility function has to take into account everything the machine can affect in order to guarantee we avoid catastrophe. An AGI that can affect things at a global scale needs to take everyone into consideration, otherwise it might, say, drain all electricity from the Eastern seaboard (including hospitals and emergency facilities) in order to solve a math problem. It won’t “know” not to do that, unless it’s programed to (by properly defining its utility function to make it take those things into consideration). Otherwise it will just do everything it can to solve the math problem and pay no attention to anything else. This is why keeping the utility function intact is extremely important. Since only a few groups, SIAI, Oxford’s FHI and the Oxford Martin Programme on the Impacts of Future Technologies, seem to be working on this, and it’s an incredibly difficult problem, I would much rather have SIAI develop the first AGI than anywhere else I can think of.
Hopefully that helps without getting too mired in details :)
I am not entirely sure I understood what was meant by those two paragraphs. Is a rough approximation of what you’re saying “an AI doesn’t need to be conscious, an AI needs code that will allow it to adapt to new environments and understand data coming in from its sensory modules, along with a utility function that will tell it what to do”?
Yeah, I’d say that’s a fair approximation. The AI needs a way to compress lots of input data into a hierarchy of functional categories. It needs a way to recognize a cluster of information as, say, a hammer. It also needs to recognize similarities between a hammer and a stick or a crow bar or even a chair leg, in order to queue up various policies for using that hammer (if you’ve read Hofstadter, think of analogies) - very roughly, the utility function guides what it “wants” done, the statistical inference guides how it does it (how it figures out what actions will accomplish its goals). That seems to be more or less what we need for a machine to do quite a bit.
If you’re just looking to build any AGI, he hard part of those two seems to be getting a nice, working method for extracting statistical features from its environment in real time. The (significantly) harder of the two for a Friendly AI is getting the utility function right.
Interestingly, hypothetical UFAI (value drift) risk is something like other existential risks in its counterintuitive impact, but more so, in that (compared to some other risks) there are many steps where you can fail, that don’t appear dangerous beforehand (because nothing like that ever happened), but that might also fail to appear dangerous after-the-fact, and therefore as properties of imagined scenarios where they’re allowed to happen. The grave implications aren’t easy to spot. Assuming soft takeoff, a prototype AGI escapes to the Internet—would that be seen as a big deal if it didn’t get enough computational power to become too disruptive? In 10 years it grown up to become a major player, and in 50 years it controls the whole future…
Even without assuming intelligence explosion or other extraordinary effects, the danger of any misstep is absolute, and yet arguments against these assumptions are taken as arguments against the risk.