I currently don’t believe that psychology/cognitive science/neuroscience can be directly helpful for advancing the state of FAI, it seems more like a philosophy/math problem. It could be strategically helpful in developing arguments for complexity of human value. Other than that, if psychology is a necessary constraint, there’s money, the unit of caring, so look for higher-income directions.
I’m thinking of majoring in Math and minoring in computational neuroscience if I get into my top choice university. Is it your opinion that that minor would be a waste of time and effort?
If you expect to be able to directly contribute to FAI, my current very tentative advice (which I’m following myself) is to focus on pure math, assuming you understand the philosophical side (problem statement) well enough (which would be the limiting factor for people who are already mathematicians). Take this with a lot of salt, since it’s more of a hunch following from the character of seemingly relevant informal ideas, and I can’t exhibit actual FAI-relevant math (other than the decision theory that was previously published, which is an important foothold and whose meaning I still don’t quite grasp, which is also one motivation).
I currently don’t believe that psychology/cognitive science/neuroscience can be directly helpful for advancing the state of FAI
I’m much less convinced about this. Knowing more about how the human brain actually measures value and what value is for humans seems hugely useful, and some of the stuff in that direction that lukeprog has been posting has seemed potentially very helpful. It does not seem implausible to presume that FAI is simply impossible without that knowledge.
Yeah. This is one of the issues on which my intuitions seem to differ the most with Nesov’s, but I think we each concede that the other might be correct.
Right. If the philosophy/math path turns out to be genuinely impossible in some sense, neuroscience, if correctly interpreted, would be the next best source of detailed knowledge about human value. (I don’t expect this to be feasible without a human-powered WBE singleton vastly stretching the available time before a global catastrophe.)
What do you mean? Are you objecting to seeing a non-math path as even a barely feasible direction (i.e. it must be something handable to an AI, hence not neuroscience), or to the necessity of solving this problem when we already have CEV (i.e. all we need is to hand this problem over to an AI)? Both interpretations seem surprising, so I probably didn’t catch the right one.
Closer to the latter, but I’m not really objecting to anything, so much as just being confused. I assume you’re not saying we have to map out the “thousand shards of desire” before running CEV. I’m not saying there aren’t various aspects of human value that would be useful to understand before we run CEV, but I’m curious what sort of details you had in mind, such that you think they could be figured out with neuroscience and without philosophy but such that you expect doing so to take a very long time.
CEV is way too far from a useful design, or even a big-picture sketch. It’s a non-technical vague description of something that doesn’t automatically fail for obvious reasons, as compared to essentially all other descriptions of human decision problem (friendliness content) published elsewhere. But “running CEV” is like running the sketch of da Vinci’s flying machine.
CEV is a reasonable description on the level where a sketch of a plane doesn’t insist on the plane having a beak or being made entirely out of feathers. It does look very good in comparison to other published sketches. But we don’t have laws of aerodynamics or wind tunnels. “Running the sketch” is never a plan (which provokes protestations such as this). One (much preferable) way of going forward is to figure out the fundamental laws, that’s decision theory/philosophy/math path. Another is to copy a bird in some sense, collecting all of its properties in as much detail as possible (metaphorically speaking, so that it’s about copying goals and not about emulating brains); that’s the neuroscience path, which I expect isn’t viable no matter how much time is given, since we don’t really know how to learn about goals by looking at brains or behavior.
(Perhaps when we figure out the fundamental laws, it’ll turn out that we want a helicopter, and to the dustbin goes the original sketch.)
I agree that CEV needs conceptual and technical fleshing-out; when I said “run CEV”, I meant “run some suitably fleshed-out version of CEV”. You seem to be saying that to do this fleshing-out, we will need knowledge of some large subset of the details of human value. I’m not saying that’s false, but I’m trying to get at what sort of details you think those are; what variables we’re trying to find out the value of. Again, surely it’s not all the details, or we wouldn’t need to run CEV in the first place.
I’m fairly sure WBE will have rather limited usefulness for AGI. The human brain is an evolved organ that looks like a patchwork and it isn’t streamlined for intelligence—our numerous biases give testimony to that. There are a million mistakes we’re making while we think (or rather while we think we think) and we’re crap at things like self-control and giving reason preference over our feelings.
The human brain is deeply obsessed with social status and it’s a hypocrite with antisocial tendencies when it knows that no one is looking. Emulating a whole human brain and somehow adding more speed or some kind of über-rationality module doesn’t sound like a clean design/promising base for AGI at all. I for one wouldn’t trust it.
Pursuing WBE for the purpose of building AGI is like trying to replicate a bird’s feathers and movements in order to get people into the air, instead of building an airplane which is better suited for that purpose. Artificial intelligence probably won’t need to borrow all that much from the way biological brains work.
EDIT: Actually, I just realized that you wrote AGI, not FAI (friendly AI). If your goal is to emulate human-level intelligence along with emotions and everything else a human brain entails, then WBE is the way to go for AGI (General Intelligence). You could also play around and adapt the emulation to suit some specific purpose, but that has serious ethical and safety-ramifications. If you are talking about creating a maximally intelligent agent, then you can probably do better than building on top of WBE.
From an utilitarian point of view, why would you want to risk creating an uFAI if you have WBE and hence virtual reality and hence the ability to create a utopia? Is it worth the risks?
Sorry for not replying earlier, I have lots of stuff going on right now.
So basically you are describing a scenario where humanity goes digital via WBE and lives inside a utopian virtual reality (either entirely or most of the time) thus solving all or most of our current problems without the need to create FAI and risk utter destruction in the process. Not a bad question to have asked as far as I am concerned—it is a scenario I considered as well.
However, there are many good objections to this virtual reality idea, but I’ll just name the most obvious that comes to mind:
If anyone but a FAI is running this utopia, it seems very plausible that it would result in a dystopia, or at best something vaguely resembling our current world. Without some kind of superior intelligence monitoring the virtual reality, mere humans would have to make decisions about how Utopiaworld is run. This human fiddling makes it vulnerable to all kinds of human stupidities. Now, if the timeline this “solution” is supposed to work approaches near-infinity, then at some point collapse due to human error seems inevitable. Moreover, a virtual world opens up possibilities (and problems!) we can hardly imagine right now—possibilities and problems vast enough, that a human mind may simply not be adequate in order to solve and control them.
So unless you are willing to change the human mind in order to adapt it to a utopian virtual reality, it seems to be no viable option. Why mandatory brain-rewiring before uploading a human mind into a virtual reality run by human-like agents would be grotesque I won’t elaborate further.
I guess what my objection really comes down to is that humans are stupid, and thus we cannot be trusted to do anything right in the long run. Or at least we certainly can’t do things right enough to build a lasting and well-operating utopia. For that to work we would need extensive brain enhancements and if you go down this path, I think you run into the exact same problems I described already:
We humans are self-obsessed shits and bolting on a rationality module won’t change that. It would make us better at getting the things we want, but the things humans seem to want most of the time (judged by our current actions) are not working utopias but things typical for self-obsessed evolved agents: Power, social status and sex. Could this change by virtue of becoming more rational and intelligent? There’s a good chance it would. People would certainly think more (and more efficiently) about the future once they are smarter, but if you make humans sufficiently smart to produce something like a virtual utopia, you have already taken the first step to creating the uFAI you hoped to avoid in the first place. Humans would want to become even smarter than they (and all those other enhanced humans) already are, thus starting a race to the top. The result may be something terrible that has been endlessly envisioned throughout the ages: Powerful gods with the petty goals of humans.
So my current conclusion after thinking about the virtual reality scenario is that it’s not a viable long-term solution for humanity. That is not to say that I’ve extensively studied and thought about this option, but I think the objection I detailed is pretty convincing.
In a sentence: We’re not smart enough to directly construct a utopia (whether real or virtual) and if we actually made ourselves smart enough to do that, we’d probably raise the odds for uFAI. So risk-wise we would be worse off than trying to build a FAI from scratch, which would be more competent at solving our problems than humans could ever be (whether intelligence-enhanced or not).
Also, I realize that a virtual reality is a great way to absolve humanity of resource scarcity and restricted bodily abilities/requirements, but virtually all other problems (moral, social etc.) are left rather untouched by the virtual reality solution. A virtual reality is not automatically a utopia. In a way it would be like living in a world where everyone is immortal and filthy rich—everyone has huge power over material (or as in our case virtual) “things”, yet this doesn’t solve all your social or personal problems.
Thank you for taking the time to write out such a thoughtful reply. I will be taking the time to return the favor shortly.
EDIT: Here’s the (long) reply:
If anyone but a FAI is running this utopia
What does it mean to run a utopia? In order to run something, one must make decisions. What sort of decisions would this person or FAI be making? I realize it’s hard to predict what exact scenario it would be like, but we can speculate all we like (and then see which one’s sound most realistic/probable). Also, who said that anyONE would be “running” this virtual reality? It could be democratic.
Also, who said that utopias have to be with other real people/uploads rather than with non-conscious programs (like in a video game or a dream)? I can understand people’s desire to be in “reality” with “real people”. But this wouldn’t be necessary for video-game type virtual reality.
Now, if the timeline this “solution” is supposed to work approaches near-infinity, then at some point collapse due to human error seems inevitable
I think it is probably less inevitable than the heat-death of the universe. I think that the transition to hardware would permit people to survive on substrates that could exist in space with no need to inhabit a planet. There is no longer a need for food, only electricity (which would be easily available in the form of solar energy). Spreading this substrate in every direction reduce the risk of the collapse of “society”.
Why mandatory brain-rewiring before uploading a human mind into a virtual reality run by human-like agents would be grotesque I won’t elaborate further.
It won’t necessarily be mandatory to rewire oneself to be smarter, kinder, more emotionally stable. But, if one had the petty desires for power, sex, and status (as you claim), then they would willingly choose to rewire themselves (or risk being left behind).
We humans are self-obsessed shits and bolting on a rationality module won’t change that. It would make us better at getting the things we want, but the things humans seem to want most of the time (judged by our current actions) are not working utopias but things typical for self-obsessed evolved agents: Power, social status and sex. Could this change by virtue of becoming more rational and intelligent? There’s a good chance it would
Today, power is measured in terms of wealth or influence. Wealth seems like it would cease to be a relevant factor as economics is dependent on scarcity (have you ever had to buy air?) and in an age in which everything is digital, the only limitation is computational capacity.
Although this is hardly certain, I hypothesize that (“actual”) sex would cease to be a relevant motivator of uploads. Sex in a virtual reality would be free, clean, and offer the user the ability to simulate situations that wouldn’t be available to them in real life.
Status is usually sought today in order to have sex (see above) and by means of acquiring wealth (see above).
Personally, I believe that once we become uploads, the chemical imbalances and irrational beliefs that drive our behavior (for evolutionary purposes) will dissipate and we will be infinitely happier than we have ever been.
Powerful gods with the petty goals of humans.
Agreed that it is frightening. Nice way of putting it.
So risk-wise we would be worse off than trying to build a FAI from scratch, which would be more competent at solving our problems than humans could ever be (whether intelligence-enhanced or not).
This is the key question. What is riskier? I acknowledge that P(utopia|FAI+WBE) > P (utopia|WBE). But, I don’t acknowledge that P(utopia|AGI+WBE) > P (utopia|WBE).
Also, I realize that a virtual reality is a great way to absolve humanity of resource scarcity and restricted bodily abilities/requirements, but virtually all other problems (moral, social etc.) are left rather untouched by the virtual reality solution
I believe these problems are caused by scarcity problems (scarcity of intelligence, money, access to quality education). And as I’ve pointed out earlier, I think that seeking sex, power, and status will disappear.
A virtual reality is not automatically a utopia. In a way it would be like living in a world where everyone is immortal and filthy rich—everyone has huge power over material (or as in our case virtual) “things”, yet this doesn’t solve all your social or personal problems.
Personal problems are caused by unhappiness, craving, addiction, etc. These can all be traced back to brain states. These brain states could be “fixed” (voluntarily) by altering the digital settings of the digital neurochemical levels. (though I believe that we will have a much better idea of how to alter the brain than simply altering chemical levels. The current paradigm in neuroscience has a hammer (drugs) and so it tends to look at all the problems as nails).
I currently don’t believe that psychology/cognitive science/neuroscience can be directly helpful for advancing the state of FAI, it seems more like a philosophy/math problem. It could be strategically helpful in developing arguments for complexity of human value. Other than that, if psychology is a necessary constraint, there’s money, the unit of caring, so look for higher-income directions.
I’m thinking of majoring in Math and minoring in computational neuroscience if I get into my top choice university. Is it your opinion that that minor would be a waste of time and effort?
If you expect to be able to directly contribute to FAI, my current very tentative advice (which I’m following myself) is to focus on pure math, assuming you understand the philosophical side (problem statement) well enough (which would be the limiting factor for people who are already mathematicians). Take this with a lot of salt, since it’s more of a hunch following from the character of seemingly relevant informal ideas, and I can’t exhibit actual FAI-relevant math (other than the decision theory that was previously published, which is an important foothold and whose meaning I still don’t quite grasp, which is also one motivation).
I’m much less convinced about this. Knowing more about how the human brain actually measures value and what value is for humans seems hugely useful, and some of the stuff in that direction that lukeprog has been posting has seemed potentially very helpful. It does not seem implausible to presume that FAI is simply impossible without that knowledge.
Yeah. This is one of the issues on which my intuitions seem to differ the most with Nesov’s, but I think we each concede that the other might be correct.
Right. If the philosophy/math path turns out to be genuinely impossible in some sense, neuroscience, if correctly interpreted, would be the next best source of detailed knowledge about human value. (I don’t expect this to be feasible without a human-powered WBE singleton vastly stretching the available time before a global catastrophe.)
I’m confused. Isn’t “detailed knowledge about human value” the part of the problem that we’d hand off to an AI implementing CEV?
What do you mean? Are you objecting to seeing a non-math path as even a barely feasible direction (i.e. it must be something handable to an AI, hence not neuroscience), or to the necessity of solving this problem when we already have CEV (i.e. all we need is to hand this problem over to an AI)? Both interpretations seem surprising, so I probably didn’t catch the right one.
Closer to the latter, but I’m not really objecting to anything, so much as just being confused. I assume you’re not saying we have to map out the “thousand shards of desire” before running CEV. I’m not saying there aren’t various aspects of human value that would be useful to understand before we run CEV, but I’m curious what sort of details you had in mind, such that you think they could be figured out with neuroscience and without philosophy but such that you expect doing so to take a very long time.
CEV is way too far from a useful design, or even a big-picture sketch. It’s a non-technical vague description of something that doesn’t automatically fail for obvious reasons, as compared to essentially all other descriptions of human decision problem (friendliness content) published elsewhere. But “running CEV” is like running the sketch of da Vinci’s flying machine.
CEV is a reasonable description on the level where a sketch of a plane doesn’t insist on the plane having a beak or being made entirely out of feathers. It does look very good in comparison to other published sketches. But we don’t have laws of aerodynamics or wind tunnels. “Running the sketch” is never a plan (which provokes protestations such as this). One (much preferable) way of going forward is to figure out the fundamental laws, that’s decision theory/philosophy/math path. Another is to copy a bird in some sense, collecting all of its properties in as much detail as possible (metaphorically speaking, so that it’s about copying goals and not about emulating brains); that’s the neuroscience path, which I expect isn’t viable no matter how much time is given, since we don’t really know how to learn about goals by looking at brains or behavior.
(Perhaps when we figure out the fundamental laws, it’ll turn out that we want a helicopter, and to the dustbin goes the original sketch.)
I agree that CEV needs conceptual and technical fleshing-out; when I said “run CEV”, I meant “run some suitably fleshed-out version of CEV”. You seem to be saying that to do this fleshing-out, we will need knowledge of some large subset of the details of human value. I’m not saying that’s false, but I’m trying to get at what sort of details you think those are; what variables we’re trying to find out the value of. Again, surely it’s not all the details, or we wouldn’t need to run CEV in the first place.
I do hope the value problem turns out to be solvable with philosophy/math instead of cognitive science. The philosophy/math path is much preferred.
Sorry if this has been discussed before, but isn’t an open question whether WBE or AGI comes first? Couldn’t WBE be a path to AGI?
I’m fairly sure WBE will have rather limited usefulness for AGI. The human brain is an evolved organ that looks like a patchwork and it isn’t streamlined for intelligence—our numerous biases give testimony to that. There are a million mistakes we’re making while we think (or rather while we think we think) and we’re crap at things like self-control and giving reason preference over our feelings.
The human brain is deeply obsessed with social status and it’s a hypocrite with antisocial tendencies when it knows that no one is looking. Emulating a whole human brain and somehow adding more speed or some kind of über-rationality module doesn’t sound like a clean design/promising base for AGI at all. I for one wouldn’t trust it.
Pursuing WBE for the purpose of building AGI is like trying to replicate a bird’s feathers and movements in order to get people into the air, instead of building an airplane which is better suited for that purpose. Artificial intelligence probably won’t need to borrow all that much from the way biological brains work.
EDIT: Actually, I just realized that you wrote AGI, not FAI (friendly AI). If your goal is to emulate human-level intelligence along with emotions and everything else a human brain entails, then WBE is the way to go for AGI (General Intelligence). You could also play around and adapt the emulation to suit some specific purpose, but that has serious ethical and safety-ramifications. If you are talking about creating a maximally intelligent agent, then you can probably do better than building on top of WBE.
From an utilitarian point of view, why would you want to risk creating an uFAI if you have WBE and hence virtual reality and hence the ability to create a utopia? Is it worth the risks?
Sorry for not replying earlier, I have lots of stuff going on right now.
So basically you are describing a scenario where humanity goes digital via WBE and lives inside a utopian virtual reality (either entirely or most of the time) thus solving all or most of our current problems without the need to create FAI and risk utter destruction in the process. Not a bad question to have asked as far as I am concerned—it is a scenario I considered as well.
However, there are many good objections to this virtual reality idea, but I’ll just name the most obvious that comes to mind:
If anyone but a FAI is running this utopia, it seems very plausible that it would result in a dystopia, or at best something vaguely resembling our current world. Without some kind of superior intelligence monitoring the virtual reality, mere humans would have to make decisions about how Utopiaworld is run. This human fiddling makes it vulnerable to all kinds of human stupidities. Now, if the timeline this “solution” is supposed to work approaches near-infinity, then at some point collapse due to human error seems inevitable. Moreover, a virtual world opens up possibilities (and problems!) we can hardly imagine right now—possibilities and problems vast enough, that a human mind may simply not be adequate in order to solve and control them.
So unless you are willing to change the human mind in order to adapt it to a utopian virtual reality, it seems to be no viable option. Why mandatory brain-rewiring before uploading a human mind into a virtual reality run by human-like agents would be grotesque I won’t elaborate further.
I guess what my objection really comes down to is that humans are stupid, and thus we cannot be trusted to do anything right in the long run. Or at least we certainly can’t do things right enough to build a lasting and well-operating utopia. For that to work we would need extensive brain enhancements and if you go down this path, I think you run into the exact same problems I described already:
We humans are self-obsessed shits and bolting on a rationality module won’t change that. It would make us better at getting the things we want, but the things humans seem to want most of the time (judged by our current actions) are not working utopias but things typical for self-obsessed evolved agents: Power, social status and sex. Could this change by virtue of becoming more rational and intelligent? There’s a good chance it would. People would certainly think more (and more efficiently) about the future once they are smarter, but if you make humans sufficiently smart to produce something like a virtual utopia, you have already taken the first step to creating the uFAI you hoped to avoid in the first place. Humans would want to become even smarter than they (and all those other enhanced humans) already are, thus starting a race to the top. The result may be something terrible that has been endlessly envisioned throughout the ages: Powerful gods with the petty goals of humans.
So my current conclusion after thinking about the virtual reality scenario is that it’s not a viable long-term solution for humanity. That is not to say that I’ve extensively studied and thought about this option, but I think the objection I detailed is pretty convincing.
In a sentence: We’re not smart enough to directly construct a utopia (whether real or virtual) and if we actually made ourselves smart enough to do that, we’d probably raise the odds for uFAI. So risk-wise we would be worse off than trying to build a FAI from scratch, which would be more competent at solving our problems than humans could ever be (whether intelligence-enhanced or not).
Also, I realize that a virtual reality is a great way to absolve humanity of resource scarcity and restricted bodily abilities/requirements, but virtually all other problems (moral, social etc.) are left rather untouched by the virtual reality solution. A virtual reality is not automatically a utopia. In a way it would be like living in a world where everyone is immortal and filthy rich—everyone has huge power over material (or as in our case virtual) “things”, yet this doesn’t solve all your social or personal problems.
Thank you for taking the time to write out such a thoughtful reply. I will be taking the time to return the favor shortly.
EDIT: Here’s the (long) reply:
What does it mean to run a utopia? In order to run something, one must make decisions. What sort of decisions would this person or FAI be making? I realize it’s hard to predict what exact scenario it would be like, but we can speculate all we like (and then see which one’s sound most realistic/probable). Also, who said that anyONE would be “running” this virtual reality? It could be democratic. Also, who said that utopias have to be with other real people/uploads rather than with non-conscious programs (like in a video game or a dream)? I can understand people’s desire to be in “reality” with “real people”. But this wouldn’t be necessary for video-game type virtual reality.
I think it is probably less inevitable than the heat-death of the universe. I think that the transition to hardware would permit people to survive on substrates that could exist in space with no need to inhabit a planet. There is no longer a need for food, only electricity (which would be easily available in the form of solar energy). Spreading this substrate in every direction reduce the risk of the collapse of “society”.
It won’t necessarily be mandatory to rewire oneself to be smarter, kinder, more emotionally stable. But, if one had the petty desires for power, sex, and status (as you claim), then they would willingly choose to rewire themselves (or risk being left behind).
Today, power is measured in terms of wealth or influence. Wealth seems like it would cease to be a relevant factor as economics is dependent on scarcity (have you ever had to buy air?) and in an age in which everything is digital, the only limitation is computational capacity.
Although this is hardly certain, I hypothesize that (“actual”) sex would cease to be a relevant motivator of uploads. Sex in a virtual reality would be free, clean, and offer the user the ability to simulate situations that wouldn’t be available to them in real life.
Status is usually sought today in order to have sex (see above) and by means of acquiring wealth (see above).
Personally, I believe that once we become uploads, the chemical imbalances and irrational beliefs that drive our behavior (for evolutionary purposes) will dissipate and we will be infinitely happier than we have ever been.
Agreed that it is frightening. Nice way of putting it.
This is the key question. What is riskier? I acknowledge that P(utopia|FAI+WBE) > P (utopia|WBE). But, I don’t acknowledge that P(utopia|AGI+WBE) > P (utopia|WBE).
I believe these problems are caused by scarcity problems (scarcity of intelligence, money, access to quality education). And as I’ve pointed out earlier, I think that seeking sex, power, and status will disappear.
Personal problems are caused by unhappiness, craving, addiction, etc. These can all be traced back to brain states. These brain states could be “fixed” (voluntarily) by altering the digital settings of the digital neurochemical levels. (though I believe that we will have a much better idea of how to alter the brain than simply altering chemical levels. The current paradigm in neuroscience has a hammer (drugs) and so it tends to look at all the problems as nails).