Show me how that is going to work out. Or at least outline how a smarter-than-human AI is supposed to take over the world. Why is nobody doing that?
People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything. The scenarios will never be enough for critics because until they are actually executed there will always be some doubt that they would work—at which point there would be no need to discuss them any more. Just like in cryonics (if you already had the technology to revive someone, there would be no need to discuss whether it would work). This is intrinsic to any discussion of threats that have not already struck or technologies which don’t already exist.
I am reminded of the quote, “‘Should we trust models or observations?’ In reply we note that if we had observations of the future, we obviously would trust them more than models, but unfortunately observations of the future are not available at this time.”
100 people are not enough to produce and employ any toxic gas or bombs in a way that would defeat a wide-stretched empire with many thousands of people.
Because that’s the best way to take over...
I said that it is highly speculative that there exists a simple algorithm that would constitute a consequentialist AI with simple values that could achieve the same as aforementioned society of minds and therefore work better than evolution. You just turned that into “XiXiDu believes that simple algorithms can’t exhibit creativity.”
That is not what you said. I’ll requote it:
Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution...An AI with simple values will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue. Which will allow an AI to solve some well-defined narrow problems, but it will be unable to make use of the broad range of synergetic effects of cultural evolution. Cultural evolution is a result of the interaction of a wide range of utility-functions.
If a singleton lacks feedback from diversity and something which is the ‘cornerstone’ of diversity is something a singleton cannot have… This is actually even stronger a claim than simple algorithms, because a singleton could be a very complex algorithm. (You see how charitable I’m being towards your claims? Yet no one appreciates it.)
And that’s not even getting into your claim about spectrum of research, which seems to impute stupidity to even ultraintelligent agents.
(‘Let’s see, I’m too dumb to see that I am systematically underinvesting in research despite the high returns when I do investigate something other than X, and apparently I’m also too dumb to notice that I am underperforming compared to those oh-so-diverse humans’ research programs. Gosh, no wonder I’m failing! I wonder why I am so stupid like this, I can’t seem to find any proofs of it.’)
People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything. The scenarios will never be enough for critics because until they are actually executed there will always be some doubt that they would work...
Speaking as one of the critics, I’ve got to say that these scenarios are “not enough” for me not because there’s “some doubt that they would work”, but because there’s massive doubt that they would work. To use an analogy, I look both ways before crossing the street because I’m afraid of being hit by a car; but I don’t look up all the time, despite the fact that a meteorite could, theoretically, drop out of the sky and squash me flat. Cars are likely; meteorites are not.
I could elaborate regarding the reasons why I doubt some of these world takeover scenarios (including “hacking the Internet” and “eating everything”), if you’re interested.
I could elaborate regarding the reasons why I doubt some of these world takeover scenarios (including “hacking the Internet” and “eating everything”), if you’re interested.
Not really. Any scenario presented in any level of detail can be faulted with elaborate scenarios why it would not work. I’ve seen this at work with cryonics: no matter how detailed a future scenario is presented or how many options are presented in a disjunctive argument, no matter how many humans recovered from death or how many organs preserved and brought back, there are people who just never seem to think it has a non-zero chance of working because it has not yet worked.
For example, if I wanted to elaborate on the hacking the Internet scenario, I could ask you your probability on the possibility and then present information on Warhol worm simulations, prevalence of existing worms, number of root vulnerabilities a year, vulnerabilities exposed by static analysis tools like Coverity, the early results from fuzz testers, the size of the computer crime blackmarket, etc. until I was blue in the face, check whether you had changed your opinion and even if you said you changed it a little, you still would not do a single thing in your life differently.
Because, after all, disagreements are not about information. There’s a lot of evidence reasoning is only about arguing and disproving other people’s theories, and it’s increasingly clear to me that politics and theism are strongly heritable or determined by underlying cognitive properties like performance on the CRT or personality traits; why would cryonics or AI be any different?
The point of writing is to assemble useful information for those receptive, and use those not receptive to clean up errors or omissions. If someone reads my modafinil or nicotine essays and is a puritan with regard to supplements, I don’t expect them to change their minds; at most, I hope they’ll have a good citation for a negative point or mention a broken hyperlink.
Any scenario presented in any level of detail can be faulted with elaborate scenarios why it would not work.
That is a rather uncharitable interpretation of my words. I am fully willing to grant that your scenarios are possible, but are they likely ? If you showed me a highly detailed plan for building a new kind of skyscraper out of steel and concrete, I might try and poke some holes in it, but I’d agree that it would probably work. On the other hand, if you showed me a highly detailed plan for building a space elevator out of candy-canes, I would conclude that it would probably fail to work. I would conclude this not merely because I’ve never seen a space elevator before, but also because I know that candy-canes make a poor construction material. Sure, you could postulate super-strong diamondoid candy-canes of some sort, but then you’d need to explain where you’re going to get them from.
there are people who just never seem to think it has a non-zero chance of working because it has not yet worked.
For the record, I believe that cryonics has a non-zero chance of working.
...until I was blue in the face, check whether you had changed your opinion and even if you said you changed it a little, you still would not do a single thing in your life differently
I think this would depend on how much my opinion had, in fact, changed. If you’re going to simply go ahead and assume that I’m a disingenuous liar, then sure, there’s no point in talking to me. Is there anything I can say or do (short of agreeing with you unconditionally) to prove my sincerity, or is the mere fact of my disagreement with you evidence enough of my dishonesty and/or stupidity ?
The point of writing is to assemble useful information for those receptive, and use those not receptive to clean up errors or omissions.
And yet, de-converted atheists as well as converted theists do exist. Perhaps more importantly, the above sentence makes you sound as though you’d made up your mind on the topic, and thus nothing and no one could persuade you to change it in any way—which is kind of like what you’re accusing me of doing.
People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything.
But none of them make any sense to me, see below.
That is not what you said. I’ll requote it:
Wait, your quote said what I said I said you said I didn’t say.
Because that’s the best way to take over...
I have no idea. You don’t have any idea either or you’d have told me by now. You are just saying that magic will happen and the world will be ours. That’s the problem with risks from AI.
Let’s see, I’m too dumb to see that I am systematically underinvesting in research despite the high returns when I do investigate something other than X, and apparently I’m also too dumb to notice that I am underperforming compared to those oh-so-diverse humans’ research programs.
See, that’s the problem. The AI can’t acquire the resources that are necessary to acquire resources in the first place. It might figure out that it will need to pursue various strategies or build nanoassemblers, but how does it do that?
Taking over the Internet is no answer, because the question is how. Building nanoassemblers is no answer, because the question is how.
I have no idea. You don’t have any idea either or you’d have told me by now. You are just saying that magic will happen and the world will be ours. That’s the problem with risks from AI.
We have plenty of ideas. Yvain posted a Discussion thread filled with ideas how. “Alternate history” is an old sub-genre dating back at least to Mark Twain (who makes many concrete suggestions about how his Connecticut yankee would do something similar).
But what’s the point? See my reply to Bugmaster—it’s impossible or would defeat the point of the discussion to actually execute the strategies, and anything short of execution is vulnerable to ‘that’s magic!11!!1’
The AI can’t acquire the resources that are necessary to acquire resources in the first place. It might figure out that it will need to pursue various strategies or build nanoassemblers, but how does it do that?
By reading the many discussions of what could go wrong and implementing whatever is easiest, like hacking computers. Oh the irony!
People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything. The scenarios will never be enough for critics because until they are actually executed there will always be some doubt that they would work—at which point there would be no need to discuss them any more. Just like in cryonics (if you already had the technology to revive someone, there would be no need to discuss whether it would work). This is intrinsic to any discussion of threats that have not already struck or technologies which don’t already exist.
I am reminded of the quote, “‘Should we trust models or observations?’ In reply we note that if we had observations of the future, we obviously would trust them more than models, but unfortunately observations of the future are not available at this time.”
Because that’s the best way to take over...
That is not what you said. I’ll requote it:
If a singleton lacks feedback from diversity and something which is the ‘cornerstone’ of diversity is something a singleton cannot have… This is actually even stronger a claim than simple algorithms, because a singleton could be a very complex algorithm. (You see how charitable I’m being towards your claims? Yet no one appreciates it.)
And that’s not even getting into your claim about spectrum of research, which seems to impute stupidity to even ultraintelligent agents.
(‘Let’s see, I’m too dumb to see that I am systematically underinvesting in research despite the high returns when I do investigate something other than X, and apparently I’m also too dumb to notice that I am underperforming compared to those oh-so-diverse humans’ research programs. Gosh, no wonder I’m failing! I wonder why I am so stupid like this, I can’t seem to find any proofs of it.’)
Speaking as one of the critics, I’ve got to say that these scenarios are “not enough” for me not because there’s “some doubt that they would work”, but because there’s massive doubt that they would work. To use an analogy, I look both ways before crossing the street because I’m afraid of being hit by a car; but I don’t look up all the time, despite the fact that a meteorite could, theoretically, drop out of the sky and squash me flat. Cars are likely; meteorites are not.
I could elaborate regarding the reasons why I doubt some of these world takeover scenarios (including “hacking the Internet” and “eating everything”), if you’re interested.
Not really. Any scenario presented in any level of detail can be faulted with elaborate scenarios why it would not work. I’ve seen this at work with cryonics: no matter how detailed a future scenario is presented or how many options are presented in a disjunctive argument, no matter how many humans recovered from death or how many organs preserved and brought back, there are people who just never seem to think it has a non-zero chance of working because it has not yet worked.
For example, if I wanted to elaborate on the hacking the Internet scenario, I could ask you your probability on the possibility and then present information on Warhol worm simulations, prevalence of existing worms, number of root vulnerabilities a year, vulnerabilities exposed by static analysis tools like Coverity, the early results from fuzz testers, the size of the computer crime blackmarket, etc. until I was blue in the face, check whether you had changed your opinion and even if you said you changed it a little, you still would not do a single thing in your life differently.
Because, after all, disagreements are not about information. There’s a lot of evidence reasoning is only about arguing and disproving other people’s theories, and it’s increasingly clear to me that politics and theism are strongly heritable or determined by underlying cognitive properties like performance on the CRT or personality traits; why would cryonics or AI be any different?
The point of writing is to assemble useful information for those receptive, and use those not receptive to clean up errors or omissions. If someone reads my modafinil or nicotine essays and is a puritan with regard to supplements, I don’t expect them to change their minds; at most, I hope they’ll have a good citation for a negative point or mention a broken hyperlink.
That is a rather uncharitable interpretation of my words. I am fully willing to grant that your scenarios are possible, but are they likely ? If you showed me a highly detailed plan for building a new kind of skyscraper out of steel and concrete, I might try and poke some holes in it, but I’d agree that it would probably work. On the other hand, if you showed me a highly detailed plan for building a space elevator out of candy-canes, I would conclude that it would probably fail to work. I would conclude this not merely because I’ve never seen a space elevator before, but also because I know that candy-canes make a poor construction material. Sure, you could postulate super-strong diamondoid candy-canes of some sort, but then you’d need to explain where you’re going to get them from.
For the record, I believe that cryonics has a non-zero chance of working.
I think this would depend on how much my opinion had, in fact, changed. If you’re going to simply go ahead and assume that I’m a disingenuous liar, then sure, there’s no point in talking to me. Is there anything I can say or do (short of agreeing with you unconditionally) to prove my sincerity, or is the mere fact of my disagreement with you evidence enough of my dishonesty and/or stupidity ?
And yet, de-converted atheists as well as converted theists do exist. Perhaps more importantly, the above sentence makes you sound as though you’d made up your mind on the topic, and thus nothing and no one could persuade you to change it in any way—which is kind of like what you’re accusing me of doing.
But none of them make any sense to me, see below.
Wait, your quote said what I said I said you said I didn’t say.
I have no idea. You don’t have any idea either or you’d have told me by now. You are just saying that magic will happen and the world will be ours. That’s the problem with risks from AI.
See, that’s the problem. The AI can’t acquire the resources that are necessary to acquire resources in the first place. It might figure out that it will need to pursue various strategies or build nanoassemblers, but how does it do that?
Taking over the Internet is no answer, because the question is how. Building nanoassemblers is no answer, because the question is how.
We have plenty of ideas. Yvain posted a Discussion thread filled with ideas how. “Alternate history” is an old sub-genre dating back at least to Mark Twain (who makes many concrete suggestions about how his Connecticut yankee would do something similar).
But what’s the point? See my reply to Bugmaster—it’s impossible or would defeat the point of the discussion to actually execute the strategies, and anything short of execution is vulnerable to ‘that’s magic!11!!1’
By reading the many discussions of what could go wrong and implementing whatever is easiest, like hacking computers. Oh the irony!