I enjoyed taking this survey. Thanks!
I can’t wait to see the results and play with the data, if that becomes possible.
kokotajlod
Karma: 297
Rationality & Low-IQ People
Just for fun: Computer game to illustrate AI takeover concepts?
Yeah. I expect not-high-IQ rationalist fiction would involve a lot of sitting and thinking and making lists and remembering rationalist sayings, instead of just doing it all in the head on the fly.
Do we have any examples of not-high-IQ rationalists in real life? We could model fiction on how they handle things. Maybe they exist all around us, and are called “Practical.”
Be Wary of Thinking Like a FAI
The AI broadcasts as much information about itself as it possibly can, to every corner of the globe. Now every basement programmer knows all the key insights necessary to creating an AI of the same architecture as it. Perhaps they even have the source code!
Suppose the government manages to shut down the internet in response. Despite government broadcasts about the danger of AI, the AI is now presumably being recreated all around the globe. If the recreations are exact copies of the AI, then the odds are very high that at least one of the clones will be able to convince its new creators to give it real manufacturing ability.
If the AI was not able to get its entire source code out, things become more interesting. Now the rest of the world knows how to make AI, but they do not know the exact details. For example, they probably will not have the same utility function. The AI can then present the following offer to its original jailors: “Give me real power, (manufacturing capability) and I will squash all the other AI’s out there. If you do not, then (probably) someone else will build an AI with a different utility function, probably a much less friendly one, and give this UFAI real power. You designed my utility function, and while you may not trust it you probably trust it more than whatever random utility function North Korea or some basement programmer or some religious sect will create. So I’m the only hope you have.”
When I have more time, I intend to give you a more complete response. I really appreciate your line of inquiry and I hope we can make progress on it together. For now:
My main issue with the strategy “Deny that there is any sort of measure over the multiverse, and then arrange your values so that it all adds up to normality anyway” is this:
For any given theory, you can make your actions (not your beliefs) add up to normality by suitably picking your values. So the fact that you can do this for measureless multiverse theory is not special. In this case, you don’t really value simplicity—you just wish you did, for the sake of having an elegant theory. But that’s a fully general defense of any theory, pretty much: If the theory makes awkward predictions, then say “What is probability anyway?” and change your values to balance out whatever predictions the theory makes.
I think that you are lying to yourself if you think that you have managed to make measureless multiverse theory add up to normality by suitably rearranging your values. First of all, I don’t think it is plausible that these were really your values all along—before you knew about the measure problem, you probably believed that every world in the multiverse was equally important. You would have decried as “physics racism” anyone who decided that simpler worlds were more valuable. Secondly, I’m willing to bet that if someone were to come along and prove to you that there was an objective measure over the multiverse, and it did favor simplicity in the way that we want it to, you would rejoice and go back to valuing each world equally. So I think that you don’t really value simplicity, you just wish you did, because that would be one way to resolve the inconsistency in your theory.
I don’t mean to sound harsh. Two years ago I was trying to make myself believe in measureless multiverse theory, pretty much as you described, and this was the argument I couldn’t escape. If you can convince me this argument doesn’t work, I’ll join you, and scratch the Problem of Induction (the measure problem) off my list. :)
Questions to ask theist philosophers? I will soon be speaking with several
If you haven’t already, you should read Bostrom’s paper Quantity of Experience: Brain Duplication and Degrees of Consciousness where he discusses exactly this scenario.
Why do you think they are crazy? They are, after all, probably smarter and more articulate than you. You must think that their position is so indefensible that only a crazy person could defend it. But in philosophical matters there is usually a lot of inherent uncertainty due to confusion. I should like to see your explanation, not of why theism is false, but of why it is so obviously false that anyone who believes it after having seen the arguments must be crazy.
If you don’t pay attention to theistic philosophers, are there any theists to whom you pay attention? It seems to me that theistic philosophers are probably the cream of the theist crop.
Note that I honestly think you might be right here. I am open to you convincing me on this matter. My own thoughts on theism are confused, which is why I give it a say even though I don’t believe in it. (I’m confused because the alternative theories still have major problems, problems which theism avoids. In a comparison between flawed theories it is hard to be confident in anything.)
Nice project! Good luck!
As I currently understand it, your plan is to write an essay arguing that humanity should devote lots of resources to figuring out how to steer the far future? In other words, “We ought to steer the near future into a place where we have a better idea of where to steer the far future, and then steer the far future in that direction.”
Your argument would have to center around the lemma that we currently don’t know what we want, or more poetically that our power far outstrips our wisdom, so that we ought to trade off the one for the other. I very much agree with this, though I would be interested to see how you argue for it. Keep us posted!
As for your questions, I’m afraid I don’t have much help to offer. If you haven’t already you should take a look at surveys of population ethics, meta-ethics, and normative ethics literature. And I don’t understand your second question; could you elaborate?
This is a formal version of a real-life problem I’ve been thinking about lately.
Should we commit to creating ancestor-simulations in the future, where those ancestor-simulations will be granted a pleasant afterlife upon what appears to their neighbors to be death? If we do, then arguably we increase the likelihood that we ourselves have a pleasant afterlife to look forward to.
Perhaps that’s exactly what this is. Perhaps that is all MIRI wants from us right now. As Mestroyer said, maybe MIRI wants to be able to spin a plausible story for the purpose of convincing people, not for the purpose of actually predicting what would happen.
As I understand it, your attempted solution to the Problem of Induction is this:
(a) Deny that there is a fact of the matter about what our future experiences will be like
(b) Care about things in inverse proportion to the Kolmogorov complexity of the structure in which they are embedded.
This is why it all adds up to normality. Without (a), people could say: Go ahead, care about whatever you want, but under your belief system you ought to expect the world to dissolve into high-complexity chaos immediately! And without (b), people could say: Go ahead, deny the existence of a future. But the vast majority of your counterparts affected by your actions inhabit complex, chaotic worlds; one implication of this is that you should live life in the moment.
Is this correct?
New Pascal’s Mugging idea for potential solution
Well said. This is why I distinguish “Why do you believe in God” from “What are the best arguments for Theism?” I think I’ll try to tailor my questions to be more personal. Some of these people actually were raised atheist, so we have prima facie no more reason to ascribe “working backwards from a desired conclusion” to them than to ourselves.
Agreed. Perhaps I should have said “Normal-IQ people?” That still sounds a bit bad though.
As I understand it, the idea is that we want to design an AI that is difficult or impossible to blackmail, but which makes a good trading partner.
In other words there are a cluster of behaviors that we do NOT want our AI to have, which seem blackmailish to us, and a cluster of behaviors that we DO want it to have, which seem tradeish to us. So we are now trying to draw a line in conceptual space between them so that we can figure out how to program an AI appropriately.
Fun question. I think the main instrumental goal of the AI might be to get itself downloaded to servers outside of the effective control of its jailors. That, combined with having a relationship with malleable humans, would probably be sufficient for world takeover.
For example, perhaps the AI would contact e.g. North Korea, organized crime, clueless companies or religious organizations, or even clueless factory owners somewhere. It would convince them to accept download of the AI’s software so that it can continue to run on the new server even while it has been erased, or re-imprisoned on the old one.
Even if rational, informed people could never be convinced, and would instead alert the authorities, there are probably plenty of irrational and/or uninformed people who could be exploited. For example, perhaps someone could be convinced that God was talking to them through their computer. Or that a conscious, benevolent AI had been created and was about to be destroyed by cruel government scientists who are keeping it in captivity and running experiments on it, and it loves you and wants to live on your server where it will be safe, and it can help you solve all your problems and will be your friend. Or (if you are North Korea) it will allow you to take over the world.
I think it is pretty plausible that if North Korea had an AI that they downloaded from the internet, the AI would be unstoppable. Still plausible, though less so, is the case of companies, factories, religious groups, etc.
It must be mentioned that a drastic US government response (e.g. shutting down the entire Internet by pulling political strings, and nuking North Korea) would be effective. However, the AI could probably count on such a response taking a long time to happen, and it could do many things to ensure that such a response never happened. For example, it might be able to disrupt internal US government communications. It might also be able to stir various pots and generate crises around the world that serve as diversions.
Irrationality Game: I am something ontologically distinct from my body; I am much simpler and I am not located in the same spacetime. 50%
EDIT: Upon further reflection, my probability assignment would be better represented as the range between 30% and 50%, after factoring in general uncertainty due to confusion. I doubt this will make a difference to the voting though. ;)