Thank you for your well-formulated comment. I agree that more details/precision could be much appreciated.
I am confused by the title, and the conclusion.
Not understanding the title and the conclusion is a natural/expected reaction. I wanted to write this Meetup summary for a long time and only thought of this funny headline for a title and I guess the conclusion might seem like a weird way to come back on feet. I was also short on time so I had to be overly implicit. I will nonetheless try to answer your comment the best as I can.
If an ASI sees a Segway, a single time, would it be able to infer what is does, what’s it for, how to build it, etc.? I think so! The purpose of one-shot learning models is to provide a context, a structure, that can be augmented with a new concept based on a single example. This is far simpler than coming up with said new concept from scratch.
I also think so! I totally agree that providing a structure/context is much simpler to truly innovate by creating a completely new idea (such as general relativity for Einstein).
See, on efficient use of sensory data, That Alien Message.
Totally relevant reference, thank you.
I interpret your post as « no, an ASI shouldn’t build the telescope, because it’s a waste of resources and it wouldn’t even need it » but I’m not sure this was the message you wanted to send.
I think I was not clear enough about the message. Thank you for asking for clarifications.
Actually, I believe the ASI should build the telescope (and it might not even be a waste of resource if it knows physics well enough to optimize it in a smart way).
The Segway is not, in itself, a complicated engineering product. An ASI could, in principle, generalize the concept of a Segway from seeing it only once (as you mentioned) and understand the usage humans would have of it (if it had some prior knowledge about humans, of course).
What I meant by “Intergalactic Segway” is an ad hoc engineering product made by some strange intergalactic empire we have never met. Segways seem really convenient for humans, but they are so because they fit our biological bodies which are very specific and adapted from natural selection (which, in turn, adapted from planet Earth).
I believe aliens might have different needs and engineering features, and would end up building “Intergalactic Segways” to suit their needs, and that we would have not a single clue about what those “Intergalactic Segways” even look like.
Furthermore, even if for the ASI it was more resource efficient to generate 10^30 simulations of the Universe to know how other aliens behave, I think it is not enough.
I think the search space for alien civilizations (if we assume that human-level-intelligence civilizations are rare in the universe) is huge, and that to run sufficiently precise physical simulations in this incredibly huge space would prove impossible, and that building a telescope (or just send von Neumann probes at the edges of the observable universe) would be the only efficient solution.
This is all I have to say for now (had not thought more about it).
If you have more critics/questions I would be happy to discuss it further.
Thanks again for this piece. I’ll follow your daily posts and comment on them regularly!
Very kind of you!
if an AGI could simulate quasi-perfectly a human brain, with human knowledge encoded inside, would your utility function be satisfied?
Interesting thought experiment. I would say no, but it depends on how it simulates the human brain.
If brain scans allow to quasi-perfectly get the position of every cell in a brain, and that we know how to model their interactions, we could have an electric circuit without knowing much about the information inside it.
So we would have the code, but we would have nothing about the meaning, so it would not “understand how the knowledge is encoded”.
is the goal of understanding all there is to the utility function? What would the AGI do, once able to model precisely the way humans encode knowledge? If the AGI has the keys to the observable universe, what does it do with it?
You’re absolutely right in the sense that it does not constitute a valid utility function for the alignment problem, or for anything really useful, if it is the one used for the final goal.
My point was that Yudkowsky showed how the encoding utility function was limited because of the simple way of maximizing it, but if we changed it to my “understanding the encoding” it could be much more interesting, enough to lead to AGI.
Once the AGI knows how humans encode knowledge, it can basically have, at least, the same model of reality of humans (assuming it has the same Input sensors (e.g. things like eyes, ears, etc.) , which is not very difficult) by encoding knowledge the same way. And then, because it is in Silico (and not in a biological prison), it can do everything humans do but much faster (basically an ASI).
I guess if it reaches this point, it would be the same as humans living for thousands of subjective years, and so it would be able to do whatever humans would believe useful if we were given more time to think about it.
Should we implement ought statements inside it in addition to the “understanding the encoding” utility function? Or the “faster human” is enough? I don’t know.
Yes, basically communicating is understanding how information is encoded in the receiver’s model of reality and encoding your message without hurting their feelings (if not necessary).
Here is my feedback on your post:
Tact: Interesting and funny!
Nerd speaking: I think I already understood your point with the tact filters quote, so what went after was redundant.
More Tact: But I nonetheless liked the diagrams.
First, let me thank you for taking the time of writing down the premises/arguments. I think you summarized the argumentation sufficiently well to allow a precise discussion.
Ancestor simulations are defined in the simulation argument as “A single such a computer could simulate the entire mental history of humankind (call this an ancestor-simulation) [...].”. I agree with you that the argument deals with the probability of ancestor simulations, and not first-person simulations. Therefore, I cannot infer the probability of a first-person simulation from the simulation argument.
However, such simulations are mentioned here:
“In addition to ancestor-simulations, one may also consider the possibility of more selective simulations that include only a small group of humans or a single individual. The rest of humanity would then be zombies or “shadow-people” – humans simulated only at a level sufficient for the fully simulated people not to notice anything suspicious. It is not clear how much cheaper shadow-people would be to simulate than real people. It is not even obvious that it is possible for an entity to behave indistinguishably from a real human and yet lack conscious experience. Even if there are such selective simulations, you should not think that you are in one of them unless you think they are much more numerous than complete simulations. There would have to be about 100 billion times as many “me-simulations” (simulations of the life of only a single mind) as there are ancestor-simulations in order for most simulated persons to be in me-simulations.” (https://www.simulation-argument.com/simulation.html)
In short, I should only believe that I live in a me-simulations if I think they are 100 billion times more probable than ancestor-simulations.
Let me try to estimate the probability of those me-simulations, and compare it with the probability of ancestor simulations.
First, I claim that one of these two assertions is true (assuming the existence of post-humans who can run ancestor-simulations):
i) Post-humans will run ancestor-simulations because they don’t observe other forms of (conscious) intelligence in the universe (Fermi Paradox), and are therefore trying to understand if there was indeed a great filter before them.
ii) Post-humans will observe that other forms of conscious intelligence are abundant and the universe, and have low interest in running Ancestor simulations.
Second, even with the premise of physical consciousness, I claim that me-simulations could be made at lest 100 billion times less computationally expensive than full simulations. Here are my reasons to believe so:
1) Even though it would be necessary to generate consciousness to mimic human processes, it would only be necessary for the humans you directly interact to, so maybe 10 hours of human consciousness other than yours every day.
2) The physical density needed to simulate a me-simulation would be at most the size of your room (about 20 meters squared * the height of your room). If you are in room it is trivially true, and if you are in the outer world I believe you are less self-aware of the rest of the physical world, so the “complexity of reality” necessary so that you to believe the world is real is about the same as if you were in your room. However, Earth’s Surface is about 500 million km squared, so 2.5 * 10^13 times greater. It follows that it would be at least 100 billion times less computationally intensive to run a me-simulation, assuming you would want to simulate at least the same height for the ancestor-simulation.
3) You would only need to run one ancestor civilization to run an infinitely large number of me-simulation: if you know about the environment and have in memory how the conscious humans behaved, you can easily run a me-simulation where a bunch of the other characters are just a copy of what they did in the past (when they are in your focus), but only one (or a small number of people) is conscious. A bit like in Westworld there are some plots where robots are really convincing, but in general they are not.
I am only starting to answer your comment and have already written a lot, so I might just create a post about selective simulations latter today. If so, you could reply to this part there.
I think I understand your point. If I got it right, you are saying that I am not contradicting Effective Altruism, but only applying an empirical reasoning with EA’s principles? If so, I agree with your claim. I guess I tried to apply Effective Altruism’s principles, and in particular the Utilitarian view (which might be controversial,even inside of EA, I don’t know) to the described world (a video-game-like life) to show that it resulted in what I called ethical egoism.
If, counterfactually, there were other conscious beings in the world, would you think that they also had moral worth?
I don’t share deeply the Utilitarian view. To describe it shortly I believe I am a Solipsist who values the perception of complexity. So I value my own survival because I may not have any proof of any kind of the complexity of the Universe if I stop to exist, but also the survival of Humanity (because I believe humans are amazingly complex creatures), but I don’t value positive subjective perceptions of other conscious human beings. I value my own positive subjective perceptions because it maximizes my utility function of maximizing my perception of complexity.
Let’s suppose I had answered yes.
To come back to what we said about physicalism and the simulation of other conscious agents in a first-person view. You said:
“[...] even in a first-person simulation, the people you were interacting with would be conscious as long as they were within your frame of awareness (otherwise the simulation couldn’t be accurate), it’s just that they would blink out of existence once they left your frame of awareness.”
I claim that even though they would be conscious in the “frame of awareness”, they would not deserve any altruism, even considering Expected Value. The reason is that if you give them sporadic consciousness, it greatly lacks of what I consider as a conscious human’s subjective experience. In particular, if the other simulated humans I connect to do not have any continuity in their consciousness, and the rest is just false memories (e.g. WestWorld), I would give a much greater value to my own subjective experience (at least 1000 times more valuable I would say).
So if I have 10$ in my pocket, I would still use it to buy me an icecream and I would not buy it to some random guys in the street, even if my 10$ could buy them 100 icereams (but I might hesitate with like 10000 for instance). The issue here is the probability of consciousness. If I assume there is a 1⁄1 000 000 chance someone is conscious and that I value my subjective experiences 10 000 times more than theirs, I would need to be able to buy like 10 000 * 1 000 000 = 10 000 000 000 icecreams (for more people than on Earth) to not buy me an icecream.
Anyway, I am very glad you clarified my hypothesis with your comment, asked for clarification and objected courteously. My post was not explicit at all and lacked rational/details arguments, what you did. Answering you made me think a lot. Thank you.
Feel free to let me know what you think. PS: I might do a full post on the probability of me-simulations in less than 10h as I said above.
As a general feeling, I am very confused because I don’t know if this is a joke, or if you are really blaming the people you mention.
Tell me if I get it right:
1) There are other sets of qualities than the list you establish. Hence, it is a shame that people from the community only have those qualities.
2) In general, activities involving sports, martial arts or anything that strengthens the body/survival skills is valuable.
3) Problem: people intellectually flex without being neither “tough” nor “wise”.
Assuming the people you mention all exhibit certain qualities from your list. What would be the cause?
What exactly would a rationalist gain from martial/survival skills? Would it be different from what an average “unfit white male” , as you describe it, would gain?
What exactly makes you think that people flex? Do you have specific examples?
Thank you for reading me.
1- In this post I don’t really mention “non-me-simulations”. I try to compare the probability of only one full-time conscious being (me-simulation) to what Bostrom calls ancestor-simulations, as those full-scale simulations where one could replay “the entire mental history of humankind”.
For any simulation consisting of N individuals (e.g. N = 7 billion), there could in principle exist simulations where 0, 1, 2, … or N of those individuals are conscious.
When the number k of individuals being conscious satisfies k << N then I call the simulation selective.
I think your comment points out to the following apparent conjunction fallacy: I am trying to estimate the probability of the event “simulation of only one conscious individual” instead of “simulation of a limited number of individuals k << N” of greater probability (first problem)
The point I was trying to make is the following: 1) ancestor-simulations (i.e. full-scale and computationally-intensive simulations to understand ancestor’s history) would be motivated by more and more evidence of a Great Filter behind the posthuman civilization. 2) the need for me-simulation (which would be the most probable type of selective simulation because it only needs one player (e.g. a guy in his spaceship)) do not appear to rely on the existence of a Great Filter behind the posthuman civilization. They could be like cost-efficient single consciousness play for fun, or prisoners are condemned to.
I guess the second problem with my argument for the probablity of me-simulations is that I don’t give any probability of being in a me-simulation, whereas in the original simulation argument, the strength of Bostrom’s argument is that whenever an ancestor-simulation is generated, 100 billion conscious lives are created, which greatly improves the probability of being in such a simulation. Here, I could only estimate the cost-effectiveness of me-simulation in comparison with ancestor simulation.
2- I think you are assuming I believe in Utilitarianism. Yes, I agree that if I am Utilitarian I may want to act altruistically, even with some very small non-zero probability of being in a non-me-simulation or in reality.
I already answered to this question Yesterday in the effective egoist post (cf. comment to Ikaxas) and I am realizing that my answer was wrong because I didn’t assume that other people could be full-time-conscious.
My argument (supposing I am Utilitarian, for the sake of argument), essentially, was that if I had 10$ in my pocket and wanted to buy me an icecream (utility of 10 for me let’s say) I would need to provide an utility of 10*1000 to someone being full-time conscious to consider giving him the icecream (his utility would rise to 10 000 for instance). In the absence of some utility monster, I believe this case to be extremely unlikely and would end up eating icecreams all by myself.
[copy paste from Yesterday’s answer to Ikaxas] In practice, I don’t share deeply the Utilitarian view. To describe it shortly I believe I am a Solipsist who values the perception of complexity. So I value my own survival because I may not have any proof of any kind of the complexity of the Universe if I stop to exist, but also the survival of Humanity (because I believe humans are amazingly complex creatures), but I don’t value positive subjective perceptions of other conscious human beings. I value my own positive subjective perceptions because it maximizes my utility function of maximizing my perception of complexity.
Anyway, I don’t want to enter the debate of highly-controversial Effective Egoism inside what I wanted to be a more scientific probability-estimation post about a particular kind of simulation.
Thank you for your comment. I hope I answered you well. Feel free to ask any other clarification or point out to other fallacies in my reasoning.
Thank you for the link.
Few questions that come to mind:
What would be the differences / improvements from the “Concrete problems in AI Safety” paper? (https://arxiv.org/abs/1606.06565)
What would be the most important concrete problems to work on (for instance for a thesis)?
More generally, does anyone know if someone already made some kind of graph of dependencies (e.g. this problem must be solved before that one) ?
humans don’t seem to have continuity of consciousness, in that we sleep
Yes, humans do sleep. Let’s suppose that the consciousness “pauses” or “vanishes” during sleep. Is it how would you define a discontinuity in consciousness? An interval of time delta_t without consciousness separating two conscious processes?
Bergson defines in Time and Free Will: An Essay on the Immediate Data of Consciousness duration as inseparable from consciousness. What would it mean to change consciousness instantaneously with teleportation? Would we need a minimum delta_t for it to make sense (maybe we could infer it from physical constraints given by general relativity?).
Also, it seems plausible that you could fake the subjective continuity of consciousness in a sim.
The question is always how do you fake it. Assuming physicalism, there would be some kind of threshold of cerebral activity which would lead to consciousness. At what point in faking “the subjective continuity of consciousness” do we reach this threshold?
I think my intuition behind the fact that (minimum cerebral activity + continuity of memories) is what lead to the human-like “consciousness” is the first season of WestWorld, where Maeve, a host, becomes progressively self-aware by trying to connect her memories during the entire season (same with Dolores).
Yes, I agree that cost-effectiveness does not mean me-simulations would be useful.
What is needed are specific/empiric reasons why such posthuman civilization would want to run those me-simulations, which I haven’t done in this article. However, I tried to give some reasons in the next post (where you also commented) : https://www.lesswrong.com/posts/8fSaiJX7toRixR6Ee/are-you-living-in-a-me-simulation
Interesting question! I don’t have any clue. Maybe you could answer your own question, or give more information about those stories or your work on prehistorical technology?
Aside from that. you seem to think that when I am talking about halting a sim, I am emulating some gradual process like fallign asleep. I’m not.
I was not thinking that you were talking about a gradual process.
I think you are just conflating consciousness (conscious experience) and sense-of-self. It is quite possible to have the one without the other. eg severe amnesiacs are not p-zombies.
I agree that I am not being clear enough (with myself and with you) and appear to be conflating two concepts. With you’re example of amnesiacs and p-zombies two things come to my mind:
1) p-zombies: when talking about ethics (for instance in my Effective Egoist article) I was aiming at qualias, instead of just sim. conscious agents (with conscious experience as you say). To come back to your first comment, I wanted to say that “identity and continuity of consciousness” contribute to qualia, and make p-zombies less probable.
2) amnesiacs: in my video game I don’t want to play with a world full of amnesiacs. If whenever I ask questions about their past they’re being evasive, it does not feel real enough. I want them to have some memories. Here is a claim:
(P) “For memories to be consistent, the complexity needed would be the same as the complexity needed to emulate the experience which would produce the memory”
I am really unsure about this claim (one could produce fake memories just good enough for people not to notice anything. We don’t have great memories ourselves). However, I think it casts light on what I wanted to express with “The question is always how do you fake it.” Because it must be real/complex enough for them not to notice anything (and the guy in the me-sim. too) but also not too complex (otherwise you could just run full-simulations).
Dear MIRI Team,
You mentioned the 25th of April that you were still accepting applications.
I submitted an application to your AI Summer Fellow Application about one week ago.
I would like to know (before the 14th of May if possible) if my Application was either accepted (as part of a “finalist” group?), or if it did not fit with your expectations.
[PS: If anyone reading this has already received an answer, would it be possible to let me know? Thank you]
I feel that what shapes the behaviour is not the belief in itself, but what this belief implies.
It’s more an empiric law like “This guy believes in A, so it’s improbable that he also believes in B, given that he is smart enough to not have contradicting views” (e.g. Solipsism and Utilitarianism).
Thank you. Will try to give more day-to-day rationality applications if I can.
EDIT: about the “write more things down”, I think writing specific to LessWrong stuff (like when your beliefs change) might prove useful. However, just a lot of personal journaling or thought-crystallization-on-the-internet is enough.
Thank you for your feedback. I used to write Medium articles for publications (e.g. this one). For some publications, the guideline was to use bold at least once every paragraph.
When a friend of mine (LW reader) read one of those articles, he gently commented that he felt he was reading buzzfeed (overly distracting formatting).
That’s why I tried to switch to undeline/italic in my writing (besides, I have a general feeling of simplicity/minimalism here).
Will edit this post with a better formatting soon (cf. Elo’s google doc below)
EDIT: just realized there was only bold and italic in Medium, no underline...