Don’t humans have to give up on doing their own science then (at least fundamental physics) ?
I guess I can have the FAI make me a safe “real physics box” to play with inside the system; something that emulates what it finds out about real physics.
Kyre
Karma: 531
Alpha Go Zero comments
If you failed you’d want to distinguish between (a) rationalism sucking, (b) your rationalism sucking, or (c) EVE already being full of rationalists.
Whether or not success in Eve is relevant outside Eve is debatable, but I think the complexity, politics and intense competition means that it would be hard to find a better online proving ground.
Good advice, but I would go further. Don’t use your inbox as a to-do list at all. I maintain a separate to-do list for roughly three reasons.
(1) You can’t have your inbox in chronological and priority order. Keeping an inbox and email folders in chronological order is good for searching and keeping track of email conversations.
(2) Possibly just my own psychological quirk, but inbox emails feel like someone waiting for me and getting impatient. I can’t seem to get away from my inbox fundamentally representing a communications channel with people on the other end. Watching me.
(3) When I “do email”, I know I’m done when I have literally inbox zero, and I get the satisfaction of that several times a day.
I have found that I need scrupulous email and task accounting though. Every email gets deleted (and that advice on unsubscribing is good), or handled right away (within say 2 minutes), or gets a task on a to-do list and the email goes into a subject folder for when it comes to be dealt with.
Not just the environment in which you share your goals, but also how you suspect you will react to the responses you get.
When reading through these two scenarios, I can just as easily imagine someone reacting in exactly the opposite way. That is, in the first case, thinking “gosh, I didn’t know I had so many supportive friends”, “I’d better not let them down”, and generally getting a self-reinforcing high when making progress.
Conversely, say phase 1 had failed and got the responses stated above. I can imagine someone thinking “hey my friends are a bunch of jerks” and “they’re right, I’m probably going to fail again”, and then developing a flinch thinking about weight loss, and losing interest in trying.
My five minutes thoughts worth.
Metrics that might useful (on the grounds that in hindsight people would say that they made bad decisions): traffic accident rate, deaths due to smoking, bankruptcy rates, consumer debt levels.
Experiments you could do if you could randomly sample people and get enough of their attention: simple reasoning tests (e.g. confirmation bias), getting people to make some concrete predictions and following them up a year later.
Maybe something measuring people’s level of surprise at real vs fake facebook news (on the grounds people should be more surprised at fake news) ?
Doing theoretical research that ignores practicalities is sometimes turns out to be valuable in practice. It can open a door to something you assumed to be impossible; or save a lot of wasted effort on a plan that turns out to have an impossible sub-problem.
A concrete example of first category might be something like quantum error correcting codes. Prior to that theoretical work, a lot of people thought that quantum computers were not worth pursuing because noise and decoherence would be an insurmountable problem. Quantum fault tolerance theorems did nothing to help solve the very tough practical problems of building a quantum computer, but it did show people that it might be worth pursuing—and here we are 20 years later closing in on practical quantum computers.
I think source code based decision theory might have something of this flavour. It doesn’t address all those practical issues such as how one machine comes to trust that another machine’s source code is what it says. That might indeed scupper the whole thing. But it does clarify where the theoretical boundaries of the problem are.
You might have thought “well, two machines could co-operate if they had identical source code, but that’s too restrictive to be practical”. But it turns out that you don’t need identical source code if you have the source code and can prove things about it. Then you might have though “ok, but those proofs will never work because of non-termination and self-reference” … and it turns out that that is wrong too.
Theoretical work like this could inform you about what you could hope to achieve if you could solve the practical issues; and conversely what problems are going to come up that you are absolutely going to have to solve.
Will second “Good and Real” as worth reading (haven’t read any of the others).
Maybe translating AI safety literature into Japanese would be a high-value use of your time ?
That’s true, 20 years wouldn’t necessarily bring to light a delayed effect.
However the GMO case is interesting because we have in effect a massive scale natural experiment, where hundreds of millions of people on one continent have eaten lots of GMO food while hundreds of millions on another continent have eaten very little, over a period of 10-15 years. There is also a highly motivated group of people who bring to the public attention even the smallest evidence of harm from GMOs.
While I don’t rule out a harmful long-term effect, GMOs are a long way down on my list of things to worry about, and dropping further over time.
Heh, that was really just me trying to come up with a justification for shoe-horning a theory of identity into a graph formalism so that Konig’s Lemma applied :-)
If I were to try to make a more serious argument it would go something like this.
Defining identity, whether two entities are ‘the same person’ is hard. People have different intuitions. But most people would say that ‘your mind now’ and ‘your mind a few moments later’ are do constitute the same person. So we can define a directed graph with verticies as mind states (mind states would probably have been better than ‘observer moments’) with outgoing edges leading to mind states a few moments later.
That is kind of what I meant by “moment-by-moment” identity. By itself it is a local but not global definition of identity. The transitive closure of that relation gives you a global definition of identity. I haven’t thought about whether its a good one.
In the ordinary course of events these graphs aren’t very interesting, they’re just chains coming to a halt upon death. But if you were to clone a mind-state and put it into two different environments, they that would give you a vertex with out-degree greater than one.
So mind-uploading would not break such a thing, and in fact without being able to clone a mind-state, the whole graph-based model is not very interesting.
Also, you could have two mind states that lead to the same successor mind state—for example where two different mind states only differ on a few memories, which are then forgotten. The possibility of splitting and merging gives you a general (directed) graph structured identity.
(On a side-note, I think generally people treat splitting and merging of mind states in a way that is way too symmetrical. Splitting seems far easier—trivial once you can digitize a mind-state. Merging would be like a complex software version control problem, and you’d need very carefully apply selective amnesia to achieve it.)
So, if we say “immortality” is having an identity graph with an infinite number of mind-states all connected through the “moment-by-moment identity” relation (stay with me here), and mind states only have a finite number of successor states, then there must be at least one infinite path, and therefore “eternal existence in linear time”.
Rather contrived, I know.
If we take “immortality” to mean “infinitely many distinct observer moments that are connect to me through moment-to-moment identity”, then yes, by Konig’s Lemma.
(Every infinite graph with finite-degree verticies has an infinite path)
(edit: hmmm, does many-worlds give you infinite-branching into distinct observer moments ?)
Procedural universes seemed to see a real resurgence from around 2014, with e.g. Elite Dangerous, No Man’s Sky, and a quite a few others that have popped up since.
I love a beautiful procedural world, but I think things will get more interesting when games appear with procedural plot structures that are cohesive and reactive.
Then multiplayer versions will appear that weave all player actions into the plot, and those games will suck people in and never let go.
For 5 minutes suspension versus dreamless deep sleep—almost exactly the same person. For 3 hours dreamless deep sleep I’m not so sure. I think my brain does something to change state while I’m deep asleep, even if I don’t consciously experience or remember anything. Have you ever woken up feeling different about something, or with a solution to a problem you were thinking about as you dropped off ? If that’s not all due to dreaming, then you must be evolving at least slightly while completely unconscious.
Would a slow cell by cell, or thought by thought / byte by byte, transfer of my mind to another medium: one at a time every new neural action potential is received by a parallel processing medium which takes over? I want to say the resulting transfer would be the same consciousness as is typing this but then what if the same slow process were done to make a copy and not a transfer? Once a consciousness is virtual, is every transfer from one medium or location to another not essentially a copy and therefore representing a death of the originating version?
I would follow this line of questioning. For example, say someone does an incremental copy process to you, but the consciousness generated does not know whether or not the original biological consciousness has been destroyed, and has to choose which one to keep. If it chooses the biological one and the biology has been destroyed, bad luck you are definitely gone. What does your consciousness, running either just on the silicon, or identically on the silicon and in the biology, choose ?
Let’s say you are informed that there is 1% chance that the biological version has been destroyed. Well, you’re almost certainly fine then, you keep the biological version, the silicon version is destroyed, and you live happily ever after until you become senile and die.
On the other hand, say you are informed that the biological version has definitely been destroyed. On your current theory, this means that that the consciousness realises that it has been mistaken about its identity, and is actually only a few minutes old. It’s sad that the progenitor person is gone, but it is not suicidal, so it chooses the silicon version.
At what point on the 1% to 100% slider would your consciousness choose the silicon version ?
(Hearing the though-experiment of incremental transfer (or alternatively duplication) was one of the things that changed my mind to pattern-identity from some sort of continuity-identity theory. I remember hearing an interview with Marvin Minsky where he described an incremental transfer on a radio program).
Not sure if it’s a scientific or engineering achievement, but this Nature letter stuck in my mind:
An aqueous, polymer-based redox-flow battery using non-corrosive, safe, and low-cost materials
Oh, I think I see what you mean. No matter how many or how detailed the simulations you run, if your purpose is to learn something from watching them, then ultimately you are limited by your own ability to observe and process what you see.
Whoever is simulating you only has to run the simulations that you launch to the level of fidelity such that you can’t tell if they’ve taken shortcuts. The deeper the nested simulation people are, the harder it is for you to pay attention to them all, and the coarser their simulations can be.
If you are running simulations to answer psychological questions, that should work. And if you are running simulations to answer physics questions … why would you fill them with conscious people ?
Of course, if I were specifically trying to crash the simulation that I was in, I might come up with some physical laws that would eat up a lot of processing power to calculate for even one person’s local space, but between the limitations of computing as they exist in the base simulation, the difficulty in confirming that these laws have been properly executed in all of their fully-complex glory
I was going to say that if you want to be a pain you could launch some NP hard problems that you can manually verify solutions to with a pencil and paper … except your simulators control your random-number generators.
I was thinking more like a random power surge, programming error,or political coup within our simulation that happened to shut down the aspect of our program that was hogging resources. If the programmers want the program to continue, it can.
You’re right—branch (2) should be “we don’t keep running run more than one”. We can launch as many as we like.
The single actor is not going to experience every aspect of the simulation in full fidelity, so a low-res simulation is all that is needed. (The actor might think that it is a full simulation, and may have correctly programmed a full simulation, but there is simply no reason for it to actually replicate either the whole universe or the whole actor, as long as it gives output that looks valid).
That would buy you some time. If a single-agent simulation is say 10^60 times cheaper than a whole universe (roughly the number of elementary particles in the observable universe ?), then that gives you about 200 doubling generations before those single-agent simulations cost as much as much as a universe.
Unless the space of all practically different possible lives of the agent is actually much smaller … maybe your choices don’t matter that much and you end up playing out a relatively small number or attractor scripts. You might be able to map out that space efficiently with some clever dynamic programming.
That’s the unbounded computation case.
Very nice. This is the cleanest result on cognitive (or rationality) costs in co-operative systems that I’ve seen. Modal combat seems kind of esoteric compared to, say, iterated prisoners’ dilemma tournaments with memory, but it pays off nicely here. It gives you the outcomes of a set of other-modelling agents (without e.g. doing a whole lot of simulation), and the box-operator depth then plugs in as a natural modelling-cost measure.
Did you ever publish any of your modal combat code (I have a vague recollection that you had some Haskell code ?) ?