That’s a good point. I probably wouldn’t want to give up my thread of subjective experience either. But unless I switch (or someone comes up with a better solution than UDT), when mind copying/merging becomes possible I’m probably going to start making some crazy decisions.
I’m not sure what the solution is, but here’s one idea. An UDT agent doesn’t use anticipation or continuity of experience to make decisions, but perhaps it can run some computations on the side to generate the qualia of anticipation and continuity.
Another idea, which may be more intuitively acceptable, is don’t make the switch yourself. Create a copy, and have the copy switch to UDT (before it starts running). Then give most of your resources to the copy and live a single-threaded life under its protection. (I guess the copy in this case isn’t so much a copy but more of a personal FAI.)
That’s what I was thinking, too. You make tools the best way you can. The distinction between tools that are or aren’t part of you will ultimately become meaningless anyway. We’re going to populate the galaxy with huge Jupiter brains that are incredibly smart and powerful but whose only supergoal is to protect a tiny human-nugget inside.
That’s a good point. I probably wouldn’t want to give up my thread of subjective experience either. But unless I switch (or someone comes up with a better solution than UDT), when mind copying/merging becomes possible I’m probably going to start making some crazy decisions.
I’m not sure what the solution is, but here’s one idea. An UDT agent doesn’t use anticipation or continuity of experience to make decisions, but perhaps it can run some computations on the side to generate the qualia of anticipation and continuity.
Another idea, which may be more intuitively acceptable, is don’t make the switch yourself. Create a copy, and have the copy switch to UDT (before it starts running). Then give most of your resources to the copy and live a single-threaded life under its protection. (I guess the copy in this case isn’t so much a copy but more of a personal FAI.)
That’s what I was thinking, too. You make tools the best way you can. The distinction between tools that are or aren’t part of you will ultimately become meaningless anyway. We’re going to populate the galaxy with huge Jupiter brains that are incredibly smart and powerful but whose only supergoal is to protect a tiny human-nugget inside.