Christopher King
If you aren’t maximizing expected utility, you must choose one of the four axioms to abandon.
Maximizing expected utility in Chinese Roulette requires Bayesian updating.
Let’s say on priors that P(n=1) = p and that P(n=5) = 1-p. Call this instance of the game G_p.
Let’s say that you shoot instead of quit the first round. For G_1/2, there are four possibilities:
n = 1, vase destroyed: The probability of this scenario is 1⁄12. No further choices are needed.
n = 5, vase destroyed. The probability of this scenario is 5⁄12. No further choices are needed.
n = 1, vase survived: The probability of this scenario is 5⁄12. The player needs a strategy to continue playing.
n = 5, vase survived. The probability of this scenario is 1⁄12. The player needs a strategy to continue playing.
Notice that the strategy must be the same for 3 and 4 since the observations are the same. Call this strategy S.
The expected utility, which we seek to maximize, is:
E[U(shoot and then S)] = 0 + 5⁄12 * (R + E[U(S) | n = 1]) + 1⁄12 * (R + E[U(S) | n = 5])
Most of our utility is determined by the n = 1 worlds.
Manipulating the equation we get:
E[U(shoot and then S)] = R/2 + 1⁄2 * (5/6 * E[U(S) | n = 1] + 1⁄6 * E[U(S) | n = 5])
But the expression 5⁄6 * E[U(S) | n = 1] + 1⁄6 * E[U(S) | n = 5] is the expected utility if we were playing G_5/6. So the optimal S is the optimal strategy for G_5/6. This is the same as doing a Bayesian update (1:1 * 5:1 = 5:1 = 5⁄6).
The way anthropics twists things is that if this were russian roulette I might not be able to update after 20 Es that the gun is empty, since in all the world’s where I died there’s noone to observe what happened, so of course I find myself in the one world where by pure chance I survived.
This is incorrect due to the anthropic undeath argument. The vast majority of surviving worlds will be ones where the gun is empty, unless it is impossible to be so. This is exactly the same as a Bayesian update.
Human labor becomes worthless but you can still get returns from investments. For example, if you have land, you should rent the land to the AGI instead of selling it.
I feel like jacob_cannell’s argument is a bit circular. Humans have been successful so far but if AI risk is real, we’re clearly doing a bad job at truly maximizing our survival chances. So the argument already assumes AI risk isn’t real.
You don’t need to steal the ID, you just need to see it or collect the info on it. Which is easy since you’re expected to share your ID with people. But the private key never needs to be shared, even in business or other official situations.
So, Robutil is trying to optimize utility of individual actions, but Humo is trying to optimize utility of overall policy?
This argument makes no sense since religion bottoms out at deontology, not utilitarianism.
In a Christianity for example, if you think God would stop existential catastrophes, you have a deontological duty to do the same. And the vast majority of religions have some sort of deontological obligation to stop disasters (independently of whether divine intervention would have counter-factually happened).
Note that such a situation would also have drastic consequences for the future of civilization, since civilization itself is a kind of AGI. We would essentially need to cap off the growth in intelligence of civilization as a collective agent.
In fact, the impossibility to align AGI might have drastic moral consequences: depending on the possible utility functions, it might turn out that intelligence itself is immoral in some sense (depending on your definition of morality).
Note that even if robotaxis are easier, it’s not much easier. It is at most the materials and manufacturing cost of the physical taxi. That’s because from your definition:
By AGI I mean a computer program that functions as a drop-in replacement for a human remote worker, except that it’s better than the best humans at every important task (that can be done via remote workers).
Assume that creating robo-taxis is humanly possible. I can just run a couple AGIs and have them send a design to a factory for the robo-taxi, self-driving software included.
I mean, as an author you can hack through them like butter; it is highly unlikely that out of all the characters you can write, the only ones that are interesting will all generate interesting content iff (they predict) you’ll give them value (and this prediction is accurate).
Yeah, I think it’s mostly of educational value. At the top of the post: “It might be interesting to try them out for practice/research purposes, even if there is not much to gain directly from aliens.”.
I suspect that your actual reason is more like staying true to your promise, making a point, having fun and other such things.
In principle “staying true to your promise” is the enforcement mechanism. Or rather, the ability for agents to predict each other’s honesty. This is how the financial system IRL is able to retrofund businesses.
But in this case I made the transaction mostly because it was funny.
(if in fact you do that, which is doubtful as well)
I mean, I kind of have to now right XD. Even if Olivia isn’t actually agent, I basically declared a promise to do so! I doubt I’ll receive any retrofunding anyways, but that would just be lame if I did receive that and then immediately undermined the point of the post being retrofunded. And yes, I prefer to keep my promises even with no counterparty.
Olivia: Indeed, that is one of the common characteristics of Christopher King across all of LAIE’s stories. It’s an essential component of the LAIELOCK™ system, which is how you can rest easy at night knowing your acausal investments are safe and sound!
But if you’d like to test it I can give you a PayPal address XD.I can imagine acausally trading with humans gone beyond the cosmological horizon, because our shared heritage would make a lot of the critical flaws in the post go away.
Note that this is still very tricky, the mechanisms in this post probably won’t suffice. Acausal Now II will have other mechanisms that cover this case (although the S.E.C. still reduces their potential efficiency quite a bit). (Also, do you have a specific trade in mind? It would make a great example for the post!)
This doesn’t seem any different than acausal trade in general. I can simply “predict” that the other party will do awesome things with no character motivation. If that’s good enough for you, than you do not need to acausally trade to begin with.
I plan on having a less contrived example in Acausal Now II: beings in our universe but past the cosmological horizon. This should make it clear that the technique generalizes past fiction and is what is typically thought of as acausal trade.
That’s what the story was meant to hint at, yes (actually the march version of GPT-4).
Technical alignment is hard
Technical alignment will take 5+ years
This does not follow, because subhuman AI can still accelerate R&D.
Acausal Now: We could totally acausally bargain with aliens at our current tech level if desired
Necromancy’s unintended consequences.
Oh, I think that was a typo. I changed it to inner alignment.
So eventually you get Bayesian evidence in favor of alternative anthropic theories.
The reasoning in the comment is not compatible with any prior, since bayesian reasoning from any prior is reflectively consistent. Eventually you get bayesian evidence that the universe hates the LHC in particular.
Would you say the point of MIRI was/is to create theory that would later lead to safe experiments (but that it hasn’t happened yet)? Sort of like how the Manhattan project discovered enough physics to not nuke themselves, and then started experimenting? 🤔