justify our reasoning
It seems more practical to develop reasoning systems that work well enough to explain what the confusing systems that were never designed happen to be doing. The useful sense of justification is not about describing what’s already happening, but about choosing well, making use of what you have available. There is no point in figuring out if what you have available is well-designed, apart from an act of redesign.
Consider the distinction between a low level detailed simulation of a world where you are making a decision, and high level reasoning about your decision making. How would you know which one is being applied to you, from within? If there is a way of knowing that, you can act differently in these scenarios, so that the low level simulation won’t show the same outcome as the prediction made with high level reasoning. A good process of making predictions by high level reasoning won’t allow there to be a difference.
The counterfactual world I’m talking about does not have to exist in any way similar to the real world, such as by being explicitly simulated. It only needs the implied existence of worldbuilding of a fictional story. The difference from a fictional story is that the story is not arbitrary, there is a precise purpose that shapes the story needed for prediction. And for a fictional character, there is no straightforward way of noticing the fictional nature of the world.
Therefore, the world where the events take place is not real, it’s a figment of past-Omega’s imagination, predicting what you’d do. Your actions in the counterfactual just convinced past-Omega not to put the $1M in the box in the real world. Usually you don’t have a way of knowing whether the world is real, or what relative weight of reality it holds. Though in this case the reality of the world was under your control.
size of the number (representing the amount of value) that can be physically encoded, which is exponential in the number of atoms
The natural numbers that can be physically encoded are not bounded by an exponent of the number of bits if you don’t have to be able to encode all smaller numbers as well in the same number of bits. If you define a number, you’ve encoded it, and it’s possible to define very large numbers indeed.
Generalizing cousin_it’s idea about ems to not-overly-superintelligent AGIs, it’s more valuable to run fewer AGIs faster than to run a larger number of AGIs at human speeds. This way AGI-as-a-service might remain in the realm of giant cheesecakes even assuming very slow takeoff.
The prediction is why you grab your coat, it’s both meaningful and useful to you, a simple counterexample to the sentiment that since correctness scope of predictions is unclear, they are no good. The prediction is not about the coat, but that dependence wasn’t mentioned in the arguments against usefulness of predictions above.
IMO, either the Oracle is wrong, or the choice is illusory
This is similar to determinism vs. free will, and suggests the following example. The Oracle proclaims: “The world will follow the laws of physics!”. But in the counterfactual where an agent takes a decision that won’t actually be taken, the fact of taking that counterfactual decision contradicts the agent’s cognition following the laws of physics. Yet we want to think about the world within the counterfactual as if the laws of physics are followed.
No, I suspect it’s a correct ingredient of counterfactuals, one I didn’t see discussed before, not an error restricted to a particular decision theory. There is no contradiction in considering each of the counterfactuals as having a given possible decision made by the agent and satisfying the Oracle’s prediction, as the agent doesn’t know that it won’t make this exact decision. And if it does make this exact decision, the prediction is going to be correct, just like the possible decision indexing the counterfactual is going to be the decision actually taken. Most decision theories allow explicitly considering different possible decisions, and adding correctness of the Oracle’s prediction into the mix doesn’t seem fundamentally different in any way, it’s similarly sketchy.
Specifically, when you say “It’s only guaranteed to be correct on the actual decision”, why does the agent not know what “correct” means for the decision?
The agent knows what “correct” means, correctness of a claim is defined for the possible worlds that the agent is considering while making its decision (which by local tradition we confusingly collectively call “counterfactuals”, even though one of them is generated by the actual decision and isn’t contrary to any fact).
In the post Chris_Leong draws attention to the point that since the Oracle knows which possible world is actual, there is nothing forcing its prediction to be correct on the other possible worlds that the agent foolishly considers, not knowing that they are contrary to fact. And my point in this thread is that despite the uncertainty it seems like we have to magically stipulate correctness of the Oracle on all possible worlds in the same way that we already magically stipulate the possibility of making different decisions in different possible worlds, and this analogy might cast some light on the nature of this magic.
Is that universal AI drive wrong, then?
Damage to AI’s implementation makes the abstractions of its design leak. If somehow without the damage it was clear that a certain part of it describes goals, with the damage it’s no longer clear. If without the damage, the AI was a consequentialist agent, with the damage it may behave in non-agentic ways. By repairing the damage, the AI may recover its design and restore a part that clearly describes its goals, which might or might not coincide with the goals before the damage took place.
No, this is an important point: the agent normally doesn’t know the correctness scope of the Oracle’s prediction. It’s only guaranteed to be correct on the actual decision, and can be incorrect in all other counterfactuals. So if the agent knows the boundaries of the correctness scope, they may play chicken and render the Oracle wrong by enacting the counterfactual where the prediction is false. And if the agent doesn’t know the boundaries of the prediction’s correctness, how are they to make use of it in evaluating counterfactuals?
It seems that the way to reason about this is to stipulate correctness of the prediction in all counterfactuals, even though it’s not necessarily correct in all counterfactuals, in the same way as the agent’s decision that is being considered is stipulated to be different in different counterfactuals, even though the algorithm forces it to be the same. So it’s a good generalization of the problem of formulating counterfactuals, it moves the intervention point from agent’s own decisions to correctness of powerful predictors’ claims. These claims act on the counterfactuals generated by the agent’s own decisions, not on the counterfactuals generated by delivery of possible claims, so it’s not about merely treating predictors as agents, it’s a novel setup.
I don’t think that contradicts my original statement strongly.
I don’t think it contradicts it at all, it’s unrelated to your original statement, only to the use of a word in it that can be steelmanned away in the obvious manner.
Some hypotheses are not beliefs (they are beliefs-in-development that aren’t yet ready for making predictions), and many constructions are not even hypotheses in this sense (they are not about the real world). I don’t believe there is a unifying concept behind the things people talk about, different concepts are salient for different statements.
When the Oracle says “The taxi will arrive in one minute!”, you may as well grab your coat.
confirms their worldview
A lot of things people talk about are not at all about “their worldview” in the sense of beliefs and values, this characterization is often enough noncentral. I’m arguing use of words in this comment, is that an element of my worldview? Perhaps I value accurate use of words, and can’t find a suitable counterexample.
Problems with software that systematically trigger hardware failure and software bugs causing data corruption can be mitigated with hardening techniques, things like building software with randomized low-level choices, more checks, canaries, etc. Random hardware failure can be fixed with redundancy, and multiple differently-randomized builds of software can be used to error-correct for data corruption bugs sensitive to low-level building choices. This is not science fiction, just not worth setting up in most situations. If the AI doesn’t go crazy immediately, it might introduce some of these things if they were not already there, as well as proofread, test, and formally verify all code, so the chance of such low-level failures goes further down. And these are just the things that can be done without rewriting the code entirely (including toolchains, OS, microcode, hardware, etc.), which should help even more.
Meta: At the time of writing this comment, before I strong-upvote the question, it sits at −3, with two non-author votes. This is not very welcoming (this is the author’s first post/comment here). The only thing wrong with the question seems to be that the author isn’t already aware of the standard way of pointing at the same cluster of issues (which is exactly the situation where asking a question should help, and it probably already did). If this is not OK, this is a disincentive for new people from getting publicly curious.
Edit: And just a few minutes after this, someone strong-downvoted again, back to 0. Possibly a strong dislike for violence in thought experiments? It does seem to be going out of style.
Such things rarely happen on their own, a natural bug would most likely crash the whole system or break something unimportant. Given that even a broken AI has an incentive to fix bugs in its cognition, it most likely has plenty of opportunity to succeed in that. It’s only if the AI wanted to hack itself that it would become a plausible problem, and my point is that it doesn’t want that, instead it wants to prevent even unlikely problems from causing trouble.
It’s not actually more general, it’s instead about a somewhat different point. The more general statement could use some sort of a notion of relative actuality, to point at the possibly counterfactual world determined by the decision made in the world where the prediction was delivered, which is distinct from the even more counterfactual worlds where the prediction was delivered but the decision was different from what it would relative-actually be had the prediction been delivered, and from the worlds where the prediction was not delivered at all.
If the prediction is not actually delivered, then it only applies to that intermediately-counterfactual world and not to the more counterfactual alternatives where the prediction was still delivered or to the less counterfactual situation where the prediction is not delivered. Saying that the prediction applies to the world where it’s delivered is liable to be interpreted as including the more-counterfactual worlds, but it doesn’t have to apply there, it only applies to the relatively-actual world. So your original framing has a necessary part of saying this carefully that my framing didn’t include, replacing it with my framing discards this correct detail. The Oracle’s prediction only has to apply to the “relatively-actual” world where the prediction is delivered.
The configuration does change, it’s just that the change is not about the amount of matter. If there are configurations absurdly more positively or negatively valuable than others, that just makes the ordinary configurations stop being decision relevant, once discerning the important configurations becomes feasible.