Regarding your comments on SPECKS preferable to TORTURE, I think that misses the argument they made. The reason you have to prefer 10N at X to N at X’ at some point, is that a speck counts as a level of torture. That’s exactly what OP was arguing against.
Andrew Jacob Sauer
Karma: 60
Seems to me that if an agent with a reasonable heuristic for logical uncertainty came upon this problem, and was confident but not certain of its consistency, it would simply cross because expected utility would be above zero, which is a reason that doesn’t betray an inconsistency. (Besides, if it survived it would have good 3rd party validation of its own consistency, which would probably be pretty useful.)
That’s the funniest thing I’ve seen all day.
The Riemann argument seems to differ from the Great Filter argument in this way: the Riemann argument depends only on the sheer number of observers, i.e. the only thing you’re taking into account is the fact that you exist. Whereas in the great filter argument you’re updating based on what kind of observer you are, i.e. you’re intelligent but not a space-travelling, uploaded posthuman.
The first kind of argument doesn’t work because somebody exists either way: if the RH or whatever is false then you are one of a small number, if it’s true than you are one of a large number, you are in a typical position either way, and the other situation simply isn’t possible. But the second kind of argument seems to hold more merit: if the great filter is behind then you are part of the extreme minority of normal humans, but if the great filter is ahead then you are rather typical of intelligent lifeforms. This might count as evidence, and it seems to be the same kind of evidence which suggests that a great filter even exists in the first place: if it doesn’t then we are very exceptional not only in being the very first humans but the very first intelligent life as well.
The proof doesn’t work on a logically uncertain agent. The logic fails here:
Examining the source code of the agent, because we’re assuming the agent crosses, either PA proved that crossing implies U=+10, or it proved that crossing implies U=0.
A logically uncertain agent does not need a proof of either of those things in order to cross, it simply needs a positive expectation of utility, for example a heuristic which says that there’s a 99% chance crossing implies U=+10.
Though you did say there’s a version which still works for logical induction. Do you have a link to where I can see that version of the argument?
Edit: Now I still see the logic. On the assumption that the agent crosses but also proves that U=-10, the agent must have a contradiction somewhere, because that, and the logical uncertainty agents I’m aware of have a contradiction upon proving U=-10 because they prove that they will not cross, and then immediately cross in a maneuver meant to prevent exactly this kind of problem.
Wait but proving crossing implies U=-10 does not mean that prove they will not cross, exactly because they might still cross, if they have a contradiction.
God this stuff is confusing. I still don’t think the logic holds though.
Because assuming Provable(C)->C as a hypothesis doesn’t allow you to prove C. Rather, the fact that a proof exists of Provable(C)->C allows you to construct a proof of C.
Rot13:
Gur vzcnpg bs na rirag ba lbh vf gur qvssrerapr orgjrra gur rkcrpgrq inyhr bs lbhe hgvyvgl shapgvba tvira pregnvagl gung gur rirag jvyy unccra, naq gur pheerag rkcrpgrq inyhr bs lbhe hgvyvgl shapgvba.
Zber sbeznyyl, jr fnl gung gur rkcrpgrq inyhr bs lbhe hgvyvgl shapgvba vf gur fhz, bire nyy cbffvoyr jbeyqfgngrf K, bs C(K)*H(K), juvyr gur rkcrpgrq inyhr bs lbhe hgvyvgl shapgvba tvira pregnvagl gung n fgngrzrag R nobhg gur jbeyq vf gehr vf gur fhz bire nyy cbffvoyr jbeyqfgngrf K bs C(K|R)*H(K). Gur vzcnpg bs R orvat gehr, gura, vf gur nofbyhgr inyhr bs gur qvssrerapr bs gubfr gjb dhnagvgvrf.
We’re talking about the impact of an event though. The very question is only asking about worlds where the event actually happens.
If I don’t know whether an event is going to happen and I want to know the impact it will have on me, I compare futures where the event happens to my current idea of the future, based on observation(which also includes some probability mass for the event in question, but not certainty).
In summary, I’m not updating to “X happened with certainty” rather I am estimating the utility in that counterfactual case.
I think that fully specifying human values may not be the best approach to an AI utopia. Rather, I think it would be easier and safer to tell the AI to upload humans and run an Archipelago-esque simulated society in which humans are free to construct and search for the society they want, free from many practical problems in the world today such as resource scarcity.
This is one of those things that seems obvious but it did cause some things to click for me that I hadn’t thought of before. Previously my idea of AGI becoming uncontrollable was basically that somebody would make a superintelligent AGI in a box, and we would be able to unplug it anytime we wanted, and the real danger would be the AGI tricking us into not unplugging it and letting it out of the box instead. What changed this view was this line: “Try to unplug Bitcoin.” Once you think of it that way it does seem pretty obvious that the most powerful algorithms, the ones that would likely first become superintelligent, would be distributed and fault-tolerant, as you say, and therefore would not be in a box of any kind to begin with.
Funny you mention AlphaGo, since the first time AlphaGo(or indeed any computer) beat a professional go player(Fan Hui), it was distributed across multiple computers. Only later did it become strong enough to beat top players with only a single computer.
But, over the lifetime of civilization, our accumulated experience led us to update this prior, and single out the complexity measure suggested by math.
I may be picking nits, here, but what exactly does it mean to “update a prior”?
And as a mathematical consideration, is it in general possible to switch your probabilities from one (limit computable) universal prior to another with a finite amount of evidence?
I think an important consideration is the degree of catastrophe. Even the asteroid strike, which is catastrophic to many agents on many metrics, is not catastrophic on every metric, not even every metric humans actually care about. An easy example of this is prevention of torture, which the asteroid impact accomplishes quite smoothly, along with almost every other negative goal. The asteroid strike is still very bad for most agents affected, but it could be much, much worse, as with the “evil” utility function you alluded to, which is very bad for humans on every metric, not just positive ones. Calling both of these things a “catastrophe” seems to sweep that difference under the rug.
With this in mind, “catastrophe” as defined here seems to be less about negative impact on utility, and more about wresting of control of utility function away from humans. Which seems bound to happen even in the best case where a FAI takes over. It seems a useful concept if that is what you are getting at but “catastrophe” seems to have confusing connotations, as if a “catastrophe” is necessarily the worst thing possible and should be avoided at all costs. If an antialigned “evil” AI were about to be released with high probability, and you had a paperclip maximizer in a box, releasing the paperclip maximizer would be the best option, even though that moves the chance of catastrophe from high probability to indistinguishable from certainty.
When “pure thought” tells you that 1 + 1 = 2, “independently of any experience or observation”, you are, in effect, observing your own brain as evidence.
I mean, yeah? You can still do that in your armchair, without looking at anything outside of yourself. Mathematical facts are indeed “discoverable by the mere operation of thought, without dependence on what is anywhere existent in the universe,” if you modify the statement a little to say “anywhere else existent” in order to acknowledge that the operation of thought indeed exists in the universe. Do mathematical facts exist independently of the universe? Maybe, maybe not, it probably depends what you mean by “exist” and it doesn’t really matter to anyone since either way, you can’t discover any mathematical facts without using your brain, which is in the universe. So there’s no observable difference between whether Platonic math exists or not.
“free will” is a useful concept which should be kept, even though it has been used to refer to nonsensical things. Just because one can’t will what he wills, doesn’t mean we shouldn’t be able to talk about willing what you do. Similarly, just because you can’t get knowledge without thinking, doesn’t mean we shouldn’t be able to use “a priori knowledge” to talk about getting knowledge without looking.
Seems to me that before a philosophical problem is solved, it becomes a problem in some other field of study. Atomism used to be a philosophical theory. Now that we know how to objectively confirm it, it (or rather, something similar but more accurate) is a scientific theory.
It seems that philosophy (at least, the parts of philosophy that are actively trying to progress) is about trying to take concepts that we have intuitive notions of, and figure out what if anything those concepts actually refer to, until we succeed at this well enough that to study then in more precise ways than, well, philosophy.
So, how many examples can we find where some vague but important-seeming idea has been philosophically studied until we learn what the idea refers to in concrete reality, and how to observe and measure it to some degree?
It’s hard to tell, since while common sense is sometimes wrong, it’s right more often than not. An idea being common sense shouldn’t count against it, even though like the article said it’s not conclusive.
Suppose you learn about physics and find that you are a robot. You learn that your source code is “A”. You also believe that you have free will; in particular, you may decide to take either action X or action Y.
My motivation for talking about logical counterfactuals has little to do with free will, even if the philosophical analysis of logical counterfactuals does.
The reason I want to talk about logical counterfactuals is as follows: suppose as above that I learn that I am a robot, and that my source code is “A”(which is presumed to be deterministic in this scenario), and that I have a decision to make between action X and action Y. In order to make that decision, I want to know which decision has better expected utility. The problem is that, in fact, I will either choose X or Y. Suppose without loss of generality that I will end up choosing action X. Then worlds in which I choose Y are logically incoherent, so how am I supposed to reason about the expected utility of choosing Y?
From an omniscient point of view, yes. From my point of view, probably not, but there are still problems that arise relating to this, that can cause logic-based agents to get very confused.
Let A be an agent, considering options X and not-X. Suppose A |- Action=not-X → Utility=0. The naive approach to this would be to say: if A |- Action=X → Utility<0, A will do not-X, and if A |- Action=X → Utility>0, A will do X. Suppose further that A knows its source code, so it knows this is the case.
Consider the statement G=(A |- G) → (Action=X → Utility<0). It can be constructed by using Godel-numbering and quines. Present A with the following argument:
Suppose for the sake of argument that A |- G. Then A |- (A |- G), since A knows its source code. Also, by definition of G, A |- (A |- G) → (Action=X → Utility<0). By modus ponens, A |- (Action=X → Utility<0). Therefore, by our assumption about A, A will do not-X: Action!=X. But, vacuously, this means that (Action=X → Utility<0). Since we have proved this by assuming A |- G, we know that (A |- G) → (Action=X → Utility<0), in other words, we know G.
The argument then goes, similarly to above:
A |- G
A |- (A |- G)
A |- (A |- G) → (Action=X → Utility<0)
A |- (Action=X → Utility<0)
Action=Not-X
We proved this without knowing anything about X. This shows that naive logical implication can easily lead one astray. The standard solution to this problem is the chicken rule, making it so that if A ever proves which action it will take, it will immediately take the opposite action, which avoids the argument presented above, but is defeated by Troll Bridge, even when the agent has good logical uncertainty.
These problems seem to me to show that logical uncertainty about the action one will take, paired with logical implications about what the result will be if you take a particular action, are insufficient to describe a good decision theory.
What do you mean by that?
Non-Archimedean utility functions seem kind of useless to me. Since no action is going to avoid moving the probability of any outcome by more than 1/3^^^3, absolutely any action is important only insomuch as it impacts the highest lexical level of utility. So you might as well just call that your utility function.