dankane

Karma: 238

dankane 30 Sep 2017 7:44 UTC
0 points
on: A taxonomy of Oracle AIs
I feel like your discussion of predictors makes a few not-necessarily-warranted assumptions about how the predictor deals with self-reference. Then again, I guess anything that doesn’t do this fails as a predictor in a wide range of useful cases. It predicts a massive fire will kill 100 people, and so naturally this prediction is used to invalidate the original prediction.

But there is a simple-ish fix. What if you simply ask it to make predictions about what would happen if it (and say all similar predictors) suddenly stopped functioning immediately before this prediction was returned?

dankane 14 Sep 2017 16:49 UTC
0 points
in reply to: a gently pricked vein’s comment on: Markets are Anti-Inductive
Unless you can explain to me how prediction markets are going to break the pattern that two different shares of the same stock have correlated prices.

I’m actually not sure how prediction markets are supposed to have an effect on this issue. My issue is not that people have too much difficulty recognizing patterns. My issue is that some patterns once recognized do not provide incentives to make that pattern disappear. Unless you can tell me how prediction markets might fix this problem, your response seems like a bit of a non-sequitur.

dankane 6 Sep 2017 1:16 UTC
1 point
on: Markets are Anti-Inductive
This seems like too general a principle. I agree that in many circumstances, public knowledge of a pattern in pricing will lead to effects causing that pattern to disappear. However, it is not clear to me that this is always to case, or that the size of the effect will be sufficient to complete cancel out the original observation.

For example, I observe that two different units of Google stock have prices that are highly correlated with each other. I doubt that this observation will cause separate markets to spring up giving wildly divergent prices to different shares of the same stock. I also note that stock prices are always non-negative. I also doubt that this will cease to be the case any time soon.

Although these are somewhat tautological, one can imagine non-tautological observations that will not disappear. If stocks A and B are known to be highly correlated, this may well lead to a larger gap as hedge funds who predict a small difference in expected returns will buy one and short the other. However, if they are correlated for structural reasons part of this might be that it is hard to detect effects that will cause their prices to diverge significantly, so the observation of the effect will likely not be enough to actually remove all of the correlation.

One can also imagine general observations about the market itself, like the approximate frequency of crashes, or log normality of price changes that might not disappear simply because they are known. In order for an effect to disappear there needs to be a way to make a profit off of it.

dankane 23 May 2016 7:49 UTC
1 point
in reply to: kboon’s comment on: The Strangest Thing An AI Could Tell You

We probably couldn’t even talk ourselves out of this box.

I don’t know… That sounds a lot like what an AI trying to talk itself out of a box would say.

dankane 19 Nov 2015 8:59 UTC
3 points
on: The Mystery of the Haunted Rationalist
Hmm… I would probably explain the threshold for staying in the house not as an implicit expected probability computation, but an evaluation of the price of the discomfort associated with staying in a location that you find spooky. At least for me, I think that the part of my mind that knows that ghosts do not exist would have no trouble controlling whether or not I remain in the house or not. However, it might well decide that it is not worth the $10 that I would receive to spend the entire night in a place where some other piece of my mind is constantly yelling at me to run away screaming.

dankane 14 Jul 2015 16:55 UTC
4 points
in reply to: Sniffnoy’s comment on: An overall schema for the friendly AI problems: self-referential convergence criteria

It’s just that such self-referential criteria as reflective equilibrium are a necessary condition

Why? The only example of adequately friendly intelligent systems that we have (i.e. us) don’t meet this condition. Why should reflective equilibrium be a necessary condition for FAI?

dankane 8 Jun 2015 6:20 UTC
2 points
in reply to: drethelin’s comment on: Taking Effective Altruism Seriously
That may be true (at least to the degree to which it is sensible to assign a specific cause to a given util). However, it is not very good evidence that investment in first world economies is the most effective way to generate utils in Africa.

dankane 5 Jun 2015 23:24 UTC
3 points
on: Taking Effective Altruism Seriously
OK. So suppose that I grant your claim that donations to sub-Saharan Africa will not substantially affect the size of the future economic pie, but that other investments will. I claim that there may still be reason to donate there.

I grant that such a donation will produce fewer dollars of value than investing in capitol infrastructure. On the other hand dollars is not the objective, utils are. We can reasonably assume that marginal utility of an extra dollar for a given person is decreasing as that person’s wealth increases. We can reasonably expect that world GDP per capita will be much higher in 100 years, and know that GDP per capita is much higher in the US than in sub-Saharan Africa. Thus, even if an investment in first-world infrastructure produces more total dollars of value, these dollars are going to much wealthier people than dollars donated to people today in sub-Saharan Africa, and thus might well produce fewer total utils.

dankane 4 Apr 2015 6:40 UTC
0 points
on: The Truly Iterated Prisoner’s Dilemma
[I realize that I missed the train and probably very few people will read this, but here goes]

So in non-iterated prisoner’s dilemma, defect is a dominant strategy. No matter what the opponent is doing, defecting will always give you the best possible outcome. In iterated prisoner’s dilemma, there is no longer a dominant strategy. If my opponent is playing Tit-for-Tat, I get the best outcome by cooperating in all rounds but the last. If my opponent ignores what I do, I get the best outcome by always defecting. It is true that all defects is the unique Nash equilibrium strategy, but this is a much weaker reason for playing it, especially given that evidence shows that when playing among people who are trying to win, Tit-for-Tat tends to achieve much better outcomes.

There seems to be a lot of discussion in the comments about this or that being the rational thing to do, and I think that this is a big problem that gets in the way of clear thinking about the issue. The problem is that people are using the word “rational” here without having a clear idea as to what exactly that means. Sure, it’s the thing that wins, but wins when? Provably, there is no single strategy that achieves the best possible outcome against all possible implementations of Clippy. So what do you mean? Are you trying to optimize your expected utility under a Kolmogorov prior? If so how come nobody seems to be trying to do computations of the posterior distribution? Or discussing exactly what side data we know about the issue that might inform this probability computation? Or even wondering which universal Turing machine we are using to define our prior? Unless you want to give a more concrete definition of what you mean by “rational” in this context, perhaps you should stop arguing for a moment about what the rational thing to do is.

dankane 26 Mar 2015 18:46 UTC
0 points
in reply to: orthonormal’s comment on: An Introduction to Löb’s Theorem in MIRI Research
I think that the way that humans predict other humans is the wrong way to look at this, and instead consider how humans would reason about the behavior of an AI that they build. I’m not proposing simply “don’t use formal systems”, or even “don’t limit yourself exclusively to a single formal system”. I am actually alluding to a far more specific procedure:
- Come up with a small set of basic assumptions (axioms)
- Convince yourself that these assumptions accurately describe the system at hand
- Try to prove that the axioms would imply the desired behavior
- If you cannot do this return for the first step and see if additional assumptions are necessary
Now it turns out that for almost any mathematical problem that we are actually interested in, ZFC is going to be a sufficient set of assumptions, so the first few steps here are somewhat invisible, but they are still there. Somebody need need to come up with these axioms for the first time, and each individual who wants to use them should convince themselves that they are reasonable before relying on them.

A good AI should already do this to some degree. It needs to come up with models of a system that it is interacting with before determining its course of action. It is obvious that it might need to update what assumptions it’s using the model physical laws, why shouldn’t it just do the same thing for logical ones?

dankane 26 Mar 2015 5:16 UTC
0 points
in reply to: hairyfigment’s comment on: An Introduction to Löb’s Theorem in MIRI Research
Yes, obviously. We solve the Lobstacle by not ourselves running on formal systems and sometimes accepting axioms that we were not born with (things like PA). Allowing the AI to only do things that it can prove will have good consequences using a specific formal system would make it dumber than us.

dankane 26 Mar 2015 4:46 UTC
4 points
on: An Introduction to Löb’s Theorem in MIRI Research
Actually, why is it that when the Lobian obstacle is discussed that it seem to always be in reference to an AI trying to determine if a successor AI is safe, and not an AI trying to determine whether or not it, itself, is safe?

dankane 26 Mar 2015 4:11 UTC
2 points
on: An Introduction to Löb’s Theorem in MIRI Research
Question: If we do manage to build a strong AI, why not just let it figure this problem out on its own when trying to construct a successor? Almost definitionally, it will do a better job of it than we will.

dankane 25 Sep 2014 17:59 UTC
2 points
in reply to: dankane’s comment on: Newcomblike problems are the norm
Relatedly, with your interview example, I think that perhaps a better model is that whether a person is confident or shy is not depending on whether they believe that they will be bold or not, but upon the degree to which they care about being laughed at. If you are confident, you don’t care about being laughed at and might as well be bold. If you are afraid of being laughed at, you already know that you are shy and thus do not gain anything by being bold.

dankane 25 Sep 2014 17:44 UTC
2 points
in reply to: dankane’s comment on: Newcomblike problems are the norm
I think my bigger point is that you don’t seem to make any real argument as to which case we are in. For example, consider the following model of how people’s perception of my trustworthiness might be correlated to my actual trustworthiness: There are two causal chains: My values → Things I say → Peoples’ perceptions My values → My actions So if I value trustworthiness, I will not, for example talk much about wanting to avoid being sucker (in contexts where it would refer to be doing trustworthy things). This will influence peoples’ perceptions of whether or not I am trustworthy. Furthermore, if I do value trustworthiness, I will want to be trustworthy.

This setup makes things look very much like the smoking lesion problem. A CDT agent that values trustworthiness will be trustworthy because they place intrinsic value in it. A CDT agent that does not value trustworthiness will be perceived as being untrustworthy. Simply changing their actions will not alter this perception, and therefore they will fail to be trustworthy in situations where it benefits them, and this is the correct decision.

Now you might try to break the causal link: My values → Things that I say And doing so is certainly possible (I mean you can have spies that successfully pretend to be loyal for extended periods without giving themselves away). On the other hand, it might not happen often for several possible reasons: A) Maintaining a facade at all times is exhausting (and thus imposes high costs) B) Lying consistently is hard (as in too computationally expensive) C) The right way to lie consistently, is to simulate the altered value set, but this may actually lead to changing your values (standard advice for become more confident is pretending to be confident, right?).

So yes, in this model an non-trust-valuing and self-modifying CDT agent will self-modify, but it will need to self-modify its values rather than its decision theory. Using a decision theory that is trustworthy despite not intrinsically valuing it doesn’t help.

dankane 25 Sep 2014 16:20 UTC
10 points
on: Newcomblike problems are the norm

Newcomblike problems occur whenever knowledge about what decision you will make leaks into the environment. The knowledge doesn’t have to be 100% accurate, it just has to be correlated with your eventual actual action.

This is far too general. The way in which information is leaking into the environment is what separates Newcomb’s problem from the smoking lesion problem. For your argument to work you need to argue that whatever signals are being picked up on would change if the subject changed their disposition, not merely that these signals are correlated with the disposition.
What links here?
- Vaniver's comment on Newcomblike problems are the norm by So8res (25 Sep 2014 17:13 UTC; 2 points)

dankane 19 Sep 2014 1:37 UTC
0 points
in reply to: [deleted]’s comment on: Causal decision theory is unsatisfactory
Sorry. I’m not quite sure what you’re saying here. Though, I did ask for a specific example, which I am pretty sure is not contained here.

Though to clarify, by “reading your mind” I refer to any situation in which the scenario you face (including the given description of that scenario) depends directly on which program you are running and not merely upon what that program outputs.

dankane 19 Sep 2014 0:59 UTC
0 points
in reply to: hairyfigment’s comment on: Causal decision theory is unsatisfactory
Well, yes. Then again, the game was specified as PD against BOT^CDT not as PD against BOT^{you}. It seems pretty clear that for X not equal to CDT that it is not the case that X could achieve the result CC in this game. Are you saying that it is reasonable to say that CDT could achieve a result that no other strategy could just because it’s code happens to appear in the opponent’s program?

I think that there is perhaps a distinction to be made between things that happen to be simulating your code and this that are causally simulating your code.

dankane 19 Sep 2014 0:42 UTC
0 points
in reply to: nshepperd’s comment on: Causal decision theory is unsatisfactory
OK. Fine. Point taken. There is a simple fix though.

MBOT^X(Y) = X’(MBOT^X) where X’ is X but with randomized irrelevant experiences.

In order to produce this properly, MBOT only needs to have your prior (or a sufficiently similar probability distribution) over irrelevant experiences hardcoded. And while your actual experiences might be complicated and hard to predict, your priors are not.

dankane 18 Sep 2014 14:50 UTC
0 points
in reply to: nshepperd’s comment on: Causal decision theory is unsatisfactory
No. BOT^CDT = DefectBot. It defects against any opponent. CDT could not cause it to cooperate by changing what it does.

If it cooperated, it would get CC instead of DD.

Actually if CDT cooperated against BOT^CDT it would get $3^^^3. You can prove all sorts of wonderful things once you assume a statement that is false.

Depending on the exact setup, “irrelevant details in memory” are actually vital information that allow you to distinguish whether you are “actually playing” or are being simulated in BOT’s mind.

OK… So UDT^Red and UDT^Blue are two instantiations of UDT that differ only in irrelevant details. In fact the scenario is a mirror matchup, only after instantiation one of the copies was painted red and the other was painted blue. According to what you seem to be saying UDT^Red will reason:

Well I can map different epistemic states to different outputs, I can implement the strategy cooperate if you are painted blue and defect if you are painted red.

Of course UDT^Blue will reason the same way and they will fail to cooperate with each other.