kokotajlod

Karma: 299

kokotajlod 22 Mar 2020 14:23 UTC
3 points
0
in reply to: Ben Pace’s comment on: Blog Post Day II
Sweet. I too will write something not about coronavirus.

kokotajlod 23 Dec 2016 18:45 UTC
2 points
0
in reply to: Lumifer’s comment on: Extortion and trade negotiations
“It’s a badly formulated question, likely to lead to confusion.” Why? That’s precisely what I’m denying.

“So, can you specify what this cluster is? Can you list the criteria by which a behaviour would be included in or excluded from this cluster? If you do this, you have defined blackmail.”

That’s precisely what I (Stuart really) am trying to do! I said so, you even quoted me saying so, and as I interpret him, Stuart said so too in the OP. I don’t care about the word blackmail except as a means to an end; I’m trying to come up with criteria by which to separate the bad behaviors from the good.

I’m honestly baffled at this whole conversation. What Stuart is doing seems the opposite of confused to me.

kokotajlod 23 Dec 2016 18:36 UTC
0 points
0
in reply to: 3p1cd3m0n’s comment on: Open Problems Related to Solomonoff Induction
Dunno what Username was thinking, but here’s the answer I had in mind: “Why is it obvious? Because the Problem of Induction has not yet been solved.”

kokotajlod 21 Dec 2016 15:23 UTC
3 points
0
in reply to: Dagon’s comment on: Extortion and trade negotiations
You make it sound like those two things are mutually exclusive. They aren’t. We are trying to define words so that we can understand and manipulate behavior.

“I don’t know what blackmail is, but I want to make sure an AI doesn’t do it.” Yes, exactly, as long as you interpret it in the way I explained it above.* What’s wrong with that? Isn’t that exactly what the AI safety project is, in general? “I don’t know what bad behaviors are, but I want to make sure the AI doesn’t do them.”

*”In other words there are a cluster of behaviors that we do NOT want our AI to have, which seem blackmailish to us, and a cluster of behaviors that we DO want it to have, which seem tradeish to us. So we are now trying to draw a line in conceptual space between them so that we can figure out how to program an AI appropriately.”

kokotajlod 20 Dec 2016 14:43 UTC
1 point
0
in reply to: Dagon’s comment on: Extortion and trade negotiations
“You want to understand and prevent some behaviors (in which case, start by tabooing culturally-dense words like “blackmail”)”

In a sense, that’s exactly what Stuart was doing all along. The whole point of this post was to come up with a rigorous definition of blackmail, i.e. to find a way to say what we wanted to say without using the word.

kokotajlod 19 Dec 2016 16:31 UTC
4 points
0
in reply to: Dagon’s comment on: Extortion and trade negotiations
As I understand it, the idea is that we want to design an AI that is difficult or impossible to blackmail, but which makes a good trading partner.

In other words there are a cluster of behaviors that we do NOT want our AI to have, which seem blackmailish to us, and a cluster of behaviors that we DO want it to have, which seem tradeish to us. So we are now trying to draw a line in conceptual space between them so that we can figure out how to program an AI appropriately.

kokotajlod 29 Oct 2016 2:09 UTC
0 points
0
in reply to: ike’s comment on: New Pascal’s Mugging idea for potential solution
OH ok I get it now: “But clearly re-arranging terms doesn’t change the expected utility, since that’s just the sum of all terms.” That’s what I guess I have to deny. Or rather, I accept that (I agree that EU = infinity for both A and B) but I think that since A is better than B in every possible world, it’s better than B simpliciter.

The reshuffling example you give is an example where A is not better than B in every possible world. That’s the sort of example that I claim is not realistic, i.e. not the actual situation we find ourselves in. Why? Well, that was what I tried to argue in the OP—that in the actual situation we find ourselves in, the action A that is best in the simplest hypothesis is also better.… well, oops, I guess it’s not better in every possible world, but it’s better in every possible finite set of possible worlds such that the set contains all the worlds simpler than its simplest member.

I’m guessing this won’t be too helpful to you since, obviously, you already read the OP. But in that case I’m not sure what else to say. Let me know if you are still interested and I”ll try to rephrase things.

Sorry for taking so long to get back to you; I check this forum infrequently.

kokotajlod 18 Aug 2016 15:24 UTC
0 points
0
in reply to: ike’s comment on: New Pascal’s Mugging idea for potential solution
Again, thanks for this.

“The problem with your solution is that it’s not complete in the formal sense: you can only say some things are better than other things if they strictly dominate them, but if neither strictly dominates the other you can’t say anything.”

As I said earlier, my solution is an argument that in every case there will be an action that strictly dominates all the others. (Or, weaker: that within the set of all hypotheses of probability less than some finite N, one action will strictly dominate all the others, and that this action will be the same action that is optimal in the most probable hypothesis.) I don’t know if my argument is sound yet, but if it is, it avoids your objection, no?

I’d love to understand what you said about re-arranging terms, but I don’t. Can you explain in more detail how you get from the first set of hypotheses/choices (which I understand) to the second?

kokotajlod 10 Aug 2016 17:37 UTC
0 points
0
in reply to: ike’s comment on: New Pascal’s Mugging idea for potential solution
This was helpful, thanks!

As I understand it, you are proposing modifying the example so that on some H1 through HN, choosing A gives you less utility than choosing B, but then thereafter choosing B is better, because there is some cost you pay which is the same in each world.

It seems like the math tells us that any price would be worth it, that we should give up an unbounded amount of utility to choose A over B. I agree that this seems like the wrong answer. So I don’t think whatever I’m proposing solves this problem.

But that’s a different problem than the one I’m considering. (In the problem I’m considering, choosing A is better in every possible world.) Can you think of a way they might be parallel—any way that the “I give up” which I just said above applies to the problem I’m considering too?

kokotajlod 6 Aug 2016 14:17 UTC
0 points
0
in reply to: Val’s comment on: New Pascal’s Mugging idea for potential solution
It’s arbitrary, but that’s OK in this context. If I can establish that this works when the ratio is 1 in a billion, or lower, then that’s something, even if it doesn’t work when the ratio is 1 in 10.

Especially since the whole point is to figure out what happens when all these numbers go to extremes—when the scenarios are extremely improbable, when the payoffs are extremely huge, etc. The cases where the probabilities are 1 in 10 (or arguably even 1 in a billion) are irrelevant.

kokotajlod 5 Aug 2016 17:03 UTC
−1 points
0
in reply to: ike’s comment on: New Pascal’s Mugging idea for potential solution
Update: The conclusion of that article is that the expected utilities don’t converge for any utility function that is bounded below by a computable, unbounded utility function. That might not actually be in conflict with the idea I’m grasping at here.

The idea I’m trying to get at here is that maybe even if EU doesn’t converge in the sense of assigning a definite finite value to each action, maybe it nevertheless ranks each action as better or worse than the others, by a certain proportion.

Toy model:

The only hypotheses you consider are H1, H2, H3, … etc. You assign 0.5 probability to H1, and each HN+1 has half the probability of the previous hypothesis, HN.

There are only two possible actions: A or B. H1 says that A gives you 2 utils and B gives you 1. Each HN+1 says that A gives you 10 times as many utils as it did under the previous hypothesis, HN, and moreover that B gives you 5 times as many utils as it did under the previous hypothesis, HN.

In this toy model, expected utilities do not converge, but rather diverge to infinity, for both A and B.

Yet clearly A is better than B...

I suppose one could argue that the expected utility of both A and B is infinite and thus that we don’t have a good reason to prefer A to B. But that seems like a problem with our ability to handle infinity, rather than a problem with our utility function or hypothesis space.

kokotajlod 5 Aug 2016 16:55 UTC
−2 points
0
in reply to: ChristianKl’s comment on: New Pascal’s Mugging idea for potential solution
True. So maybe this only works in the long run, once we have more than 30 bits to work with.

kokotajlod 5 Aug 2016 13:07 UTC
−2 points
0
in reply to: entirelyuseless’s comment on: New Pascal’s Mugging idea for potential solution
Yes, but I don’t think that’s relevant. Any use of complexity depends on the language you specify it in. If you object to what I’ve said here on those grounds, you have to throw out Solomonoff, Kolmogorov, etc.

kokotajlod 5 Aug 2016 13:05 UTC
−2 points
0
in reply to: ike’s comment on: New Pascal’s Mugging idea for potential solution
Yes, I’ve read it, but not at the level of detail where I can engage with it. Since it is costly for me to learn the math necessary to figure this out for good, I figured I’d put the basic idea up for discussion first just in case there was something obvious I overlooked.

Edit: OK, now I think I understand it well enough to say how it interacts with what I’ve been thinking. See my other comment .

kokotajlod 4 Aug 2016 20:14 UTC
0 points
0
on: The map of p-zombies
I disagree with your characterization of 0. You say that it is incompatible with physicalism, but that seems false. Indeed it seems to be a very mainstream physicalist view to say “I am a physical object—my brain. So a copy of me would have the same experiences, but it would not be me.”

kokotajlod 11 Oct 2015 0:37 UTC
0 points
0
on: Experiment: Changing minds vs. preaching to the choir
How do I do step II? I can’t seem to find the relevant debates. I found one debate with the same title as the minimum wage one I argued about, but I don’t see my argument appearing there.

kokotajlod 26 Jul 2015 20:35 UTC
−1 points
0
in reply to: snarles’s comment on: Steelmaning AI risk critiques
Yep. “The melancholy of haruhi suzumiya” can be thought of as an example of something in the same reference class.

kokotajlod 5 Jun 2015 19:56 UTC
2 points
0
on: Counterfactual trade
This is an interesting idea! Some thoughts:

Doesn’t acausal trade (like all trade) depend on enforcement mechanisms? I can see how two AI’s might engage in counterfactual trade, since they can simulate each other and see that they self-modify to uphold the agreement, but I don’t think a human would be able to do it.

Also, I’d like to hear more about motivations for engaging in counterfactual trade. I get the multiverse one, though I think that’s a straightforward case of acausal trade rather than a case of counterfactual trade, since you would be trading with a really existing entity in another universe. But can you explain the second motivation more?

kokotajlod 30 Sep 2014 10:48 UTC
1 point
0
in reply to: lackofcheese’s comment on: Simulation argument meets decision theory
The point you raise is by far the strongest argument I know of against the idea.

However, it is a moral objection rather than a decision-theory objection. It sounds like you agree with me on the decision theory component of the idea: that if we were anthropically selfish, it would be rational for us to commit to making ancestor-simulations with afterlives. That’s an interesting result in itself, isn’t it? Let’s go tell Ayn Rand.

When it comes to the morality of the idea, I might end up agreeing with you. We’ll see. I think there are several minor considerations in favor of the proposal, and then this one massive consideration against it. Perhaps I’ll make a post on it soon.

kokotajlod 28 Sep 2014 22:34 UTC
7 points
0
on: Simulation argument meets decision theory
This is a formal version of a real-life problem I’ve been thinking about lately.

Should we commit to creating ancestor-simulations in the future, where those ancestor-simulations will be granted a pleasant afterlife upon what appears to their neighbors to be death? If we do, then arguably we increase the likelihood that we ourselves have a pleasant afterlife to look forward to.