I’d be glad to add a few. I’ll have to do some thinking but one immediately popped to mind so I’ll add it now. Good luck with the project. Sounds interesting.

Which I suppose could be termed “infinitely confused”, but that feels like a mixing of levels. You’re not confused about a given probability, you’re confused about how probability works.

Or alternatively, it’s a clever turn of phrase: “infinitely confused” as in confused about infinities.

This is a good point, but perhaps precommitting to give in/not give in vs. precommitting to blackmail/not blackmail is a simultaneous choice.

“If I have an action that I can take that would help me but hurt you and I ask you for some compensation for refraining from taking this action, then this is more like a value trade than a blackmail”—Maybe. What about if an action gives you 1 utility, but costs me a 100 and you demand 90. That sounds a lot like blackmail!

Using epsilons can in principle allow you to update. However, the situation seems slightly worse than jimrandomh describes. It looks like you need P(E|h), or the probability if H is false, in order to get a precise answer. Also, the missing info that jim mentioned is already enough in principle to let the final answer be any probability whatsoever.

If we use log odds (the framework in which we could literally start with “infinite certainty”) then the answer could be anywhere on the real number line. We have infinite (or at least unbounded) confusion until we make our assumptions more precise.

This math is exactly why we say a rational agent can never assign a perfect 1 or 0 to any probability estimate. Doing so in a universe which then presents you with counterevidence means you’re not rational.

Which I suppose could be termed “infinitely confused”, but that feels like a mixing of levels. You’re not confused about a given probability, you’re confused about how probability works.

In practice, when a well-calibrated person says 100% or 0%, they’re rounding off from some unspecified-precision estimate like 99.9% or 0.000000000001.

This mapping does not match any actual decisions in blackmail. First, it’s not a simultaneous choice, it’s a branching multi-turn decision tree. Second, there are more than 2 actions available at various stages. Either of these would make prisoner’s dilemma analysis suspect, together it becomes much more like multi-street multi-bet poker than like PD.

The “victim” first makes choices (or is born into a situation) susceptible to blackmail. The blackmailer learns of this, and has at least 3 choices: publish the information, threaten to publish, or bury the information. The “victim” in the threaten-to-publish (blackmail) case offers incentives (which may be the same as the requested fee, or may not) to bury rather than publish, and the blackmailer chooses which action to take. Even leaving out true defection cases (accept the money and publish anyway, or killing the blackmailer), this is a fairly complex payout tree, and the correct choices are specific to the situation. In fact, since parts of the payout tree are unknown to one or both players, it’s likely that mixed strategies come into play, to prevent exploitation of the unknowns.

In other words, the agent assigned zero probability to an event, and then it happened.

I see. so -

If P(H) = 1.0 - ϵ1

And P(E|H) = 0 + ϵ2

Then it equals “infinite confusion”.

Am i correct?

and also, when you use epsilons, does it mean you get out of the “dogma” of 100%? or you still can’t update down from it?

And what i did in my post may just be another example of why you don’t put an actual 1.0 in your prior, cause then even if you get evidence of the same strength in the other direction, that would demand that you divide zero by zero. right?

This is a great post! Thank you in particular for the Stephen Boyd recommendation.

~~~~

In terms of frameworks and ML, you might be interested in this blog post announcing the release of the DiffEqFlux.jl package in the Julia programming language. I was reminded of this because it seems to be an example of

*merging*already-mature frameworks, specifically a machine learning framework (Flux) and a differential equation framework (DifferentialEquations.jl).A recent paper called Neural Ordinary Differential Equations explored the problem of parameterizing a term in an ODE instead of specifying a sequence of layers, using ML techniques. The benefit of the combined framework is that it allows us to incorporate what we already know about the structure of the problem via the ODE, and apply neural networks to the parts we don’t.

I’m not deeply familiar with either framework, but I have been following DifferentialEquations.jl for a while because I had hoped it would let me tackle a few problems I have had my eye on.

~~~~

I feel like Geometric Algebra is pointing in a similar direction to what you describe. It is not directly an applied framework, but rather a kind of ur-framework for teaching and developing other frameworks on top of it. The idea is that they make geometric objects elements of the algebra, which really boils down to making the math match the structure of space and in this way preserve geometric

*intuitions*throughout. Its advocates hope it will serve as a unified framework for physics and engineering applications. The primary theorist is David Hestenes, and his argument for why this was necessary is found here; there are surveys/tutorials from other people who have done work in the field here and here.Geometric Algebra and its extension Geometric Calculus are both still pretty firmly in the ‘reorganizing math’ kind of endeavor, with lots of priority put on proving results and developing algorithms. But there are a few areas where they are growing into an applied framework of the kind you describe, specifically robotics, computer graphics, and computer vision.

It was through the robotics applications that I found it; they promised an intuitive explanation of quaternions, which otherwise were just this scary way to calculate motion.

The latter; it could be anything, and by saying the probabilities were 1.0 and 0.0, the original problem description left out the information that would determine it.

Thanks for the answer! i was somewhat amused to see that it ends up being a zero divided by zero.

Does the ratio between 1epsilon over 2epsilon being undefined means that it’s arbitrarily close to half (since 1 over two is half, but that wouldn’t be

*exactly*it)? or means that we get the same problem i specified in the question, where it could be anything from (almost) 0 to (almost) 1 and we have no idea what exactly?

Hi Iwan,

I personally find this subject (pragmatism generally) interesting, but others here might not. If you’re going to link to an external source, could you please write a post detailing what is being explained in the external source and why people here might find it relevant plus why they should care about it? I strong-downvoted your post because simply link-posting with no context or explanation about why the topic is relevant/why people should care, is almost always a non-useful thing for people, and if one is going to post things here, they ought to be useful for the community.

Cheers

If you do out the algebra, you get that P(H|E) involves dividing zero by zero:

There are two ways to look at this at a higher level. The first is that the algebra doesn’t really apply in the first place, because this is a domain error: 0 and 1 aren’t probabilities, in the same way that the string “hello” and the color blue aren’t.

The second way to look at it is that when we say and , what we really meant was that and ; that is, they aren’t

*precisely*one and zero, but they differ from one and zero by an unspecified, very small amount. (Infinitesimals are like infinities; is arbitrarily-close-to-zero in the same sense that an infinity is arbitrarily-large). Under this interpretation, we don’t have a contradiction, but we do have an underspecified problem, since we need the ratio and haven’t specified it.

AI systems end up controlled by a group of humans representing a small range of human values (ie. an ideological or religious group that imposes values on everyone else). While not caused only by AI design, it is possible that design decisions could impact the likelihood of this scenario (ie. at what point are values loaded into the system/how many people’s values are loaded into the system), and is relevant for overall strategy.

Failure to learn how to deal with alignment in the many-humans, many-AIs case even if single-human, single-AI alignment is solved (which I think Andrew Critch has talked about). For example, AIs negotiating on behalf of humans take the stance described in https://arxiv.org/abs/1711.00363 of agreeing to split control of the future according to which human’s priors are most accurate (on potentially irrelevant issues) if this isn’t what humans actually want.

Is this future AI catastrophe? Or is this just a description of current events being a general gradual collapse?

This seems like what is happening now, and has been for a while. Existing ML systems are clearly making Type-I problems, already quite bad before ML was a thing at all, much worse, to the extent that I don’t see much ability left of our civilization to get anything that can’t be measured in a short term feedback loop—even in spaces like this, appeals to non-measurable or non-explicit concerns are a near-impossible sell.

Part II problems are not yet coming from ML systems, exactly, But we certainly have algorithms that are effectively optimized and selected for the ability to gain influence; the algorithm gains influence, which causes people to care about it and feed into it, causing it to get more. If we get less direct in the metaphor we get the same thing with memetics, culture, life strategies, corporations, media properties and so on. The emphasis on choosing winners, being ‘on the right side of history’, supporting those who are good at getting support. OP notes that this happens in non-ML situations explicitly, and there’s no clear dividing line in any case.

So if there is another theory that says, this has already happened, what would one do next?

On the “a day in hell cannot be outweighed” question, do you have any analysis of that intuition? Are you assuming that you’ll remember that day and be broken by it, or is there some other negative value you’re putting on it. Do you evaluate “a day in hell, 1000 years in heaven, then termination” differently from “1000 years in heaven, then a day in hell, then termination”? How about “a day in hell, mind-reset to prior state, then 1000 years in heaven”?

The reason I ask is that I’m trying to understand what’s being evaluated. Are you comparing value of instantaneous experience integrated over time, or are you comparing value of effect that experiences have on your identity?

Who are you and how is it that we don’t we know each other yet?