Do you run into a distinction between benign and malign tampering at any point? For example, if humans can never tell the difference between the tampered and non-tampered result, and their own sanity has not been compromised, it is not obvious to me that the tampered result is worse than the non-tampered result.
It might be easier to avoid compromising human sanity + use hold-out sensors than to solve ELK in general (though maybe not? I haven’t thought about it much).
So, granting the assumption of not corrupting the humans (which is maybe what you are denying), doesn’t this imply that we can go on adding sensors after the fact until, at some point, the difference between fooling them all and being honest becomes unproblematic?
I’ve previously argued that the concept of counterfactuals can only be understood from within the counterfactual perspective.
I think this goes too far. We can give an account of counterfactuals from assumptions of symmetry. This account is unsatisfactory in many ways—for one thing, it implies that counterfactuals exist much more rarely than we want them to. Nonetheless, it seems to account for some properties of a counterfactual and is able to stand up without counterfactual assumptions to support it. I think it also provides an interesting lens for examining decision theory paradoxes.
Determinism is not a defining feature of counterfactuals, you can make a stochastic theory of counterfactuals that is a strict generalisation of SEM-style deterministic counterfactuals. See Pearl, Causality (2009), p. 220 “counterfactuals with intrinsic nondeterminism” for the basic idea. It’s a brief discussion and doesn’t really develop the theory but, trust me, such a theory is possible. The basic idea is contained in “the mechanism equations lose their deterministic character and hence should be made stochastic.”
I’m pretty sure that picture is from the Book of Why
Pearl is distinguishing “intrinsically nondeterministic” from “ordinary” Bayesian networks, and he is saying that we shouldn’t mix up the two (though I think it would be easier to avoid this with a clearer explanation of the difference).
Three questions:
Do we need determinism to define counterfactuals?
No
Is uncertainty represented in causal Bayesian networks typically used in social science limited to “intrinsic nondeterminism”?
No, and so we should be careful not to mix them up with “intrinsically nondeterministic” Bayesian networks
Is there no intrinsic nondeterminism in any causal Bayesian network relevant to social science?
I doubt it
More importantly, the thing you can do with counterfactual models that you can’t do with “ordinary” causal Bayesian networks is you can condition on the results of an action and then change the action (called “abduction”; this is why it would be helpful to have a better explanation of the difference between the two!). This will usually leave a bunch of uncertainty about stuff, which may or may not be intrinsic. You definitely don’t need determinism to do abduction, and I submit that our attitude towards the question of whether the leftover uncertainty is “intrinsic” should often be the same as our usual attitude toward this question: who cares?
This is an old post, but my idea of CDT is that it’s a rule for making decisions, not for setting beliefs. Thus the agent never believes in the outcome given by CDT, just that it should choose according to the payoffs it calculates. This is a seemingly weird way to do things, but apart from that is there a reason I should think about CDT as a prescription for forming beliefs while I am acting?
My confusion was: even “when the agent is acting”, I think it would still be appropriate to describe its beliefs according to EDT. However, I was confused by thinking about ”...and then offering a bet”. As far as I can tell, this is just an unnecessary bit of storytelling set around a two step decision problem, and a CDT agent has to evaluate the prospects of each decision according to CDT.
Γ=Σ^R, it’s a function from programs to what result they output. It can be thought of as a computational universe, for it specifies what all the functions do.
Should this say “elements are function… They can be thought of as...?”
Can you make a similar theory/special case with probability theory, or do you really need infra-bayesianism? If the second, is there a simple explanation of where probability theory fails?
how to respond to the temptation to shift from utilising actual peers to potential peers which then seems to reraise the specter of circularity.
I think you might be able to say something like “actual peers is why the rule was learned, virtual peers is because the rule was learned”.
(Just to be clear: I’m far from convinced that this is an actually good theory of counterfactuals, it’s just that it also doesn’t seem to be obviously terrible)
Let’s suppose I face Newcomb’s problem in a yellow shirt and you face it in a red shirt. They ought to be comparable because my shirt color doesn’t matter.
The definition of peers given is in terms of what actually happens, and the question of whether you think someone is your peer is put down to “induction somehow”. I think this approach has serious problems, but it does answer your question: whether we are peers depends on the results we get. How we should include the colour of our shirts in our decision making on the other hand is a question of what inductive assumptions we’re willing to make when assessing likely peers.
Actually, there’s an important asymmetry in terms of inductive assumptions in just about every situation that involves “I see you do x, then I do x”. The thing is, I almost always know more about why I am doing x than I do about why you are doing x. You might be two boxing because you signed a contract with the predictor beforehand where you agreed to two-box in exchange for an under the table payment of $2m, while I am two-boxing because I decided the dominance theory is compelling. I don’t know why you act and you don’t know why I act, but I know why I act and you know why you act.
Thus the case for epistemic indifference between “me about to choose, from my perspective” and “you about to choose, from my perspective” seems to be quite compromised, before we even consider what shirts we are wearing. And this is as it should be! Inferring causation from correlation is usually a bad move. Nonetheless, it seems unreasonable to think that I am so different to everyone else that I should two box in Newcomb’s problem w/a predictor that has a perfect track record.
The notion that “some of these people were in the same position I am in, though I don’t know who they are” seems pretty plausible, though I feel like it’s resting on some more fundamental assumptions that I’m not quite sure of right now.
Peers get the same results from the same actions. It’s not exactly clear what “same action” or “same result” means—is “one boxing on the 100th run” the same as “one boxing on the 101st run” or “box 100 with $1m in it” the same as “box 101 with $1m in it”? I think we should think of peers as being defined with respect to a particular choice of variables representing actions and results.
I think the definitions of these things aren’t immediately obvious, but it seems like we might be able to figure them out sometimes. Given a decision problem, it seems to me that “the things that I can do” and “the things that I care about” might often be known to me. It seems to be the case that I can also define some variables that represent copies of these things from your point of view, although it’s a bit less obvious how to do that.
If we think about predictions vs outcomes, we judge good predictions to be ones that have a good match to outcomes. Similarly, a “peer inference” is a bit like a prediction—I think this group of action-outcome pairs will be similar to my own—and the outcome that can be assessed at the end is whether they actually are similar to my own action and outcome. I can’t assess whether they “would have been” peers “had I taken a different action”, but maybe I don’t need to. For example: if I assess some group of people to be my peers relative to a particular decision problem, and all the people who take action 1 end up winners while all the people who take action 2 end up losers, and I take action 1 and end up a winner then I have done well relative to the group of people I assessed to be my peers, regardless of “what would have happened had I taken action 2″.
There is a sense in which I feel like peers ought to also be a group that it is relevant to judge myself against—I want to take actions that do better than my peers, in some sense. Maybe defining actions and outcomes addresses this concern? I’m not sure.
I think a substantial problem with this theory is the fact that I may often find that the group of peers for some problem contains only me, which leaves us without a useful definition for a counterfactual.
My 3yo: “a bowl of museli because it’s heavy”
Also “an apple, plum and peach and a bit of wax”
Regarding
”Strategy: train a reporter which isn’t useful for figuring out what the human will believe/Counterexample: deliberately obfuscated human simulator”.
If you put the human-interpreter-blinding before the predictor instead of between the predictor and the reporter, then whether or not the blinding produces an obfuscated human simulator, we know the predictor isn’t making use of human simulation.
An obfuscated human simulator would still make for a rather bad predictor.
I think this proposal might perform slightly better than a setup where we expand the training environment to situations the supervisor doesn’t understand and train the controller to take no actions when the human supervisor reports that they don’t understand the data. I think the behaviour you’d get in this setup is similar to what you’d get with a perfectly obfuscated human simulator, and in practice you probably wouldn’t get perfectly obfuscated simulator. Maybe it could nevertheless achieve a similar level of safety, in terms of guarantees against exploiting gaps in the supervisor’s knowledge. I could be wrong on either point, I’ve only thought about it a little.
This gets good log loss because it’s trained in the regime where the human understands what’s going on, correct?
Taking a cue from the wiki article “you will be hanged tomorrow, and you will not be able to derive from this statement whether or not you will be hanged tomorrow”
Seems kind of weird because it is self-contradictory, and yet true.
I got “masks are like assholes” from the sentence, even before I read the Ben’s analysis.
The Legg-Hutter definition of intelligence is counterfactual (“if it had X goal, it would do a good job of achieving it”). It seems to me that the counterfactual definition isn’t necessary to capture the idea above. The LH definition also needs a measure over environments (including reward functions), and it’s not obvious how closely their proposed measure corresponds to things we’re interested in, while influentialness in the world we live in seems to correspond very closely.
The mesa-optimizer paper also stresses (not sure if correctly) that they’re not talking about input-output properties.
Influentialness in the real world definitely screens off training environment performance WRT impact in the real world, because that’s what the definition of influentialness is. If training environment performance fully screens off real world influentialness then this means that the outcome of training is essentially deterministic, which is not obviously always going to be true. Even if it is, kind of real world influence any given training environment maps to will be a bit random, which might ultimately yield a similar result.
When you say “we’ll reward intelligence”, your meaning is similar to when I say “training environment and real world might be highly compatible”.
I think an idea like Legg-Hutter intelligence could be useful for arguing that the environments are likely to be compatible.
We both have a similar intuition about the kinds of optimizers we’re interested in. You say they optimize things that are “far away”, I say they affect “big pieces of the environment”. One difference is that I think of big as relative to the size of the agent, but something can be “far away” even if the agent is itself quite large, and it seems that agent size doesn’t necessarily matter to your scheme because the information lost over a given distance doesn’t depend on whether there’s a big agent or a small one trying to exert influence over this distance.
I think agent size (in the sense I’m thinking about it) is mainly relevant from the point of view of “how likely is it for such an agent to come about?” (which suggest something like “large measure, given initial conditions + dynamics” instead of “small size”).
Here are some of my thoughts on the issue: https://www.lesswrong.com/posts/me34KqMLwJNYAZKbs/is-evolutionary-influence-the-mesa-objective-that-we-re
I think my scheme needs some distinction between “microstates” and “macrostates” in order to offer a reasonable definition of “big features”. Your setup seems to have this fairly naturally in terms of the telephone theorem, though the precise analogy (if there is one) isn’t striking me immediately.
I am a forecaster on that question: the main doubt I had was if/when someone would try to do wordy things + game playing on a “single system”. Seemed plausible to me that this particular combination of capabilities never became an exciting area of research, so the date at which an AI can first do these things would then be substantially after this combination of tasks would be achievable with focused effort. Gato was a substantial update because it does exactly these tasks, so I no longer see much reason possibility that the benchmark is achieved only after the capabilities are substantially overshot.
I also tend to defer somewhat to the community.
I was at 2034 when the community was at 2042, and I updated further to 2026 on the Gato news.
I’m a bit curious about what job “dimension” is doing here. Given that I can map an arbitrary vector in Rn to some point in R via a bijective measurable map (https://en.wikipedia.org/wiki/Standard_Borel_space#Kuratowski’s_theorem), it would seem that the KPD theorem is false. Is there some other notion of “sufficient statistic complexity” hiding behind the idea of dimensionality, or am I missing something?