If you are going to read just one thing I wrote, read The Problem of the Criterion.
More AI related stuff collected over at PAISRI
G Gordon Worley III(G Gordon Worley III)
Karma: 5,358
Folks sometimes talk about the human alignment problem, which I think is what you’re getting at. I think the earliest instance of it can be found in this post. Searching for “human alignment problem” on LW will turn up more stuff, although I don’t think anyone has done an exhaustive post summarizing what we really mean by this, though it’s generally one of two things:
humans aligning themselves to goals (getting yourself to do something)
aligning multiple humans to a goal (like in an organization)
Thanks for this review and reminding me of GEB. Definitely still the most important book I ever read. If I hadn’t read it I think my life would have turned out very different indeed!
“Swimming animals” vs. “fish” seems like a fairly neutral term choice, so it’s somewhat boring as an example. It’s a much bigger deal to choose between terms where one option has negative connotations and the other positive (or even just very different connotations!). Cf. various political debates over terms that shall not be named here.
Interesting. Selection theorems seem like a way of identifying the purposes or source of goal directness in agents that seems obvious to us yet hard to pin down. Compare also the ground of optimization.
I know I keep leaving comments pointing to this post of mine (I would stop if I felt like its contents had entered the collective consciousness, but they haven’t and I wrote the post to change that so here we are), but all three of these enigmas are instances of the problem of the criterion. Reading through the comments, I feel like most of them come down to folks continuing to struggle to come to terms with the problem and its implications.
The linked post is pretty uninteresting to me. I skimmed it, but seems mostly to be some biographies of folks who were/are unusually productive and comes away with not much of a theory of why these people are that way or how one could become like them (or why you can’t!). Seems like a reasonable first step towards exploring the issue, but would have found more value in some information I could use beyond what I already knew (some people are extremely productive beyond what everyone else seems able to do).
Given that you’re idea of a deliberate optimizer depends on an ability to generalize, I think there’s a missing bit here about the scope of the optimizer. From your examples, I think the only thing telling accidental and deliberative optimizers apart is the scope you chose to impose to make the decision, so this is not a property natural to the optimizer but to what the observer doing the categorization cares about.
I don’t really have a whole picture that I think says more than what others have. I think there’s something to knowing as the act of operationalizing information, by which I mean a capacity to act based on information.
To make this more concrete, consider a simple control system like a thermostat or a steam engine governor. These systems contain information in the physical interactions we abstract away to call “signal” that’s sent to the “controller”. If we had only signal there’d be no knowledge because that’s information that is not used to act. The controller creates knowledge by having some response it “knows” to perform when it gets the signal.
This view then doesn’t really distinguish knowledge from purpose in a cybernetic sense, and I think that seems reasonable at first blush. This let’s us draw a hard line between “dead” information like words in a book and “live” information like words being read.
Of course this doesn’t necessarily make all the distinctions we’d hope to make, since this makes no difference between a thermostat and a human when it comes to knowledge. Personally I think that’s correct. There’s perhaps some interesting extra thing to say about the dynamism of these two systems (the thermostat is an adaption executor only, the human is that and something capable of changing itself intentionally), but I think that’s separate from the knowledge question.
Obviously this all hinges on a particular sort of deflationary approach to these terms to have them make sense with the weakest possible assumptions and covering the broadest classes of systems. Whether or not this sort of “knowledge” I’m proposing here is useful for much is another question.
Quick thought: reading this I get a sense that some of our collective confusion here revolves around “knowledge” as a quantifiable noun rather than “knowing” as a verb, and if we give up on the idea that knowledge is first a quantifiable thing (rather than a convenient post hoc reification) we open up new avenues of understanding knowledge.
How do ordinary people, who largely profess good values, and who have no particular interest in doing evil things, nonetheless become instrumental in horrible crimes?
From the story shared just above this quote, this sounds like we can mostly explain it as coordination failure, i.e. a failure to hunt stag instead of rabbit.
FWIW, sounds like you’re pointing at what Chalmers calls the meta problem of consciousness: why do we think there is a hard problem of consciousness?
Small insight why reading this: I’m starting to suspect that most (all???) unintuitive things that happen with Oracles are the result of them violating our intuitions about causality because they actually deliver no information, in that nothing can be conditioned on what the Oracle says because if we could then the Oracle would fail to actually be an Oracle, so we can only condition on the existence of the Oracle and how it functions and not what it actually says, e.g. you should still 1-box but it’s mistaken to think anything an Oracle tells you allows you to do anything different.
FWIW I actually recently updated Wikipedia to reflect this fact (that Alcor has been in continuous operation since the 70s).
To what extent do you find the above explanation unsatisfying? And if you do not find it unsatisfying, then (I repeat): what is the use of talking about the “problem of the criterion”, beyond (perhaps) the fact that it allows you to assert fun and quirky and unintuitive (and false) things like “facts don’t exist”?
To me this is like asking what’s the point in talking about a Theory of Everything when trying to talk about physics. You might complain you can do a lot of physics without it, yet we still find it useful to have a theory that unifies physics at a fundamental level (even if we keep failing to find one). I argue that the problem of the criterion fills a conceptually similar niche in epistemology: it’s the fundamental thing to be understood in order to be able to say anything else meaningful and not inconsistent about how we know or what we know, which is itself fundamental to most activity. Thus it is often quite useful to appeal to because lots of deconfusion research, like this post, are ultimately consequences of the problem of the criterion, and so I find most object-level arguments, like those found in this post, a certain kind of wasted motion that could be avoided if only the problem of the criterion were better understood.
I agree that this is a real difficulty that people run into. I disagree with [what I see as] your [implicit] claim that the “problem of the criterion” framing provides any particular tools for addressing this problem, or that it’s a useful framing in general. (Indeed, the sequence I just linked constitutes what I would characterize as a “real” attempt to confront the issue, and you will note the complete absence of claims like “there is no such thing as knowledge” in any of the posts in question; in the absence of such claims, you will instead see plenty of diagrams and mathematical notation.)
I think the thing the framing of the sequence you link and the way most people approach this is missing something fundamental about epistemology that lets one get confused, specifically by easily forgetting that one’s knowledge is always contingent on some assumption that one may not even be able to see, and so mistakes one’s own perspective for objectivity. As for what tools understanding the problem of the criterion provides, I’d say it’s more like a mindset of correctly calibrated epistemic humility. Not to say that grokking the problem of the criterion makes you perfectly calibrated in making predictions, but to say it requires adopting a mindset that is sufficient to achieve the level of epistemic humility/update fluidity necessary to become well calibrated or, dare we say, Bayesian rational.
(Note: I realize my claim about grokking the problem of the criterion sets up a potential “no true Scotsman” situation where anyone who claims to grok the problem of the criterion and then seems to lack this capacity for update fluidity can be dismissed as not really grokking it. I’m not really looking to go that far, but I want to say that this claim is, I believe, predictive enough to make useful inferences.)
It should probably be obvious by now that I view the latter approach as far superior to the former.
Maybe it is for some people (you wouldn’t be the first person to make this claim). Others do seem to find my approach useful. Perhaps the whole point of this should be that not everyone is necessarily reasoning from the same base assumptions, and thus the ground of truth is unstable enough that what seem like reasonable explanations cannot be sure to be arbitrarily convincing. To be fair, Eliezer doesn’t miss this point, but it seems poorly enough appreciated that I often find cause to remind people of it.
If I really wanted to be pointed about it, I think you’d be less annoyed with me if you grokked the point both Eliezer and I are trying to make in different ways about how epistemology grounds out, since taken to its extreme we must accept that the same lines of reasoning don’t work for everyone on a practical level, by which I mean that even if you show someone correct math, they may yet not be convinced by it, and this is epistemically relevant and not to be dismissed since we are each performing our own reckoning of what to accept as true, no matter how much we may share in common (which, for what it’s worth, brings us right back to the core of the intentional stance and the object level concerns of the OP!).
And as for the problemist’s recursive ladder of justification? It runs straight into the hard brick wall called “evolutionary hardcoding”, and proceeds no further than that: the buck stops immediately. Evolution neither needs nor provides justification for the things it does; it merely optimizes for inclusive genetic fitness. Even attempting to apply the problemist’s tried-and-true techniques to the alien god produces naught but type and category errors; the genre of analysis preferred by the problemist finds no traction whatsoever. Thus, the problemist of the criterion is defeated, and with him so too vanishes his problem.
This feels like violent agreement with my arguments in the linked post, so I think you’re arguing against some different reading of the implications of the problem of the criterion than what it does imply. It doesn’t imply there is literally no way to ground knowledge, but that ground is not something especially connected to traditional notions of truth or facts, but rather in usefulness to the purpose of living.
This mostly comes up when we try to assess things like what does it mean for something to “be” an “agent”. We then run headlong into the grounding problem and this becomes relevant, because what it means for something to “be” an “agent” ends up connected to what end we need to categorize the world, rather than how the world actually “is”, since the whole point is that there is no fact of the matter about what “is”, only a best effort assessment of what’s useful (and one of the things that’s really useful is predicting the future, which generally requires building models that correlate to past evidence).
That’s part of the problem, not part of the solution. If evolution is optimising for genetic fitness, then it is not optimising for the ability to achieve a correct ontology … after all , a wrong but predictive model is good enough for survival.
In many ways this is the crux of things. The problem of the criterion does mean that we can’t ground knowledge in the ways we had hoped to, and that we can still ground knowledge, just in something quite a bit different from the objective: namely, in some practical purpose to which we use knowledge.
It’s not been mentioned enough, since the point has not generally sunk in.
For what it’s worth, I feel exactly the same way about the robustness of Goodhart and I’ll keep beating that drum as long as I have to. Luckily no one much objects that Goodharting is a problem, whereas everyone seems to be annoyed with epistemology that makes a fairly simple point that is counterintuitive to the experience of being able to make use of knowledge to do useful things, and thinking this is somehow contrary to the epistemological point being made that becomes relevant when you try to ground concepts rather than just use them as you find them.
There’s no observer-independent fact of the matter about whether a system “is” an agent[9]
Worth saying, I think, that this is fully generally true that there’s no observer-independent fact of the matter about whether X “is” Y. That this is true of agents is just particularly relevant to AI.
There is some relevant difference between the practical and the philosophical, where for some particular purpose the lack of “objective” being possible doesn’t matter because you’re not asking for resolution past the point where it’s relevant. As long as you are comfortable with the assumptions you’ve made and don’t wish to question them, this won’t be a concern, and you can make great practical progress despite the lack of firm grounding (as evidenced by the existence of the modern world).
Since I don’t see you mention it here, a reasonable solution to the meaninglessness of a view from nowhere yet still finding it useful is to replace it with intersubjectivity: an abstracted view of reality that aggregates over reports of many subjective experiences. For many purposes the intersubjective and objective views are functionality the same, hence you end up with people defending the metaphysics of objectivity even though they don’t make sense because they don’t have the handle for intersubjectivity they would need not make unusual metaphysical claims.