Alex Turner, Oregon State University PhD student working on AI alignment. Reach me at turneale[at]oregonstate[dot]edu.
TurnTrout
Karma: 7,316
I’ve edited the post to incorporate a bit of wisdom I’ve gained over the last year.
Use cloze deletions for Anki cards.
Cloze deletions hide a small part of the text, and make you remember it given the rest of the content. They are fast to make and fast to review.
The vast, vast majority of my new cards are cloze now, and they weren’t when I first wrote this post. I think that was a mistake.
In the post, I used my representation theory cards as examples.
What a bad choice. Those cards were too long. I ended up forgetting them (although not really many other cards from that point in time). Out of all the examples I could have chosen, why those?!
The problem was that the cards were too long. I made myself remember too much on each pass.
I now read using MarginNote on iPad with a nice cloze deletion → Anki export pipeline.
I find it concerning that you felt the need to write “This is not at all a criticism of the way this post was written. I am simply curious about my own reaction to it” (and still got downvoted?).
For my part, I both believe that this post contains valuable content and good arguments, and that it was annoying / rude / bothersome in certain sections.
The biggest disconnect is that this post is not a proposal for how to solve corrigibility. I’m just thinking about what corrigibility is/should be, and this seems like a shard of it—but only a shard. I’ll edit the post to better communicate that.
So, your points are good, but they run skew to what I was thinking about while writing the post.
I skip over those pragmatic problems because this post is not proposing a solution, but rather a measurement I find interesting.
Flag: even being less worried is an additional inference not demanded by the form of the contrapositive of “if you’re worried, you should get your booster” . But that’s just a nitpick over how literally we’re meaning logical equivalence.
Formalizing Policy-Modification Corrigibility
I’m not sure how often I subvocalize to think thoughts. Often I have trouble putting a new idea into words just right, which means the raw idea essence came before the wordsmithing. But other times it feels like I’m synchronously subvocalizing as I brainstorm
Reading EY’s dath ilan glowfics, I can’t help but think of how poor English is as a language to think in. I wonder if I could train myself to think without subvocalizing (presumably it would be too much work to come up with a well-optimized encoding of thoughts, all on my own, so no new language for me). No subvocalizing might let me think important thoughts more quickly and precisely.
OK, I’ll bite on EY’s exercise for the reader, on refuting this “what-if”:
Humbali: Then here’s one way that the minimum computational requirements for general intelligence could be higher than Moravec’s argument for the human brain. Since, after, all, we only have one existence proof that general intelligence is possible at all, namely the human brain. Perhaps there’s no way to get general intelligence in a computer except by simulating the brain neurotransmitter-by-neurotransmitter. In that case you’d need a lot more computing operations per second than you’d get by calculating the number of potential spikes flowing around the brain! What if it’s true? How can you know?
Let’s step back and consider what kind of artifact the brain is. The human brain was “found” by evolution via via a selection process over a rather limited amount of time (between our most recent clearly-dumb ancestor, and anatomically modern humans). We have a local optimization process which optimizes over a relatively short timescale. This process found a brain which implements a generally intelligent algorithm.
In high-dimensional non-convex optimization, we have a way to describe algorithms found by a small amount of local optimization: “not even close to optimal.” (Humans aren’t even at a local optimum for inclusive-genetic-fitness due to our being mesa-optimizers.) But if the brain’s algorithm isn’t optimal, it trivially can’t be the only algorithm that can produce general intelligence. Indeed, I would expect the fact that evolution found our algorithm at all to indicate that there were many possible such algorithms.
There are many generally intelligent algorithms, and our brain only implements one, and it’s just not going to be true that all of the others—or even the ones most likely to be discovered by AI researchers—are only implementable using (simulated) neurotransmitters.
no-one has the social courage to tackle the problems that are actually important
I would be very surprised if this were true. I personally don’t feel any social pressure against sketching a probability distribution over the dynamics of an AI project that is nearing AGI.
I would guess that if people aren’t tackling Hard Problems enough, it’s not because they lack social courage, but because 1) they aren’t running a good-faith search for Hard Problems to begin with, or 2) they came up with reasons for not switching to the Hard Problems they thought of, or 3) they’re wrong about what problems are Hard Problems. My money’s mostly on (1), with a bit of (2).
But it does imply that you should not expect that this community will ever be willing to agree that corrigibility, or any other alignment problem. has been solved.
Noting that I strongly disagree but don’t have time to type out arguments right now, sorry. May or may not type out later.
I’m not done with the post yet, but this part really jumped out at me.
Second point is a doozy, and it’s that you can’t look at intent when diagnosing frame control. As in, “what do they mean to do” should be held separate from “what are the effects of what they’re doing”—which I know is counter to almost every good lesson about engaging with people charitably.
I think you’re right in a narrow way but mostly wrong here. The narrow way in which you seem right is that (someone’s intent) and (someone’s impact) are indeed separate quantities. But someone having good intent—or someone seeming to have good intent, if one can generally discern this with above-random accuracy—means that their actions are more likely optimized to have good effects, and so these two quantities are generally correlated.
In this section, your language conflates two possible scenarios (by my reading). In the first, we condition on “leader X seems to have good intent.” In the second, we condition on “me and my friends are talking about how leader X is deeply flawed and perceptive, and the things he did that hurt people were either for their own good, or an unintentional byproduct of him genuinely trying to do good.” The second scenario is very different.
In the first scenario, we surely have P(bad frame-controller | leader X seems to have good intent) < P(bad frame controller | leader X does not seem to have good intent).
In the second scenario, P(bad frame-controller | leader X is flawed but seems well-meaning, [other red flags]) should still be less than P(bad frame-controller | leader X is flawed but does not seem well-meaning, [other red flags]), but I think that the probability of something bad going on should be high either way.
And so as far as I can tell, intent does matter for the beliefs you arrive at, and I am very very wary that this post claims otherwise in. “Frame control” has the potential to be an argumentative superweapon.
This all might sound pretty dark, like I’m painting a reality where you might go around squinting at empathetic, open, caring people who have zero ill intent whatsoever and trying to figure out how they are ‘actually bad.’ And this is kind of true, but if only because “I am an empathetic, open, caring person with zero ill intent” is exactly the kind of defense actual frame-controllers inhabit.
Surely we have P(awful frame control | They seem like an empathetic, open, caring person with zero ill intent) << P(awful frame control | They do not seem like an empathetic, open, caring person with zero ill intent)?
The vast majority of good people with good intent aren’t doing any significant kind of frame control; my point is just that “good person with good intent” should not be considered a sufficient defense if there seems to be other elements of frame control present.
But then you back off the original claim that you can’t look at intent, and merely say that good intent is not sufficient to conclude that they aren’t doing this horrible frame control. OK. I agree. But this feels like a motte and bailey.
Rereading the portion in question to make sure I’m not missing something, you write:
And so, when evaluating frame control, you have to throw out intent. The question is not “does this person mean to control my frame,” the question is “is this person controlling my frame?”. This is especially true for diagnosing frame control that you’re inside of, because the first defense a frame controller uses is the empathy you hold for them.
I think “be on the lookout for ways people can weaponize your empathy” seems wiser to me than “throw out intent.”
5x Delta transmissibility seems insane. I… Don’t have gears level models which prohibit such a thing, but wouldn’t that give it a reproduction rate between 25 and 45?! Even chickenpox is only 12.
Why should we think a lifetime limit exists?
I think you should distrust the other person for other reasons. They may have ulterior motives, for example—as you wrote, they may be lying. There are many social reasons for them to make claims like that. Whatever lies we tell ourselves, they are usually not conscious and explicit.
Which is why I found it strange that you stated this claim without further explanation.
This fallible-other logic should apply equally to the self.
Why?
What’s up with biological hermaphrodite species? My first reaction was, “no way, what about the specialization benefits from sexual dimorphism?”
There are apparently no hermaphrodite mammal or bird species, which seems like evidence supporting my initial reaction. But there are, of course, other hermaphrodite species—maybe they aren’t K-strategists, and so sexual dimorphism and role specialization isn’t as important?
Addendum: One lesson to take away is that quantilization doesn’t just depend on the base distribution being safe to sample from unconditionally. As the theorems hint, quantilization’s viability depends on base(plan | plan doing anything interesting) also being safe with high probability, because we could (and would) probably resample the agent until we get something interesting. In this post’s terminology, A := {safe interesting things}, B := {power-seeking interesting things}, C:= A and B and {uninteresting things}.
I’ve started commenting on this discussion on a Google Doc. Here are some excerpts:
During this step, if humanity is to survive, somebody has to perform some feat that causes the world to not be destroyed in 3 months or 2 years when too many actors have access to AGI code that will destroy the world if its intelligence dial is turned up.
Contains implicit assumptions about takeoff that I don’t currently buy:
Well-modelled as binary “has-AGI?” predicate;
(I am sympathetic to the microeconomics of intelligence explosion working out in a way where “Well-modelled as binary “has-AGI?” predicate is true, but I feel uncertain about the prospect)
Somehow rules out situations like: We have somewhat aligned AIs which push the world to make future unaligned AIs slightly less likely, which makes the AI population more aligned on average; this cycle compounds until we’re descending very fast into the basin of alignment and goodness.
This isn’t my mainline or anything, but I note that it’s ruled out by Eliezer’s model as I understand it.
Some other internal objections are arising and I’m not going to focus on them now.
Every AI output effectuates outcomes in the world.
Right but the likely domain of cognitive discourse matters. Pac-Man agents effectuate outcomes in the world, but their optimal policies are harmless. So the question seems to hinge on when the domain of cognition shifts to put us in the crosshairs of performant policies.
This doesn’t mean Eliezer is wrong here about the broader claim, but the distinction deserves mentioning for the people who weren’t tracking it. (I think EY is obviously aware of this)If you knew about the things that humans are using to reuse their reasoning about chipped handaxes and other humans, to prove math theorems, you would see it as more plausible that proving math theorems would generalize to chipping handaxes and manipulating humans.
Could we have observed it any other way? Since we surely wouldn’t have been selected for proving math theorems, we wouldn’t have a native cortex specializing in math. So conditional on considering things like theorem-proving at all, it has to reuse other native capabilities.
More precisely, one possible mind design which solves theorems also reasons about humans. This is some update from whatever prior, towards EY’s claim. I’m considering whether we know enough about the common cause (evolution giving us a general-purpose reasoning algorithm) to screen off/reduce the Theorems → Human-modelling update.
So here’s one important difference between humans and neural networks: humans face the genomic bottleneck which means that each individual has to rederive all the knowledge about the world that their parents already had. If this genetic bottleneck hadn’t been so tight, then individual humans would have been significantly less capable of performing novel tasks.
Thanks, Richard—this is a cool argument that I hadn’t heard before.
You will systematically overestimate how much easier, or how far you can push the science part without getting the taking-over-the-world part, for as long as your model is ignorant of what they have in common.
OK, it’s a valid point and I’m updating a little, under the apparent model of “here’s a set of AI capabilities, linearly ordered in terms of deep-problem-solving, and if you push too far you get taking-over-the-world.” But I don’t see how we get to that model to begin with.
While this may help in that specific scenario, I’m worried about e.g. self-induced value drift from self-consistency pressure (“I’m a little more of the kind of person who is willing to type things I know to be misleading”).