(The underlying HTML is <a href="http://I think that working on truthful LMs has a comparative advantage in worlds where: We have around 10-40 years until transformative AI Transformative AI is built using techniques that resemble modern deep learning There is a slow takeoff Alignment does not require vastly more theoretical insight (but may require some) Our current picture of the risks posed by transformative AI is incomplete">this thread</a>. I am also surprised that this results in clicking being a no-op, rather than a “functional” link that leads to your browser’s could-not-resolve-host page.)
Zack_M_Davis(Zack M. Davis)
Karma: 10,239
If I was Scott Alexander or Zvi I’d comb through those papers and wring out insight.
Huh? What’s stopping you? How are Scott or Zvi relevant here, at all?
The idea of “general education” is that it’s good for ordinary people to learn lots of things that were discovered by specialists: partially because we value knowledge for its own sake, but also because it’s hard to tell in advance what knowledge will end up being useful. In principle, you could reject the idea that general education is generally good, but if you’re going to be consistent about what that entails, I don’t think anyone who reads this website actually wants to go there. Do I really need to know that matter is made of “atoms”, that have a “nucleus” composed of “uncharged” “neutrons” and “positively-charged” “protons”, surrounded by “negatively-charged” “electrons”? When have I ever used any of this stuff to make money? Do I really need to know that the world is round? &c.
(Incidentally, Dick Grayson is the best Robin.)
I don’t think we disagree about anything substantive, and I don’t expect Daniel to disagree about anything substantive after reading this. It’s just—
I agree that the OP’s talking of PONR as a point in time doesn’t make sense; a charitable read is that [...]
I don’t think we should be doing charitable readings at yearly review time! If an author uses a toy model to clarify something, we want the post to say “As a clarifying toy model [...]” rather than making the readers figure it out.
(Thanks for this—it’s important that critiques get counter-critiqued, and I think that process is stronger when third parties are involved, rather than it just being author vs. critic.)
The reason that doesn’t satisfy me is because I expect the actual calculus of “influence” and “control” in real-world settings to be sufficiently complicated that there’s probably not going to be any usefully identifiable “point of no return”. On the contrary, if there were an identifiable PONR as a natural abstraction, I think that would be a surprising empirical fact about the world in demand of deeper explanation—that the underlying calculus of influence would just happen to simplify that way, such that you could point to an event and say, “There—that’s when it all went wrong”, rather than there just being (say) a continuum of increasingly detailed possible causal graphs that you can compute counterfactuals with respect to (with more detailed graphs being more expensive to learn but granting more advanced planning capabilities).
If you’re pessimistic about alignment—and especially if you have short timelines like Daniel—I think most of your point-of-no-return-ness should already be in the past. When, specifically? I don’t see any reason to expect there to be a simple answer. You lost some measure when OpenAI launched; you lost some measure when Norbert Weiner didn’t drop everything to work on the alignment problem in 1960; you lost some measure when Samuel Butler and Charles Babbage turned out to not be the same person in our timeline; you lost some measure when the ancient Greeks didn’t discover natural selection …
The post does have a paragraph mentioning continuous loss of influence and already-lost influence in the past (“Of course, influence over the future might not disappear all on one day …”), but the reason this doesn’t satisfy me as a critic is because it seems to be treated as an afterthought (“We should keep these possibilities in mind as well”), rather than being the underlying reality to which any putative “PONR” would be a mere approximation. Instead, the rhetorical emphasis is on PONR as if it were an event: “The Date of AI Takeover Is Not the Day the AI Takes Over”. (And elsewhere, Daniel writes about “PONR-inducing tasks”.)
But in my philosophy, “the date” and “the day” of the title are two very different kinds of entities that are hard to talk about in the same sentence. The day AI takes over actually is a physical event that happens on some specific, definite date: nanobots disassemble the Earth, or whatever. That’s not subjective; the AI historian-subprocess of the future will record a definitive timestamp of when it happened. In contrast, “the date” of PONR is massively “subjective” depending on further assumptions; the AI historian-subprocesses of the future will record some sort of summary of the decision-relevant results of a billion billion ancestor simulations, but the answer is not going to fit in a 64-bit timestamp.
Maybe to Daniel, this just looks like weirdly unmotivated nitpicking (“not super relevant to the point [he] was trying to make”)? But it feels like a substantive worldview difference to me.
We rationalists are very good at making predictions, and the best of us, such as Scott Alexander
This weird self-congratulatory tribalism is a completely unnecessary distraction from an otherwise informative post. Are “we” unusually good at making predictions, compared to similarly informed and motivated people who haven’t pledged fealty to our social group? How do you know?
Scott Alexander is a justly popular writer, and I’ve certainly benefitted from reading many of his posts, but it seems cultish and bizarre to put him on a pedestal as “the best” of “us” like this as the first sentence of a post that has nothing to do with him.
while my post may have given the misleading impression [...] I didn’t fall for that fallacy myself.
I reach for this “bad writing” excuse sometimes, and sometimes it’s plausible, but in general, I’m wary of the impulse to tell critics after the fact, “I agree, but I wasn’t making that mistake,” because I usually expect that if I had a deep (rather than halting, fragmentary, or inconsistent) understanding of the thing that the critic was pointing at, I would have anticipated the criticism in advance and produced different text that didn’t provide the critic with the opportunity, such that I could point to a particular sentence and tell the would-be critic, “Didn’t I already adequately address this here?”
lays out his reasoning very clearly [...] the bar for Peer Review
I mean, it’s not really my reasoning. This academic-paper-summary-as-blog-post was basically me doing “peer” review for Synthese (because I liked the paper, but was annoyed that someone would publish a paper based on computer simulations without publishing their code).
This post is making a valid point (the time to intervene to prevent an outcome that would otherwise occur, is going to be before the outcome actually occurs), but I’m annoyed with the mind projection fallacy by which this post seems to treat “point of no return” as a feature of the territory, rather than your planning algorithm’s map.
(And, incidentally, I wish this dumb robot cult still had a culture that cared about appreciating cognitive algorithms as the common interest of many causes, such that people would find it more natural to write a post about “point of no return”-reasoning as a general rationality topic that could have all sorts of potential applications, rather than the topic specifically being about the special case of the coming robot apocalypse. But it’s probably not fair to blame Kokotajlo for this.)
The concept of a “point of no return” only makes sense relative to a class of interventions. A 1 kg ball is falling at 9.8 m/s². When is the “point of no return” at which the ball has accelerated enough such that it’s no longer possible to stop it from hitting the ground?
The problem is underspecified as stated. If we add the additional information that your means of intervening is a net that can only trap objects falling with less than X kg⋅m/s² of force, then we can say that the point of no return happens at X/9.8 seconds. But it would be weird to talk about “the second we ball risk reducers lose the ability to significantly reduce the risk of the ball hitting the ground” as if that were an independent pre-existing fact that we could use to determine how strong of a net we need to buy, because it depends on the net strength.
Maybe I’m explaining it badly? I’m trying to point to the Judea Pearl thing in my own words. The claim is not that causality “just is” conditional independence relationships. (Pearl repeatedly explicitly disclaims that causal concepts are different from statistical concepts and require stronger assumptions.) Do you have an issue with the graph formalism itself (as an explanation of the underlying reality of how causality and counterfactuals work), separate from practical concerns about how one would learn a particular graph?
I think this is a solved problem. Are you familiar with the formalization of causality in terms of Bayesian networks? (You have enough history on this website that you’ve probably heard of it!)
Make observations using sensors. Abstract your sensory data into variables: maybe you have a
weathervariable with possible valuesRAINYandSUNNY, asprinklervariable with possible valuesONandOFF, and asidewalkvariable with possible valuesWETandDRY. As you make more observations, you can begin to learn statistical relationships between your variables: maybeweatherandsprinklerare independent, but conditionally dependent given the value ofsprinkler. It turns out that you can summarize this kind of knowledge in the form of a directed graph: weather → sidewalk ← sprinkler. (I’m glossing over a lot of details: a graph represents conditional-independence relationships in the joint distribution over your variables, but the distribution doesn’t uniquely specify a graph.)But once you’ve learned this graphical model representing the probabilistic relationships between variables which represent abstractions over your sensory observations, then you can construct a similar model that fixes a particular variable to have a particular value, but keeps everything else the same.
Why would you do that? Because such an altered model is useful for decisionmaking if the system-that-you-are is one of the variables in the graph. The way you compute which decision to output is based on a model of how the things in your environment depend on your decision, and it’s possible to learn such a model from previous observations, even though you can’t observe the effects of your current decision in advance of making it.
And that’s what counterfactuals are! I don’t think this is meaningfully circular: we’ve described how the system works in terms of lower-level components. (I’ve omitted a lot of details, but we can totally write computer programs that do this stuff.)
As a starting point, it might help to understand exactly where people’s naïve intuitions about why corrigibility should be easy, clash with the technical argument that it’s hard.
For me, the intuition goes like this: if I wanted to spend some fraction of my effort helping dolphins in their own moral reference frame, that seems like something I could do. I could give them gifts that I can predict that they’d like (like tasty fish or a water purifier), and be conservative when I couldn’t figure out what dolphins “really wanted”, and be eager to accept feedback when the dolphins wanted to change how I was trying to help. If my superior epistemic vantage point let me predict that the way dolphins would respond to gifts, would depend on the details like what order the gifts were presented in, I might compute an average over possible gift-orderings, or I might try to ask the dolphins to clarify, but I definitely wouldn’t tile the lightcone with tiny molecular happy-dolphin sculptures, because I can tell that’s not what dolphins want under any sensible notion of “want”.
So what I’d like to understand better is, where exactly does the analogy between “humans being corrigible to dolphins (in the fraction of their efforts devoted to helping dolphins)” and “AI being corrigible to humans” break, such that I haven’t noticed yet because empathic inference between mammals still works “well enough”, but won’t work when scaled to superintelligence? When I try to think of gift ideas for dolphins, am I failing to notice some way in which I’m “selfishly” projecting what I think dolphins should want onto them, or am I violating some coherence axiom?
Thanks for this analysis! However—
if you start with a uniform prior over [0,1] for the probability of the coin coming up heads
I’m not. The post specifies “a coin that is either biased to land Heads 2/3rds of the time, or Tails 2/3rds of the time”—that is (and maybe I should have been more explicit), I’m saying our prior belief about the coin’s bias is just the discrete distribution {”1/3 Heads, 2⁄3 Tails”: 0.5, “2/3 Heads, 1⁄3 Tails”: 0.5}.
I agree that a beta prior would be more “realistic” in the sense of applying to a wider range of scenarios (your uncertainty about a parameter is usually continuous, rather than “it’s either this, or it’s that, with equal probability”), but I wanted to make the math easy on myself and my readers.
Comment on “Deception as Cooperation”
I’m not advocating lying.
I understand that. I cited a Sequences post that has the word “lies” in the title, but I’m claiming that the mechanism described in the cited posts—that distortions on one topic can spread to both adjacent topics, and to people’s understanding of what reasoning looks like—can apply more generally to distortions that aren’t direct lies.
Omitting information can be a distortion when the information would otherwise be relevant. In “A Rational Argument”, Yudkowsky gives the example of an election campaign manager publishing survey responses from their candidate, but omitting one question which would make their candidate look bad, which Yudkowsky describes as “cross[ing] the line between rationality and rationalization” (!). This is a very high standard—but what made the Sequences so valuable, is that they taught people the counterintuitive idea that this standard exists. I think there’s a lot of value in aspiring to hold one’s public reasoning to that standard.
Not infinite value, of course! If I knew for a fact that Godzilla will destroy the world if I cite a book that I would otherwise would have cited as genuinely relevant, then fine, for the sake of the sake of the world, I can not cite the book.
Maybe we just quantitatively disagree on how tough Godzilla is and how large the costs of distortions are? Maybe you’re happy to throw Sargon of Akkad under the bus, but when Steve Hsu is getting thrown under the bus, I think that’s a serious problem for the future of humanity. I think this is actually worth a fight.
With my own resources and my own name (and a pen name), I’m fighting. If someone else doesn’t want to fight with their name and their resources, I’m happy to listen to suggestions for how people with different risk tolerances can cooperate to not step on each other’s toes! In the case of the shared resource of this website, if the Frontpage/Personal distinction isn’t strong enough, then sure, “This is on our Banned Topics list; take it to /r/TheMotte, you guys” could be another point on the compromise curve. What I would hope for from the people playing the sneaky consequentialist image-management strategy, is that you guys would at least acknowledge that there is a conflict and that you’ve chosen a side.
might fill their opinion vacuum with false claims from elsewhere, or with true claims
For more on why I think not-making-false-claims is vastly too low of a standard to aim for, see “Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think” and “Heads I Win, Tails?—Never Heard of Her”.
I agree that offense-takers are calibrated against Society-in-general, not particular targets.
As a less-political problem with similar structure, consider ransomware attacks. If an attacker encrypts your business’s files and will sell you the encryption key for 10 Bitcoins, do you pay (in order to get your files back, as common sense and causal decision theory agree), or do you not-pay (as a galaxy-brained updateless-decision-theory play to timelessly make writing ransomware less profitable, even though that doesn’t help the copy of you in this timeline)?
It’s a tough call! If your business’s files are sufficiently important, then I can definitely see why you’d want to pay! But if someone were to try to portray the act of paying as pro-social, that would be pretty weird. If your Society knew how, law-abiding citizens would prefer to coordinate not to pay attackers, which is why the U.S. Treasury Department is cracking down on facilitating ransomware payments. But if that’s not an option …
our behavior [...] punishment against us [...] some other entity that we shouldn’t care much about
If coordinating to resist extortion isn’t an option, that makes me very interested in trying to minimize the extent to which there is a collective “us”. “We” should be emphasizing that rationality is a subject matter that anyone can study, rather than trying to get people to join our robot cult and be subject to the commands and PR concerns of our leaders. Hopefully that way, people playing a sneaky consequentialist image-management strategy and people playing a Just Get The Goddamned Right Answer strategy can at least avoid being at each other’s throats fighting over who owns the “rationalist” brand name.
But there are systemic reasons why Society gets told that hypotheses about genetically-mediated group differences are offensive, and mostly doesn’t (yet?) get told that technological forecasting is offensive. (If some research says Ethnicity E has higher levels of negatively-perceived Trait T, then Ethnicity E people have an incentive to discredit the research independently of its truth value—and people who perceive themselves as being in a zero-sum conflict with Ethnicity E have an incentive to promote the research independently of its truth value.)
Steven and his coalition are betting that it’s feasible to “hold the line” on only censoring the hypotheses are closely tied to political incentives like this, without doing much damage to our collective ability to think about other aspects of the world. I don’t think it works as well in practice as they think it does, due to the mechanisms described in “Entangled Truths, Contagious Lies” and “Dark Side Epistemology”—you make a seemingly harmless concession one day, and five years later, you end up claiming with perfect sincerity that dolphins are fish—but I don’t think it’s right to dismiss the strategy as fantasy.
The relevant actors aren’t consciously being strategic about it, but I think their emotions are sensitive to whether the threat of being offended seems to be working. That’s what the emotions are for, evolutionarily speaking. People are innately very good at this! When I babysit a friend’s unruly 6-year-old child who doesn’t want to put on her shoes, or talk to my mother who wishes I would call more often, or introspect on my own rage at the abject cowardice of so-called “rationalists”, the functionality of emotions as a negotiating tactic is very clear to me, even if I don’t have the same kind of deliberative control over my feelings as my speech (and the child and my mother don’t even think of themselves as doing game theory at all).
(This in itself doesn’t automatically negate your concerns, of course, but I think it’s an important modeling consideration: animals like Godzilla may be less incentivizable than Homo economicus, but they’re more like Homo economicus than a tornado or an avalanche.)
What’s with the neglect of Richard J. Herrnstein?! His name actually comes first on the cover!
“Kill everyone” seems like it should be “easy”, because there are so many ways to do it: humans only survive in environments with a specific range of temperatures, pressures, atmospheric contents, availability of human-digestible food, &c.