If this post is selected, I’d like to see the followup made into an addendum—I think it adds a very important piece, and it should have been nominated itself.
I think this post (and similarly, Evan’s summary of Chris Olah’s views) are essential both in their own right and as mutual foils to MIRI’s research agenda. We see related concepts (mesa-optimization originally came out of Paul’s talk of daemons in Solomonoff induction, if I remember right) but very different strategies for achieving both inner and outer alignment. (The crux of the disagreement seems to be the probability of success from adapting current methods.)
Strongly recommended for inclusion.
It’s hard to know how to judge a post that deems itself superseded by a post from a later year, but I lean toward taking Daniel at his word and hoping we survive until the 2021 Review comes around.
I can’t think of a question on which this post narrows my probability distribution.
The content here is very valuable, even if the genre of “I talked a lot with X and here’s my articulation of X’s model” comes across to me as a weird intellectual ghostwriting. I can’t think of a way around that, though.
That being said, I’m not very confident this piece (or any piece on the current state of AI) will still be timely a year from now, so maybe I shouldn’t recommend it for inclusion after all.
Ironically enough for Zack’s preferred modality, you’re asserting that even though this post is reasonable when decoupled from the rest of the sequence, it’s worrisome when contextualized.
I agree about the effects of deep learning hype on deep learning funding, though I think very little of it has been AGI hype; people at the top level had been heavily conditioned to believe we were/are still in the AI winter of specialized ML algorithms to solve individual tasks. (The MIRI-sphere had to work very hard, before OpenAI and DeepMind started doing externally impressive things, to get serious discussion on within-lifetime timelines from anyone besides the Kurzweil camp.)
Maybe Demis was strategically overselling DeepMind, but I expect most people were genuinely over-optimistic (and funding-seeking) in the way everyone in ML always is.
This is a retroactively obvious concept that I’d never seen so clearly stated before, which makes it a fantastic contribution to our repertoire of ideas. I’ve even used it to sanity-check my statements on social media. Well, I’ve tried.
This reminds me of That Alien Message, but as a parable about mesa-alignment rather than outer alignment. It reads well, and helps make the concepts more salient. Recommended.
This makes a simple and valuable point. As discussed in and below Anna’s comment, it’s very different when applied to a person who can interact with you directly versus a person whose works you read. But the usefulness in the latter context, and the way I expect new readers to assume that context, leads me to recommend it.
I liked the comments on this post more than I liked the post itself. As Paul commented, there’s as much criticism of short AGI timelines as there is of long AGI timelines; and as Scott pointed out, this was an uncharitable take on AI proponents’ motives.
Without the context of those comments, I don’t recommend this post for inclusion.
I’ve referred and linked to this post in discussions outside the rationalist community; that’s how important the principle is. (Many people understand the idea in the domain of consent, but have never thought about it in the domain of epistemology.)
As mentioned in my comment, this book review overcame some skepticism from me and explained a new mental model about how inner conflict works. Plus, it was written with Kaj’s usual clarity and humility. Recommended.
I stand by this piece, and I now think it makes a nice complement to discussions of GPT-3. In both cases, we have significant improvements in chunking of concepts into latent spaces, but we don’t appear to have anything like a causal model in either. And I’ve believed for several years that causal reasoning is the thing that puts us in the endgame.
(That’s not to say either system would still be safe if scaled up massively; mesa-optimization would be a reason to worry.)
I never found a Coherence Therapy practitioner, but I found a really excellent IFS practitioner who’s helped me break down many of my perpetual hangups in ways compatible with this post.
In particular, one difference between the self-IFS I’d attempted before is that I’d previously tried to destroy some parts as irrational or hypocritical, where the therapist was very good at being non-judgmental towards them. That approach paid better dividends.
Can’t update on #4. Of course a rapidly growing new strain will have a negligible impact on total numbers early on; it’s a question of whether it will dominate the total numbers in a few months.
Remind me which bookies count and which don’t, in the context of the proofs of properties?
If any computable bookie is allowed, a non-Bayesian is in trouble against a much larger bookie who can just (maybe through its own logical induction) discover who the bettor is and how to exploit them.
[EDIT: First version of this comment included “why do convergence bettors count if they don’t know the bettor will oscillate”, but then I realized the answer while Abram was composing his response, so I edited that part out. Editing it back in so that Abram’s reply has context.]
Not the most important thing, but Adler and Colbert’s situations feel rather different to me.
Colbert is bubbled with a small team in order to provide mass entertainment to the nation… just like sports teams, which you endorse.
Adler is partying for his own benefit.
To steelman the odds’ consistency (though I agree with you that the market isn’t really reflecting careful thinking from enough people), Biden is farther ahead in the 538 projection now than he was before, but on the other hand, Trump has completely gotten away with refusing to commit to a peaceful transfer of power. Even if that’s not the most surprising thing in the world (how far indeed we have fallen), it wasn’t at 100% two months ago.