Communications lead at MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s.
Rob Bensinger
Karma: 17,624
I disagree with the reasoning in this reply to Tom (and in nostalgebraist’s reply). If stuff like this is net-positive to post about on LW, the chain of reasoning to arrive at that conclusion seems to me like it has to look different from the reasoning in these comments. E.g.:
“It seems unlikely that comments on lesswrong speed up capabilities research”—If “unlikely” here meant “only 40% likely”, then it would obviously be a bad idea to post a capabilities insight. The degree of unlikeliness obviously matters, and it has to be weighed against the expected benefit of sharing the insight.
At the policy level, “How does this weigh against the expected benefits?” has to take into account that the quality and rarity of LWers’ insights is likely to vary a lot by individual and across time; and it has to take into account that the risk level of LW posts is very correlated with the benefit level. In the worlds where there’s not much future risk of a LWer someday posting a dangerous capabilities insight, there’s also less future benefit to LW posts, since we’re probably not generating many useful ideas in general (especially about AGI and AGI alignment).
“it seems important to know which problems in capabilities research can be alleviate”—What specific safety progress does this enable? (Maybe there’s something, but ‘it seems safety-relevant because it’s a fact about ML’ seems to prove too much. What’s the actual implicit path by which humanity ends up safely navigating the AGI transition?)
‘We should require a high bar before we’re willing to not-post potentially-world-destroying information to LW, because LW has a strong commitment to epistemic rationality’ seems like an obviously terrible argument to me. People should not post stuff to the public Internet that destroys the world just because the place they’re posting is a website that cares about Bayesianism and belief accuracy.
(Note that this is a high-level point I’m making about the kind of arguments being made here, not about the object-level question.)
Thanks for the update, Ajeya! I found the details here super interesting.
I already thought that timelines disagreements within EA weren’t very cruxy, and this is another small update in that direction: I see you and various MIRI people and Metaculans give very different arguments about how to think about timelines, and then the actual median year I tend to hear is quite similar.
(And also, all of the stated arguments on all sides continue to seem weak/inconclusive to me! So IMO there’s not much disagreement, and it would be very easy for all of us to be wrong simultaneously. My intuition is that it would be genuinely weird if AGI is much more than 70 years away, but not particularly weird if it’s 1 year away, 10 years away, 60 years away, etc.)
I think the main value of in-depth timelines research and debate has been that it reveals disagreements about other topics (background views about ML, forecasting methodology, etc.).
A blurb for the book “The Feeling of Value”:
This revolutionary treatise starts from one fundamental premise: that our phenomenal consciousness includes direct experience of value. For too long, ethical theorists have looked for value in external states of affairs or reduced value to a projection of the mind onto these same external states of affairs. The result, unsurprisingly, is widespread antirealism about ethics.
In this book, Sharon Hewitt Rawlette turns our metaethical gaze inward and dares us to consider that value, rather than being something “out there,” is a quality woven into the very fabric of our conscious experience, in a highly objective way. On this view, our experiences of pleasure and pain, joy and sorrow, ecstasy and despair are not signs of value or disvalue. They are instantiations of value and disvalue. When we feel pleasure, we are feeling intrinsic goodness itself. And it is from such feelings, argues Rawlette, that we derive the basic content of our normative concepts—that we understand what it means for something to be intrinsically good or bad.
Rawlette thus defends a version of analytic descriptivism. And argues that this view, unlike previous theories of moral realism, has the resources to explain where our concept of intrinsic value comes from and how we know when it objectively applies, as well as why we sometimes make mistakes in applying it. She defends this view against G. E. Moore’s Open Question Argument as well as shows how these basic facts about intrinsic value can ground facts about instrumental value and value “all things considered.” Ultimately, her view offers us the possibility of a robust metaphysical and epistemological justification for many of our strongest moral convictions.
My reply: Sounds descriptively false? I prefer lots of things that aren’t a matter of valenced experience.
You don’t need to have “direct experience” of all moral properties in order to be a moral realist, any more than you need direct experience of porcupines in order to be a porcupine realist. You can just acknowledge that moral knowledge is inferred indirectly, the same as our knowledge of most things. Seems like a classic case of philosophers reaching weird conclusions because they’re desperate for certainty (rather than embracing Bayesian/probabilistic ways of thinking about stuff).
Likewise, you don’t need direct experience or certainty in order to reconcile “is” and “ought”. Just accept that “ought” facts look like hypothetical imperatives that we happen to care about a lot, or look like rules-of-a-game that we happen to deeply endorse everyone always playing. No deep riddles are created by the fact that “what is a legal move in chess?” is not reducible to conjunctions of claims about our universe’s boundary conditions and laws of physics; we just treat chess-claims like math/logic claims and carry on with our lives. Treating our moral claims (insofar as they’re coherent and consistent) in the same way raises no special difficulties.
This also feels to me like an example of the common philosopher-error “notice a super important fact about morality, and rush (in your excitement) to conclude that this must therefore be the important fact about morality, the one big thing morality is About”.
Human morality is immensely complicated, because human brains are a complicated, incoherent mishmash of innumerable interacting preferences and experiences. Valenced experience indeed seems to be a super important piece of that puzzle, but we can acknowledge that without pretending that it Solves Everything or Exhausts The Phenomenon.
I suspect moral philosophy would have made a lot more progress by now if philosophers spent more of their time adding to the pool of claims about morality (so we can build a full understanding of what phenomenon we need to explain / account for in the first place), and less time trying to reduce all of those claims to a single simple principle.
In principle, I love theorizing and philosophizing about this stuff. In practice, seems like people have a strong tendency to fall in love with the first Grand Theory of Everything they discover in this domain, causing progress to stagnate (and unrealistic views to proliferate) relative to if we had more modest goals. Less “try to reduce all of morality to virtue cultivation”, more “try to marginally improve our understanding of what virtue cultivation consists in”.
Some added context for this list: Nate and Eliezer expect the first AGI developers to encounter many difficulties in the “something forces you to stop and redesign (and/or recode, and/or retrain) large parts of the system” category, with the result that alignment adds significant development time.
By default, safety-conscious groups won’t be able to stabilize the game board before less safety-conscious groups race ahead and destroy the world. To avoid this outcome, humanity needs there to exist an AGI group that…
is highly safety-conscious.
has a large resource advantage over the other groups, so that it can hope to reach AGI with more than a year of lead time — including accumulated capabilities ideas and approaches that it hasn’t been publishing.
has adequate closure and opsec practices, so that it doesn’t immediately lose its technical lead if it successfully acquires one.
The magnitude and variety of difficulties that are likely to arise in aligning the first AGI systems also suggests that failure is very likely in trying to align systems as opaque as current SotA systems; and suggests an AGI developer likely needs to have spent preceding years deliberately steering toward approaches to AGI that are relatively alignable; and it suggests that we need to up our game in general, approaching the problem in ways that are closer to the engineering norms at (for example) NASA, than to the engineering norms that are standard in ML today.
I’d be more interested in a project to review the Sequences, have savvy people weigh in on which parts they think are more probable vs. less probable (indexed to a certain time, since views can change), and display their assessments in a centralized, easy-to-navigate way.
I want to say that trying to purify the Sequences feels… boring? Compared to engaging with the content, debating it, vetting it, updating it, improving it, etc.
And I worry that attempts to cut out the less-certain parts will also require cutting out the less-legible parts, even where these contain important content to interact with. The Sequences IMO are a pretty cohesive thing; many insights can be extracted without swallowing the thing wholesale, but a large portion of it will be harder to understand if you don’t read the full thing at all. (Or don’t read enough of the core ‘kinda-illegible’ pieces.)
Maybe what I want is more like a new ‘Critiques of the Sequences’ sequence, or a ‘Sequences + (critiques and responses)’ alt-sequence. Since responding to stuff and pointing at problems seems obviously productive to me, whereas I’m more wary of purification.
Yeah, this might be the comment I would have written if I were better at articulating my thoughts.
❤
That makes sense to me, and it updates me toward your view on the kid-having thing. (Which wasn’t the focus of my comment, but is a thing I was less convinced of before.) I feel sad about that having happened. :( And curious about whether I (or other people I know) are making a similar mistake.
(My personal state re kids is that it feels a bit odd/uncanny when I imagine myself having them, and I don’t currently viscerally feel like I’m giving something up by not reproducing. Though if I lived for centuries, I suspect I’d want kids eventually in the same way I’d want to have a lot of other cool experiences.)
I feel kinda confused about how “political” my AGI-beliefs are. The idea of dying to AGI feels very sensorily-real to me — I feel like my brain puts it in the same reference class as ‘dying from a gunshot wound’, which is something I worry about at least a little in my day-to-day life (even though I live in a pretty safe area by US megalopolis standards), have bad dreams about, semi-regularly idly imagine experiencing, etc. I don’t know how that relates to the “is this belief political?” question, or how to assess that.
Regardless, I like this:
I am worried that people may say this, intend it with some portion of themselves, but have their animal hear “I’m going to abandon these goals and live in belief-in-beliefs.”
I’d also assume by default that ‘the animal discounts more heavily than the philosopher does’ is a factor here...? And/or ‘the animal is better modeled on this particular question as an adaptation-executer rather than a utility-maximizer, such that there’s no trade you can make that will satisfy the animal if it involves trading away having-kids-before-age-45’?
It could be wise to have kids, for the sake of harmony between your parts/subgoals, even if you judge that having kids isn’t directly x-risk-useful and your animal seems to have a pretty visceral appreciation for x-risk—just because that is in fact what an important part of you wants/needs/expects/etc.
I agree if the criterion is to get us to utopia, it’s a problem (maybe not even that, but whatever), but if we instead say that it has to avoid x-risk
This seems to misunderstand my view? My goal is to avoid x-risk, not to get us to utopia. (Or rather, my proximate goal is to end the acute risk period; ultimate goal utopia, but I want to pint nearly all of the utopia-work to the period after we’ve ended the acute risk period.
“When I say that alignment is lethally difficult, I am not talking about ideal or perfect goals of ‘provable’ alignment, nor total alignment of superintelligences on exact human values, nor getting AIs to produce satisfactory arguments about moral dilemmas which sorta-reasonable humans disagree about, nor attaining an absolute certainty of an AI not killing everyone. When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask that we are not on course to get. So far as I’m concerned, if you can get a powerful AGI that carries out some pivotal superhuman engineering task, with a less than fifty percent change of killing more than one billion people, I’ll take it. Even smaller chances of killing even fewer people would be a nice luxury, but if you can get as incredibly far as “less than roughly certain to kill everybody”, then you can probably get down to under a 5% chance with only slightly more effort. Practically all of the difficulty is in getting to “less than certainty of killing literally everyone”. Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don’t care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI ‘this will not kill literally everyone’. Anybody telling you I’m asking for stricter ‘alignment’ than this has failed at reading comprehension. The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors.”
mathematical work like provable alignment will do much
I don’t know what you mean by “do much”, but if you think theory/math work is about “provable alignment” then you’re misunderstanding all (or at least the vast majority?) of the theory/math work on alignment. “Is this system aligned?” is not the sort of property that admits of deductive proof, even if the path to understanding “how does alignment work?” better today routes through some amount of theorem-proving on more abstract and fully-formalizable questions.
(Of course, some people won’t read the post because they’ve seen it in the past, and also won’t want to upvote something based only on a vague years-old recollection. That makes total sense to me. But if you do read it, I wouldn’t personally hold back on upvotes just because it’s old.)
If this is the canonical reference for one of the most important concepts of all time on LW (and I think it is), then I’d bid in favor of it getting upvoted on that basis even if it’s not novel. My thinking is that LW is meant to mostly be a timeless repository of knowledge, where content continues to be used ten years from now rather than just being used for a month. If someone stumbles on this page in five or ten years, I want the upvotes to provide a signal about how useful the post is to read.
(The karma only matters as a signal for people who aren’t familiar with the post/concept; if you are familiar, you can just decide whether to reread on that basis.)
‘Karma as a signal that I want to see more things like this’ matters a little, but less than usual since CFAR!Duncan is a dummy account and the handbook-posting is a scheduled thing.
The physical world has phase transitions, but it doesn’t have magisteria. ‘Non-overlapping magisteria’, as I’m using the term, is a question about literary genres; about which inferences are allowed to propagate or transfer; about whether a thing feels near-mode or far-mode; etc.
The idea of “going to college” post-AGI sounds silly for two distinct reasons:
The post-singularity world will genuinely be very different from today’s world, and institutions like college are likely to be erased or wildly transformed on relatively short timescales.
The post-singularity world feels like an inherently “far-mode world” where everything that happens is fantastic and large-scale; none of the humdrum minutiae of a single person’s life, ambitions, day-to-day routine, etc. This includes ‘personal goals are near, altruistic goals are far’.
1 is reasonable, but 2 is not.
The original example was about “romantic and reproductive goals”. If the AGI transition goes well, it’s true that romance and reproduction may work radically differently post-AGI, or may be replaced with something wild and weird and new.
But it doesn’t follow from this that we should think of post-AGI-ish goals as a separate magisterium from romantic and reproductive goals. Making the transition to AGI go well is still a good way to ensure romantic and reproductive success (especially qua “long-term goals/flourishing”, as described in the OP), or success on goals that end up mattering even more to you than those things, if circumstances change in such a way that there’s now some crazy, even better posthuman opportunity that you prefer even more.
(I’m assuming here that we shouldn’t optimize goals like “kids get to go to college if they want” in totally qualitatively different ways than we optimize “kids get to go to college if they want, modulo the fact that circumstances might change in ways that bring other values to the fore instead”. I’m deliberately choosing an adorably circa-2022 goal that seems especially unlikely to carry over to a crazy post-AGI world, “college”, because I think the best way to reason about a goal like that is similar to the best way to reason about other goals where it’s more uncertain whether the goal will transfer over to the new phase.)
I chose the college example because it’s especially jarring / especially disrespectful of trying to separate the world into two “pre-AGI versus post-AGI” magisteria.
A more obvious way to see that x-risk matters for ordinary day-to-day goals is that parents want their kids to have long, happy lives (and nearly all of the variance in length and happiness is, in real life, dependent on whether the AGI transition goes well or poorly). It’s not a separate goal; it’s the same goal, optimized without treating ‘AGI kills my kids’ as though it’s somehow better than ‘my kids die in a car accident’.
Your talk of a pendulum makes it sound like you want to create a social environment that incentivizes things that look like optimizing for regardless of whether they’re actually in anyone’s best interest.
I and my kids not being killed by AGI is in my best interest!
A successful AI Safety project probably looks a lot more like a network of very different people figuring out how to collaborate for mutual benefit than a cadre of self-sacrificing idealists.
Not letting AGI kill me and everyone I love isn’t the “self-sacrificing” option! Allowing AGI to kill me is the “self-sacrificing” option — it is literally allowing myself to be sacrificed, albeit for ~zero gain. (Which is even worse than sacrificing yourself for a benefit!)
I’m not advocating for people to pretend they’re more altruistic than they are, and I don’t see myself as advocating against any of the concrete advice in the OP. I’m advocating for people to stop talking/thinking as though post-AGI life is a different magisterium from pre-AGI life, or as though AGI has no effect on their ability to realize the totally ordinary goals of their current life.
I think this would help with shrugging-at-xrisk psychological factors that aren’t ‘people aren’t altruistic enough’, but rather ‘people are more myopic than they wish they were’, ‘people don’t properly emotionally appreciate risks and opportunities that are novel and weird’, etc.
I want to have a dialog about what’s true, at the level of piece-by-piece reasoning and piece-by-piece causes. I appreciate that you Rob are trying to do this; “pedantry” as you put it is great, and seems to me to be a huge chunk of why LW is a better place to sort some things out than is most of the internet.
Yay! I basically agree. The reason I called it “pedantry” was because I said it even though (a) I thought you already believed it (and were just speaking imprecisely / momentarily focusing on other things), (b) it’s an obvious observation that a lot of LWers already have cached, and (c) it felt tangential to the point you were making. So I wanted to flag it as a change of topic inspired by your word choice, rather than as ordinary engagement with the argument I took you to be making.
and that you talk of my post as trying to push the pendulum in a particular way, and “people trying to counter burnout,” and whether this style of post “works” for others.
I think I came to the post with a long-term narrative (which may have nothing to do with the post):
There are a bunch of (Berkeley-ish?) memes in the water related to radical self-acceptance, being kind to yourself, being very wary of working too hard, staying grounded in ordinary day-to-day life, being super skeptical and cautious around Things Claiming To Be Really Important and around moralizing, etc.
I think these are extremely important and valuable memes that LW would do well to explore, discuss, and absorb much more than it already has. I’ve found them personally extremely valuable, and a lot of my favorite blog posts to send to new EAs/rats make points like ‘be cautious around things that make big moralistic demands of you’, etc.
But I also think that these kinds of posts are often presented in ways that compete against the high value I (genuinely, thoughtfully) place on the long-term future, and the high probability I (genuinely, thoughtfully) place on AI killing me and my loved ones, as though I need to choose between the “chill grounded happy self-loving unworried” aesthetic or the “working really hard to try to solve x-risk” aesthetic.
This makes me very wary, especially insofar as it isn’t making an explicit argument against x-risk stuff, but is just sort of vaguely associating not-worrying-so-much-about-human-extinction with nice-sounding words like ‘healthy’, ‘grounded’, ‘relaxed’, etc. If these posts spent more time explicitly arguing for their preferred virtues and for why those virtues imply policy X versus policy Y, rather than relying on connotation and implicature to give their arguments force, my current objection would basically go away.
If more of the “self-acceptance, be kind to yourself, be vary wary of working too hard, etc.” posts were more explicit about making space for possibilities like ‘OK, but my best self really does care overwhelmingly more about x-risk stuff than everything else’ and/or ‘OK, but making huge life-changes to try to prevent human extinction really is the psychologically healthiest option for me’, I would feel less suspicious that some of these posts are doing the dance wrong, losing sight of the fact that both magisteria are real, are part of human life.
I may have been primed to interpret this post in those terms too much, because I perceived it to be a reaction to Eliezer’s recent doomy-sounding blog posts (and people worrying about AI more than usual recently because of that, plus ML news, plus various complicated social dynamics), trying to prevent the community from ‘going too far’ in certain directions.
I think the post is basically good and successful at achieving that goal, and I think it’s a very good goal. I expect to link to the OP post a lot in the coming months. But it sounds like I may be imposing context on the post that isn’t the way you were thinking about it while writing it.
Yeah, that makes sense to me. I’m complaining about a larger class of posts, so maybe this one isn’t really an example and I’m just pattern-matching. I do still wish there existed more posts that were very obviously examples of the ‘both-and’ things I was pointing at. (Both dentist appointments and Dyson spheres; both embrace slack and embrace maxipok; etc.)
It might be that if my thinking were clearer here, I’d be able to recognize more posts as doing ‘both-and’ even if they don’t explicitly dwell on it as much as I want.
I love this post a lot.
Also, I am excited about people trying to follow paths to all of their long-term goals/flourishing, including their romantic and reproductive goals, and I am actively not excited about people deciding to shelve that because they think AI risk demands it.
I’m going to reply in a pedantic way, then say why I did so.
Pedantic response: Trying to solve AI risk is one way to increase the odds you’ll achieve goals like that. (E.g., if you want to support your kids going to college, you should consider putting aside money for them and keeping an eye out for ways to reduce the risk you and your kids die from rogue AI disassembling the inner solar system before they’re of college age.)
Explanation:
I make this point not to argue against finding love or starting a family, but to argue against a mindset that treats AGI and daily life as more or less two different magisteria.
I think there’s a pendulum-swing-y thing going on: people trying to counter burnout, freakouts, etc. heavily emphasize mundane, non-sci-fi-sounding reality, so as to push back a pendulum they see as having swung too far toward ‘my mental universe is entirely about Dyson swarms, and not at all about dental appointments’.
I find this mostly doesn’t work for me personally. I come away feeling like half the blog posts I read are disassociated from ordinary day-to-day life, and the other half are disassociated from ‘the portion of my life that will occur during and after the invention of this wild technology’, such that I end up wary of both kinds of post.
Your overall post is a whole lot better than most on this dimension. It doesn’t just treat ‘be psychologically happy and healthy this year’ as obviously lexicographically more important than preventing your and everyone else’s deaths in twenty-five years (or whatever). It gives specific arguments for why the happy-and-healthy stuff is super important even if you’re confident your personal CEV’s terminal values assign overwhelmingly more weight to x-risk stuff than to your happiness over the next fifty years. It acknowledges the “‘Emergencies’ as wake-up calls” thing.
But it still feels to me like it’s a post trying to push the pendulum in a particular direction, rather than trying to fully and openly embody the optimal-by-your-lights Balancing Point.
(Where de-prioritizing pendulum pushing indeed risks worsening the current problem we face, insofar as EAs today do over-extend more than they under-extend. Maybe I just selfishly prefer reading posts like that, even if they’re not optimal for the community.)
It still doesn’t feel to me like it’s fully speaking as though the two worlds as one world, or fully embracing that a lot of people probably need to hear something closer to the opposite advice. (‘You’re asking too little of yourself; there are great things you could achieve if you were more strategic and purposeful with your time, rather than just following the nearest hedonic gradient; you are not fragile, or a ticking time bomb that will surely flame out if you start doing qualitatively harder things; you can output good-by-maxipok-lights things without waiting to first resolve all your personal issues or get your life fully in order; the alignment problem may sound crazy and intense to you from a distance, and it still possibly be true that you could do good work without sacrificing a flourishing, well-rounded pre-AGI life, and perhaps even flourish more pre-AGI as a consequence.’)
That might just be because I’m not the target audience and this style works great for others, but I wanted to mention it.
One caveat to the claim that we should prioritize serial alignment work over parallelizable work, is that this assumes an omniscient and optimal allocator of researcher-hours to problems.
Why do you think it assumes that?
Later in the article you seem positive about ELK, so maybe we agree on this fact.
? Where does he seem positive about ELK?
This isn’t a coincidence; the state of alignment knowledge is currently “we have no idea what would be involved in doing it even in principle, given realistic research paths and constraints”, very far from being a well-specified engineering problem. Cf. https://intelligence.org/2013/11/04/from-philosophy-to-math-to-engineering/.
If you succeed at the framework-inventing “how does one even do this?” stage, then you can probably deploy an enormous amount of engineering talent in parallel to help with implementation, small iterative improvements, building-upon-foundations, targeting-established-metrics, etc. tasks.
From A central AI alignment problem: capabilities generalization, and the sharp left turn:
Suppose that the fictional team OpenMind is training up a variety of AI systems, before one of them takes that sharp left turn. Suppose they’ve put the AI in lots of different video-game and simulated environments, and they’ve had good luck training it to pursue an objective that the operators described in English. “I don’t know what those MIRI folks were talking about; these systems are easy to direct; simple training suffices”, they say. At the same time, they apply various training methods, some simple and some clever, to cause the system to allow itself to be removed from various games by certain “operator-designated” characters in those games, in the name of shutdownability. And they use various techniques to prevent it from stripmining in Minecraft, in the name of low-impact. And they train it on a variety of moral dilemmas, and find that it can be trained to give correct answers to moral questions (such as “in thus-and-such a circumstance, should you poison the operator’s opponent?”) just as well as it can be trained to give correct answers to any other sort of question. “Well,” they say, “this alignment thing sure was easy. I guess we lucked out.”
Then, the system takes that sharp left turn,[4][5] and, predictably, the capabilities quickly improve outside of its training distribution, while the alignment falls apart.
[...]
[5] “Hold on, isn’t this unfalsifiable? Aren’t you saying that you’re going to continue believing that alignment is hard, even as we get evidence that it’s easy?” Well, I contend that “GPT can learn to answer moral questions just as well as it can learn to answer other questions” is not much evidence either way about the difficulty of alignment. I’m not saying we’ll get evidence that I’ll ignore; I’m naming in advance some things that I wouldn’t consider negative evidence (partially in hopes that I can refer back to this post when people crow later and request an update). But, yes, my model does have the inconvenient property that people who are skeptical now, are liable to remain skeptical until it’s too late, because most of the evidence I expect to give us advance warning about the nature of the problem is evidence that we’ve already seen. I assure you that I do not consider this property to be convenient.
As for things that could convince me otherwise: technical understanding of intelligence could undermine my “sharp left turn” model. I could also imagine observing some ephemeral hopefully-I’ll-know-it-when-I-see-it capabilities thresholds, without any sharp left turns, that might update me. (Short of “full superintelligence without a sharp left turn”, which would obviously convince me but comes too late in the game to shift my attention.)
Alex Lintz’s take: https://forum.effectivealtruism.org/posts/eggdG27y75ot8dNn7/three-pillars-for-avoiding-agi-catastrophe-technical