Epistemic status: A possibly annoying mixture of straightforward reasoning and hard-to-justify personal opinions.
It is often stated (with some justification, IMO) that AI risk is an “emergency.” Various people have explained to me that they put various parts of their normal life’s functioning on hold on account of AI being an “emergency.” In the interest of people doing this sanely and not confusedly, I’d like to take a step back and seek principles around what kinds of changes a person might want to make in an “emergency” of different sorts.
Principle 1: It matters what time-scale the emergency is on
There are plenty of ways we can temporarily increase productivity on some narrow task or other, at the cost of our longer-term resources. For example:
Skipping meals
Skipping sleep
Ceasing to clean the house or to exercise
Accumulating credit card debt
Calling in favors from friends
Skipping leisure time
If I would strongly prefer to address some situation x before time t, I may sometimes want to “borrow from the future” like this. But the time-scales matter. If I’m trying to address x as much as possible in the next five hours, skipping sleep may make sense. If I’m trying to address x as much as possible over the next year, I’ll probably do better to get my usual amount of sleep tonight. Something similar (with different, resource-specific timescales) will hold for other resources.
So, in short time-scale emergencies, it’ll often make sense to suspend a great deal of normal functioning for a short period of time. In longer time-scale emergencies, your life should mostly look closer to normal.
Principle 2: It matters how much we know how to address the emergency
Much of what we do in daily life – especially when we’re feeling “free” and “unstuck,” and as though there is nothing in particular that we “have to” do – has the effect of making us well-resourced and capable in general. For example, by default, a lot of us would spend a lot of time reading interesting books of varied sorts, nerding out about interesting topics, trying our hands at new skills and crafts, etc. Also, we often like making our living spaces nicer (and more functional), forming new friendships, and so on.
If there is a particular thingy that matters hugely, and if you have an accurate model of how exactly to change that thingy, it may make sense to sacrifice some of your general-purpose capacities in trade for increased ability to address that thingy. (E.g., if you know for a fact that you’ll lose your home unless you pay the mortgage, and if keeping your home is important, it may make sense to trade general-purpose capacities for the ability to make mortgage payments by e.g. working over-long hours at a mind-numbing job that leaves you stupid.)
However, if you don’t have an accurate map of how to address a given thingy, then, even if the thingy is very important, and even if its time-scale is short, you’ll probably mostly want to avoid sacrificing general-purpose capacities. (In a low-information context, your general-purpose capacities are perhaps more likely to turn out helpful for your very important thingy than whatever you’d be trading them off to buy.) Thus, in “emergencies” where you do not have an accurate map of how to solve the emergency, your behavior should probably be more like normal than in better-mapped emergencies.
Side-note: “Emergencies” as wake-up calls
A different way an “emergency” can sometimes rearrange priorities is by serving as a “wake-up call” that helps people peel away accidental accumulation of habits, “obligations,” complacency, etc. For example, I’m told that near encounters with death sometimes leave people suddenly in touch with what matters in life. (I’ve heard this from one friend who had cancer, and seen accounts from strangers in writing; I’m not sure how common this is or isn’t really.)
I non-confidently suspect some gravitate toward AI risk or other emergencies in the hopes that it’ll help them peel away the cruft, notice a core of caring within themselves, and choose activities that actually make sense. (See also: Something to Protect.) If “AI risk as wake-up call” works out, I could imagine AI risk helping a person rearrange their actions in a way that boosts long-term capacities. (E.g., finding courage; trying new things and paying attention to what happens; cultivating friendships right now rather than in the possibly-non-existent future; facing up to minor social conflicts; attempting research that might yield fresh insights instead of research that feels predictable.)
This sort of “post wake-up call” change is almost the opposite of the kinds of “borrowing from the future” changes that typify short-term emergency responses. You should be able to tell the difference by seeing whether a person’s actions are unusually good for their long-term broad-based capacities (e.g., do they keep their house in a pattern that is good for them, get exercise, engage in the kinds of leisure and study that boost their ability to understand the world, appear unusually willing and able to cut through comfortable rationalizations, etc.?), or unusually bad for their long-term broad-based capacities (e.g., do they live on stimulants, in a house no one would want to live in, while saying they ‘don’t have time’ for exercise or for reading textbooks and seeming kind of burn-out-y and as though they don’t have enough free energy to fully take an interest in something new, etc.?).
What sort of “emergency” is AI risk?
It seems to me that AI risk is a serious problem in the sense that it may well kill us (and on my personal models, may well not, too). In terms of time-scales, I am pretty ignorant, but I personally will not be too surprised if the highest risk period is in only a couple years, nor if it is in more than thirty years. In terms of how accurate our maps of what to do are, it seems to me that our maps are not accurate; most people who are currently burning themselves out to try to help with AI risk on some particular path might, for all I know, contribute at least as well (even on e.g. a 5-year timescale) if they built general capacities instead.
I therefore mostly suspect that we’ll have our best shot at AI risk if we, as a community, cultivate long-term, robust capacities. (For me, this is hinging more on believing we have poor maps of how to create safety, and less on beliefs about time-scale.) Our best shot probably does mean paying attention to AI and ML advances, and directing some attention that way compared to what we’d do in a world where AI did not matter. It probably does mean doing the obvious work and the obvious alignment experiments where we know what those are, and where we can do this without burning out our long-term capacities. But it mostly doesn’t mean people burning themselves out, or depleting long-term resources in order to do this.
Some guesses at long-term, robust-in-lots-of-scenarios resources that may help with AI risk
Briefly, some resources I’d like us to have, as AI approaches:
Accurate trust in one another’s words. (Calibration, honesty, accurate awareness of one another’s honesty, valuing of truth over comfort or rationalizations. Practice seeing one another get things right and wrong in varied domains.)
Integrity. The ability to reason, tell the truth, and do what matters in the face of pain, fear, etc.
Practice having pulled off a variety or projects (not necessarily AI projects). (In my book, we get points for e.g. creating: movies; books; charter cities or other small or large-scale political experiments; buildings and communities and software; and basically anything else that involves reusable skills and engagement with the world.)
Practice accomplishing things in groups.
Time. Having AI not as advanced as in alternate scenarios. Having chip manufacture not as advanced as in alternate scenarios.
Spiritual health. Ability to love, to care about that which matters, to laugh and let go of rationalizations, to hope and to try, to form deep friendships.
Deep STEM knowledge, ability to do science of varied sorts, ability to do natural philosophy. (Not necessarily all AI.)
Engineering skill and practice.
An understanding of the wider cultural context we’re in, and of how to interact with it and what to expect.
All kinds of ML-related skills and resources, although this is in some tension with wanting time.
Given the above, I am excited about people in our larger orbit following their interests, trying to become scientists or writers or engineers or other cool things, exploring and cultivating. I am excited about people attempting work on AI alignment, or on other angles on AI safety, while also having hobbies and interests and friendships. For the most part I am not excited about people burning themselves out working super-long hours on alignment tasks in ways that damp their ability to notice new things or respond to later challenges, although I am excited about people pushing their limits doing work they’re excited by, and these things can blur together.
Also, I am excited about people trying to follow paths to all of their long-term goals/flourishing, including their romantic and reproductive goals, and I am actively not excited about people deciding to shelve that because they think AI risk demands it. This is probably the hardest to justify of my opinions, but, like Divia in this tweet, I am concerned (based partly on personal experience, partly on observations of others, and partly on priors/models) that when people try to table their deepest personal goals, this messes up their access to caring and consciousness in general.
Why do we see burnout in AI safety efforts and in effective altruism?
IMO, I see burnout (people working hard at the expense of their long-term resources and capacities) more often than I expect is optimal for helping with AI risk. I’m not sure why. Some of it is in people who (IMO) have a better shot and a better plan than most for reducing AI risk, which makes it more plausibly actually-helpful for those folks to be doing work at the expense of long-term capacity, though I have my doubts even there. I suspect much of it is for confused/mistaken reasons, though; I sometimes see EAs burning themselves out doing e.g. random ML internships that they think they should work hard on because of AI risk, and AFAICT this trade does not make sense. Also, when I look at the wider world (e.g. less-filtered chunks of Twitter; conversations with my Lyft drivers) I see lots of people acting as though lots of things are emergencies worth burning long-term resources for, in ways I suspect do not make sense, and I suspect that whatever dynamics lead to that may also be involved here. I’ve also heard a number of people tell me that EA or AI safety efforts caused them to lose the ability to have serious hobbies, or serious intellectual interests, and I would guess this was harmful to long-term AI safety potential in most cases. (This “lots of people lose the ability to have serious hobbies when they find EA” is something I care very much about. Both as damage to our movements’ potential, and as a symptom of a larger pattern of damage. I’ve heard this, again, from many, though not everyone.)
I’d be curious for y’all’s takes on any of the above!
What should you change in response to an “emergency”? And AI risk
This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn.
Related to: Slack gives you the ability to notice/reflect on subtle things
Epistemic status: A possibly annoying mixture of straightforward reasoning and hard-to-justify personal opinions.
It is often stated (with some justification, IMO) that AI risk is an “emergency.” Various people have explained to me that they put various parts of their normal life’s functioning on hold on account of AI being an “emergency.” In the interest of people doing this sanely and not confusedly, I’d like to take a step back and seek principles around what kinds of changes a person might want to make in an “emergency” of different sorts.
Principle 1: It matters what time-scale the emergency is on
There are plenty of ways we can temporarily increase productivity on some narrow task or other, at the cost of our longer-term resources. For example:
Skipping meals
Skipping sleep
Ceasing to clean the house or to exercise
Accumulating credit card debt
Calling in favors from friends
Skipping leisure time
If I would strongly prefer to address some situation x before time t, I may sometimes want to “borrow from the future” like this. But the time-scales matter. If I’m trying to address x as much as possible in the next five hours, skipping sleep may make sense. If I’m trying to address x as much as possible over the next year, I’ll probably do better to get my usual amount of sleep tonight. Something similar (with different, resource-specific timescales) will hold for other resources.
So, in short time-scale emergencies, it’ll often make sense to suspend a great deal of normal functioning for a short period of time. In longer time-scale emergencies, your life should mostly look closer to normal.
Principle 2: It matters how much we know how to address the emergency
Much of what we do in daily life – especially when we’re feeling “free” and “unstuck,” and as though there is nothing in particular that we “have to” do – has the effect of making us well-resourced and capable in general. For example, by default, a lot of us would spend a lot of time reading interesting books of varied sorts, nerding out about interesting topics, trying our hands at new skills and crafts, etc. Also, we often like making our living spaces nicer (and more functional), forming new friendships, and so on.
If there is a particular thingy that matters hugely, and if you have an accurate model of how exactly to change that thingy, it may make sense to sacrifice some of your general-purpose capacities in trade for increased ability to address that thingy. (E.g., if you know for a fact that you’ll lose your home unless you pay the mortgage, and if keeping your home is important, it may make sense to trade general-purpose capacities for the ability to make mortgage payments by e.g. working over-long hours at a mind-numbing job that leaves you stupid.)
However, if you don’t have an accurate map of how to address a given thingy, then, even if the thingy is very important, and even if its time-scale is short, you’ll probably mostly want to avoid sacrificing general-purpose capacities. (In a low-information context, your general-purpose capacities are perhaps more likely to turn out helpful for your very important thingy than whatever you’d be trading them off to buy.) Thus, in “emergencies” where you do not have an accurate map of how to solve the emergency, your behavior should probably be more like normal than in better-mapped emergencies.
Side-note: “Emergencies” as wake-up calls
A different way an “emergency” can sometimes rearrange priorities is by serving as a “wake-up call” that helps people peel away accidental accumulation of habits, “obligations,” complacency, etc. For example, I’m told that near encounters with death sometimes leave people suddenly in touch with what matters in life. (I’ve heard this from one friend who had cancer, and seen accounts from strangers in writing; I’m not sure how common this is or isn’t really.)
I non-confidently suspect some gravitate toward AI risk or other emergencies in the hopes that it’ll help them peel away the cruft, notice a core of caring within themselves, and choose activities that actually make sense. (See also: Something to Protect.) If “AI risk as wake-up call” works out, I could imagine AI risk helping a person rearrange their actions in a way that boosts long-term capacities. (E.g., finding courage; trying new things and paying attention to what happens; cultivating friendships right now rather than in the possibly-non-existent future; facing up to minor social conflicts; attempting research that might yield fresh insights instead of research that feels predictable.)
This sort of “post wake-up call” change is almost the opposite of the kinds of “borrowing from the future” changes that typify short-term emergency responses. You should be able to tell the difference by seeing whether a person’s actions are unusually good for their long-term broad-based capacities (e.g., do they keep their house in a pattern that is good for them, get exercise, engage in the kinds of leisure and study that boost their ability to understand the world, appear unusually willing and able to cut through comfortable rationalizations, etc.?), or unusually bad for their long-term broad-based capacities (e.g., do they live on stimulants, in a house no one would want to live in, while saying they ‘don’t have time’ for exercise or for reading textbooks and seeming kind of burn-out-y and as though they don’t have enough free energy to fully take an interest in something new, etc.?).
What sort of “emergency” is AI risk?
It seems to me that AI risk is a serious problem in the sense that it may well kill us (and on my personal models, may well not, too). In terms of time-scales, I am pretty ignorant, but I personally will not be too surprised if the highest risk period is in only a couple years, nor if it is in more than thirty years. In terms of how accurate our maps of what to do are, it seems to me that our maps are not accurate; most people who are currently burning themselves out to try to help with AI risk on some particular path might, for all I know, contribute at least as well (even on e.g. a 5-year timescale) if they built general capacities instead.
I therefore mostly suspect that we’ll have our best shot at AI risk if we, as a community, cultivate long-term, robust capacities. (For me, this is hinging more on believing we have poor maps of how to create safety, and less on beliefs about time-scale.) Our best shot probably does mean paying attention to AI and ML advances, and directing some attention that way compared to what we’d do in a world where AI did not matter. It probably does mean doing the obvious work and the obvious alignment experiments where we know what those are, and where we can do this without burning out our long-term capacities. But it mostly doesn’t mean people burning themselves out, or depleting long-term resources in order to do this.
Some guesses at long-term, robust-in-lots-of-scenarios resources that may help with AI risk
Briefly, some resources I’d like us to have, as AI approaches:
Accurate trust in one another’s words. (Calibration, honesty, accurate awareness of one another’s honesty, valuing of truth over comfort or rationalizations. Practice seeing one another get things right and wrong in varied domains.)
Integrity. The ability to reason, tell the truth, and do what matters in the face of pain, fear, etc.
Practice having pulled off a variety or projects (not necessarily AI projects). (In my book, we get points for e.g. creating: movies; books; charter cities or other small or large-scale political experiments; buildings and communities and software; and basically anything else that involves reusable skills and engagement with the world.)
Practice accomplishing things in groups.
Time. Having AI not as advanced as in alternate scenarios. Having chip manufacture not as advanced as in alternate scenarios.
Spiritual health. Ability to love, to care about that which matters, to laugh and let go of rationalizations, to hope and to try, to form deep friendships.
Deep STEM knowledge, ability to do science of varied sorts, ability to do natural philosophy. (Not necessarily all AI.)
Engineering skill and practice.
An understanding of the wider cultural context we’re in, and of how to interact with it and what to expect.
All kinds of ML-related skills and resources, although this is in some tension with wanting time.
Given the above, I am excited about people in our larger orbit following their interests, trying to become scientists or writers or engineers or other cool things, exploring and cultivating. I am excited about people attempting work on AI alignment, or on other angles on AI safety, while also having hobbies and interests and friendships. For the most part I am not excited about people burning themselves out working super-long hours on alignment tasks in ways that damp their ability to notice new things or respond to later challenges, although I am excited about people pushing their limits doing work they’re excited by, and these things can blur together.
Also, I am excited about people trying to follow paths to all of their long-term goals/flourishing, including their romantic and reproductive goals, and I am actively not excited about people deciding to shelve that because they think AI risk demands it. This is probably the hardest to justify of my opinions, but, like Divia in this tweet, I am concerned (based partly on personal experience, partly on observations of others, and partly on priors/models) that when people try to table their deepest personal goals, this messes up their access to caring and consciousness in general.
Why do we see burnout in AI safety efforts and in effective altruism?
IMO, I see burnout (people working hard at the expense of their long-term resources and capacities) more often than I expect is optimal for helping with AI risk. I’m not sure why. Some of it is in people who (IMO) have a better shot and a better plan than most for reducing AI risk, which makes it more plausibly actually-helpful for those folks to be doing work at the expense of long-term capacity, though I have my doubts even there. I suspect much of it is for confused/mistaken reasons, though; I sometimes see EAs burning themselves out doing e.g. random ML internships that they think they should work hard on because of AI risk, and AFAICT this trade does not make sense. Also, when I look at the wider world (e.g. less-filtered chunks of Twitter; conversations with my Lyft drivers) I see lots of people acting as though lots of things are emergencies worth burning long-term resources for, in ways I suspect do not make sense, and I suspect that whatever dynamics lead to that may also be involved here. I’ve also heard a number of people tell me that EA or AI safety efforts caused them to lose the ability to have serious hobbies, or serious intellectual interests, and I would guess this was harmful to long-term AI safety potential in most cases. (This “lots of people lose the ability to have serious hobbies when they find EA” is something I care very much about. Both as damage to our movements’ potential, and as a symptom of a larger pattern of damage. I’ve heard this, again, from many, though not everyone.)
I’d be curious for y’all’s takes on any of the above!