Upgrading the AI Safety Community
Special thanks to Justis Mills for pointing out important caveats.
Intelligence Enhancement
Yes, you recently did a post about that to aggregate people’s findings.
Recently, Genesmith and kman’s post opening the Overton window on adult human gene editing and Richard Ngo’s post on Moloch being as much about inadequate technology/understandings as coordination failures, I feel like people here have a better sense of what historically unprecedented things are possible in a transforming world. Including things like upgrading the AI safety community, or the AI safety community leading the world on human upgrading in general.
Also recently, Yudkowsky wrote:
I currently guess that a research community of non-upgraded alignment researchers with a hundred years to work, picks out a plausible-sounding non-solution and kills everyone at the end of the hundred years.”
I’m much more focused on intelligence enhancement for the purpose of making the AI safety community better positioned to save the world, e.g. intelligence-augmented people researching and facilitating AI policy/AI pause, instead of technical alignment.
But either way, the value here is obviously intense. And we can get impressive results, quick, and keep them too. The 2020s will probably move faster, even without slow takeoff or AI transformations. And for intelligence amplification, the AI safety community is one of the world’s best candidates to figure things out as we go along and be the first to ride each wave. We’re exactly the kind of people to reap the benefits as they come, no matter how strange, and use them to make things happen on our terms.
Woah, I just read this section from that article you sent about ultrasound:
it’s a huge opportunity for research. If you can “turn up” or “turn down” any brain region at will, safely enough to mess around with it on healthy human subjects, you can develop a functional atlas of the brain, where you find out exactly what each part is doing.
That absolutely blew my mind. It uses the same causality analysis as Zack M Davis’s Optimized Propaganda with Bayesian Networks, except by sensing and modulating activity in brain tissue, rather than surveys given to large numbers of people, but both of them run interpretability on the human brain. Neuromodulation is bottlenecked on safety trials, of course, so it might take too long before it’s valuable for AI safety.
Maybe it would be easier to do it safely if it used nonlinear acoustics to resonate with skull bones, instead of at the frequency that affects neural activity. That way, you aren’t messing with the neural activity at all, you’re only vibrating the inner skull; just have it automatically turn it down as they approach the right direction, so that thinking in beneficial directions feels more comfortable.
Either way, you could have smart people doing Squirrel’s Tuning in an fMRI machine, or having moments of insight in the machine, if you map the brain well enough at key moments via the fMRI machines, you could even find out how to suppress the parts of the brain that inhibit cognitive strategy tuning, or to halt the feedback loops that prevent or conclude moments of insight.
such technologies need to have good UX in mind, which is extra-important when the medium is potentially your whole mind. E.g. you don’t want your deep thinking interrupted by 20 modalities of distracting HUD elements.
The recent iPhone’s “dynamic island” is great for this. They also have variable refresh rates, and if you oscillate those at different times, then that would also be a safe stimulus e.g. cause discomfort and ratchet down the discomfort as they approach the correct brain state. You can stack weaker versions of these stimuli together, to create multi-source discomfort that eases as they move in the correct direction. Maybe also discomfort from barely-audible or inaudible high or low frequency sounds.
That’s a lot of sources of relief, possibly stackable. Plenty of safe ways to deploy the positive reinforcement at precisely the right moments, no need for any heroin or juice reward.
Becomes:
Frankly, unhelpful/wasteful brain states should be uncomfortable, and helpful/gainful brain states should be the comfortable ones.
We just have to set up the positive and negative reinforcements correctly, instead of cluelessly ignoring them and getting reinforced in random directions, like our hunter gatherer ancestors.
Figuring things out as we go along won’t be easy, these Chesterton-Schelling fences will drop all sorts of bizarre curve balls on us, but we’re already starting out with some of the best people for the job, and as we go along we’ll be creating even better people.
Megaprojects
dath ilan
What about the Megaproject of “moving most AI safety people to the Bay Area”? Getting everyone in one place seems like a good idea, similar to dath ilan. More in-person conversations, different kinds of thinkers working on the same whiteboard, etc.
Bay Area people have been setting up group houses, buying the Rose Garden Inn and conference venues. There’s obvious problems e.g. various social dynamics, but the groundwork (the land) has been laid.
My thinking is that if we had more people just realized “wow, billions of people are going to die, my family and friends are going to die”, they’d be more comfortable with Spartan lifestyles, except with galaxy-brained ways to maximize the number of people working in one place (e.g. these Capsule hotels give a sense of what can be discovered with sufficiently outside-the-box thinking). That lowers the cost of people visiting for just a month, which provides intense value for things like sourcing talent for specific projects:
The global situation with AI really is plenty harsh enough to justify outside-the-box solutions like this; according to this joke tweet from Yudkowsky:
my friends who are parents don’t seem very impressed with my latest theory of childrearing, that you should start out your kids in a giant dirt pit under regular attack by robotic velociraptors… so that afterwards ordinary Earth standards of living will seem to them like supreme luxury because their baseline brain expectations will have been set very low
Maybe not everyone is OK with the EA ascetic/extremophile lifestyle, but spending a year like that (while your talent is being evaluated) isn’t too harsh, it’s a great way to increase the talent per square foot, and is a great way of filtering people who are serious about saving the world. Those are the things that have worked for me, anyway; there are tons of options to mitigate the harm caused by San Francisco’s layout and prices, including building a space-optimized city within a city.
Better options for people to physically move to a better space also seems like an important part of the solution for a really interesting talent-sourcing problem raised by Habryka about a week ago:
The world is adversarial in the sense that if you are smart and competent, there are large numbers of people and institutions optimizing to get you to do things that are advantageous to them, ignoring your personal interests.
Most smart people “get got” and end up orienting their lives around some random thing they don’t even care about that much, because they’ve gotten their OODA loop captured by some social environment that makes it hard for them to understand what is going on or learn much about what they actually want to do with their lives.
I think that being anchored a specific space, e.g. a house and community in Michigan or Miami, is exactly the kind of thing that would suppress people’s ability to think about the global situation or decide what they want to do about it, same as a high-paying and interesting job at Facebook or Wall Street or maybe even OpenAI. A sleepy boring life is an attractor state.
Adaptability
“In times of change, learners inherit the earth, while the learned find themselves beautifully equipped to deal with a world that no longer exists”
—Eric Hoffer
Extracting much more value from the Sequences
What’s your thoughts on increasing the readership rates of Yudkowsky’s Sequences, the CFAR handbook, Scott Alexander’s Codex, Projectlawful/HPMOR, Tuning cognitive strategies/feedbackloopfirst rationality, etc? What’s the bottleneck preventing people from reading and growing from these, and what can we do about that bottleneck?
They offer so many unique ways to get better habits into your brain so you can go out and get what you want out of life, instead of being confused and constantly betraying yourself; just like getting groups of people into better Nash Equilibria where they aren’t constantly betraying each other.
Intelligence amplification technology is great, but a superintelligent AI could write a manual that uses true knowledge to show you how to exploit the full reaches of your mind, probably a couple orders of magnitude more effectively than the CFAR handbook.
And stuff like the CFAR handbook is not only something we already have, it’s something that the entire human race has barely bothered to try at, for the entirety of human history. We don’t know how big the gap is between the CFAR Handbook and a superintelligence-written true-instruction-manual-for-being-a-human-brain that lets us think optimally to maximize results.
That gap might be very small, maybe we need ten smart people working on it instead of five, maybe five was more than enough but they didn’t have SquirrelInHell’s Tuning Your Cognitive Strategies to work off of, and now they do. Maybe all five used social media and other entertainment media so frequently that they lost the ability to measure effectiveness of independent thought. Everything in the AI safety community boils down to inadequacy and growing pains.
I’d argue that a big part of the value from intelligence augmentation tech just comes from making people better at applying the advice that’s already described by the Sequences, CFAR handbook, and the others.
I think a big part of it might be the lack of positive reinforcement, it’s basically work, e.g. like this thing Yudkowsky retweeted:
And if I could get everyone in AI alignment to spend 2 hours a week thinking about solutions for AI governance, that would be great too.
In response to the OpenAI conflict, Yudkowsky wrote:
The Twitterati are vastly overestimating how much focus I spend on individual AI companies. I currently see their collective as an inevitable downhill slide, whose leaders could not change anything even if they wanted to. It’s international treaties or bust.
And for me that was a pretty big update in favor of tapping into the alignment researcher’s talent base to solve the impossible-seeming global problems like US-China coordination on AI issues, which I’ve been researching for years without knowing that AI governance would become so significant relative to all other work areas.
Like, plenty of people have resigned themselves to being “words and stats people”, and think they can’t join “ML and/or maths people”.
Yes, I totally agree on the maths issue. I think it goes both ways though; the impression I’ve gotten from spending years in DC is that the people here rarely have the quant skills to apply Bayesian thinking to AI governance at all. But it might make more sense to have the AI governance people get the alignment researchers in the Bay Area up to speed so that the alignment researchers can think up galaxy-brained solutions to the problems, rather than trying to boost the quant skills of the AI governance people so they can start playing catch-up with the people in the bay area who have been pioneering Bayesian thinking for decades.
I think the bottleneck is the top of the funnel right now. Everyone’s heard of HP, but “only” HP fanfic nerds (and/or existing rationalists) have heard of HPMOR. The Sequences and related works have even narrower funnel-tops.
There are ways this could be done more or less efficiently/directly, but RationalAnimations is a decent example of “basic AI alignment ideas can be presented compellingly-enough to reach lots of people”.
I agree here. When I imagine my 2014 or 2013 self, before I had heard of AI safety, I can easily imagine myself watching a 10 minute video and thinking “jesus christ” and bookmarking it. And then, later on, if I was the type of person to be pragmatic making the world a better place, I’d be more likely to remember it again in a month or a week or so, I’d be like “wait, what the fuck was up with that?” and watch it again. Ideally, people would read the Sequences or WWOTF and there wouldn’t be any other onramp for AI safety, but that’s not the world we live in.
Maybe the podcasts that Yud and others are doing is a better version of that, better at filtering for smart insightful people due to covering a lot of ground on interesting futurist topics, or better for intuition flooding making the current situation intuitively understandable by approaching it from a ton of different angles, e.g. spending 10 minutes chatting about theory of mind and then another 10 on instrumental convergence and then another 10 on AI race economics.
I was actually thinking more about increasing the Sequences and CFAR handbook reading rate within AI Safety, rather than getting the rest of the world on board. That’s also a huge issue though, and inextricably linked to getting the new talent required for the AI safety community to thrive at all.
I think that rereading the Sequences, for example, could have averted a lot of harm, a la this tweet (I have https://twitter.com/ESYudkowsky bookmarked and you should too):
I think that rereading the Sequences every 5-10 years, or even more frequently, makes sense for most people doing AI safety, but there’s a better way (and we should really be getting deep into the mindset of finding a better way).
I think that we should be reading the Sequences like the Bible. Specifically, reading various Yudkowsky and other great rationalist quotes every day, like Bible quotes. There’s tons of apps, websites, newsletters, and paper calendars that give you one bible quote per day.
There’s some pretty decent stuff in the Bible and those quotes were searched and found by the optimization pressure of lots of people looking. It works very well, for millions of non-priests. But Yudkowsky’s writings are far better for this than the Bible.
It would plausibly make sense to have some kind of feed that gives people random Yudkowsky quotes every day.
This is similar to reading the Sequences in random order, which I also highly recommend; if you look at entertainment media or social media after drinking coffee in the morning, you are stacking two strong addiction/behavior reinforcing things, and you should really substitute the media with reading a randomly selected Sequence so your brain associates that with the rush instead.
But aside from the badass EA ascetic extremophiles, most people want to do quick and easy things, hence using great rationalist quotes like Bible quotes. It’s a great way to start the day, or in the middle of the day.
There could be different quotes per person per day, in order to maximize the multiplier effect via people being boosted in unique ways and then talking with each other; of course, this requires a large enough portion of AI safety would do this, and for the AI safety community this is like herding cats.
Meanwhile, the incredible CFAR handbook sequence is somewhat long (~20,000 words) but explains difficult and important concepts really well, and the Hammertime sequence is much shorter and reread-able, but that comes with the cost of weaker intuitiveness and making it easier to forget.
Ideally someone like Duncan could do a really great job creating a middle ground between the two, although it’s possible that the CFAR handbook sequence was already pretty intensely shortened given the complexity of the underlying concepts. Duncan’s writing and explanations are pretty great, and my prior probability is that I’d have a hard time preserving the value if I tried to distill it to ~75% of the original length.
Converting a shortened version into flashcards would conceivably be very valuable, but the main thing is that people just need to read it, and get their friends to read it, because they’re all still well worth reading, even at their current length.
When people debate the value or effectiveness of the Sequences, they’re generally thinking, like, slap the book down in front of someone and tell them to read it. Maybe see what they’re like 5-10 years after finishing. But the bigger question is whether there’s ways to do this orders of magnitude more effectively.
Are there any organizations or research groups that are specifically working on improving the effectiveness of the alignment research community? E.g.
Reviewing the literature on intellectual progress, metascience, and social epistemology and applying the resulting insights to this community
Funding the development of experimental “epistemology software”, like Arbital or Mathopedia
The classic one is Lightcone Infrastructure, the team that runs LessWrong and the Alignment Forum.
Note: CFAR has had been a MIRI hiring pipeline for years, and they also seemed to function as a way of upskilling people in CFAR-style rationality, which CFAR thought was the load-bearing bits required to turn someone into a world saver.
I don’t think anyone is saying this outright so I suppose I will—pushing forward the frontier on intelligence enhancement as a solution to alignment is not wise. The second order effects of pushing that particular frontier (both the capabilities and overton window) are disastrous, and our intelligence outpacing our wisdom is what got us into this mess in the first place.
I absolutely agree. Since lots of things are happening in the brain, you can’t amplify intelligence without tearing down lots of Chesterton-Schelling fences. Making a community wealthy or powerful will make all the people and structures and norms inside it OOD.
But at the same time, we need nuanced calculations comparing the expected costs and the expected benefits. We will need to do those calculations as we go along, so we can update based on which tech and projects turn out to be low hanging fruit. Staying the course also doesn’t seem to be a winning strategy.
You’re not going to just be able to stop the train at the moment the costs outweigh the benefits. The majority of negative consequences will most likely come from grey swans that won’t show up in your nuanced calculations of costs and benefits.
EY particularly mean “intelligence in broad sense”, including wisdom. One of his proposals was “identify structures that makes people rationalize, disable them”.
This is not what I mean by wisdom.
Double-click? I’m wondering what you mean by “cold sleep” here.