Elon Musk is a real-life epic tragic hero, authored by someone trying specifically to impart lessons to EAs/rationalists:
--Young Elon thinks about the future, is worried about x-risk. Decides to devote his life to fighting x-risk. Decides the best way to do this is via developing new technologies, in particular electric vehicles (to fight climate change) and space colonization (to make humanity a multiplanetary species and thus robust to local catastrophes)
--Manages to succeed to a legendary extent; builds two of the worlds leading tech giants, each with a business model notoriously hard to get right and each founded on technology most believed to be impossible. At every step of the way, mainstream expert opinion is that each of his companies will run out of steam and fail to accomplish whatever impossible goal they have set for themselves at the moment. They keep meeting their goals. SpaceX in particular brought cost to orbit down by an order of magnitude, and if Starship works out will get one or two more OOMs on top of that. Their overarching goal is to make a self-sustaining city on mars and holy shit it looks like they are actually succeeding. Did all this on a shoestring budget compared to various rivals, made loads of enemies who employed various dirty tricks against him, etc. Succeeded anyway.
--Starts to worry more about AI x-risk. Tries to convince people to take it more seriously. People don’t listen to him. Doesn’t like Demis Hassabis’ plan for how to handle the situation. Founds OpenAI with what seems to be a better plan.
--Oops! Turns out the better plan was actually a worse plan. Also turns out AI risk is a bigger deal than he initially realized; it’s big enough that everything else he’s doing won’t matter (unaligned AI can follow us to Mars...). Oh well. All the x-risk-reduction accomplished by all the amazing successes Elon had, undone in an instant, by a single insufficiently thought-through decision.
This story is hitting us over the head with morals/lessons.
--Heavy Tails Hypothesis: Distribution of interventions by impact is heavy-tailed, you’ll do a bunch of things in life and one of them will be the most important thing and if it’s good, it’ll outweigh all the bad stuff and if it’s bad, it’ll outweigh all the good stuff. This is true even if you are doing MANY very important, very impactful things.
--Importance of research and reflection: It’s not obvious in advance what the most important thing is, or whether it’s good or bad. You need to do research and careful analysis, and even that isn’t a silver bullet, it just improves your odds.
I agree with you completely and think this is very important to emphasize.
I also think the law of equal and opposite advice applies. Most people act too quickly without thinking. EAs tend towards the opposite, where it’s always “more research is needed”. This can also lead to bad outcomes if the results of the status quo are bad.
I can’t find it, but recently there was a post about the EU policy on AI and the author said something along the lines of “We often want to wait to advise policy until we know what would be good advice. Unfortunately, the choice isn’t give suboptimal advice now or great advice in 10 years. It’s give suboptimal advice now or never giving advice at all and politicians doing something much worse probably. Because the world is moving, and it won’t wait for EAs to figure it all out.”
I think this all largely depends on what you think the outcome is if you don’t act. If you think that if EAs do nothing, the default outcome is positive, you should err on extreme caution. If you think that the default is bad, you should be more willing to act, because an informed, altruistic actor increases the value of the outcome in expectation, all else being equal.
--Importance of research and reflection: It’s not obvious in advance what the most important thing is, or whether it’s good or bad. You need to do research and careful analysis, and even that isn’t a silver bullet, it just improves your odds.
It’s not clear that would have been sufficient to change the outcome (above).
I feel optimistic that if he had spent a lot more time reading, talking, and thinking carefully about it, he would have concluded that founding OpenAI was a bad idea. (Or else maybe it’s actually a good idea and I’m wrong.)
Can you say more about what you have in mind here? Do you think his values are such that it actually was a good idea by his lights? Or do you think it’s just so hard to figure this stuff out that thinking more about it wouldn’t have helped?
How much thinking/researching would have been necessary to avoid the failure?
5 hours? 5 days? 5 years? 50? What does it take to not make a mistake? (Or just, that one in particular?)
Expanding on what you said:
Or do you think it’s just so hard to figure this stuff out that thinking more about it wouldn’t have helped?
Is it a mistake that wouldn’t have been solved that way? (Or...solved that way easily? Or another way that would have fixed that problem faster?)
For research to trivially solve a problem, it has...someone pointing out it’s a bad idea. (Maybe talking with someone and having them say _ is the fix.)
Elon Musk is a real-life epic tragic hero, authored by someone trying specifically to impart lessons to EAs/rationalists:
--Young Elon thinks about the future, is worried about x-risk. Decides to devote his life to fighting x-risk. Decides the best way to do this is via developing new technologies, in particular electric vehicles (to fight climate change) and space colonization (to make humanity a multiplanetary species and thus robust to local catastrophes)
--Manages to succeed to a legendary extent; builds two of the worlds leading tech giants, each with a business model notoriously hard to get right and each founded on technology most believed to be impossible. At every step of the way, mainstream expert opinion is that each of his companies will run out of steam and fail to accomplish whatever impossible goal they have set for themselves at the moment. They keep meeting their goals. SpaceX in particular brought cost to orbit down by an order of magnitude, and if Starship works out will get one or two more OOMs on top of that. Their overarching goal is to make a self-sustaining city on mars and holy shit it looks like they are actually succeeding. Did all this on a shoestring budget compared to various rivals, made loads of enemies who employed various dirty tricks against him, etc. Succeeded anyway.
--Starts to worry more about AI x-risk. Tries to convince people to take it more seriously. People don’t listen to him. Doesn’t like Demis Hassabis’ plan for how to handle the situation. Founds OpenAI with what seems to be a better plan.
--Oops! Turns out the better plan was actually a worse plan. Also turns out AI risk is a bigger deal than he initially realized; it’s big enough that everything else he’s doing won’t matter (unaligned AI can follow us to Mars...). Oh well. All the x-risk-reduction accomplished by all the amazing successes Elon had, undone in an instant, by a single insufficiently thought-through decision.
This story is hitting us over the head with morals/lessons.
--Heavy Tails Hypothesis: Distribution of interventions by impact is heavy-tailed, you’ll do a bunch of things in life and one of them will be the most important thing and if it’s good, it’ll outweigh all the bad stuff and if it’s bad, it’ll outweigh all the good stuff. This is true even if you are doing MANY very important, very impactful things.
--Importance of research and reflection: It’s not obvious in advance what the most important thing is, or whether it’s good or bad. You need to do research and careful analysis, and even that isn’t a silver bullet, it just improves your odds.
I agree with you completely and think this is very important to emphasize.
I also think the law of equal and opposite advice applies. Most people act too quickly without thinking. EAs tend towards the opposite, where it’s always “more research is needed”. This can also lead to bad outcomes if the results of the status quo are bad.
I can’t find it, but recently there was a post about the EU policy on AI and the author said something along the lines of “We often want to wait to advise policy until we know what would be good advice. Unfortunately, the choice isn’t give suboptimal advice now or great advice in 10 years. It’s give suboptimal advice now or never giving advice at all and politicians doing something much worse probably. Because the world is moving, and it won’t wait for EAs to figure it all out.”
I think this all largely depends on what you think the outcome is if you don’t act. If you think that if EAs do nothing, the default outcome is positive, you should err on extreme caution. If you think that the default is bad, you should be more willing to act, because an informed, altruistic actor increases the value of the outcome in expectation, all else being equal.
It wasn’t clear what this meant.
This made it seem like it was a word for a type of company.
Thanks, made some edits. I still don’t get your second point though I’m afraid.
The second point isn’t important, it’s an incorrect inference/hypothesis, predicated on the first bit of information being missing. (So it’s fixed.)
It’s not clear that would have been sufficient to change the outcome (above).
I feel optimistic that if he had spent a lot more time reading, talking, and thinking carefully about it, he would have concluded that founding OpenAI was a bad idea. (Or else maybe it’s actually a good idea and I’m wrong.)
Can you say more about what you have in mind here? Do you think his values are such that it actually was a good idea by his lights? Or do you think it’s just so hard to figure this stuff out that thinking more about it wouldn’t have helped?
My point was just:
How much thinking/researching would have been necessary to avoid the failure?
5 hours? 5 days? 5 years? 50? What does it take to not make a mistake? (Or just, that one in particular?)
Expanding on what you said:
Is it a mistake that wouldn’t have been solved that way? (Or...solved that way easily? Or another way that would have fixed that problem faster?)
For research to trivially solve a problem, it has...someone pointing out it’s a bad idea. (Maybe talking with someone and having them say _ is the fix.)