Wei Dai(Wei Dai)
It was easier for Eliezer Yudkowsky to reformulate decision theory to exclude time than to buy a new watch.
Eliezer Yudkowsky’s favorite sport is black hole diving. His information density is so great that no black hole can absorb him, so he just bounces right off the event horizon.
God desperately wants to believe that when Eliezer Yudkowsky says “God doesn’t exist,” it’s just good-natured teasing.
Never go in against Eliezer Yudkowsky when anything is on the line.
(Tangentially) If users are allowed to ban other users from commenting on their posts, how can I tell when the lack of criticism in the comments of some post means that nobody wanted to criticize it (which is a very useful signal that I would want to update on), or that the author has banned some or all of their most prominent/frequent critics? In addition, I think many users may be mislead by lack of criticism if they’re simply not aware of the second possibility or have forgotten it. (I think I knew it but it hasn’t entered my conscious awareness for a while, until I read this post today.)
(Assuming there’s not a good answer to the above concerns) I think I would prefer to change this feature/rule to something like allowing the author of a post to “hide” commenters or individual comments, which means that those comments are collapsed by default (and marked as “hidden by the post author”) but can be individually expanded, and each user can set an option to always expand those comments for themselves.
The option I bought is up 700% since I bought them, implying that as of 2/10/2020 the market thought there was less than 1⁄8 chance things would be as bad as they are today. At least for me this puts a final nail in the coffin of EMH.
Added on Mar 24: Just in case this thread goes viral at some point, to prevent a potential backlash against me or LW (due to being perceived as caring more about making money than saving lives), let me note that on Feb 8 I thought of and collected a number of ideas for preventing or mitigating the pandemic that I foresaw and subsequently sent them to several people working in pandemic preparedness, and followed up with several other ideas as I came across them.
- The EMH Aten’t Dead by 15 May 2020 20:44 UTC; 194 points) (
- Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis by 11 May 2020 6:00 UTC; 72 points) (
- Coronavirus: Justified Key Insights Thread by 13 Apr 2020 22:40 UTC; 50 points) (
- 13 Mar 2020 21:59 UTC; 33 points) 's comment on March Coronavirus Open Thread by (
- 11 Oct 2020 0:01 UTC; 17 points) 's comment on The Treacherous Path to Rationality by (
I think AI risk is disjunctive enough that it’s not clear most of the probability mass can be captured by a single scenario/story, even as broad as this one tries to be. Here are some additional scenarios that don’t fit into this story or aren’t made very salient by it.
AI-powered memetic warfare makes all humans effectively insane.
Humans break off into various groups to colonize the universe with the help of their AIs. Due to insufficient “metaphilosophical paternalism”, they each construct their own version of utopia which is either directly bad (i.e., some of the “utopias” are objectively terrible or subjectively terrible according to my values), or bad because of opportunity costs.
AI-powered economies have much higher economies of scale because AIs don’t suffer from the kind of coordination costs that humans have (e.g., they can merge their utility functions and become clones of each other). Some countries may try to prevent AI-managed companies from merging for ideological or safety reasons, but others (in order to gain a competitive advantage on the world stage) will basically allow their whole economy to be controlled by one AI, which eventually achieves a decisive advantage over the rest of humanity and does a treacherous turn.
The same incentive for AIs to merge might also create an incentive for value lock-in, in order to facilitate the merging. (AIs that don’t have utility functions might have a harder time coordinating with each other.) Other incentives for premature value lock-in might include defense against value manipulation/corruption/drift. So AIs end up embodying locked-in versions of human values which are terrible in light of our true/actual values.
I think the original “stereotyped image of AI catastrophe” is still quite plausible, if for example there is a large amount of hardware overhang before the last piece of puzzle for building AGI falls into place.
- What Failure Looks Like: Distilling the Discussion by 29 Jul 2020 21:49 UTC; 81 points) (
- Persuasion Tools: AI takeover without AGI or agency? by 20 Nov 2020 16:54 UTC; 81 points) (
- What would we do if alignment were futile? by 14 Nov 2021 8:09 UTC; 75 points) (
- 26 May 2020 9:48 UTC; 22 points) 's comment on AGIs as collectives by (
- Persuasion Tools: AI takeover without AGI or agency? by 20 Nov 2020 16:56 UTC; 15 points) (EA Forum;
- 26 May 2020 8:59 UTC; 9 points) 's comment on AGIs as collectives by (
- 28 Aug 2023 2:10 UTC; 9 points) 's comment on A list of core AI safety problems and how I hope to solve them by (
- 18 Dec 2021 10:36 UTC; 5 points) 's comment on Persuasion Tools: AI takeover without AGI or agency? by (
Shouldn’t someone (some organization) be putting a lot of effort and resources into this strategy (quoted below) in the hope that AI timelines are still long enough for the strategy to work? With enough resources, it should buy at least a few percentage of non-doom probability (even now)?
Given that there are known ways to significantly increase the number of geniuses (i.e., von Neumann level, or IQ 180 and greater), by cloning or embryo selection, an obvious alternative Singularity strategy is to invest directly or indirectly in these technologies, and to try to mitigate existential risks (for example by attempting to delay all significant AI efforts) until they mature and bear fruit (in the form of adult genius-level FAI researchers).
After the truth destroyed everything it could, the only thing left was Eliezer Yudkowsky.
In his free time, Eliezer Yudkowsky likes to help the Halting Oracle answer especially difficult queries.
Eliezer Yudkowsky actually happens to be the pinnacle of Intelligent Design. He only claims to be the product of evolution to remain approachable to the rest of us.
Omega did its Ph.D. thesis on Eliezer Yudkowsky. Needless to say, it’s too long to be published in this world. Omega is now doing post-doctoral research, tentatively titled “Causality vs. Eliezer Yudkowsky—An Indistinguishability Argument”.
It seems like a large part of the problem is not that our brains unconsciously optimize for prestige per se, but they incorrectly optimize for prestige. Surely, having to take extra years to graduate and damaging one’s own cause are not particularly prestigious. Helping Eliezer write a book will at least net you an acknowledgement, and you also get to later brag about how you were willing to do important work that nobody else was.
I don’t have much empirical data to support this, but I suspect it might help (or at least might be worth trying to see if it helps) if you consciously optimized for prestige and world-saving simultaneously (as well as other things that you unconsciously want, like leisure), instead of trying to fight yourself. I have a feeling that in the absence of powerful self-modification technologies, trying to fight one’s motivation to seek prestige will not end well.
- 20 Nov 2011 20:26 UTC; 5 points) 's comment on The curse of identity by (
We have a lot of experience and knowledge of building systems that are broadly beneficial and safe, while operating in the human capabilities regime.
What? A major reason we’re in the current mess is that we don’t know how to do this. For example we don’t seem to know how to build a corporation (or more broadly an economy) such that its most powerful leaders don’t act like Hollywood villains (race for AI to make a competitor ‘dance’)? Even our “AGI safety” organizations don’t behave safely (e.g., racing for capabilities, handing them over to others, e.g. Microsoft, with little or no controls on how they’re used). You yourself wrote:
Unfortunately, given that most other actors are racing for as powerful and general AIs as possible, we won’t share much in terms of technical details for now.
How is this compatible with the quote above?!
Metaculus had only a 8% chance of “Russian Troops in Kyiv in 2022” as late as Feb 11. (It’s now at 97%.) Why did everyone do so badly on this prediction?
I strongly downvoted this post because I want to discourage this type of post on LW (at least for now), as it’s currently impossible to discuss these issues honestly in public (from certain perspectives) without incurring unacceptable levels of political and PR risk, both for individual commenters and LW / the rationality community as a whole. (I strongly upvoted one of your other posts to even out your karma.) I wish the LW team would prioritize thinking about how to enable such discussions to happen more safely on LW, but until that’s possible, I’d rather not see LW host discussions where only some perspectives can be represented.
(If you disagree with this, perhaps one compromise could be to post this kind of content on another forum or your personal blog, link to it from LW, and encourage people to only comment on the original post, to put more distance between the two and reduce risks.)
Is it just me, or do Luke and Eliezer’s initial responses appear to send the wrong signals? From the perspective of an SI critic, Luke’s comment could be interpreted as saying “for us, not being completely incompetent is worth bragging about”, and Eliezer’s as “we’re so arrogant that we’ve only taken two critics (including Holden) seriously in our entire history”. These responses seem suboptimal, given that Holden just complained about SI’s lack of impressive accomplishments, and being too selective about whose feedback to take seriously.
- 11 May 2012 18:26 UTC; 39 points) 's comment on Thoughts on the Singularity Institute (SI) by (
You already know that you know how to compute “Awesomeness”, and it doesn’t feel like it has a mysterious essence that you need to study to discover.
I wish! Both metaethics and normative ethics are still mysterious and confusing to me (despite having read Eliezer’s sequence). Here’s a sample of problems I’m faced with, none of which seem to be helped by replacing the word “right” with “awesome”: 1 2 3 4 5 6 7 8 9. I’m concerned this post might make a lot of people feel more clarity than they actually possess, and more importantly and unfortunately from my perspective, less inclined to look into the problems that continue to puzzle me.
So sharing evidence the normal way shouldn’t be necessary. Asking someone “what’s the evidence for that?” implicitly says, “I don’t trust your rationality enough to take your word for it.”
I disagree with this, and explained why in Probability Space & Aumann Agreement . To quote the relevant parts:
There are some papers that describe ways to achieve agreement in other ways, such as iterative exchange of posterior probabilities. But in such methods, the agents aren’t just moving closer to each other’s beliefs. Rather, they go through convoluted chains of deduction to infer what information the other agent must have observed, given his declarations, and then update on that new information. (The process is similar to the one needed to solve the second riddle on this page.) The two agents essentially still have to communicate I(w) and J(w) to each other, except they do so by exchanging posterior probabilities and making logical inferences from them.
Is this realistic for human rationalist wannabes? It seems wildly implausible to me that two humans can communicate all of the information they have that is relevant to the truth of some statement just by repeatedly exchanging degrees of belief about it, except in very simple situations. You need to know the other agent’s information partition exactly in order to narrow down which element of the information partition he is in from his probability declaration, and he needs to know that you know so that he can deduce what inference you’re making, in order to continue to the next step, and so on. One error in this process and the whole thing falls apart. It seems much easier to just tell each other what information the two of you have directly.
In other words, when I say “what’s the evidence for that?”, it’s not that I don’t trust your rationality (although of course I don’t trust your rationality either), but I just can’t deduce what evidence you must have observed from your probability declaration alone even if you were fully rational.
- 16 Nov 2019 8:20 UTC; 16 points) 's comment on I’m Buck Shlegeris, I do research and outreach at MIRI, AMA by (EA Forum;
That’s the path the world seems to be on at the moment. It might end well and it might not, but it seems like we are on track for a heck of a roll of the dice.
I agree with almost everything you’ve written in this post, but you must have some additional inside information about how the world got to this state, having been on the board of OpenAI for several years, and presumably knowing many key decision makers. Presumably this wasn’t the path you hoped that OpenAI would lead the world onto when you decided to get involved? Maybe you can’t share specific details, but can you at least talk generally about what happened?
(In additional to satisfying my personal curiosity, isn’t this important information for the world to have, in order to help figure out how to get off this path and onto a better one? Also, does anyone know if Holden monitors the comments here? He apparently hasn’t replied to anyone in months.)
We’re all living in a figment of Eliezer Yudkowsky’s imagination, which came into existence as he started contemplating the potential consequences of deleting a certain Less Wrong post.
- 20 Jul 2014 10:06 UTC; 15 points) 's comment on [LINK] Another “LessWrongers are crazy” article—this time on Slate by (
- 3 Nov 2021 2:24 UTC; 6 points) 's comment on [Book Review] “The Bell Curve” by Charles Murray by (
Another detail: My grandmother planed to join the Communist Revolution together with two of her classmates, who made it farther than she did. One made it all the way to Communist controlled territory (Yan’an) and later became a high official in the new government. She ended up going to prison in one of the subsequent political movements. Another one almost made it before being stopped by Nationalist authorities, who forced her to write a confession and repentance before releasing her back to her family. That ended up being dug up during the Cultural Revolution and got her branded as a traitor to Communism.
Is there a standard name for the logical fallacy where you attempt a reductio ad absurdum but fail to notice that you’re deriving the absurdity from more than one assumption? Why conclude that it’s the caring about far-away strangers that is crazy, as opposed to the decision algorithm that says you should give in to extortions like this?
- 7 Jun 2011 19:55 UTC; 0 points) 's comment on St. Petersburg Mugging Implies You Have Bounded Utility by (
The three examples from The Art of Strategy don’t seem to measure up to the book’s reputation.
The Evil Plutocrat
Meanwhile, in their meeting, the Republicans think the same thing. The vote ends with all members of Congress supporting your bill, and you don’t end up having to pay any money at all.
This isn’t a Nash equilibrium. If I’m a congressman and I expect everyone else to vote for the bill, then I should vote against it since in that case doing so has no bad financial consequences and I get to represent my constituents.
The Hostile Takeover
Suppose the investor believes you will succeed in taking over the company. That means your competitor will not take over the company, and its $101 offer will not apply. That means that the new value of the shares will be $90, the offer you’ve made for the second half of shares.
It was mentioned that the stock is currently worth $100 on the open market, so why should I sell my shares to you for $90 instead of selling them on the open market? Is it assumed that there aren’t legal protections for minority shareholders so whoever buys 50% of the shares can plunder the company and diminish its value for the other shareholders?
The Hostile Takeover, Part II
if the motion passes 3-2, then the two ‘no’ voters get no compensation and the three ‘yes’ voters may remain on the board and will also get a spectacular prize—to wit, our company’s 51% share in your company divided up evenly among them.
Presumably the board has no power to split your shares among themselves, so what you’re doing is making a promise that if the board votes that way, then you will split you shares among them. But if the board members reason backwards, they would think that if they did vote that way, you’d have no reason to actually fulfill your promise. So what the author seems to be doing is exempting the “evil villain” from backwards reasoning (or equivalently, giving him alone the power of making credible promises).
Reposting a Twitter reply to Eliezer:
Eliezer: I no longer believe that this civilization as it stands would get to alignment with 30-50 years of hard work. You’d need intelligence augmentation first. This version of civilization and academic bureaucracy is not able to tell whether or not alignment work is real or bogus.
Wei: Given this, why put all your argumentative eggs in the “alignment is hard” basket? (If you’re right, then policymakers can’t tell that you’re right.) Why not talk more about some of the other reasons to slow down or pause AI? https://www.lesswrong.com/posts/WXvt8bxYnwBYpy9oT/the-main-sources-of-ai-risk
Expanding on this, I’m worried that if MIRI focused all its communications efforts on “alignment is hard” and there’s apparent alignment progress in the next few years, which may be real progress (I have a wider distribution on alignment difficulty than Eliezer) or fake progress that most people can’t distinguish from real progress, MIRI and MIRI-adjacent people would lose so much credibility that it becomes impossible to pivot to other messages about AI risk.
- 7 Jan 2024 2:10 UTC; 38 points) 's comment on MIRI 2024 Mission and Strategy Update by (
Lessons I draw from this history:
To predict a political movement, you have to understand its social dynamics and not just trust what people say about their intentions, even if they’re totally sincere.
Short term trends can be misleading so don’t update too much on them, especially in a positive direction.
Lots of people who thought they were on the right side of history actually weren’t.
Becoming true believers in some ideology probably isn’t good for you or the society you’re hoping to help. It’s crucial to maintain empirical and moral uncertainties.
Risk tails are fatter than people think.