Wei Dai(Wei Dai)
A tale from Communist China
[Question] Have epistemic conditions always been this bad?
UDT shows that decision theory is more puzzling than ever
Morality is Scary
Forum participation as a research strategy
Meta Questions about Metaphilosophy
The Nature of Offense
Beyond Astronomical Waste
AI Safety “Success Stories”
Shut Up and Divide?
[Question] Where are people thinking and talking about global coordination for AI safety?
A broad basin of attraction around human values?
It was easier for Eliezer Yudkowsky to reformulate decision theory to exclude time than to buy a new watch.
Eliezer Yudkowsky’s favorite sport is black hole diving. His information density is so great that no black hole can absorb him, so he just bounces right off the event horizon.
God desperately wants to believe that when Eliezer Yudkowsky says “God doesn’t exist,” it’s just good-natured teasing.
Never go in against Eliezer Yudkowsky when anything is on the line.
[Question] Why is so much discussion happening in private Google Docs?
(Tangentially) If users are allowed to ban other users from commenting on their posts, how can I tell when the lack of criticism in the comments of some post means that nobody wanted to criticize it (which is a very useful signal that I would want to update on), or that the author has banned some or all of their most prominent/frequent critics? In addition, I think many users may be mislead by lack of criticism if they’re simply not aware of the second possibility or have forgotten it. (I think I knew it but it hasn’t entered my conscious awareness for a while, until I read this post today.)
(Assuming there’s not a good answer to the above concerns) I think I would prefer to change this feature/rule to something like allowing the author of a post to “hide” commenters or individual comments, which means that those comments are collapsed by default (and marked as “hidden by the post author”) but can be individually expanded, and each user can set an option to always expand those comments for themselves.
The option I bought is up 700% since I bought them, implying that as of 2/10/2020 the market thought there was less than 1⁄8 chance things would be as bad as they are today. At least for me this puts a final nail in the coffin of EMH.
Added on Mar 24: Just in case this thread goes viral at some point, to prevent a potential backlash against me or LW (due to being perceived as caring more about making money than saving lives), let me note that on Feb 8 I thought of and collected a number of ideas for preventing or mitigating the pandemic that I foresaw and subsequently sent them to several people working in pandemic preparedness, and followed up with several other ideas as I came across them.
- The EMH Aten’t Dead by 15 May 2020 20:44 UTC; 194 points) (
- Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis by 11 May 2020 6:00 UTC; 72 points) (
- Coronavirus: Justified Key Insights Thread by 13 Apr 2020 22:40 UTC; 50 points) (
- 13 Mar 2020 21:59 UTC; 33 points) 's comment on March Coronavirus Open Thread by (
- 11 Oct 2020 0:01 UTC; 17 points) 's comment on The Treacherous Path to Rationality by (
Two Neglected Problems in Human-AI Safety
Tips and Tricks for Answering Hard Questions
I think AI risk is disjunctive enough that it’s not clear most of the probability mass can be captured by a single scenario/story, even as broad as this one tries to be. Here are some additional scenarios that don’t fit into this story or aren’t made very salient by it.
AI-powered memetic warfare makes all humans effectively insane.
Humans break off into various groups to colonize the universe with the help of their AIs. Due to insufficient “metaphilosophical paternalism”, they each construct their own version of utopia which is either directly bad (i.e., some of the “utopias” are objectively terrible or subjectively terrible according to my values), or bad because of opportunity costs.
AI-powered economies have much higher economies of scale because AIs don’t suffer from the kind of coordination costs that humans have (e.g., they can merge their utility functions and become clones of each other). Some countries may try to prevent AI-managed companies from merging for ideological or safety reasons, but others (in order to gain a competitive advantage on the world stage) will basically allow their whole economy to be controlled by one AI, which eventually achieves a decisive advantage over the rest of humanity and does a treacherous turn.
The same incentive for AIs to merge might also create an incentive for value lock-in, in order to facilitate the merging. (AIs that don’t have utility functions might have a harder time coordinating with each other.) Other incentives for premature value lock-in might include defense against value manipulation/corruption/drift. So AIs end up embodying locked-in versions of human values which are terrible in light of our true/actual values.
I think the original “stereotyped image of AI catastrophe” is still quite plausible, if for example there is a large amount of hardware overhang before the last piece of puzzle for building AGI falls into place.
- What Failure Looks Like: Distilling the Discussion by 29 Jul 2020 21:49 UTC; 81 points) (
- Persuasion Tools: AI takeover without AGI or agency? by 20 Nov 2020 16:54 UTC; 81 points) (
- What would we do if alignment were futile? by 14 Nov 2021 8:09 UTC; 75 points) (
- 26 May 2020 9:48 UTC; 22 points) 's comment on AGIs as collectives by (
- Persuasion Tools: AI takeover without AGI or agency? by 20 Nov 2020 16:56 UTC; 15 points) (EA Forum;
- 26 May 2020 8:59 UTC; 9 points) 's comment on AGIs as collectives by (
- 28 Aug 2023 2:10 UTC; 9 points) 's comment on A list of core AI safety problems and how I hope to solve them by (
- 18 Dec 2021 10:36 UTC; 5 points) 's comment on Persuasion Tools: AI takeover without AGI or agency? by (
Lessons I draw from this history:
To predict a political movement, you have to understand its social dynamics and not just trust what people say about their intentions, even if they’re totally sincere.
Short term trends can be misleading so don’t update too much on them, especially in a positive direction.
Lots of people who thought they were on the right side of history actually weren’t.
Becoming true believers in some ideology probably isn’t good for you or the society you’re hoping to help. It’s crucial to maintain empirical and moral uncertainties.
Risk tails are fatter than people think.