Mischaracterizations, misleading language, and false dichotomies count as misinformation. Just because it’s prevalent on the modern internet doesn’t change the fact that it misdirects people in a tangential direction away from having accurate models of reality.
What makes internet content misinformation is about how manipulative and misleading their piece is, not about plausible deniability that the author could have unintentionally gotten something wrong or thinking suboptimal thoughts. Real life interactions have lower standards for misinformation, because the internet contains massive billion-dollar industries for lying to people at large scale, with the industry systematically being optimized to make the authors and outlets immune to accusations of outright lying.
Eliezer also hereby gives a challenge to the reader: Eliezer and Nate are thinking about writing up their thoughts at some point about OpenAI’s plan of using AI to aid AI alignment. We want you to write up your own unanchored thoughts on the OpenAI plan first, focusing on the most important and decision-relevant factors, with the intent of rendering our posting on this topic superfluous.Our hope is that challenges like this will test how superfluous we are, and also move the world toward a state where we’re more superfluous / there’s more redundancy in the field when it comes to generating ideas and critiques that would be lethal for the world to never notice.
Eliezer also hereby gives a challenge to the reader: Eliezer and Nate are thinking about writing up their thoughts at some point about OpenAI’s plan of using AI to aid AI alignment. We want you to write up your own unanchored thoughts on the OpenAI plan first, focusing on the most important and decision-relevant factors, with the intent of rendering our posting on this topic superfluous.
Our hope is that challenges like this will test how superfluous we are, and also move the world toward a state where we’re more superfluous / there’s more redundancy in the field when it comes to generating ideas and critiques that would be lethal for the world to never notice.
I strongly endorse this, based on previous personal experience with this sort of thing. Crowdsourcing routinely fails at many things, but this isn’t one of them (it does not routinely fail).
It’s a huge relief to see that there are finally some winning strategies, lately there’s been a huge scarcity of those.
I don’t use twitter because social media use seems like a losing strategy for living in the world over the next 10 years. Is there any alternative such as an email newsletter?
Does AI policy count, or is this more technical AI safety? I do a specific niche in AI policy so my knowledge is somewhat lopsided, and I have the credentials to prepare people for AI policy but I’d definitely like to meet people and get industry advice.
Yes, that was me too! But it’s a bit long.
This is a really good way of putting it, and I’ve never thought about it this effectively before. Before reading this, my best thoughts like this on crypto were comparing cryptocoins to gold reserves or corn or oil, which are difficult to flood the market with. But, like, you might end up disappointed if a new cold war starts and world gold reserves get sold off. But that risk is nothing compared to crypto exchanges tanking the industry or everything related to hacking. If AI becomes a fire hose of cognitive labor that makes everything topsy turvy then that’s even worse for crypto.
I don’t know about big reasoning machines, but I’ve heard a lot of rumors about LLMs being integrated into an extremely wide variety of extant ML systems that were already commercially viable on their own. It seems pretty intuitive to me that LLMs can provide some very good layers to support other systems. What have people heard about that?
That’s actually the reason why I put the disclaimer on, since it’s definitely annoying for some people but also extremely helpful for people who do AI policy, foreign affairs, and nuclear risk analysis. I understand that the Trump title might repel some people from the forum, but the post is pretty unambiguously for professional purposes and not for political discussion. I don’t expect it to be on the front page for more than 12 hours anyway.
Ideally I’d put it on EAforum since this is mainly a rationalist forum not an AI policy forum, but there’s some serious problems with that for a variety of reasons. I can trust the forecasting/policy community on Lesswrong the way I can’t trust the people on EAforum. Also, it might be AI policy but I’m definitely not putting it on alignment forum, for exactly the reasons you’ve stated above.
A prediction market is only as good as the participants. People with the insider information and highly lucrative information necessary to forecast FTX-like catastrophes generally don’t participate in prediction markets, especially not free ones, and probably won’t any time soon. That’s something that financial and analyst firms can do, because they have ways to guarantee that their org’s members are safe, and that they won’t get targeted by all sorts of vicious people out there.
Startups are fundamentally incapable of this. Wall street is highly capable of this, possibly also legal and government contractors that I’m not aware of, but it’s only possible because those systems are fundamentally closed and ingrained inside large nontransparent institutions that have built up networks of trust for decades.
I just want to clarify that there are also “create more alignment researchers” people, not just “buy time” people and “technical alignment” people. I am legally and morally obligated to avoid anything related to “buying time”. And I also don’t touch it with a ten foot pole because it seems much much much easier and safer to double the number of people working on alignment than to halve the annual R&D of the global AI industry.
It seems like we’re all getting distracted from the main point here. It doesn’t even matter whether SBF did it, let alone why. What matters is what this says about the kind of world we live in, for the last 20 years, and, now, for the last 7 days:
I strongly suspect that in ten years from now, conventional wisdom will hold the above belief as being basically cannon, regardless of further evidence in either direction. This is because it presents an intrinsically interesting, almost Hollywood villain-esque narrative, one that will surely evoke endless “hot takes” which journalists, bloggers, etc. will have a hard time passing over. Expect this to become the default understanding of what happened (from outsiders at least), and prepare accordingly.
The fact that Lesswrong is vulnerable to this, let alone EA, is deeply disturbing. Smart people are supposed to automatically coordinate around this sort of thing, because that’s what agents do, and that’s not what’s happening right now. This is basically a Quirrell moment in real life; a massive proportion of people on LW are deferring their entire worldview to obvious supervillains.
I think to maximize legibility to different kinds of people, it helps to include “instrumental convergence usually involves all things converging on the path of accumulating resources (including information) and self-preservation, and, by extension, anything that contributes to self-preservation and accumulating resources”.
Instrumental Convergence might seem like common language, but it certainly is not, and it really should be. Evocative examples increase word count, but they’re so helpful that cutting them out in order to lower word count is basically Goodharting. At minimum, it will help an unfamiliar reader quickly and conveniently explain it to others in many casual conversations.
Also, this post is very helpful, I have archived it in my open-source intelligence/research folder and I plan to cite it in the future. The way this post was written is actually the ideal way to explain AI safety to someone for the first time since it allows a wide variety of intelligent people to really dive into the problem in a way they understand. I recommend submitting it to the AI Safety Public Materials Contest which I recently confirmed will still read, evaluate, and implement good papers for explaining AI safety to people for the first time. Regardless of whether they have any money to award, they will still credit the author with a ranking that will build credibility.