Tenoke
has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national
This wasn’t even supposed to be general which I can see sense in but only non-Americans? It’s hard to hate the American government and the people who elected this administration during the most critical period in history enough. I know how that sounds but..
anyone working on e.g. making models more honest in prod models
I don’t really think people working on ‘what instruciton can I add to the system prompt’ or equivalents are meaningfully working on the kind of endgame alignment the post is talking about.
Edit: Nothing wrong with that kind of work for current alignment, just doesn’t apply to endgame alignment all that much in my opinion.
This is true, and to be fair it’s a bit harder to even see how much outside organizations can even help at this point. The main companies have grown so much and share so much less, that outsiders have less influence, as well as in many cases less access to big models, and especially to being able to train them.
Some do have some access but it still seems limited compared to what an in-Anthropic team and in-OpenAI team could do. Of course, you also can end up with a result so good as an outsider that influences them but, again, it just seems like limited impact from the get go.
It’s probably time to truly start considering simulating some exemplar past people from their writings, with a truly large model finetuned to that exact task. Von-Neumann is a favorite, but I expect there’s even better candidates (in terms of quantity of output we can use for training and finetuning, character, morality, who’d likely consent to it, etc.). Who would be the best candidate? Gemini argues Bertrand Russell due to writing (including personal letters) so much well into his 90s.
In the case where LLMs are conscious or pseudo-conscious, I am unsure whether adding ‘you take joy in completing this task’ to the system prompt does more good or bad for them.
Again I want to preface that I don’t even dislike him but It’s simply that engaging with a community that he is a part of means engaging with him, and engaging with him gets tiresome. At best it’s technical contributions people kind of enjoy and at worst It’s kind of like talking to r/SneerClub if they were on ‘your’ side. And no single thing crosses a line, it is more a death by a thousand cuts that results in me valuing not communicating with him more than I value being there.
(Sorry for this when you read it, not that I imagine you care much about this type of comment)
No comment on the rest, but there’s a community I’ve quite liked throughout the last decade (and more) but rarely participate in anymore because he is active there—and that’s as someone who has less of a negative reaction to his kind of abrasiveness than others. I can see how a lot of people have had that experience with him but in regards to LessWrong.
I feel bad making a negative comment about him, but I understand why Habryka et al. would’ve made that decision in the end. I don’t know if that means he should be banned, but I understand it.
If roon is saying it, especially about Anthropic, my prior is that it is biased, or optimized for clicks/fame and not truth-seeking. Reading some of this, some of it rings directionally true, but it’s probably counter-productive to engage with it under the exaggerated framing he lays out in particular.
There used to be a lot of arguments about AI Timelines 5+ years ago of the sort ’if AI is coming why are the markets not reacting”. We’re now on the other side—by already being within the time horizon that markets react to—where the markets themselves are pointing in the directon of AGI, and people instead wonder how to undercount that (e.g. by saying it is a bubble, or that trends must slow).
separately, I am not sure what your comment is supposed to be doing.
My comment is a pretty neutral response to the central claim ‘Dario probably doesn’t believe in superintelligence’ which you specify you believe in and isn’t just a clickbait headline, and to the arguments for it. Do you react like this to all comments which disagree with you? May I suggest to just comment on the arguments in the comment than to have such a kneejerk reaction?
> Unless you mean to claim that I am wrong about how the thing he’s describing in MoLG is actually compatible with the kind of superintelligence I’m imagining?
Your definition is:
> Roughly speaking, that the returns to intelligence past the human level are large, in terms of the additional affordances they would grant for steering the world, and that it is practical to get that additional intelligence into a system.
As I said I think ‘a country of supergeniuses in a datacenter’ fits, yes:
>A country of geniuses in a datacenter is pretty clearly “Superintelligence” and he pretty clearly believes in it. He seems to rather belive that Superintelligence wouldn’t solve everything quite as quick as others think.
We could summarize this as a “country of geniuses in a datacenter”.
A country of geniuses in a datacenter is pretty clearly “Superintelligence” and he pretty clearly believes in it. He seems to rather belive that Superintelligence wouldn’t solve everything quite as quick as others think.
You can also use more recent sources—e.g. 2 months ago here where he discusses the topic with Demis Hassabis—they differ on timelines, how fast it’d achieve things etc. but clearly they both believe in it.
I suspect you wouldn’t believe a claim that it has improved it 4x (or whatever) months down the line either, and I struggle to see under what scenario you’d believe them, just how you dont trust this survey.
Relatedly, what do you believe the current improvement is for them from using their pre-Mythos models compared to if they used no AI? Is it close to nothing?
Also, don’t your estimates that if it was 4x timelines will be shortened by X forget that this survey compares to 0 AI, and the current timelines are (I hope) based on them already using AI?
This from June lists a lot of people who have read it, including Stephen Fry, Grimes, professors etc. Seperately on Twitter seemingly anyone who was someone in the scene had given their opinion after having read it.
Any thread from the first announcement onward had people saying they’ve read it already. From the same thread (and that was early on)
Many people (like >100 is my guess), with many different view points, have read the book and offered comments.
Note that IFP (a DC-based think tank) recently had someone deliver 535 copies of their new book to every US Congressional office.
More endorsements and there’s also a lot of twitter personalities that had mentioned reading it, which I wont hunt. It definitely felt like a lot more than 50. I’m not arguing it’s a bad or good strategy, just that it’s felt a bit off to wait for months for a ‘pre-order’ when anyone who I might see on Twitter and would’ve been interested to have read it already has.
While I mostly did it out of support and reducing x-risk, pre-ordering “If Anyone Builds It, Everyone Dies” has been one of the more frustrating book order experiences I’ve had. The main purpose of the order looks to be successful enough and the actual book experience doesn’t matter all that much but still:
I pre-ordered in mid May as soon as I heard about it, and since then it’s been months of nearly everyone on the Internet having already read it, then later pre-order prices (barely relevant) were lowered which seems a bit backwards, and now that it’s been ‘out’, I still don’t have the book (or even estimated shipping—from Amazon, Germany) while everyone else who hadn’t posted about it has now been posting reviews etc.
This is kind of annoying, as I’m not reading any of the commentary now—reading the book firsthand when I’ve already pre-ordered it would seem to make more sense, but by the time I even get it, It’d be far after most of the initial conversation happened so at this point I’m having a worse experience for having pre-ordered it.
Again, that experience is not that important, I’ve benefited a lot from Eliezer’s other writting before etc. but it’s disappointing enough to vent in at least one comment before taking the L and moving on.
While I believe SC2 and Dota would fail today with sufficient effort, the models didn’t quite perform superhuman, and as far as I am aware no community bots do either.
One of the reasons why it’s plausible that today’s or tomorrow’s LLMs can result in brief simulations of consciousness or even qualia is that it happens with dreams in humans. Dreams are likely some sort of processing of information/compression/garbage collection, yet they still result in (badly) simulated experiences as a clear side-effect of trying to work with human experience data.
I still want something even closer to Givewell but for AI Safety (though it is easier to find where to donate now than before). Hell, I wouldn’t mind if LW itself had recommended charities in a prominent place (though I guess LW now mostly asks for Lightcone donations instead).
Thanks for sharing this. Based on the About page, my ‘vote’ as a EU citizen working in an ML/AI position could conceivably count for a little more, so it seems worth doing it. I’ll put it in my backlog and aim to get to it on time (it does seem like a lengthy task).
If you don’t know who to believe then falling back on prediction markets or at least expert consensus is not the worst strategy.
It’d be profoundly sad if this leads to only Americans being allowed to legally access frontier models. It helps safety very little, while it hurts people like me a lot.