Researcher at MIRI
peterbarnett
Karma: 3,492
Great point! This possibly makes my proposal a Bad idea. I would need to know more about how the labs respond to this kind of incentive to actually know.
Model providers often don’t provide the full CoT, and instead provide a summary. I think this is a fine/good thing to do to help prevent distillation.
However, I think it would be good if the summaries provided a flag for when the CoT contained evaluation awareness or scheming (or other potentially concerning behavior).
I worry that currently the summaries don’t really provide this information, and this probably makes alignment and capability evaluations less valid.
What they don’t do is filter out every web page that has the canary string. Since people put them on random web pages (like this one), which was not their intended use, they get into the training data.
As others have mentioned, this seems kinda crazy and bad. I was surprised you didn’t think this.
“Unrelated” question, but are you under a non-disparagement agreement with GDM that would prevent you from criticizing things like their data-filtering practices?
Considerations for setting the FLOP thresholds in our example international AI agreement
New Report: An International Agreement to Prevent the Premature Creation of Artificial Superintelligence
There’s a funny and bad incentive where I want to upvote posts I haven’t read to push them past the 30 Karma threshold and make them appear on the podcast feed.
I expect the line to blur between introspective and extrospective RSI. For example, you could imagine AIs trained for interp to doing interp on themselves, directly interpretting their own activations/internals and then making modifications while running.
Someone please explain
I think that one of the ways in which ice cream probably easily beats bear fat (+honey and salt), is that you can eat a bowl of icec ream and not feel terrible. It’s very plausible that the bear fat is awesome for a couple of bites, but if you try eat half a cup you will probably want to vomit.
I guess that influencing P(future goes extremely well | no AI takeover) maybe pretty hard, and plagued by cluelessness problems. Avoiding AI takeover is a goal that I have at least some confidence is good.
That said, I do wish more people were thinking about to make the future go well. I think my favorite thing to aim for is increasing the probability that we do a Long Reflection, although I haven’t really thought at all about how to do that.
I would love for someone to tell me how big a deal these vulnerabilities are, and how hard people had previously been trying to catch them. The blog post says that two were severity “Moderate”, and one was “Low”, but I don’t really know how to interpret this.
I would guess that this is mainly due to there being much more limited FLOP data for the closed models (especially for recent models), and for closed models focusing much less on small training FLOP models (eg <1e25 FLOP)
I think that the proposal in the book would “tank the global economy”, as defined by a >10% drop in the S&P 500, and similar index funds, and I think this is a kinda reasonable definition. But I also think that other proposals for us not all dying probably have similar (probably less severe) impacts because they also involve stopping or slowing AI progress (eg Redwood’s proposed “get to 30x AI R&D and then stop capabilities progress until we solve alignment” plan[1]).
- ^
I think this is an accurate short description of the plan, but it might have changed last I heard.
- ^
I think it’s useful to think about the causation here.
Is it:
Intervention → Obvious bad effect → Good effect
For example: Terrible economic policies → Economy crashes → AI capability progress slows
Or is it:
Obvious bad effect ← Intervention → Good effect
For example: Patient survivably poisoned ← Chemotherapy → Cancer gets poisoned to death
AI Generated Podcast of the 2021 MIRI Conversations
The Arbital link (Yudkowsky, E. – “AGI Take-off Speeds” (Arbital 2016)) in there is dead, I briefly looked at the LW wiki to try find the page but didn’t see it. @Ruby?
I first saw it in the this aug 10 WSJ article: https://archive.ph/84l4H
I think it might have been less public knowledge for like a year
Carl Shulman is working for Leopold Aschenbrenner’s “Situational Awareness” hedge fund as the Director of Research. https://whalewisdom.com/filer/situational-awareness-lp
Thanks!