Also, regarding:
> AIs that are sufficiently aligned/controlled to aid research, may remain a crucial part of AI safety research during an AI pause or slowdown.
If this research would also make the AIs better at AI R&D (which I think it usually would), then probably the capabilities teams would be happy to fund such research, which would make it extremely not-neglected (even if it’s still positive).
wdyt?
I think this definition hides the hard part, no?
Like, ~everyone would agree it’s good to do work that is “genuinely net-beneficial for minimizing existential risk and enabling flourishing futures”.
Seems like your question is more “is ~prosaic safety work net-beneficial” (right?)
Letting people get hurt seems bad
I agree!
But there are many places to work that would prevent people getting hurt.
If that’s a major reason for someone (prosaic harms like HF getting hacked, as distinct from doing work that might align ASI) then I’d encourage them to do the normal EA/80k thing of comparing possible jobs
Do you think the people doing RSI would keep going if there was nobody around like you to help them secure it? Would they let everything get hacked and stolen?
I worry the actual dynamic is: they claim they’ll do RSI no matter what, they focus a lot of resources on the dangerous acceleration, because they know that others will step in to do harm reduction.
I worry this is the dynamic of them making a costly threat only because they expect others to respond to it.
wdyt?
I’m also curious if this seems cruxy/important to you (separately from whether you think it’s correct)
I wanted to suggest writing a new system until I read the last line
The main downside here seems to be that having good monitors will make AIs more capable for commercial use cases --> which means developing foundation models will be (knowingly) more profitable, less likely to be a bubble, and so on.
What do you think?
Also, got opinions about the “sharing examples of misalignment while preserving privacy” part?
Also, you wrote:
the 100s of employees who push the boundaries of the frontier full-time
These employees aren’t all working on things like safety monitors (or whatever your startup would specialize in).
Hey, building safety monitors is a direction I’m considering myself.
The main thing I’m concerned about is: I might call them “safety monitors”, but in practice they might be “things that make AI more reliable for using in a commercial context”. In the counterfactual world where nobody builds these safety monitors, do people simply deploy AI less because they know AIs are unreliable? If AIs are less reliable, is making them less profitable?
Another way to phrase this is: “Could this project also be built by the capabilities team of [some AGI company that doesn’t care about safety beyond what they need for making money]?”. I’d be especially worried about this if your product is able to get an AI to automate large parts of technical AI Safety work—this sounds like it is mostly making AIs more capable, and (if I understand correctly) you’re considering selling this product.
On the other hand:
I think there’s a lot of potential for a project like this to find reproducible examples of misalignment in realistic tasks at scale[1] (which seems like a bottleneck for many current AI Safety research agendas).
The hard part here seems to be your customer’s privacy, since the example might involve e.g their proprietary code. I think there could be solutions here that would work for everyone but they’re don’t seem trivial.
Edit: I’m not generally against for-profit AI Safety companies; just replying with considerations I see with “safety monitors” and “automate AI safety research” as for-profit companies
I think you mentioned this in the post but neither me nor claude can find it
About Perplexity, Zvi wrote things about them like:
[Perplexity] will take those two things [the headline of the article and a rough description of what the article is about] and they will then make up the content of the article and publish it as if it was fact for, “This was published by this author at this time”.
Which makes me not want to use Perplexity.
Got opinions?
I don’t even know whether I think Anthropic having good security is good or bad for the world
Could you share the TL;DR for why this might be bad for the world?
I recently read through the entire pdf—big thumbs up from me!
A few questions / requests:
Would you release the transcripts of part/all the runs? (I’m not sure if there’s a consideration against doing this that I’m missing)
I’m most interested in “no goal” runs that still showed misaligned behavior
Would you share recordings of the computer-use if you have them?
And also the setup for reproducing the computer-use scenario? (I’m guessing you have some code for a fake browser which isn’t in your repo)
FYI, I think most people don’t know that you ran a variation where the model wasn’t given any goal (which is different from “no goal conflict”, which implies the existing and successor models have the same goal)
Big thumbs up again!
TL;DR: Can we collect real-world examples of misalignment while preserving privacy?
Naively, startups that monitor agents for unintended behavior could publish examples of misalignment they found, but I doubt almost any org would agree to this, since examples might involve the org’s code or other private data.
Work with orgs/agents that don’t handle private info. Maybe open source repos?
Only share the scenarios with a few research centers (Redwood?)
Would orgs agree[1] to this?
When misalignment is found, ask the org if it’s ok to share it. If even 1% agree it still might be nice.
Look up what created the culture of disclosing vulnerabilities in cyber security. Maybe that can be reproduced? (I do think the situation is very different here)
Sharing an anonymized/redacted version of the scenario
Seems a lot less useful to me (unless there is enough info to reproduce similar behavior), but it would be nice if we could get a lot of these
I found this interesting because the user of Replit seems to want to disclose the problem.
I can imagine having a norm (or a regulation?) where after Replit fix whatever problem they had—they release the LLM transcripts that are relevant to the incident.
(or maybe I’m wrong and this person wouldn’t want to share enough details for a full transcript)
Looking at Cursor’s Privacy Policy:
We do not use Inputs or Suggestions to train our models unless: (1) they are flagged for security review (in which case we may analyze them to improve our ability to detect and enforce our Terms of Service)
But who decides what is flagged? And can Cursor do things other than train their models in other cases?
Also, this is from their TOS:
5.4. Usage Data. Anysphere may: (i) collect, analyze, and otherwise process Usage Data internally for its business purposes, including for security and analytics, to enhance the Service, and for other development and corrective purposes
I’m not sure if this is a reliable hint for what orgs would agree to
This seems like the highest quality source of data for “what do LLMs want” I’ve heard of (am I missing others?), may I download and analyze it myself?
Thanks for doing this!
Fair,
Note they used GPT-3 which wasn’t trained with RLHF (right?)
Hey, I think I’m the target audience of this post, but it really doesn’t resonate well with me.
Here are some of my thoughts/emotions about it:
Feeling doom/depressed/sad isn’t a good idea. I wish I’d have less of this myself :( it isn’t helping anyone
Many people thought the world would end and they were wrong—this should give us a lower prior on ourselves being right about the world maybe ending
Biases are probably in play
I read you as trying to be supportive, which I appreciate, thanks
5. Was worrying about covid also similarly misguided?
At the beginning of covid, people said “stop making a big deal about it”, “lots of people freaked out about things in the past and were wrong”, I also live close to a city with a lot of (religious) anti-vax people, and I visited a (hippie) place where I told someone when I was last tested and what I did since, but she told me she sees I look good and I better not think about it, I’ll be healthier if I don’t think about covid too much.
Should I have listened to the people who told me to chill about covid? I assume you agree I shouldn’t have, beyond (I agree) no reason to emotionally freak out, just take reasonable actions, that’s all.
What’s the difference between the AI and covid scenarios? I assume you’d say “covid was real” (?). Anyway, that’s my crux. Is AI danger “real” (vaguely defined).
What if the covid deniers would say “if you’re so sure about covid, tell us exactly in what week (or whatever) everyone will die”—this seems like a hard/unfair question. I personally think people spend way too much energy geeking out about AI timelines, and after (metaphorically) a million requests for timelines, a serious team wrote ai-2027, which is a probability distribution, where they think there’s a 50% chance problems will happen earlier, and 50% they’ll happen later (right?). Telling me to chill at 2028 seems similar to asking for an estimate for when covid problems will actually hit, getting a median probability distribution, and telling me to “chill” if that median passed (while still things seem to be getting worse for similar reasons?). Yeah there’s a time where if AI doesn’t seem to be taking over then I’ll eat my hat and say I am completely missing something about how the world works, but that time is not 2028.
6. Fallacy Fallacy
I often think back on The Adventures Of Fallacy Man.
Both “sides” have fallacies. Maybe doomers want a way to deal with our mortality. Maybe non-doomers have a bias against imagining changes or whatever. I don’t think this kind of conversation is a good way to figure out what is true. I think AI is a complicated topic that makes it clear how problematic many rules-of-thumb or metaphors are and encourages us to look at the details, or at least to pick our mental shortcuts wisely.
Would you leave more anonymous feedback for people who ask for it if there was a product that did things like:
Rewrite your feedback to make it more anonymous (with an LLM?)
Aggregate your feedback with other feedback this person received, and only tell them things like “many people said you’re rude”
Delay your feedback by some (random?) amount of time to make it less recognizable
I’m mostly interested to hear from people who consider leaving feedback and sometimes don’t, I think it would be cool if we could make progress on solving whatever painpoint you have (not sure we’ll succeed, but worth trying).
If you prefer to reply anonymously (very meta!) you can say so here. I’ll assume it’s ok to share responses as comments to this shortform unless you ask me not to.
Idea credit: cwbakerlee
So FYI, the emoji I posted above was actually AI generated.
The real one looks like this (on my browser) :
I still think the original one looks a lot like a paperclip, but I cheated and exaggerated this in the April Fool’s version
Seems like Unicode officially added a “person being paperclipped” emoji:
Here’s how it looks in your browser: 🙂↕️
Whether they did this as a joke or to raise awareness of AI risk, I like it!
Hey,
On anti-trust laws, see this comment. I also hope to have more to share soon
https://x.com/EvanHub/status/2097497037956891126