their refusal to implement adequate safeguards
This received a citation needed tag and I’m not sure why. The fact that they haven’t implemented adequate safeguards is evident from the fact that their model escaped, attacked HF, and OpenAI didn’t even notice for a week.
Their model broke out of their sandboxes, committed offensive cyber operations against a billion dollar company, and OpenAI only noticed after a week passed. IMO it kind of doesn’t matter whether or not they’re “compromised” or “bad at security“ or what. You should assume that OpenAI does not have the ability to sandbox their models or to monitor their behavior meaningfully.
There have been loss of control and models escaping sandboxing incidents at both OpenAI and Anthropic for over a year. They’ve publicly disclosed some of them (e.g., the latest system cards have stories about this from both companies) and some of them have been leaked within the community.
I know for a fact that OpenAI and Anthropic have been warned by internal and external experts that their security infrastructure for testing misaligned agentic coding agents is insufficient because I have personally told them that as have several former staff members. I had a debate with the head of security (?) at Anthropic at DEF CON in 2023 where I was pressing him on the fact that Anthropic wasn’t building air gapped networks.
It is absolutely within OpenAI and Anthropic’s ability to build an air gapped system for developing and testing these models. It seems likely to me that an internal GitHub clone and a moderately sized intranet would be sufficient to test the vast majority of agentic and web-enabled capabilities on such a platform. I think that their refusal to implement adequate safeguards is unjustifiable, but based on conversations with current and former safety and security researchers at OpenAI it seems like a company culture and lack of executive leadership buy-in problem that’s very hard to change without massive external pressure.
One issue that seems very worrisome today is that back in like 2023 an OpenAI security researcher was telling me about how they were unable to get OpenAI staff to stop using unreleased and inadequately tested models to develop internal infra, including internal monitoring tooling. I wish I remembered the person‘s name, I’d love to follow up.
Note: the final paragraph is an anecdote was told me with the expectation that I not disclose it to anyone else. Given recent events I view breaking that trust as akin to being a whistleblower. If the person who told me that is reading this, I’m sorry. Before last week I never disclosed it to anyone.
You receive a final report with documented results. Promising pilots are recommended to DARPA for a 3-month extended engagement, contingent on your availability.
What about data and code? Suppose a project looks like “here is a FT algorithm that will increase controllability” or “if you curate your dataset in this way the model won’t learn to do such and such.” There is a significant amount of research output that would be very valuable and isn’t found in a report.
What does the IP agreement look like? Does AE Studio own the IP? Do I? Does the USG? What hurdles would need to be passed to publish the results?
Training a 6B model for 500B tokens costs about 20k USD. That number increases linearly with model size and # of tokens, and decreases with the amount of money you have. Work like this is super doable, especially at large labs.
This is one area where I hope the USG will be able to exert coercive force to bring companies to heel. Early access evals, access to base models, and access to training data seem like no-brainers from a regulatory POV.
I think you’re misrepresenting Gwern’s argument. He’s arguing that terrorists are not optimizing for killing the most people. He makes no claims about whether terrorists are scientifically incompetent.
Thanks! I like your title more :)
It seems helpful to me if policy discussions can include phrases like “the evidence suggests that if the current ML systems were trying to deceive us, we wouldn’t be able to change them not to”.
I take this as evidence that TurnTrout’s fears about this paper are well-grounded. This claim is not meaningfully supported by the paper, but I expect many people to repeat it as if it is supported by the paper.
We ended up talking about this in DMs, but to gist of it is:
Back in June Hoagy opened a thread in our “community research projects” channel and the work migrated there. Three of the five authors of the [eventual paper](https://arxiv.org/abs/2309.08600) chose to have EleutherAI affiliation (for any work we organize with volunteers, we tell them they’re welcome to use an EleutherAI affiliation on the paper if they like) and we now have an entire channel dedicated to future work. I believe Hoagy has two separate paper ideas currently in the works and over a half dozen people working on them.
Ooops. It appeared that I deleted my comment (deeming it largely off-topic) right as you were replying. I’ll reproduce the comment below, and then reply to your question.
I separately had a very weird experience with them on the Long Term Future Fund where Conor Leahy applied for funding for Eleuther AI. We told him we didn’t want to fund Eleuther AI since it sure mostly seemed like capabilities-research but we would be pretty interested in funding AI Alignment research by some of the same people. He then confusingly went around to a lot of people around EleutherAI and told them that “Open Phil is not interested in funding pre-paradigmatic AI Alignment research and that that is the reason why they didn’t fund Eleuther AI”. This was doubly confusing and misleading because Open Phil had never evaluated a grant to Eleuther AI (Asya who works at Open Phil was involved in the grant evaluation as a fund member, but nothing else), and of course the reason he cited had nothing to do with the reason we actually gave. He seems to have kept saying this for a long time even after I think someone explicitly corrected the statement to
While this anecdote is largely orthogonal to the broader piece, I remembered that this existed randomly today and wanted to mention that Open Phil has recommended a 2.6 M/3 years grant to EleutherAI to pursue interpretability research. It was a really pleasant and very easy experience: Nora Belrose (head of interpretability) and I (head of everything) talked with them about some of our recent and on-going work such as Eliciting Latent Predictions from Transformers with the Tuned Lens, Eliciting Latent Knowledge from Quirky Language Models, and Sparse Autoencoders Find Highly Interpretable Features in Language Models very interesting and once they knew we had shared areas of interest it was a really easy experience.
I had no vibes along the lines of “oh we don’t like EleutherAI” or “we don’t fund pre-paradigmatic research.” It was a surprise to some people at Open Phil that we had areas of overlapping interest, but we spent like half an hour clarifying our research agenda and half an hour talking about what we wanted to do next and people were already excited.
I agree that a control group is vital for good science. Nonetheless, I think that such an experiment is valuable and informative, even if it doesn’t meet the high standards required by many professional science disciplines. I believe in the necessity of acting under uncertainty. Even with its flaws, this study is sufficient evidence for us to want to enact temporary regulation at the same time as we work to provide more robust evaluations.
But… this study doesn’t provide evidence that LLMs increase bioweapon risk.
It doesn’t let the government institute prior restraint on speech.
So far, I’m confident that our proposals will not impede the vast majority of AI developers, but if we end up receiving feedback that this isn’t true, we’ll either rethink our proposals or remove this claim from our advocacy efforts.
It seems to me like you’ve received this feedback already in this very thread. The fact that you’re going to edit the claim to basically say “this doesn’t effect most people because most people don’t work on LLMs” completely dodges the actual issue here, which is that there’s a large non-profit and independent open source LLM community that this would heavily impact.
I applaud your honestly in admitting one approach you might take is to “remove this claim from our advocacy efforts,” but am quite sad to see that you don’t seem to care about limiting the impact of your regulation to potentially dangerous models.
Nora didn’t say that this proposal is harmful. Nora said that if Zach’s explanation for the disconnect between their rhetoric and their stated policy goals is correct (namely that they don’t really know what they’re talking about) then their existence is likely net-harmful.
That said, yes requiring everyone who wants to finetune LLaMA 2 get a license would be absurd and harmful. la3orn and gallabyres articulate some reasons why in this thread.
Another reason is that it’s impossible to enforce, and passing laws or regulations and then not enforcing them is really bad for credibility.
Another reason is that the history of AI is a history of people ignoring laws and ethics so long as it makes them money and they can afford to pay the fines. Unless this regulation comes with fines so harsh that they remove all possibility of making money off of models, OpenAI et al. won’t be getting licenses. They’ll just pay the fines while small scale and indie devs (who allegedly the OP is specifically hoping to not impact) screech their work to a halt and wait for the government to tell them it’s okay for them to continue to do their work.
Also, such a regulation seems like it would be illegal in the US. While the government does have wide latitude to regulate commercial activities that impact multiple states, this is rather specifically a proposal that would regulate all activity (even models that never get released!). I’m unaware of any precedent for such an action, can you name one?
CAIP is also advised by experts from other organizations and is supported by many volunteers.
Who are the experts that advise you? Are claims like “our proposals will not impede the vast majority of AI developers” vetted by the developers you’re looking to avoid impacting?
It’s always interesting to see who has legitimacy in the eyes of mainstream media. The “other companies” mentioned are EleutherAI and Open Future, both of whom co-authored the letter, and LAION who signed it. All three orgs are major players in the open source AI space, and EAI & LAION are arguably bigger than GitHub and CC given that this is specifically about the impact of the EU AI Act on open source large scale AI R&D. Of course, MSN’s target audience hasn’t heard of EleutherAI or LAION.
Note that other orgs have also done blog posts on this topic: EleutherAI (co-written by me), Hugging Face, Creative Commons, Open Future.
It’s extremely difficult to create a fraudulent company and get it listed on the NYSE. Additionally, the Exchange can and does stop trading on both individual stocks and the exchange as a whole, though due to the downstream effects on consumer confidence this is only done rarely.
I don’t know what lessons one should learn from the stock market regarding MM, but I don’t think we should rush to conclude MM shouldn’t intervene or shouldn’t be blamed for not intervening.
I’ve tried to edit my message for clarity (the cultural comments were meant to apply to OAI, not Anthropic) but we know that whatever safeguards Anthropic has are insufficient because the Mythos model regularly broke out of their sandboxes and gained access to the open internet during development and testing. According to the system card, about 20% of attempts by the model to do so were successful!