It sure would.
Would you give 3:1 odds against a major player showing the untrusted model the old CoT, conditional on them implementing a “replace suspicious outputs” flow on a model with hidden CoT and publicizing enough implementation details to determine whether or not they did that?
If the untrusted model has a private and un-audited chain of thought (like o1), and that chain of thought is preserved between rounds, the untrusted model may have a very good idea of what its original command was before the auditor replaced the command.
Your “the untrusted model is stateless between queries and only sees the command history and system state” does cover this, but wanted to call out that that assumption might not hold if current architectures are indicative of the future direction of development.
I note that if software developers used that logic for thinking about software security, I expect that almost all software in the security-by-obscurity world would have many holes that would be considered actual negligence in the world we live in.
This suggests that the threat model isn’t so much “very intelligent AI” as it is “very cheap and at least somewhat capable robots”.
“Based on your understanding of AI technical developments as of March 29, 2023, evaluate the most important known object-level predictions of Eliezer Yudkowsky on the subject, and which ones seemed true versus false. Afterwards, evaluate those predictions as a group, on a scale from ‘mostly true’ to ‘mostly false.’“
I ran this prompt but substituted in “Gary Marcus” for “Eliezer Yudkowsky”. Claude says
Overall evaluation: On a scale from ‘mostly true’ to ‘mostly false,’ I would rate Gary Marcus’s predictions as a group as “Mostly True.”
Many of Marcus’s predictions about the limitations of current AI systems and the challenges ahead have proven accurate. His concerns about reasoning, abstract thinking, and the need for more sophisticated knowledge representation align with ongoing challenges in the field. His emphasis on AI safety and alignment has also been prescient.
However, it’s worth noting that some of his predictions might be seen as overly pessimistic by some in the AI community. The rapid progress in LLMs and their applications has surprised many, including some skeptics. Nonetheless, many of the fundamental challenges he pointed out remain relevant.
It’s also important to remember that the field of AI is rapidly evolving, and assessments of such predictions can change quickly as new breakthroughs occur. As of my last update in April 2024, many of Marcus’s key points still held true, but the field continues to advance at a rapid pace.
I think Claude likes saying nice things about people, so it’s worth trying to control for that.
Another issue is that a lot of o1’s thoughts consist of vagaries like “reviewing the details” or “considering the implementation”, and it’s not clear how to even determine if these steps are inferentially valid.
If you’re referring to the chain of thought summaries you see when you select the o1-preview model in chatgpt, those are not the full chain of thought. Examples of the actual chain-of-thought can be found on the learning to reason with LLMs page with a few more examples in the o1 system card. Note that we are going off of OpenAI’s word that these chain of thought examples are representative—if you try to figure out what actual reasoning o1 used to come to a conclusion you will run into the good old “Your request was flagged as potentially violating our usage policy. Please try again with a different prompt.”
Shutting Down all Competing AI Projects is not Actually a Pivotal Act
This seems like an excellent title to me.
Technically this probably isn’t recursive self improvement, but rather automated AI progress. This is relevant mostly because
It implies that, at least through the early parts of the takeoff, there will be a lot of individual AI agents doing locally-useful compute-efficiency and improvement-on-relevant-benchmarks things, rather than one single coherent agent following a global plan for configuring the matter in the universe in a way that maximizes some particular internally-represented utility function.
It means that multi-agent dynamics will be very relevant in how things happen
If your threat model is “no group of humans manages to gain control of the future before human irrelevance”, none of this probably matters.
My argument is more that the ASI will be “fooled” by default, really. It might not even need to be a particularly good simulation, because the ASI will probably not even look at it before pre-commiting not to update down on the prior of it being a simulation.
Do you expect that the first takeover-capable ASI / the first sufficiently-internally-cooperative-to-be-takeover-capable group of AGIs will follow this style of reasoning pattern? And particularly the first ASI / group of AGIs that actually make the attempt.
Yeah, my argument was “this particular method of causing actual human extinction would not work” not “causing human extinction is not possible”, with a side of “agents learn to ignore adversarial input channels and this dynamic is frequently important”.
It does strike me that, to OP’s point, “would this act be pivotal” is a question whose answer may not be knowable in advance. See also previous discussion on pivotal act intentions vs pivotal acts (for the audience, I know you’ve already seen it and in fact responded to it).
If an information channel is only used to transmit information that is of negative expected value to the receiver, the selection pressure incentivizes the receiver to ignore that information channel.
That is to say, an AI which makes the most convincing-sounding argument for not reproducing to everyone will select for those people who ignore convincing-sounding arguments when choosing whether to engage in behaviors that lead to reproduction.
… is it possible to train a simple single-layer model to map from residual activations to feature activations? If so, would it be possible to determine whether a given LLM “has” a feature by looking at how well the single-layer model predicts the feature activations?
Obviously if your activations are gemma-2b-pt-layer-6-resid-post and your sae is also on gemma-2b-pt-layer-6-pt-post your “simple single-layer model” is going to want to be identical to your SAE encoder. But maybe it would just work for determining what direction most closely maps to the activation pattern across input sequences, and how well it maps.
Disclaimer: “can you take an SAE trained on one LLM and determine which of the SAE features exist in a separately-trained LLM which uses the same tokenizer” is an idea I’ve been kicking around and halfheartedly working on for a while, so I may be trying to apply that idea where it doesn’t belong.
Although watermarking the meaning behind text is currently, as far as I know, science fiction.
Choosing a random / pseudorandom vector in latent space and then perturbing along that vector works to watermark images, maybe a related approach would work for text? Key figure from the linked paper:
You can see that the watermark appears to be encoded in the “texture” of the image, but in a way where that texture doesn’t look like the texture of anything in particular—rather, it’s just that a random direction in latent space usually looks like a texture, but unless you know which texture you’re looking for, knowing that the watermark is “one specific texture is amplified” doesn’t really help you identify which images are watermarked.
There are ways to get features of images that are higher-level than textures—one example that sticks in my mind is [Ensemble everything everywhere: Multi-scale aggregation for adversarial robustness](https://arxiv.org/pdf/2408.05446)
I would expect that a random direction in the latent space of the adversarially robust image classifier looks less texture-like but is still perceptible by a dedicated classifier far at far lower amplitude than the point where it becomes perceptible to a human.
As you note, things like “how many words have the first e before the first a” are texture-like features of text, and a watermarking schema that used such a feature would work at all. However, I bet you can do a lot better than such manually-constructed features, and I would not be surprised if you could get watermarking / steganography using higher-level features working pretty well for language models.
That said, I am not eager to spend unpaid effort to bring this technology into the world, since I expect most of the uses for a tool that allows a company to see if text was generated by their LLM would look more like “detecting noncompliance with licensing terms” and less like “detecting rogue behavior by agents”. And if the AI companies want such a technology to exist, they have money and can pay for someone to build it.
Edit: resized images to not be giant
If your text generation algorithm is “repeatedly sample randomly (at a given temperature) from a probability distribution over tokens”, that means you control a stream of bits which don’t matter for output quality but which will be baked into the text you create (recoverably baked in if you have access to the “given a prefix, what is the probability distribution over next tokens” engine).
So at that point, you’re looking for “is there some cryptographic trickery which allows someone in possession of a secret key to determine whether a stream of bits has a small edit distance from a stream of bits they could create, but where that stream of bits would look random to any outside observer?” I suspect the answer is “yes”.
That said, this technique is definitely not robust to e.g. “translate English text to French and then back to English” and probably not even robust to “change a few tokens here and there”.
Alternatively, there’s the inelegant but effective approach of “maintain an index of all the text you have ever created and search against that index, as the cost to generate the text is at least an order of magnitude higher[1] than the cost to store it for a year or two”.
I see $0.3 / million tokens generated on OpenRouter for llama-3.1-70b-instruct, which is just about the smallest model size I’d imagine wanting to watermark the output for. A raw token is about 2 bytes, but let’s bump that up by a factor of 50 − 100 to account for things like “redundancy” and “backups” and “searchable text indexes take up more space than the raw text”. So spending $1 on generating tokens will result in something like 0.5 GB of data you need to store.
Quickly-accessible data storage costs something like $0.070 / GB / year, so “generate tokens and store them in a searchable place for 5 years” would be about 25-50% more expensive than “generate tokens and throw them away immediately”.
Super interesting work!
I like the idea of seeing if there are any features from the base model which are dead in the instruction-tuned-and-fine-tuned model as a proxy for “are there any features which fine-tuning causes the fine-tuned model to become unable to recognize”. Another related question also strikes me as interesting, which is whether an SAE trained on the instruction-tuned model has any features which are dead in the base model—these might represent new features that the instruction-tuned model learned, which in turn might give some insight into how much of instruction tuning is learning to recognize new features of the context vs how much of it is simply changing the the behaviors exhibited by the fine-tuned model in response to the context having features that the base model already recognized as important.
I don’t see any SAEs trained on Gemma 2b instruct, but I do see one on Gemma 9b instruct (residual), and there’s one on Gemma 9b base (also residual) as well, so running this experiment could hopefully be a matter of substituting those models into the code you wrote and tweaking until it runs. Speaking of which...
All code is available at: https://github.com/tommasomncttn/SAE-Transferability
I think this repo is private, it 404s when I try to look at it.
None of it would look like it was precipitated by an AI taking over.
But, to be clear, in this scenario it would in fact be precipitated by an AI taking over? Because otherwise it’s an answer to “humans went extinct, and also AI took over, how did it happen?” or “AI failed to prevent human extinction, how did it happen?”
It uses whatever means to eliminate other countries nuclear arsenals
I think the part where other nations just roll with this is underexplained.
Looking at the eSentire / Cybersecurity Ventures 2022 Cybercrime Report that appears to be the source of the numbers Google is using, I see the following claims:
$8T in estimated unspecified cybercrime costs (“The global annual cost of cybercrime is predicted to reach $8 trillion annually in 2023” with no citation of who is doing the estimation)
$20B in ransomware attacks in 2021 (source: a Cybersecurity Ventures report)
$30B in “Cryptocrime” which is e.g. cryptocurrency scams / rug pulls (source: another Cybersecurity Ventures report)
It appears to me that the report is intended to enable the collection of business email addresses as the top of a sales funnel, as evidenced by the fact that you need to provide your name, company name, role, and a business email address to download the report. As such, I wouldn’t take any of their numbers particularly seriously—I doubt they do.
As a sanity check, $8T / year in cybercrime costs is an average annual cost of $1,000 per person annually. This is not even remotely plausible.
What if both the AI group and the control group have high success rates at this test?