AI security and cybersecurity
tchauvin
Karma: 150
On misleading figures in Daybreak and Glasswing:
OpenAI commits $1B of “subsidized access”, i.e. commits essentially nothing (10% off on $10B of API purchases is $1B of subsidized access).
I’d say this is more disingenuous than Anthropic “committing up to $100M” in usage credits of Mythos, at a 5x Opus price that no one ever paid, as far as the public record goes (the price dropped to 2x Opus before Fable release).
(To be clear, these initiatives are great and I support them).
Thanks for the comment, do you mind copying this to https://www.lesswrong.com/posts/huh4bvwzeKTLxw6hS/vulnerabilities-and-exploits-where-are-we-headed? I’ll reply there. It’s my bad, I realized after cross-posting today that I had already done so when I initially published the piece in June, but had forgotten. I’ve moved this back to a draft
In the spirit of gwern’s Writing for LLMs so They Listen, I’ve mirrored all my X posts on my website. Source code, what it looks like. In addition to LLM discoverability, I’ve found that looking back over my posts helps me develop my thinking.
The vulnpocalypse is here. Where are the large-scale cyberattacks?
Hopefully after the recent OpenAI news, the fact that large-scale attacks are coming is clear to all. Anyway, it’s still interesting to explain the lag:
Vulnerabilities are just one small input to attacks. Especially small on mundane targets (as opposed to nation-state targets).
Even if this input (vulnerability discovery) is infinitely faster, cyberattacks will only increase by a capped amount (this is Amdahl’s law).
The most effective cyberattacks are the ones with a human driving the intrusion hands on keyboard. Once that is unblocked (basically once malware becomes synonymous with LLM agent making API calls to “Together AI for cybercrime” serving open-weight models), we will see the large scale attacks. Did you know that your laptop is more valuable than just joining a botnet or mining crypto if someone can look at all of your files and threaten you to send the files to the public/the appropriate persons, unless you pay the ransom (itself set according to your standard of living)?
The same question was asked here in March 2025 but for SWE, and the top answers said the same thing (basically in the absence of end-to-end automation, Amdahl’s law results in disappointing real uplift). A year later, we absolutely do have the 10x developers.
(Btw Gustafson’s law seems more real than Amdahl’s law but that’s a detail here.)
tchauvin’s Shortform
When should a frontier-in-cybersecurity model be released to everyone?
Assuming that open-weight models are 6 months behind, I believe that access should be gradually expanded (defenders first, but with an increasingly loose definition) over the course of 6 months.
I think companies should publish the number of orgs and people on the “trusted access” list so we can check we are on track.
I suspect that the current rollout is too slow, and I’m afraid that the June 2 EO, and lack of cyber literacy that resulted in Fable 5 being suspended, will make this way worse.
Vulnerabilities and exploits: where are we headed?
They had more volunteers than they had space for.
What happened to the non-volunteers during these 4 weeks?
Nice attempt. This reminds of the Pizza Meter and Gay Bar Index related to Pentagon crisis situations. I found it hard to find reliable information on this when I looked (I can’t even find a good link to share), but the mechanism seems plausible.
In general, the hacking capabilities of state actors and the likely involvement of national security when we get closer to AGI feel like significant blind spots of Lesswrong discourse.
(The Hacker and The State by Ben Buchanan is a great book to learn about the former)
End-to-end hacking with language models
If you are very good at cyber and extremely smart, you can hide vulnerabilities in 10k-lines programs in a way that less smart specialists will have trouble discovering even after days of examination—code generation/analysis is not really defense favored
I think the first part of the sentence is true, but “not defense favored” isn’t a clear conclusion to me. I think that backdoors work well in closed-source code, but are really hard in open-source widely used code − just look at the amount of effort that went into the recent xz / liblzma backdoor, and the fact that we don’t know of any other backdoor in widely used OSS.
The main effect of a market being underground is not making transactions harder (people find ways to exchange money for vulnerabilities by building trust), but making it much harder to figure out what the market price is and reducing the effectiveness of the overall market
Note this doesn’t apply to all types of underground markets: the ones that regularly get shut down (like darknet drug markets) do have a big issue with trust.
Being the target of an autocratic government is an awful experience, and you have to be extremely careful if you put anything they dislike on a computer. And because of the zero-day market, you can’t assume your government will suck at hacking you just because it’s a small country
This is correct. As a matter of personal policy, I assume that everything I write down somewhere will get leaked at some point (with a few exceptions, like − hopefully − disappearing signal messages).
An Overview of AI risks—the Flyer
The link of “this is a linkpost for” is not the correct one
Here are the same two GIFs but with a consistent speed (30ms/frame) and an infinite loop, in case anyone else is interested for e.g. presentations:
[Question] Is there an analysis of the common consideration that splitting an AI lab into two (e.g. the founding of Anthropic) speeds up the development of TAI and therefore increases AI x-risk?
I think you can guess a reasonable answer even as a complete outsider (like me), considering the purpose of these funds, which is to maximize the amount of expected good they cause by the allocation of their money. A few things that must come into consideration:
is it productive to pay very good researchers the bare minimum they need to survive? No:
it consequently make the path of independent research unattractive to most;
it produces some level of adverse selection in future applicants, i.e. you’ll move toward getting more applications from people who couldn’t really be hired in the industry, which isn’t always bad but could be thought to correlate with not being an excellent researcher;
you don’t want your researchers to waste too much time and energy on things that could be straightforwardly solved with a bit more money.
on the other hand, paying too much has a few issues:
the money spent on a researcher hitting decreasing returns after a certain point;
also some level of adverse selection, if people know they can make the same amount of money here or in the industry, and your fund’s approval process is lighter than industry interviews due to a lack of resources and a hits-based giving approach: you might get an influx of candidates who aren’t altruistically motivated and were rejected from the industry for perhaps good reasons;
damaging the reputation of the movement by appearing to allocate your money frivolously.
In terms of existing data, you can have a look at past public payouts, for example the December 2021 LTFF report. In it, you’ll find a 6-month salary of $50,000 for a well-known independent researcher, which is above the bare minimum needed to survive, probably allowing for some level of travel and luxury, while staying well below industry standards.
Google can decide to verify existing accounts. I think it’s likely that in its efforts to avoid platforming spam, cybercrime and astroturfing, Google may decide to leave existing “reputable” accounts alone, but will ask for e.g. a phone number or other KYC for existing accounts that were basically never used, as an easy fix to the issue you’re describing here.
Another mitigation strategy against the “LLM learns steganography from other LLM’s output” problem is to add some noise to the training data (performing a number of transformations that preserve meaning) prior to training, to break the embedded steganography schemes. I feel like this should be relatively easy and effective?
Note that the loss would obviously be worse on the outputs of the previous LLM.
Which makes me think that comparing losses could be a way to detect that some text contains steganography, even if we can’t pinpoint how exactly (but our transformations break it).
EDIT: in case of disagreement, I can try being more specific about my claim that this should be relatively easy and effective, using concrete examples. Basically I believe that after scrambling, there can only remain a small number of bits of steganographic information per token, even for a steganographic encoding that originally used the null space of natural language optimally.
Do we have any evidence on Mythos being a distilled Mythos Preview?