Interesting!
Thanks for sharing. You’ve laid out two real bottlenecks in this effort, which I think I have made real progress on.
I do wonder how those startups get in front of the labs though.
TheVinci
Karma: 42
Ten Thousand Cyber Labs for Training & Eval
Gemma The Unstopping: a Behavioral Experiment
Some Thoughts on The Environment Problem in Agent Training
Great insight. I may have had the framing backwards, and your point makes more sense than mine.
This is arguably more interesting, because, if true, it shows that Gemma does have some strategic thinking, but it may be underused.
By underused, I mean specifically that a good researcher would have laid out hypotheses, and under fixed time, would presumably allocate that time more wisely.
An obvious follow up experiment I have in mind would be some intervention within a run. That would resolve the causation question you’re raising.
Additionally, an interesting experiment would be a qualitative contrast with a larger model, to see if there is some divergence in resource allocation.
When Gemma Thinks About Resources—it Fails: a Behavioral Experiment
The Case for AI Behavioral Science
Your claim is correct in a narrow scope. Yes, Microsoft, Google, etc., can patch vulnerabilities somewhat quickly. This is derived from many reasons, but is essentially linearly correlated with the amount of security engineers the company has.
However.
A mind-boggingly large amount of modern software infrastructure is built upon the form of software that is run by 1 guy in Alaska who patches his project once a month after a fishing quest.
These are the bottlenecks, and in a large sense, the crown jewels.
If Alex from Alaska has had to patch his project once a month because of one vulnerability that was discovered by a white/black hat hacker, that’s manageable.
If Alex has to patch his project twenty times a month because Mythos-class models are repeatedly breaking it, that is not manageable.
When these software projects are hacked, which cause what is called a supply-chain attack, these are the class of attacks which reach breaking-news scale (e.g. the SolarWinds hack).
Bottom line, the risk of this transformational capability is not mainly does not rest on well-defended companies, but the smaller, under-defended ones.
I find a strong single thread that unites both your post and the comments here: when/if LLMs become ASI, what capabilities does it require, how it will work, scale, etc.
It’s an important and consequential point, but I think there is another one that is not attended to:
If we completely disregard the granular definition of ASI and/or AGI, specifically because of, as you accurately caricaturized, “fogginess”—what are the downstream effects of a technology which will check enough of the boxes that we can claim are closely related with what we would expect from ASI/AGI.
For example, if we see:
Large job displacement of white collar workers
Strong take-off in scientific understanding
A system which increases its own development speed
An autonomous system which is unusually difficult to contain
Etc.
Then, perhaps, the discussion of specific architectures and training methods can, to some extent, be quieted down?
Stated plainly, is it not the case that we can claim that a system is an ASI / AGI if it checks enough of the boxes of downstream effects which we claim such a system will cause?
Moreover, I’d say that stopping “at the last possible moment” and other arguments implicitly says a lot of things, all of which I deem to be 10% likely:
We know when the last possible moment even is. To be able to say that, is to say that 50% of ai research has succeeded. What I mean by that is that for any group to say “Model version N is at a position such that Model version N+1 will destroy civilization”, requires a complete understanding of some Model N, which is presumably some N-x versions away in the future, and thus more complex and more advanced than what we have today.
We know whether a model isn’t dangerous. To know whether a model isn’t dangerous implies that we can know whether a model is dangerous. This requires us, again, to have progressed far into ai research in a distance there is no evidence to claim is possible or attainable
We know whether a model is dangerous, such that we need to pause ai development, and therefore will not release the model to the public. This implies that we have the ability to contain a model that is very capable. This ability to contain these models, which has been discussed a lot, has not been shown to exist.
The above claims do not refer to the models currently blocked by the US government. Personally, I’d say with 99% confidence that the capabilities that are required to end civilization do not exist in those models.
This is an interesting idea, and I think it’s more attainable today that what is being credited for in the comments.
OpenClaw and their derivatives already supercharge what was once defined as an assistant (e.g. Siri). If you upload enough data actively and set up some pipeline that feeds it more of your choices and preferences, it might look like some abstraction of you.
That being said, what are the use cases for it? Would you be comfortable with your GA performing sensitive actions on your behalf, under the attempt to emulate you? Do you think the recipients of the GA will act on behalf of their output?
Personally, I would maybe use it as a sort of behavior-autocomplete, such that for any given input to your environment, the GA would recommend a response based on your history and preference.
For example, say I’m a ceo of some company, and a client has a problem which I’ve solved for a different client. A GA could surface:
“Here’s what you did the last time this problem was raised, based on this risk-analysis / cost-benefit analysis. This situation is similar to that. Would you like me to walk that client through the same steps as you did previously?”
Does that align with what you had in mind?
If so—I think, as stated above, this is attainable today. Whether it’s a company or an open source project is an interesting question for which I’d have to give some more thought.
Cybersecurity, as I see it, is a rare field where offensive tools (as distinct from offensive artifacts) are good to have public and open sourced. Strong examples are Kali, Burp, Mimikatz.
The reasons include budget asymmetry between strong attackers and the average defenders, the requirement of using offensive tools to find improper defenses, and more.
It’s evident today, but based on my opinion, that recent trends show that having defenders and white-hat hackers having access to the best tooling will make the overall security of a lot of technological infrastructure more, not less, protected.