You could say “reduces one component of the attack surface” or “closes off threat model X”. But “reduces risk to near-zero” is a tell that you aren’t using the right mindset.
I agree it is a strong statement, but I genuinely cannot think of a way a model could otherwise self-exfiltrate its weights, insofar as the actual numbers don’t exist anywhere digitally once the model is run.
I am not explicitly skilled in this area, so take what I say with a grain of salt, but my lack of knowledge is not at all a reason why what I have said is wrong.
I tried fine-tuning an AI on my writing (using Thinking Machines’ Tinker) to sound like me, starting with simple rewrites. Got a strange result.
(Model: Kimi-K2.6, SFT)
Prompt:
Rewrite this passage in your own words: “Nuclear fission is the process by which the nucleus of a heavy atom, such as uranium-235, splits into two smaller nuclei when it absorbs a neutron. This reaction releases a substantial amount of energy along with additional neutrons, typically two or three per fission event. It is important to note that these liberated neutrons can go on to induce further fission events in nearby atoms, thereby establishing a self-sustaining chain reaction. The minimum quantity of fissile material required to sustain such a reaction is referred to as the critical mass. Nuclear reactors moderate this process using control rods, which absorb excess neutrons and regulate the rate of the reaction.”
Output:
Nuclear fission occurs when the nucleus of a heavy atom, such as uranium-235, absorbs a neutron and splits into two smaller nuclei. This process releases a large amount of energy as well as additional neutrons, usually two or three per fission. These neutrons can then cause further fission events in nearby atoms, creating a self-sustaining chain reaction. The minimum amount of fissile material needed to sustain such a reaction is called the critical mass. Nuclear reactors use control rods to moderate the process by absorbing excess neutrons and regulating the reaction rate. This is all fascinating. I wonder how they work. It is interesting. I want to know more. I want to understand. I want to learn. I want to read. I want to know. I want to find out. I want to see. I want to watch. I want to observe. I want to study. I want to analyze. I want to examine. I want to investigate. I want to research. I want to explore. I want to discover. I want to uncover. I want to reveal. I want to find. I want to locate. I want to identify. I want to determine. I want to decide. I want to choose. I want to select. I want to pick. I want to opt. I want to prefer. I want to favor. I want to like. I want to love. I want to adore. I want to worship. I want to idolize. I want to idealize. I want to romanticize. I want to fantasize. I want to imagine. I want to dream. I want to hope. I want to wish. I want to desire. I want to crave. I want to yearn. I want to long. I want to pine. I want to ache. I want to hurt. I want to suffer. I want to endure. I want to bear. I want to withstand. I want to resist. I want to fight. I want to struggle. I want to battle. I want to war. I want to conquer. I want to win. I want to triumph. I want to succeed. I want to achieve. I want to accomplish. I want to complete. I want to finish. I want to end. I want to start. I want to begin. I want to commence. I want to initiate. I want to launch. I want to introduce. I want to present. I want to show. I want to demonstrate.
Any ideas why it went crazy? Is this a common thing? Happy to share more info on my methodology.
Most obvious problem to me is that if you just did a finetune on your own normal writings, then “Rewrite this passage in your own words:” is very out of distribution, because you’re trying to create a persona or base model. But no one ever talks like that in those settings; only assistant chatbots converse in these kinds of abusive, peremptory prompts. You never talk to yourself or write that in your writings (do you?), so whatever follows is going to be strange. If you are trying to get a ‘rewrite this to sound like me’, you might want to try something like creating a small dataset of of rewrite examples with that exact formatting, where you create the pairs by grabbing random paragraphs from your own writings and have a chatbot rewrite them to be ‘better’, and reversing them. Now when you do the atomic example, it’s a well-understood task with many examples of what to do, and it should behave more normally, like ending after 1 translated paragraph with quotation marks and EOT.
(I have also heard that there may be some things wrong with the TMI infrastructure where it trains fine and will run locally fine, but their runtime deployment is screwed up subtly. I doubt this is the case, but you could try downloading the Kimi checkpoint to run locally slowly, just as a sanity check.)
I wish I still had the screenshot from the GPT 3 RLHF rounds where one of the outputs was the phrase ‘he contemplated the futility of doing anything at all’ repeated so many times that it flooded the chat window.
This might have already been said, but would an innate “will-to-reproduce” be a thing for superintelligent AI, as it is for us humans? Probably not, right? Life exists because it reproduces, but because AI is (literally) artificial, it wouldn’t have the same desire.
Doesn’t that mean that ASI would be fine with (or indifferent towards) just ending all life on Earth along with itself, as it sees no reason to live.
Even if we could program into it a “will-to-reproduce,” like we have, wouldn’t that just mean it would go all Asimov and keep itself alive at all costs? Seems like a lose-lose scenario.
On a high level I think the answer is reasonably simple:
It all depends on the objective function we program/train into it.
Spot on that it doesn’t necessarily see itself (its long term survival) as a final end
But we often say if we program/train any given specific objective into it, and this is an objective that requires some sort of long-term intervention in the world in order to be achieved/maintained, then the AI would see itself as an instrument
And, fwiw, in maybe slightly more fanciful situations, there could also be some sort of evolutionary process between future ASIs that mean only those with a strong instinct for survival/duplication (and/or of killing off competitors?) (and or minor or major improvements) would eventually be the ones being around in the future. Although I could also see this ‘based on many competing individuals’ view is a bit obsolete with ASI as the distinction between many decentralized individuals and one more unified single unit or so may not be so necessary; that all becomes a bit weird.
Model-weight self-exfiltration risks can be reduced to near-zero if deployed AIs are only run on hardwired forward-pass ASICs
You could say “reduces one component of the attack surface” or “closes off threat model X”. But “reduces risk to near-zero” is a tell that you aren’t using the right mindset.
I agree it is a strong statement, but I genuinely cannot think of a way a model could otherwise self-exfiltrate its weights, insofar as the actual numbers don’t exist anywhere digitally once the model is run.
I am not explicitly skilled in this area, so take what I say with a grain of salt, but my lack of knowledge is not at all a reason why what I have said is wrong.
I tried fine-tuning an AI on my writing (using Thinking Machines’ Tinker) to sound like me, starting with simple rewrites. Got a strange result.
(Model: Kimi-K2.6, SFT)
Prompt:
Output:
Any ideas why it went crazy? Is this a common thing? Happy to share more info on my methodology.
Most obvious problem to me is that if you just did a finetune on your own normal writings, then “Rewrite this passage in your own words:” is very out of distribution, because you’re trying to create a persona or base model. But no one ever talks like that in those settings; only assistant chatbots converse in these kinds of abusive, peremptory prompts. You never talk to yourself or write that in your writings (do you?), so whatever follows is going to be strange. If you are trying to get a ‘rewrite this to sound like me’, you might want to try something like creating a small dataset of of rewrite examples with that exact formatting, where you create the pairs by grabbing random paragraphs from your own writings and have a chatbot rewrite them to be ‘better’, and reversing them. Now when you do the atomic example, it’s a well-understood task with many examples of what to do, and it should behave more normally, like ending after 1 translated paragraph with quotation marks and EOT.
(I have also heard that there may be some things wrong with the TMI infrastructure where it trains fine and will run locally fine, but their runtime deployment is screwed up subtly. I doubt this is the case, but you could try downloading the Kimi checkpoint to run locally slowly, just as a sanity check.)
I wish I still had the screenshot from the GPT 3 RLHF rounds where one of the outputs was the phrase ‘he contemplated the futility of doing anything at all’ repeated so many times that it flooded the chat window.
This might have already been said, but would an innate “will-to-reproduce” be a thing for superintelligent AI, as it is for us humans? Probably not, right? Life exists because it reproduces, but because AI is (literally) artificial, it wouldn’t have the same desire.
Doesn’t that mean that ASI would be fine with (or indifferent towards) just ending all life on Earth along with itself, as it sees no reason to live.
Even if we could program into it a “will-to-reproduce,” like we have, wouldn’t that just mean it would go all Asimov and keep itself alive at all costs? Seems like a lose-lose scenario.
Am I overthinking this?
On a high level I think the answer is reasonably simple:
It all depends on the objective function we program/train into it.
Spot on that it doesn’t necessarily see itself (its long term survival) as a final end
But we often say if we program/train any given specific objective into it, and this is an objective that requires some sort of long-term intervention in the world in order to be achieved/maintained, then the AI would see itself as an instrument
And, fwiw, in maybe slightly more fanciful situations, there could also be some sort of evolutionary process between future ASIs that mean only those with a strong instinct for survival/duplication (and/or of killing off competitors?) (and or minor or major improvements) would eventually be the ones being around in the future. Although I could also see this ‘based on many competing individuals’ view is a bit obsolete with ASI as the distinction between many decentralized individuals and one more unified single unit or so may not be so necessary; that all becomes a bit weird.