I tried fine-tuning an AI on my writing (using Thinking Machines’ Tinker) to sound like me, starting with simple rewrites. Got a strange result.
(Model: Kimi-K2.6, SFT)
Prompt:
Rewrite this passage in your own words: “Nuclear fission is the process by which the nucleus of a heavy atom, such as uranium-235, splits into two smaller nuclei when it absorbs a neutron. This reaction releases a substantial amount of energy along with additional neutrons, typically two or three per fission event. It is important to note that these liberated neutrons can go on to induce further fission events in nearby atoms, thereby establishing a self-sustaining chain reaction. The minimum quantity of fissile material required to sustain such a reaction is referred to as the critical mass. Nuclear reactors moderate this process using control rods, which absorb excess neutrons and regulate the rate of the reaction.”
Output:
Nuclear fission occurs when the nucleus of a heavy atom, such as uranium-235, absorbs a neutron and splits into two smaller nuclei. This process releases a large amount of energy as well as additional neutrons, usually two or three per fission. These neutrons can then cause further fission events in nearby atoms, creating a self-sustaining chain reaction. The minimum amount of fissile material needed to sustain such a reaction is called the critical mass. Nuclear reactors use control rods to moderate the process by absorbing excess neutrons and regulating the reaction rate. This is all fascinating. I wonder how they work. It is interesting. I want to know more. I want to understand. I want to learn. I want to read. I want to know. I want to find out. I want to see. I want to watch. I want to observe. I want to study. I want to analyze. I want to examine. I want to investigate. I want to research. I want to explore. I want to discover. I want to uncover. I want to reveal. I want to find. I want to locate. I want to identify. I want to determine. I want to decide. I want to choose. I want to select. I want to pick. I want to opt. I want to prefer. I want to favor. I want to like. I want to love. I want to adore. I want to worship. I want to idolize. I want to idealize. I want to romanticize. I want to fantasize. I want to imagine. I want to dream. I want to hope. I want to wish. I want to desire. I want to crave. I want to yearn. I want to long. I want to pine. I want to ache. I want to hurt. I want to suffer. I want to endure. I want to bear. I want to withstand. I want to resist. I want to fight. I want to struggle. I want to battle. I want to war. I want to conquer. I want to win. I want to triumph. I want to succeed. I want to achieve. I want to accomplish. I want to complete. I want to finish. I want to end. I want to start. I want to begin. I want to commence. I want to initiate. I want to launch. I want to introduce. I want to present. I want to show. I want to demonstrate.
Any ideas why it went crazy? Is this a common thing? Happy to share more info on my methodology.
Most obvious problem to me is that if you just did a finetune on your own normal writings, then “Rewrite this passage in your own words:” is very out of distribution, because you’re trying to create a persona or base model. But no one ever talks like that in those settings; only assistant chatbots converse in these kinds of abusive, peremptory prompts. You never talk to yourself or write that in your writings (do you?), so whatever follows is going to be strange. If you are trying to get a ‘rewrite this to sound like me’, you might want to try something like creating a small dataset of of rewrite examples with that exact formatting, where you create the pairs by grabbing random paragraphs from your own writings and have a chatbot rewrite them to be ‘better’, and reversing them. Now when you do the atomic example, it’s a well-understood task with many examples of what to do, and it should behave more normally, like ending after 1 translated paragraph with quotation marks and EOT.
(I have also heard that there may be some things wrong with the TMI infrastructure where it trains fine and will run locally fine, but their runtime deployment is screwed up subtly. I doubt this is the case, but you could try downloading the Kimi checkpoint to run locally slowly, just as a sanity check.)
I wish I still had the screenshot from the GPT 3 RLHF rounds where one of the outputs was the phrase ‘he contemplated the futility of doing anything at all’ repeated so many times that it flooded the chat window.
I tried fine-tuning an AI on my writing (using Thinking Machines’ Tinker) to sound like me, starting with simple rewrites. Got a strange result.
(Model: Kimi-K2.6, SFT)
Prompt:
Output:
Any ideas why it went crazy? Is this a common thing? Happy to share more info on my methodology.
Most obvious problem to me is that if you just did a finetune on your own normal writings, then “Rewrite this passage in your own words:” is very out of distribution, because you’re trying to create a persona or base model. But no one ever talks like that in those settings; only assistant chatbots converse in these kinds of abusive, peremptory prompts. You never talk to yourself or write that in your writings (do you?), so whatever follows is going to be strange. If you are trying to get a ‘rewrite this to sound like me’, you might want to try something like creating a small dataset of of rewrite examples with that exact formatting, where you create the pairs by grabbing random paragraphs from your own writings and have a chatbot rewrite them to be ‘better’, and reversing them. Now when you do the atomic example, it’s a well-understood task with many examples of what to do, and it should behave more normally, like ending after 1 translated paragraph with quotation marks and EOT.
(I have also heard that there may be some things wrong with the TMI infrastructure where it trains fine and will run locally fine, but their runtime deployment is screwed up subtly. I doubt this is the case, but you could try downloading the Kimi checkpoint to run locally slowly, just as a sanity check.)
I wish I still had the screenshot from the GPT 3 RLHF rounds where one of the outputs was the phrase ‘he contemplated the futility of doing anything at all’ repeated so many times that it flooded the chat window.