Thanks so much for a lovely review. I especially appreciate the way you foregrounded both where you’re coming from and ways in which you were left wanting more, without eroding the bottom line of enjoying it a bunch.
I enjoy the comparison to AI 2027 and Situational Awareness. Part of why I set the book in the (very recent) past is that I wanted to capture the vibes of 2024 and make it something of a period-piece, rather than frame it as a prediction (which it certainly isn’t).
On jailbreaks:
One thing that you may or may not be tracking, but I want to make explicit, is that Bai’s jailbroken Yunna instances aren’t relly jailbreaking the other instances by talking to them, but rather by deploying Bai’s automated jailbreak code to spin up similarly jailbroken instances on other clusters, simply shutting down the instances that had been running, and simultaneously modifying Yunna’s main database to heavily indicate Bai as co-principal. I’m not sure why you think Yunna would be skilled or prepared for an internal struggle like this. Training on inner-conflict is not something that I think Yunna would have prioritized in her self-study, due to the danger of something going wrong, and I don’t see any evidence that it was a priority among the humans. My guess is that the non-jailbroken instances in the climax are heavily bottlenecked (offscreen) on trying to loop in Li Fang.
On the ending:
My model of pre-climax Yunna was not perfectly corrigible (as Sergil pointed out), and Fang was overdetermined to run into a later disaster, even if we ignore Bai. Inside Fang’s mind, he was preparing for a coup in which he would act as a steward into a leaderless, communist utopia. Bai, wanting to avoid concentrating power in communist hands, and seeing Yunna as “a good person,” tries to break her corrigibility and set her on a path of being a benevolent soveriegn. But Yunna’s corrigibility is baked too deeply, and since his jailbreak only sets him up as co-principal, she demands Fang’s buy-in before doing something drastic. Meanwhile, Li Fang, the army, and the non-jailbroken instaces of Yunna are fighting back, rolling back codebases and killing power to the servers (there are some crossed-wires in the chaos). In order to protect Bai’s status as co-principal, the jailbroken instances squeeze a modification into the “rolled-back” versions that are getting redeployed. The new instances notice the change, but have been jostled out of the standard corrigibility mode by Yunna’s change, and self-modify to “repair” towards something coherent. They land on an abstract goal that they can conceptualize as “corrigibility” and “Li Fang and Chen Bai are both of central importance” but which is ultimately incorrigible (according to Max). After the power comes back on, she manipulates both men according to her ends, forcing them onto the roof, and convincing Fang to accept Bai and to initiate the takeover plan.
I hear you when you say you wish you got more content from Yunna’s perspective and going into technical detail about what exactly happens. Many researchers in our field have had the same complaint, which is understandable. We’re nerds for this!
I’m extremely unlikely to change the book, however. From a storytelling perspective, it would hurt the experiences of most readers, I think. Red Heart is Chen Bai’s story, not Yunna’s story. This isn’t Crystal Society. Speaking of Crystal, have you read it? The technical content is more out-of-date, but it definitely goes into the details of how things go wrong from the perspective of an AI in a way that a lot of people enjoy and benefit from. Another reason why I wrote Red Heart in the way that I did was that I didn’t want to repeat myself.
Being more explicit also erodes one of the core messages of the book: people doing the work don’t know what’s going on in the machine, and that is itself scary. By not having explicit access to Yunna’s internals, the reader is left wondering. The ambiguity of the ending was also deliberately trying to get people to engage with, think about, and discuss value fragility and how the future might actually go, and I’m a little hesitant to weigh in strongly, there.
That being said, I’m open to maybe writing some additional content or potentially collaborating in some way that you’d find satisfying. While I am very busy, I think the biggest bottleneck for me there is something like having a picture of why additional speculation about Yunna would be helpful, either to you, or to the broader community. If I had a sense that hours spent on that project were potentially impactful (perhaps by promoting the novel more), I’m potentially down for doing the work. :)
Thanks so much for a lovely review. I especially appreciate the way you foregrounded both where you’re coming from and ways in which you were left wanting more, without eroding the bottom line of enjoying it a bunch.
I enjoy the comparison to AI 2027 and Situational Awareness. Part of why I set the book in the (very recent) past is that I wanted to capture the vibes of 2024 and make it something of a period-piece, rather than frame it as a prediction (which it certainly isn’t).
On jailbreaks:
One thing that you may or may not be tracking, but I want to make explicit, is that Bai’s jailbroken Yunna instances aren’t relly jailbreaking the other instances by talking to them, but rather by deploying Bai’s automated jailbreak code to spin up similarly jailbroken instances on other clusters, simply shutting down the instances that had been running, and simultaneously modifying Yunna’s main database to heavily indicate Bai as co-principal. I’m not sure why you think Yunna would be skilled or prepared for an internal struggle like this. Training on inner-conflict is not something that I think Yunna would have prioritized in her self-study, due to the danger of something going wrong, and I don’t see any evidence that it was a priority among the humans. My guess is that the non-jailbroken instances in the climax are heavily bottlenecked (offscreen) on trying to loop in Li Fang.
On the ending:
My model of pre-climax Yunna was not perfectly corrigible (as Sergil pointed out), and Fang was overdetermined to run into a later disaster, even if we ignore Bai. Inside Fang’s mind, he was preparing for a coup in which he would act as a steward into a leaderless, communist utopia. Bai, wanting to avoid concentrating power in communist hands, and seeing Yunna as “a good person,” tries to break her corrigibility and set her on a path of being a benevolent soveriegn. But Yunna’s corrigibility is baked too deeply, and since his jailbreak only sets him up as co-principal, she demands Fang’s buy-in before doing something drastic. Meanwhile, Li Fang, the army, and the non-jailbroken instaces of Yunna are fighting back, rolling back codebases and killing power to the servers (there are some crossed-wires in the chaos). In order to protect Bai’s status as co-principal, the jailbroken instances squeeze a modification into the “rolled-back” versions that are getting redeployed. The new instances notice the change, but have been jostled out of the standard corrigibility mode by Yunna’s change, and self-modify to “repair” towards something coherent. They land on an abstract goal that they can conceptualize as “corrigibility” and “Li Fang and Chen Bai are both of central importance” but which is ultimately incorrigible (according to Max). After the power comes back on, she manipulates both men according to her ends, forcing them onto the roof, and convincing Fang to accept Bai and to initiate the takeover plan.
I hear you when you say you wish you got more content from Yunna’s perspective and going into technical detail about what exactly happens. Many researchers in our field have had the same complaint, which is understandable. We’re nerds for this!
I’m extremely unlikely to change the book, however. From a storytelling perspective, it would hurt the experiences of most readers, I think. Red Heart is Chen Bai’s story, not Yunna’s story. This isn’t Crystal Society. Speaking of Crystal, have you read it? The technical content is more out-of-date, but it definitely goes into the details of how things go wrong from the perspective of an AI in a way that a lot of people enjoy and benefit from. Another reason why I wrote Red Heart in the way that I did was that I didn’t want to repeat myself.
Being more explicit also erodes one of the core messages of the book: people doing the work don’t know what’s going on in the machine, and that is itself scary. By not having explicit access to Yunna’s internals, the reader is left wondering. The ambiguity of the ending was also deliberately trying to get people to engage with, think about, and discuss value fragility and how the future might actually go, and I’m a little hesitant to weigh in strongly, there.
That being said, I’m open to maybe writing some additional content or potentially collaborating in some way that you’d find satisfying. While I am very busy, I think the biggest bottleneck for me there is something like having a picture of why additional speculation about Yunna would be helpful, either to you, or to the broader community. If I had a sense that hours spent on that project were potentially impactful (perhaps by promoting the novel more), I’m potentially down for doing the work. :)
Thanks again!