Thanks for such a glowing review!! I’m so glad you heart the book!
The least realistic part of Red Heart is simply that there’s a near-superhuman AI in the near future at all[2].
I’d be curious for the specific ways in which you feel that Yunna is unrealistically strong or competent for a model around the size of GPT-6.5 (which is where I was aiming for in the story). LessWrong has spoiler tags in case you want to get into
the ending. (Use >! at the start of a line to black it out.)
The story actually starts in an alternate-timeline October 2023. I knew the book would be a period-piece and wanted to lampshade that it’s unrealistically early without making it distracting. Glad to hear you didn’t pick up on the exact date.
Just to defend myself about AI 2027 and timelines, I think a loss-of-control event in 2028 is very plausible, but as I explain in the piece you link in the footnote, my expectation is actually in the early 2030s, due to various bottlenecks and random slowdowns. But also, error bars are wide. I think the onus should be on people to explain why they don’t think a loss of control in 2028 is possible, given the natural uncertainty of the future and the difficulty of prediction.
It’s probably better to be safe than sorry when it comes to AI, so I’m not against AI safety research, but I do personally think doom will take longer than 3-10 years.
My reasoning is that I’m pessimistic about the underlying technology in a way that I talked about a few days ago here. I think I’ve picked up an intuition of how language models work by using them for coding, and I see their limitations. I don’t buy the benchmarks saying how awesome they are.
I don’t think Yunna is unrealistic beyond the fact that she’s superintelligent earlier than I’d predict. And assuming corrigibility turns out to not be super hard.
Thanks for such a glowing review!! I’m so glad you heart the book!
I’d be curious for the specific ways in which you feel that Yunna is unrealistically strong or competent for a model around the size of GPT-6.5 (which is where I was aiming for in the story). LessWrong has spoiler tags in case you want to get into
the ending. (Use >! at the start of a line to black it out.)
The story actually starts in an alternate-timeline October 2023. I knew the book would be a period-piece and wanted to lampshade that it’s unrealistically early without making it distracting. Glad to hear you didn’t pick up on the exact date.
Just to defend myself about AI 2027 and timelines, I think a loss-of-control event in 2028 is very plausible, but as I explain in the piece you link in the footnote, my expectation is actually in the early 2030s, due to various bottlenecks and random slowdowns. But also, error bars are wide. I think the onus should be on people to explain why they don’t think a loss of control in 2028 is possible, given the natural uncertainty of the future and the difficulty of prediction.
Regardless, thanks again. :)
It’s probably better to be safe than sorry when it comes to AI, so I’m not against AI safety research, but I do personally think doom will take longer than 3-10 years.
My reasoning is that I’m pessimistic about the underlying technology in a way that I talked about a few days ago here. I think I’ve picked up an intuition of how language models work by using them for coding, and I see their limitations. I don’t buy the benchmarks saying how awesome they are.
I don’t think Yunna is unrealistic beyond the fact that she’s superintelligent earlier than I’d predict. And assuming corrigibility turns out to not be super hard.