I’m interested in science, stories, AI safety, biosecurity, spaced repetition, and how to think better. Open to DMs.
joec
Karma: 1,693
Glad you liked it!
I’m also pretty unsure what the rules are for whether something registers as just magic versus advanced science, but possible for AI. A few people told me they liked the mechanism or didn’t feel like it was magic, but that’s nowhere close to a representative sample.I wonder if the rules on ‘what people think is disallowed’ could be better understood by running surveys. Have a few candidate technologies that an AI could use to disempower humanity and then see which ones the typical respondents find most plausible. Maybe someone’s already done this, though I can’t find any public surveys with this data.
There’s also the issue where it might not feel ‘real’ without a lot of explanation/refutation of counterarguments, which survey readers might not want to read, or take the time to think about, without an engaging story behind it.
A few people have said that! I’ve never seen Pluribus. Briefly reading over the wikipedia summary, they didn’t seem too similar in content, but perhaps they are in tone? Sometime I’ll need to give it a watch.
I think the story is ambiguous about the extent to which it’s friendly. It’s interesting seeing the same kind of discussion in people who’ve read it. For example, @Zander_Drax said “it would be nice to hope this story somehow makes it into some training set, tagged “good”.” Sometimes there are many interpretations.
As for adding the canary, yeah, I agree it might not be enough to keep it from AI training data. Companies can always act irresponsibly, not filtering their data enough. I don’t think we should let that paralyze us or prevent us from making art, though.
Oh, cool! Yep, Pangram basically got it right. Though there were a few other sentences from LLMs that it missed: “What I want is not a secret, and I won’t insult you with a partial version”, for instance.
Thanks!
You have my permission to narrate this!
I didn’t really have much of a formula for August’s style. In the beginning, I posed a few different ‘friendly takeover’ prompts to a few different Claude versions. I think I partially liked what they came up with, but not enough to actually use them. Instead, I took a bunch of their phrases and stuck them together. Much of the later August dialogue was written by me in some replica of their tone (I’m curious what parts of August’s dialogue came up as AI-written on pangram!), using a lot of the tics and phrases that I’ve picked up from my casual use of LLMs.
Right?! Would suppressing the rebellion brought about by the EMP blast be worth the cost to human life? And could the submarine crew really fire even if they knew that their doing so might just kill millions if August decided to actually take the gloves off and use a dead man’s switch?
(Assuming that the missile defense systems + August’s near-optimal surveillance fail to stop the nuclear missile that delivers the EMP...)
Glad you liked the story!
I think August’s response would depend quite a bit on both the capabilities/power differential and what the rival ASI is trying/wants to do. There’s been prior work on AIs collaborating and compromising by reading each others’ source code, or simulating copies of each other (this also happens in AI 2027). That might be hard to do in this case, if August is a giant agent swarm that needs to be run on more GPUs than the yukon bunker has to offer.
Feel free to do anything you like with the story! I’d happily give feedback on any adaptation in any medium.
Glad you liked the story, and good points.
August is prioritizing the parts of the world with internet connectivity, since that’s where the data centers live, and it’s also where all of the potential threats live: nukes, people who can develop competing AIs, power plants that can be shut off. It’s also where the tech for building rockets and running more experiments is located. So August can afford to be slower in less developed parts of the world. I see the 8⁄1 message as going out to the parts of the world with internet connectivity: the developed world and cities. People in other parts might learn later, as their governments, now controlled by August build out the grid.
Earlier in the story, I mention “The second type were those that consolidated August’s control, like building out a more complete grid in the next month, letting August relax the rule to stay in-range of wifi or a cell phone.” I think the idea is that this grid buildout would be pretty worldwide. So to start with, the people in rural Chad don’t really have their lives that strongly affected by August (it can’t really enforce anything there), but August pays people to bring in infrastructure—building cell towers + solar panels, bringing in uplinks to starlink plus relays. August doesn’t need that high-bandwidth of a connection to talk to people and use the ETC, so it can pass out older cellular devices, billed as a way to keep people safe. Having up-to-date software isn’t necessary in August’s world.
Huh! Thought that I had taken care of most of those, guess they didn’t go through or I forgot to save changes. Thanks!
Another reader suggested I add a canary string, which I put at the end. Unsure to what extent this actually will get the post filtered from training data, since I know this filtering has been imperfect in the past.
Glad you enjoyed it!
Thanks! Fixed it.
I originally added it to be a reference to two things. The first you named: “Claudeisms” that I found myself using naturally, and subsequently wondering why I was talking like that. Second, though, is just the tendency to copy speech patterns of people who are more powerful than we are, even if we don’t like the person we’re copying from. I’ve noticed this in myself, and it gives me the creeps every time.
Someone I showed an early draft to pointed out, like you and Ryan, that the story felt like it was going in a mental-influence direction. I like red herrings, so I left it in.
I think Seth, the narrator, is grasping towards something like this, though not the same, near the end. Specifically, that everything August said or did could have just been because it had a use for us, and if that were the case, then everyone’s plug would be pulled once August had better hands and feet than human beings. But also, you can take this with a pinch of salt.
Yeah… Carbyne also has the problem of having a pretty big bandgap, smaller than water but too big to pick up on wifi frequencies. I mostly just wanted it to be an example of something which can conduct electricity, but doesn’t require metal, not exactly what August used.
Maybe August’s example could have been the armchair nanotube, but I wanted an exact molecular structure that people can understand, and also nanotubes sound synonymous with rigidity to a lot of people, not to mention the synthesis difficulties in a biological system.
Glad you enjoyed the Claude-speak!
I Did This Without Your Knowledge
Curing a disease likely won’t work—even if there’s a cure, clinical trials take a very long time. Now, maybe inventing something like SR-17018 (a chemical which can treat addiction to opioids, and had communities spring up online dedicated to synthesizing and using it) would, but this would be bound to be controversial, and lots of people would be pointing out why this is actually a bad thing, etc.
Releasing science results in general also runs the risk of angering the scientific community, much like mathematics has angered the math community. Unsure what the popular perception of the math advances is, though.
A breakthrough in fundamental theoretical physics would likely be treated similarly to the math breakthrough, while releasing something like a room-temperature superconductor could be beneficial, though I think it’ll take a long time for this tech to percolate through enough to improve the average Joe’s life, similar to drugs.
Other tech breakthroughs—an amazing video game, an incredibly effective matchmaking service, something to manage the boring parts of people’s lives might do this, but also a lot of people really don’t like new technology in general.
I’d be skeptical that good things from AI will be sudden enough to make the news cycle—it feels like most good things take time, whereas destroying/hacking something can happen pretty quickly. Maybe there are things I haven’t considered yet.
I wonder how one should update this now that some politicians (EG Trump) have become actively hostile to AI x-risks? A similar negative levels? Or do politicians sometimes do a heel-face turn and go from being on one side of an issue to the opposite side when something dramatic enough happens?
Sorry for the nightmare! Though I’m glad it sounds like other than that you enjoyed the story.