There is probably a way to explain that pacing the frontier is a costly action in the blog post.
Most users aren’t going to click through to a blog post are they? I think you want to get it down to a sentence or two.
Also, you could test with small cohorts first.
That’s not a great way to measure the reaction on social media? And, if people figure out you’re A/B testing messaging that could also be bad?
I remember back before Elon Musk bought Twitter, they had a fact-checking team which would display fact-check messages over tweets. IIRC, this basically completely backfired and people treated the fact-checks as corporate BS. I expect if OP’s idea is implemented, we’ll get a huge surge of discussion on social media about how this is supposedly “terror marketing”, “a bid for regulatory capture”, “building hype for their IPO”, “distracting from the fact that they’re hitting a wall”, etc. People tend to have a cynical view of corporations as amoral profit-maximizing agents. As described there’s not much in this proposal which refutes the standard terror marketing/regulatory capture/IPO hype narratives. I think people tend to latch on to those narratives due to generic anti-AI sentiment and also fear of mentally accepting the possibility of human extinction. I worry many on this site are in a bubble and don’t realize how common these narratives already are. Adopting the narrative gives you an air of cynical superiority; un-critically repeating what BigCorp says makes you look like a rube.
Oh yeah, if you’re going to do this anyway I suppose the displayed message should be right-wing-coded? https://x.com/ESYudkowsky/status/2099213169805902203
EDIT: If people at AI companies want my help brainstorming what an actually-effective message would look like, feel free to shoot me a DM
Don’t waste energy responding to convoluted conspiracy theories. The people proposing them don’t care about your response. And it looks to outsiders like these theories are worth responding to. Your caveman brain loves to win silly internet arguments. But stay on message, present your view of the world, respond to the best criticism.
In general, thinking about online debate in terms of the person you’re arguing with is a bit of a mistake. You want to focus on people who are reading the exchange, who are far more numerous and far more likely to change their mind. They won’t announce if their mind is changed. But they might refrain from retweeting the conspiracy theory or whatever.
I don’t think outsiders will evaluate much on the basis of what’s deemed “worth responding to”. Especially if you’re a lowbie or pseudonymous account. I think outsiders are more likely to evaluate on the basis of arguments which are going unanswered.
I think the conspiracy theory angle is a pretty big deal but it’s also fairly easy to solve by behaving in a way which invalidates the conspiracy theory. https://www.lesswrong.com/posts/FzSSrx6nbmCZRoKkq/ebenezer-dukakis-s-shortform?commentId=Pcjww5Cdn5bSdoGtT
I’ve been reading a bunch of online discussion around Jacob Coxon’s whistleblowing. There seems to be a big sticking point for a ton of people around the idea that this is some sort of scheme by major AI companies, to either market their upcoming IPO or achieve regulatory capture. There’s way too much of this type of talk online for anyone to realistically go around refuting it.
Perhaps the best way to refute it is to behave in a way that makes it as clearly false as possible. Specifically, instead of talking about treaties and regulators, I think we should consider pivoting to the following near-term policy request:
Pass a law that any AI system trained on copyrighted material must be sold “at-cost” to consumers, unless the copyrighted material was licensed to the AI company.
(By “at-cost” I mean the company can’t charge more than the cost of serving the model plus a small profit margin which makes it barely worth their while.)
This policy is obviously terrible for AI company profits. It doesn’t call for some sort of opaque regulatory authority which could be co-opted by major AI companies. It will receive support from existing copyright holders, obviously a powerful constituency which includes many artists, musicians, and authors with huge social media followings. It reduces the incentive to invest in future training runs, since expected profits become less favorable. And it makes sense morally–the idea that AI companies stole their training material is resonating with many, many people on social media. Especially the hypocrisy of complaining about “illicit” distillation after having trained on copyrighted material.
Existing APIs should continue to work, and this doesn’t require sophisticated hardware controls or create burdensome regulations for “little tech”.
Once we get this policy win, we’ll hopefully get a partial slowdown of the frontier. It will establish credentials and build a coalition that lets us go for more ambitious proposals like an international treaty etc. Just talking about this policy will decrease the incentive for investors to pour more capital into Anthropic/OpenAI. Thoughts?
Do we know how people are learning about PauseAI and IABIED? I’ve been reading a bunch of the discussion generated by Coxon, and PauseAI and IABIED are usually not mentioned.
I often try to mention them when it feels appropriate. Usually instead of recommending IABIED directly, I link to this Youtube video. I feel that watching a video is a smaller ask than buying a book, and the video on its own does a decent job of communicating why alignment could be extremely hard. I’ve also been linking this post. I feel like concrete examples are more compelling than abstract theory for a lot of people.
Jacob Coxon now has a quarter million followers on X. Imagine if he was to tweet a link to PauseAI. It could generate a bunch more signups, and also send a credible signal that this isn’t just a marketing stunt.
BTW have you guys been recruiting from data center opposition groups? I feel like if PauseAI people gave talks at those protests, you could probably get some mailing list signups and stuff.
I don’t necessarily agree with this proposal on the object level, but the insights are interesting. If we want this to stay in the news, ideally instead of a boring drip-drip-drip of employee resignations, we get a boom-bang-crash of drama. If there’s a way to resign in a different, flashy way which somehow “advances the story”, adds color, etc. that’s liable to generate continuing coverage.
Furthermore, the ideal resignation would somehow address this suspicion that it’s all about creating hype for investors. For example, imagine if a resigning employee called on politicians to tax away all the profits from the AI industry, to reduce the incentive to invest in doom. Headline: “Capitalism will kill us all, resigning AI employee says”. At that point, it becomes difficult to claim that the resignation is some sort of corporate gambit, since what corporation would beg the government to punitively tax them?
Another idea is to resign from OpenAI and then sue OpenAI. There must be something you can sue them for, right? Or how about calling on securities regulators to block the upcoming IPOs? Is there any way they could do that? Essentially call for some sort of policy which will hurt the profits of these AI companies so people don’t think it’s all about investor hype. “Ethical investors should boycott upcoming IPO, says resigning AI engineer.” Simply linking to PauseAI could also be a credible signal that you actually want a pause and you’re not just marketing.
A mental model for media. There are various podcasts and broadcast news channels. Some of them might be interested in doing an interview with someone on the topic of AI safety. We should aim to saturate these interview channels to the greatest degree possible.
For example, if Coxon has more media requests than he has time to field, it might be possible to coordinate and have him say something like: “Sorry I don’t have time to chat—but now there is news that ANOTHER employee has resigned from Anthropic, for similar reasons. Would you like to talk to them?”
Or, it’s possible that certain channels are open to covering this topic, but don’t want to talk to Coxon because they feel the Coxon “scoop” has already been covered to death. So when formulating media strategy for another resignation, you could specifically target channels which did not cover Coxon, but are still “newsy”/”techy”/etc. enough that such a resignation would be within the channel’s topical scope. Then you contact them, with the subtext that: “You missed the Coxon scoop, but you could be among the first channels to cover this new scoop!”
The desired end result is that no matter where a person gets their news, they’ve seen coverage related to the big AI company resignations.
I imagine working with a publicist could be really helpful since they’re professionals at coordinating stuff like this.
The fact that this resignation got so much buzz makes the “if we don’t build it, someone else will” claim much less plausible. If you choose to “not build it” by resigning in a big public way, that could do a lot to shift the probability that it gets built!
Anthropic has thousands of engineers. The unilateral resignation of, say, 5 more engineers would do very little to shift the race vs OpenAI, but could do a lot to increase the probability of an AI pause, especially with good media strategy. I suppose a group resignation of engineers across various different AI companies would be even better if it’s possible.
weeks if not months for an enforceable pause to be implemented
That sounds like a quite optimistic timeline for a major international treaty to be negotiated etc.
If I was a funder, I might try to strike a deal with Holly to yield the “PauseAI US” brand to the international PauseAI organization, in exchange for Holly spinning out to form her own organization with a more polemical name and mission, and also Holly agrees to be like 10% more cautious about criticizing people.
Maybe there is room in the ecosystem for both a conciliatory and a confrontational approach—a sort of “good cop, bad cop” routine.
You could have a commission component to the compensation. If you flag something, it gets graded by severity by a disinterested professional, and you get a cash bonus depending on severity.
An underlying theme, in my view, is that your community is more hierarchical than you give it credit for. For example:
...many people in this space really want to pull some lever that feels important, and privilege arguments which justify that. That might come from a sense that they need to have an impact on the world; or fear about failing to fulfil their potential; or the more mundane explanation that big levers tend to be associated with money and prestige and proximity to power. Certainly the latter was a large part of why I joined OpenAI originally; I expect that most people who joined earlier were less prestige-oriented than me, but still made that decision using reasoning that was warped by similar emotional drives (and later further warped by the social dynamics of actually working there).
Note also the deference to the founders of OpenAI, and the general tendency of this post to focus a lot on what a few famous people were saying and doing. This culture seems to be one where people tend to either fawn over a famous person, or try to become one of the famous people that everyone is fawning over. I think it is partially downstream of the EA idea that a few people are going to have a disproportionate impact (and, implicitly, we can identify those individuals in advance).
My sense is there’s a sort of egalitarian professionalism which characterizes functional scientific communities that’s undersupplied in your community. Perhaps because the boundaries are so porous—in a university there’s a harder in/out boundary which relieves pressure on individuals in a university to try to determine who is worthy or important (outside of explicit admissions or hiring contexts which can be explicitly optimized).
People asked Eliezer about getting people like Terence Tao to work on alignment.
“Geniuses” with nice legible accomplishments in fields with tight feedback loops where it’s easy to determine which results are good or bad right away, and so validate that this person is a genius, are (a) people who might not be able to do equally great work away from tight feedback loops, (b) people who chose a field where their genius would be nicely legible even if that maybe wasn’t the place where humanity most needed a genius, and (c) probably don’t have the mysterious gears simply because they’re rare. You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them. They probably do not know where the real difficulties are, they probably do not understand what needs to be done, they cannot tell the difference between good and bad work, and the funders also can’t tell without me standing over their shoulders evaluating everything, which I do not have the physical stamina to do. I concede that real high-powered talents, especially if they’re still in their 20s, genuinely interested, and have done their reading, are people who, yeah, fine, have higher probabilities of making core contributions than a random bloke off the street. But I’d have more hope—not significant hope, but more hope—in separating the concerns of (a) credibly promising to pay big money retrospectively for good work to anyone who produces it, and (b) venturing prospective payments to somebody who is predicted to maybe produce good work later.
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
Setting aside the possibility of recruiting security geniuses such as DJB (because MIRI and friends seem uninterested in that, for whatever reason).
It seems to me that adding more high-IQ people to the “set of people thinking about AGI” has been tried in the past, and the effect was to accelerate AI capabilities. What you fundamentally need is dispositional/sociological, not mental horsepower.
You could argue that with reprogenetics, it will be possible to make humans so smart that they’re in a qualitatively different regime from Terence Tao. In which case they could destroy the world during a teenage tantrum. Wisdom doesn’t necessarily scale with IQ. Children don’t necessarily adopt the ideology of their parents. And it’s not ethical to “shut a person off” or read their thoughts like you could do for a rogue AI.
It seems to me that the community is taking the same “Leeroy Jenkins” approach to reprogenetics that it took to AGI, without serious effort at e.g. public red-teaming. Equally catastrophic results seem quite plausible. Feeling blackpilled on this community.
I think it matters less what you call it, and more the institutional incentives/culture/etc. which drive the process.
The trick is to pause right now, when warning shots are abundant but not deadly. The more we pause, the more warning shots we get, the more political momentum we get for further pause measures.
Plus with Chinese open-weight models, if e.g. Anthropic claimed that training new models doesn’t even look great from a profitability perspective (due to open source distillation), financiers might be reluctant to loan capital to OpenAI to train new models. In that sense, unilateral withdrawal from the AI race might conceivably trigger a broader pause.
How about posting on local subreddits, Facebook groups, etc.? Reddit posts can easily get tens of thousands of views. The leverage seems far higher than canvassing.
dry wood is mostly CO2
FWIW, you appear to be confusing two claims:
“Most mass in trees was previously part of the surrounding air”
“dry wood is mostly CO2”
From a US Forest Service website:
Wood is best defined as a three-dimensional biopolymer composite composed of an interconnected network of cellulose, hemicelluloses and lignin with minor amounts of extractives, and inorganics. The major chemical component of a living tree is water, but on a dry weight basis, all wood cell walls consist mainly of sugar-based polymers (carbohydrates, 65-75%) that are combined with lignin (18-35%).
https://research.fs.usda.gov/treesearch/42245
“Trees are mostly made of air” (direct quote from post title) is more clickbait headline than scientific fact.
I see many people on social media asking why Amodei et al developed powerful AI if they knew it was dangerous. I think this is a pretty legitimate question. One which was asked by people concerned about AI x-risk around the time that OpenAI was started, even!
I also see many people pointing out that Anthropic has received money from major EA funders who also fund various AI safety efforts. People are characterizing EA as a cult/conspiracy/etc.
For those concerned with AI x-risk, there is a simple way to address both of these complaints. Throw Anthropic (and OpenAI) under the bus. Don’t just talk about “regulation”. Talk about specific, punishing regulation which will obviously produce deep cuts to the profits of frontier AI companies. Arguably it’s both the right thing to do (the recklessness of these companies should be punished??) and also one that will make safetyists more popular and more credible.