epistemology enthusiast
Zack_M_Davis
Karma: 20,784
Anthropic and OpenAI are pulling in revenue at the rate of tens of billions of dollars per year; even if most of that is selling cheaper Opus/Sol tokens rather than expensive Fable/Astra tokens, that’s still a lot of money and therefore implied economic value. (I just used Astra as a travel agent to plan a trip to visit my girlfriend in Australia.) It would be more complicated to estimate the damages from the Hugging Face incident and RubyGems signups being down or four days, but I’m not inclined to regret my initial comment on your Opus’s say-so.
The docket is Buist v. Anthropic, PBC (3:26-cv-10693) in NorCal District Court. The complaint identifies the plaintiffs as follows:
-
Plaintiff Charles Buist is a resident of Florida. During the Class Period, he personally purchased a paid individual consumer subscription to Claude, ChatGPT, Grok, and Gemini. He continues to subscribe and intends to continue subscribing.
-
Plaintiff Cheyenne Hunt is a resident of California. During the Class Period, she per- sonally purchased a paid individual consumer subscription to Claude, ChatGPT, Grok, and Gemini. She continues to subscribe and intends to continue subscribing.
-
Plaintiff Christine Bullock is a resident of California. During the Class Period, she personally purchased a paid individual consumer subscription to Claude. She continues to subscribe and intends to continue subscribing.
-
Plaintiff Nick Spetsas is a resident of Florida. During the Class Period, he personally purchased a paid individual consumer subscription to Claude, ChatGPT, Grok, and Gemini. He continues to subscribe and intends to continue subscribing.
Best wishes, Less Wrong Reference Desk
-
I’d like Eric Schmidt to assume burden of proof that the labs are required to do the dumb thing [...] More shared knowledge to not do the dumb thing, is good
What is “the dumb thing”, specifically? I’m imagining that training or releasing GPT-6 Astra or Claude Fable 5.1 wasn’t “the dumb thing”, because those are fantastically valuable commercial products that have created enormous amounts of economic value (far in excess of the damage caused by the reported hacking incidents in evaluations). Or was it? Maybe GPT-4 was the dumb thing, and everything after that was even dumber? What is it?
To be clear, I personally agree that the near-term extinction risk from advanced AI is terrifying, and that a pause or slowdown would be good. But in order to appeal to “shared knowledge” that a pause or slowdown is the default, non-insane action (such that “pause” language is misleadingly marking what should be unmarked), you should be able to answer this kind of specific, nitpicky question. Otherwise, you don’t have “shared knowledge” or “common information”; you’re just pushing your preferred frame like any other political actor. Maybe you think that’s the winning move, but I think we might still have need for shared knowledge of the actual meaning of shared knowledge, even at this late hour.
Are you sure the humans aren’t just bad at math, rather than refusing to communicate? (Most people don’t have the training to say anything interesting about this, and a lot of the ones that do find it much more time-consuming than commenting on non-mathematical natural language prose arguments.)
I assumed the blocks were for substantively LLM-written prose (as would be flagged by Pangram), not “was an LLM consulted in any way whatsoever during the production process.”
Sorry, I see: the IA world is still an increase in inequality over our own (adding the enhanced as a new overclass), but with no AI-controlled robots, menial labor still has some negotiating power.
Even the success stories aren’t success stories, they all designate some kind of underclass with zero leverage. IA is the only path that doesn’t have that.
What? At least with AI, there’s no logical or physical barrier to handing everyone the same API access (although as you note, the politics haven’t shaken out that way). But intelligence augmentation would be an intimate experimental medical procedure (and in the case of embryo selection, only applies to the next generation), where everyone who doesn’t want to upgrade becomes part of the underclass. Did you think this through at all?
I think reputations are important for both individuals, groups, and movements, and movements are caricatured using the overstatements, distortions, and other antisocial actions of a few members.
I think “Telling the truth is important in order to have a credible reputation” is missing something critically important: you also want to tell the truth so that other people can help you figure out what actually going on, because your optimal decisions are going to depend on what’s actually going on. For more on this, see Ben Hoffman on “The Humility Argument for Honesty”.
I think the truth is strongly on the side of caution about AI.
I agree, but you want to update yourself incrementally. If it were true that the highly-persistent model in the Hugging Face attack hadn’t undergone alignment training, that would to some (possibly very small) quantitative extent be evidence for less alignment difficulty (if the biggest disaster so far had been due to lack of alignment effort rather than happening despite effort). If you block out incremental updates by rehearsing your fixed bottom line, that’s bad for your ability to orient to the changing details of the situation even if your bottom line is basically true.
I think this one is particularly clear-cut:
Or as someone advocating what I took to be modesty recently said to me, after I explained why I thought it was sometimes okay to give yourself the discretion to disagree with mainstream expertise when the mainstream seems to be screwing up, in exactly the following words: “But then what do you say to the Republican?”
[...]
This question, “But what if a crackpot said the same thing?”, I’ve never heard formalized—though it seems clearly central to the modest paradigm.
The swipe at “Republicans” was completely unnecessary to the point being made and not something we would expect to see in the record if the “tried to keep nonkilleveryoneism from being coded as right or left” claim were true.
I certainly don’t think we should distort the truth
No one thinks of thinks of themselves as “distorting the truth”, but you did literally just say that a claim “could be true” but that “[e]ither way, having this as part of the discourse is much worse than not”, suggesting that you have strong preferences about the discourse unrelated to its truth?
I’ll also nominate this kind of policing:
Anyone posting a neoreactionary concept on my Facebook wall would be instablocked and the comment deleted. It’d be like their posting creationism on my wall; somebody needs to reeducate them, but it’s not going to be me. I think that if you do argue with neoreactionaries instead of just blocking them, then you’ve been suckered into Somebody Is Wrong On The Internet syndrome and trollfeeding.
[...]
So if in the future you hear anyone on Tumblr mention “Eliezer Yudkowsky” and “neoreaction” in the same sentence and the connector isn’t something like “deletes”, then remember always that that poster is intellectually dishonest and probably lying to you about other things as well.
I don’t think the creationism analogy stands up to scrutiny. The reason to block creationists is to protect the signal-to-noise ratio: you could win the debate on the merits, but you probably wouldn’t learn anything in the process that was worth the time. I don’t think neoreaction was like that; I think I learned a lot from engaging with Curtis Yarvin and other far-right authors. Rather than trying to protect the signal-to-noise ratio, I think Yudkowsky was trying to protect his fiefdom from being punished by the left for social adjacency to heretical ideas like hereditarianism.
I am dubious to say the least of Yudkowsky’s claim to have resisted “short-term incentive gradients to try to cozy up to the left.” I could log on to Twitter and contest the claim there, but I don’t feel like logging on to Twitter today.
but I wonder, would upholding that level of truthfulness and honesty make one less effective politically?
Yes, I already said that (“the achieving-goals part could come into conflict with the map-accuracy part (because deception is often useful for achieving goals)”).
is that you could be significantly more deceptive than what you describe and still be a moral person.
I’m not interested in being a “moral person” (whatever that means); I’m interested in achieving the map that reflects the territory.
(Also, blockquotes for quotations of a full paragraph or more.)
Thanks for writing this! Before it got eaten by the “AI safety community”, this was a website about rationality—the art of achieving a map that reflects the territory and using the map to plan to achieve one’s goals. I think a central reason that the project to improve human rationality failed so abjectly is because too little attention was paid to how the achieving-goals part could come into conflict with the map-accuracy part (because deception is often useful for achieving goals): as time has gone on, epistemic rationality has increasingly been forgotten in favor of seeking the “locally optimal discursive posture” (as you so aptly put it) in the service of AI safety. And the first step towards getting the epistemic rationality back would be accounting for what changed, rather than pretending it’s always been this way—for example, by writing up the first-person intellectual history of the strategic forces that covertly shaped one’s public writing, as you’ve done here. I would love to see more “AI safety” people do the same.
I hope you will see this apology as a product of fastidiousness and high integrity, rather than some admission of a yearslong effort at deception.
I think this is a false dichotomy that conflates absolute and relative standards. For example, you write that “the notion that [you] have not acted with intellectual integrity is not quite fair” because almost none of your fellow SB 1047 opponents agreed with you about AI risk (such that even as it was, you got “EA plant” accusations). But that seems less like a defense of your integrity and more like a claim that too much integrity would have come at an unacceptable cost to your political goals. Well, sure. There’s no law of physics that says that there can’t be a social environment that punishes integrity. In such a fallen world, doing as much deception as you need to in order to achieve your political goals, but feeling vaguely bad about it such that you fess up later is a product of relative fastidiousness and high integrity—but that doesn’t mean that no deception occured. “Deception” is about a speaker sending communication signals that predictably decrease the accuracy of listeners’ beliefs; whether the speaker had no better alternatives available doesn’t play into it.
One of the most corrosive effects of this dynamic is not the harm of the deception itself, but self-deception about what higher integrity would even look like, as people faced with a conflict between honesty and winning reason, “Well, I’m a good person, and I did what I had to do given the incentives, so what I did can’t be dishonest.”
You write you “do not think anything I have ever written about AI safety or risks constitutes a lie.” But as I’ve explained in a previous essay response to Eliezer Yudkowsky on this website, not-lying turns out to be a surprisingly weak standard: natural language has so many degrees of freedom that it’s not that hard to arbitrarily push on listeners’ beliefs while only using sentences that permit a true interpretation, simply by, e.g., “[leading] with arguments that [the speaker] believe[s] would be more palatable to a given audience at the expense of arguments that [they] believe[ ] might be more important”.
At this point, some might be skeptical of the purported existence of a higher standard of integrity: what would that mean, concretely? How can there be more to honesty than just not lying? On this topic, I recommend in the strongest terms reading and meditating on two posts from this website’s founding texts: “The Bottom Line” and “A Rational Argument”.
Briefly: once you’ve decided what conclusion you want to argue for, that conclusion is already right or wrong. Searching for additional arguments for that fixed conclusion might make you more persuasive, but they can’t make the conclusion more true. The arguments that matter are the ones that determine which conclusion you’re motivated to argue for. Anything you come up afterwards that lacks the power to change your bottom line is in some important sense dishonest, even if every sentence is true. If the actual reason you care about open weights is liberty, then your arguments about geopolitics and diffusion are fake insofar as you wouldn’t be talking about them if the geopolitical or diffusion concerns had pointed the other way.
It’s a terrifyingly ambitious standard for anyone to aspire to—but just hearing it articulated has deeply changed the course of my life. Even this late in the timeline, I think it could change the world—if only anyone could remember.
But if it’s “unproductive to continue conversations with people who make claims that their opponents are self-deceived in order to win political arguments”, that would seem to be a stance against “bridging gaps with others” and “questioning self-models” in the specific case where the others with self-model questions are trying to win political arguments. Right?
To be clear, I think that’s a coherent position: in a world of good-faith actors who want the truth and bad-faith actors who want to win political arguments, probably you don’t want to waste your time talking to bad-faith actors when you can get all the information you need from good-faith actors. The reason I disagree with it is that I don’t think we live in that kind of binary world: I think that mixed motives are ubiquitous and that you lose information by refusing to talk to people who want to win political arguments against you. People trying to win political arguments selectively attend to information that supports their preferred policy, but that doesn’t make them wrong about the information they do report, which might not be available from other sources.
Kelsey freaking Piper, of all people
I see this kind of hero-worship occasionally (Piper being perhaps the most common “beneficiary” behind Alexander and Yudkowsky), and I think it’s bad: bad for you and bad for Kelsey. Your favorite writers are not gods! They—”of all people” and like all people—make mistakes sometimes, including politically motivated mistakes.
If a critic is claiming that they got something simple wrong, maybe it’s not worth your time to consider (if your favorite writer is usually right and the critic doesn’t have a track record you care about), but it shouldn’t occasion a (bold and italic!) pronouncement that the critic has gone catastophically wrong in some supposed duty to steelman your favorite writer.
(but read the room)
Sorry, I don’t understand this part. Why read the room? What information is conveyed by reading the room, specifically?
Um, to motivate the question, I’m a little worried that injunctions to “read the room” might be a bid to comply with local status hierarchies, such that people with enough power among the ingroup can get away with asking “these seem in conflict to me, what’s the deal?”, while other people asking the same question are adversarially construed as doing the “My first impression carries more weight than your lifetime” or “I am so deeply embedded in my mind-palace” things.
But that’s just my non-confident paranoid worry; I’m definitely not claiming that’s what you meant!
if you happened to not produce artifacts of your thinking when deciding where to buy cookies, the only thing you have is claims about your internal reasoning
Sure. So, there was actually a bit of unflagged dramatic irony in the (fictional, despite being told in the first person) cookie story—
Suppose “I” didn’t make any spreadsheets, but when questioned, I say (as in the post) that my internal reasoning considered the “obvious objective criteria” of “price, quality, Anzac biscuit availability, and owner’s prior familiarity with the salon’s reading list.”
It might occur to you to ask followup questions: “Wait, why specifically Australian cookies? And what does it matter if the dessert vendor has done the reading? It kind of seems like you specifically chose those criteria in order to give your girlfriend the business.” If I don’t have a surprisingly compelling response to these obvious questions, that’s also a sort of evidence about my intent.
You wouldn’t expect that, by looking at how many PhDs and money they have – and yet.
Their mad-scientist credibility doesn’t come from the Ph.D.s and the money. (Jason Arday had a Ph.D.; Paris Hilton has money.) The mad-scientist credibility comes from developing the most exciting technological advancement of the century, and the money comes from the technology being fantastically economically useful—an all-in-one marvel that answers your questions, summarizes your documents, writes your code, and generates images, all for the low, low price of $20 a month. That’s high-status according to the same standards that western Society has judged these things for generations.
It’s not a trivial achievement. I don’t think I’m stupid, exactly—I worked as a software engineer for eight years; I have a math degree; I write some blog posts about philosophy that some people like—but I’m not sure I could have done it. If I tried to claim that I could, but that I didn’t for the good of the world because I’m so much smarter than them, I think it would make sense for people to be suspicious that something other than the good of the world was at stake in me saying that.
So, you’re right that it is or should be clear to all reasonable people that the technology is on a pathway to existential danger. Obviously I agree with that.
The problem is that the argument for the existence of existential danger, as compelling as it is, is not itself (without extra steps) an argument for redefining the status of technical achievement in Society (such that you can sell the mad-scientist to stupid-criminal casting change). You can try to argue that it should be, but then you face the problem of making it credible that your actual reason for wanting to reallocate status away from people who make things-that-work is the same as your stated reason, and that’s tough, because among the people who want to demonize Sam Altman, you’re vastly outnumbered by people who would demonize Cornelius Vanderbilt for the same reasons.
What work is being done by the phrases “AI safety” and “our community” in this post?