epistemology enthusiast
Zack_M_Davis
Karma: 20,351
while posing as being responsive to what Alice is saying
I think the problem with this isn’t rudeness as such, but with deceiving people (who aren’t paying close enough attention) that Bob’s comment was relevant to Alice’s thesis. (A lot of politeness norms are about concealing or obfuscating information, and I often want to rudely defy those, but I don’t want to deceive people; the category of rudeness is lumping together different things that I want to treat very differently.)
In particular, it’s not “adopt one of your beliefs on request”, but more like “spin up a mental sandbox which explores the implications of your belief being true”
Sure; I regret the rhetorical excess. (I think it would actually be “spin up a more powerful search for reasons your belief might be true”, not a search for its implications if true.)
And so the exchange that I’m actually proposing is more like “I will spend compute on trying to figure out ways that your perspective might be more consistent and well-intentioned than I currently expect, if you do the same for me”.
The question is, why would you propose that deal? What are the circumstances under which proposing that deal would seem like a good idea to someone?
The reason I’m asking is because if we assume that you’re truthseeking, it’s kind of a weird thing to propose, right? If you think that you’re right and that my position is inconsistent and ill-intentioned, presumably that’s because you’ve already thought about the matter and this is where you ended up. How would it improve the accuracy of your beliefs to reallocate your inference compute if I do so, too? Whether a proposed compute reallocation improves your beliefs shouldn’t depend on what I’m doing with my compute!
That is, the deal only makes sense for you to propose if you have a stake in something other than the accuracy of your own beliefs—if there’s something it would benefit you for me to believe. But then if I’m truthseeking, then there’s no reason for me to accept the deal (because whether a proposed compute reallocation improves my beliefs shouldn’t depend on what you’re doing with your compute). It seems like the deal should only go through if we both have a stake in the other person’s beliefs that isn’t about the truth or falsity of those beliefs, and there’s some reason you can’t just dump the reasoning that convinced you. That’s a weird and suspicious situation to be in, you know? Why would you propose the deal instead of just crushing my arguments on the merits in the eyes of third parties? It seems like the kind of thing you would only propose if you didn’t think you could win on the merits.
I wrote more about this in July 2023:
[W]hen someone who is currently trying to persuade me of something tells me that it doesn’t look I’m making enough effort to think of reasons why they’re right, that immediately makes me think they’re more likely to be wrong. Why? Because I think that if they had an argument, they would be telling me the argument, not chastising my lack of charity. The advice to be on special lookout for reasons your interlocutor is right is good in general, but your interlocutor is the last person to be trusted to give it, because [...] they have an ulterior motive.
In fact I’m confident I’ve made this argument to you sometime in the last ~year.
Indeed, it was only 84 days ago. I’m listening if you have a response to my last comment in that thread, in which I agreed that censorship to maintain the signal-to-noise threshold is good. I then linked to a compilation of Achmiz’s favorites of his own comments and offered the judgments of myself and Jessica Taylor that Achmiz’s work is well above the threshold of being worthy of published on Less Wrong.
Now, maybe you think Taylor (former MIRI employee, inventor of quantilizers and co-author of the logical induction paper, and 7x Curated author) and I (9x Curated, 4x Best of Less Wrong, 3x Less Online invited author) have terrible judgment about what advances the state of rationality/the frontier of human knowledge/&c. Is that your position? Happy to dig into the details if you want.
Unfortunately, within the group of people trying to diagnose what went wrong with rationalism, we end up often categorizing each other as part of the problem
Right.
One axis of disagreement about the diagnosis of things going wrong seems to be: how valuable it is to have a positive vision, versus to be able to critique flaws.
I don’t think that’s the axis. Obviously, if no one had any positive vision, then there would be nothing to critique. People who care about intellectual progress should definitely be trying to have positive visions.
It’s just that people who want their positive visions to be correct should also care about fixing flaws as they’re pointed out.
For example, I’m currently working on several interrelated posts in which I articulate a positive vision about signaling theory. (It’s about the conditions under which “honest” agents can successfully distinguish themselves from “dishonest” agents, but the details aren’t important.)
I think my positive vision is basically on the right track, but that doesn’t mean I’ve got it exactly right. I ran the drafts by my friend Said Achmiz via email, and he had a lot of detailed criticisms, some of which I disagree with, but some of which I recognized as serious. It spurred me to do a lot of laborious rewriting which I’m still not done with.
I think Achmiz’s criticism was really helpful for refining my positive vision. I don’t think he was being nice to me in his comments because I’m his friend. I think he was saying the same kind of things to me that he says to everyone. [1] If some people are not only personally uninterested in reading this kind of detailed, incisive engagement with their work, but don’t even want it to be published for others to read, I think the most plausible reason for that is that they don’t care whether their positive vision is correct and don’t want anyone else to find out, either.
Sorry, I know that seems like a really mean or “uncharitable” thing to say. But, well, I’m not sure what alternative theory of people’s behavior that you (Richard) think I should be embracing instead. (More below on my understanding of others’ theories.) Just, where would someone even get the idea that “critiquing flaws” and “having a positive vision” are somehow in a zero-sum competition with each other on opposite ends of an axis, such that people who are in favor of critiquing flaws are therefore less in favor of positive visions?
As far as I can tell, it seems to have something to do with a psychological hypothesis about morale. When Elizabeth von Nostrand writes about “a general trade off between authors’ experience and improving correctness” or Raymond Arnold writes that “it’d basically be the wrong call for any platform to ignore that” “content creation is harder than critique”, the idea seems to be that if people’s attempts to formulate positive visions get critiqued too vigorously, they’ll get discouraged and give up.
The implied psychological model of Less Wrong authors reminds me of my attempts to teach chess to my five-year-old niece this week. It’s not just that I let her capture material [2] in order to keep her engaged with the endeavor of learning how to make legal moves. It’s that the idea of losing material was so aversive that she didn’t seem to be able to process my attempted instruction of the form, “Okay, I’m not playing seriously; I’m going to let you capture my pony in a moment; but if I were playing seriously, I would recapture your rook here.”
Basically, I think Less Wrong authors are capable of having more emotional maturity than a five-year-old: we don’t need to falsely “let them win” in order to induce them to participate at all. If you think I’m being unrealistically optimistic about that, then I feel like I’m not the one who doesn’t believe in intellectual progress.
There’s also something that I (and, I infer, Habryka) want to protect—as implicitly expressed in my post it was something like “intellectual progress is in fact a valuable thing which we can aim for”.
But you can’t seriously have expected me to disagree that intellectual progress is in fact a valuable thing which we can aim for.
On my end, that involves reorienting how I interpret Wei’s comments. On your end, it would ideally involve acknowledging the thing that Habyrka and I are trying to protect, and helping us figure out how to protect it with as few tradeoffs as possible.
Uh, correct me if I’m misreading this, but it seems like you’re trying to deescalate the conflict by proposing mutual concessions: you’ve accepted a belief that favors my side, so I should reciprocate and accept a belief of your choice.
That’s not how it works. Treating epistemics as social exchange—trying to see things the other guy’s way on his request, in exchange for him trying to see it your way on your request—doesn’t create true maps; it creates false maps representing a compromise between the parties’ preferred lies. I put a lot of effort into explaining why I don’t think Habryka’s observed behavior is well-described as trying to protect intellectual progress. If you think I’m wrong about that, you should be able to explain why I’m wrong. It’s not a trade!
- ↩︎
Selected excerpts: “So why would this argument work? Why should it work? It’s deceptive, isn’t it? And transparently so!”, “I am just not convinced by almost anything else you write in these two posts”, “Until you figure out just what is going on there, I don’t think that anything you write on this topic will manage to form any kind of coherent or sensible whole”, “I still think that you’re misinterpreting/misconstruing rather than miscommunicating”.
- ↩︎
It seems imprecise to say “let her win”, since I despaired of trying to explain the concept of checkmate.
- ↩︎
in order to help people understand your perspective
I think this is inappropriately applying mistake theory to what is actually a conflict. If the problem was that people just didn’t understand the perspective expressed in one of Dai’s or Achmiz’s or my comments because it wasn’t expressed clearly enough, I think the natural solution would be to reply, “I don’t understand why you think this comment is relevant, can you spell it out for me?” I’m sure Wei would be happy to elaborate, as would I, as was Achmiz when he was still permitted to speak.
The point of the hobby-horse meme and the persecution of Achmiz is that we have things to say that you don’t want others to hear, because letting our perspectives be heard on the website we’ve been using for 17 years interferes with authors being able to control discussion of their own work. That’s a conflict. Maybe you think you can serve your side in the conflict by pretending that our side is making a mistake by not writing enough posts explaining our point of view? That trick worked to let you purge Achmiz without (yet) triggering a fatal loss-of-legitimacy for the whole site, but I don’t think it’s going to work against me (9x Curated, 4x Best of Less Wrong, 3x Less Online invited author) or Dai (7x Curated, 3x Best of Less Wrong, 3x Less Online invited author).
It’s likely that you think I’m mischaracterizing you, so I should probably make the standard disclaimer here that I’m speaking in functionalist terms, looking for the simplest explanations that I think predict people’s behavior, even when that contradicts their self-report. I’m not saying you self-identify as being in a conflict.
Rather, I’m looking at the comment you just wrote recommending that Dai write more posts explaining his perspective and making a judgment call that it’s sufficiently “unserious” in an important sense such that it makes sense to suppose (as a high-probability but by no means certain hypothesis) that you’re motivatedly refusing to see the conflict you’re engaged in (again in a functionalist sense).
Specifically, Dai already writes a lot of posts explaining his perspective. Telling him to do more of that is obviously not going to help. If a dishonest person who favored authors being able to control discussions was consciously aware of being in a conflict about that and wanted to strategically undermine Dai’s position, they might insincerely suggest that Dai expend even more labor writing the kinds of posts he already does in the hopes of deceiving third parties unfamiliar with the full record into thinking that that any censorship or delegitimization Dai faces is his own fault for not being clear enough.
A big part of the reason I think it’s important to be able to talk about functionally implied conflicts is that we don’t want to create incentives for self-deception. If it would be problematic for a dishonest person to insincerely suggest that Dai needs to do more of the thing he already does in order for his complaints to have standing, then it should also be problematic (maybe not as problematic, but problematic to some nonzero degree) if someone performs the exact same speech act without being consciously insincere.
I think that some people think that we shouldn’t talk about functionally implied conflicts on the grounds that the accusation is unfalsifiable: if I’m questioning your motives, how could you possibly defend yourself except with self-reports that I’ll also doubt? The reason I disagree with this is because accusations of hidden motives aren’t fabricated from nothing: they’re inferred from contradictions between unobservable stated motives and behavior—which, crucially, is observable. If you think I’m reading this whole situation wrong when I claim that there’s a functional conflict between Dai et al. and the mod team about whether authors should be able to control discussions, that’s something you can argue on the merits and potentially embarrass me in the eyes of third parties!
reduce my engagement with LW/rationalists, and spend more of my time elsewhere
Where do you have in mind? I’ve been committed to a “stay and fight” policy out of a perceived lack of alternatives. (I have my whitelist of personal friends who I trust to be saner than the “rationalist” center of memetic gravity, but there doesn’t seem to be an alternative scene with critical mass for the type of work we do.)
pretty defensive, and a mistake for that reason (as was Zack Davis’ original reply upthread).
What do you think would have been the superior play for me in that situation? When someone is deploying a “distasteful” epithet without specifics, I think asking for specifics (forcing them to take a stand on which side of the fact/value boundary the distaste is supposedly on) is the natural follow-up. (And then after he answered that fact-beliefs themselves aren’t culpable, there was nothing to do except thank him for answering the question.)
I think the idea is that we shouldn’t have to indiscriminately divest from the whole industry when we could “just” build fiction-writing AIs and not catastrophically risky AIs (as you suggested about Gwern’s company).
If one is an altruist (if you are purely self-interested there are different calculations of the risk), this seems to me to be fallacious and counter productive in the case of fossil fuels as well as AI.
Does your altruism take into account benefits from economic growth? I think the people whose behavior you’re puzzled by have an attitude more like “AI and fossil fuels are great except for the potential for catastrophic risk, which demands targeted mitigations” (as contrasted to “AI and fossil fuels are Bad in general; only use if truly unavoidable”).
But presumably you would care on the assumption that if Cowen had considered the question a good amount and concluded that what you are doing is super harmful, it would be because he came up with a more substantive objection than “it all feels a bit cultish”? You’ve already taken into account that some people think it all feels a bit cultish.
I’m saying that Shulman already took arguments against investing in AI capabilities into account before he decided to work for Situational Awareness. Yudkowsky’s disapproval would only matter to him insofar as it reflected objections that he hadn’t already “priced in.” Sorry if that wasn’t already clear due to my inexact analogy.
This is bad propaganda because it derives all of its persuasive force from the substitution of “demons” for AI, where people already know that “demons” are supposed to be bad; the reader is encouraged to sneer at the stupidity and wickedness of AI developers for working with the mythical incarnation of the concept of evil itself. But one of the main reasons the world is in so much danger from AI is precisely because the danger isn’t that obvious.
Imagine you’d never heard of “demons” before. If, as in this story, demons liked to “[k]ill, maim, and slaughter with squealing glee and ecstatic delight” and the degree of human control over demons was “50/50” “[o]n a good day”, then, indeed, those who summon demons would be bad people—and it also wouldn’t be very hard to get an international treaty banning demon-summoning. (No one likes killing, maiming and slaughtering with ecsatic delight. We banned CFCs for much less.)
If, as with real-world AI, millions of people had lots of everyday experience with demons routinely obeying human commands to answer questions, summarize documents, write code, and generate images, the case for banning demon-summoning altogether would be an extremely hard sell. Most of the people who understood some of the theoretical arguments for why demons could be dangerous would rather work on demon safety (and increase investments in demon safety and security efforts in response to the occasional rogue demon accident) than just give up on something so useful that had already become part of their lives. If people who thought demon safety research was doomed despaired of winning the technical argument on the merits with Society’s demon experts, they might switch messaging strategies to trying to demonize demons in the eyes of policymakers and the general public, but fighting the economic incentives would be an uphill battle.
I think readers intrigued by the premise of this story should save themselves the click and instead read “Love Stays Loved”, which explores the analogy between AI and the occult with serious literary merit rather than as cheap satire.
I hope Eliezer has let Shulman know that he’s lost a lot of respect in Eliezer’s eyes and done terrible things
You think Carl doesn’t already know that?
I’m not sure you’re modeling how uncompelling moral disapproval is to people who have their own considered views on a topic (as Carl does on the machine intelligence transition and how to intervene on it) rather than deferring to the same authorities that you do. Imagine if Tyler Cowen told you that by choosing to work at Lightcone Infrastructure, you’ve lost a lot of respect in his eyes and done terrible things. Not convincing, right?
If others are interested, I would happily share my takeaways at greater length
Please do! (I find your students’ perspective alien and would be interested to read more.)
I’ve been following the NorCal case closely. (There’s a separate case in the D.C. Circuit under a different statute.)
My impression as a non-expert is that the impact of adding a single high-profile name to an amicus curiae brief is negligible. The amicus brief in question was organized by a group called Protect Democracy. One can only assume that if Turner hadn’t asked Dean, the same document would be prepared but without Dean’s name on it. Judge Lin’s preliminary injunction order in the NorCal case collectively acknowledges the various pro-Anthropic amici a few times (“Several amicus briefs detail the chilling effect”, “Several amicus briefs support this conclusion”), but there’s no indication that she thought anything like, “Well, if Jeff Dean thinks the government is wrong …” (as of course she shouldn’t!).
Charles Goodhart Elementary School
I find some of Richard’s attitudes very distasteful
Examples? What attitudes, specifically? Can you link to something Ngo wrote that expresses the attitude you find distasteful? (Same question to any of the agree-voters.)
I feel like this should be easy. (I think that when I find someone’s attitudes distasteful, I usually don’t have a problem retrieving a link; I need the links to write a convincing critical blog post explaining why the views are bad, as I so often do.)
Taking a wild guess (I don’t know if this is what you had in mind), if the distasteful attitude is that he thinks it’s plausible that there are ancestry-cluster (“racial”) differences in socially-relevant traits like cognitive ability (mentioned in footnote 2 of the OP), I think that’s more of a “hypothesis” than an “attitude”. (It’s a claim about the world that could be true or false, not a claim about how we should behave or feel.)
If the distasteful attitude is that he thinks scholars should be permitted to reason in public about decision-relevant consequences of that hypothesis while also getting paid to do decision theory research, I can see reasons why someone would find that attitude distasteful (see my discussion of “the Schelling point for preventing group conflicts” in a 2020 book review), but to the extent that those reasons aren’t about the truth or falsity of the hypothesis, I think it’s important to notice the rationality implications of silencing discussion of the implications of a hypothesis for reasons other than its truth.
But it also seems like rationalists and EAs do better at this than almost any similarly sized subculture?
What is the relevance of this? It is written of the eighth virtue of humility that “it is useless to be superior [...] There is no guarantee that adequacy is possible given your hardest effort; therefore spare no thought for whether others are doing worse.”
Yes, and those plausible worlds are exactly why I’m so suspicious of the proposed compute reallocation deal: I’m worried that “you’re being uncharitable; try to think harder about why I’m right” is something people around here say to conceal the fact that they don’t actually have an argument.
Maybe you’re right that the deal could make sense (and my argument that it doesn’t is wrong), but it gets scuttled by the adverse selection problem (where good faith actors would prefer to make such deals with each other, but they have no way to credibly distinguish themselves from bad faith actors who are just bluffing)?