Benquo

Karma: 9,918

Benquo 24 Dec 2021 2:17 UTC
5 points
in reply to: gallabytes’s comment on: Occupational Infohazards
Listing “I don’t know, some other reason we haven’t identified yet” as an “obvious source” can make sense as a null option, but giving it a virtus dormitiva type name is silly.

I think that Jessica has argued with some plausibility that her psychotic break was in part the result of taking aspects of the AI safety discourse more seriously and unironically than the people around her, combined with adversarial pressures and silencing. This seems like a gears-level model that might be more likely in people with a cognitive disposition correlated with psychosis.

Benquo 23 Dec 2021 1:02 UTC
10 points
in reply to: gallabytes’s comment on: Occupational Infohazards
When I got back into town and talked with Jessica, she was talking about how it might be wrong to take actions that might possibly harm others, i.e. pretty much any actions, since she might not learn fast enough for this to come out net positive. Seems likely to me that the content of Jessica’s anxious perseveration was partly causally upstream of the anxious perseveration itself.
I agree that a decline in bodily organization was the main legitimate reason for concern. It seems obviously legitimate for Jessica (and me) to point out that Scott is proposing a standard that cannot feasibly be applied uniformly, since it’s not already common knowledge that Scott isn’t making sense here, and his prior comments on this subject have been heavily upvoted. The main alternative would be to mostly stop engaging on LessWrong, which I have done.
I don’t fully understand what “latent tendency towards psychosis” means functionally or what predictions it makes, so it doesn’t seem like an adequate explanation. I do know that there’s correlation within families, but I have a family history of schizophrenia and Jessica doesn’t, so if that’s what you mean by latent tendency it doesn’t seem to obviously have an odds ratio in the correct direction within our local cluster.

Benquo 22 Dec 2021 21:40 UTC
6 points
in reply to: jessicata’s comment on: Occupational Infohazards
My recollection is that at that time you were articulately expressing what seemed like a level of scrupulosity typical of many Bay Area Rationalists. You were missing enough sleep that I was worried, but you seemed oriented x3. I don’t remember you talking about demons or auras at all, and have no recollection of you confusedly reifying agents who weren’t there.

Benquo 21 Dec 2021 1:40 UTC
6 points
in reply to: Scott Alexander’s comment on: Occupational Infohazards
The main thing I’d fight if I felt fighty right now is the claim that by not listening to talk about demons and auras MIRI (or by extension me, who endorsed MIRI’s decision) is impinging on her free speech.
You wrote that talking about auras and demons the way Jessica did while at MIRI should be considered a psychiatric emergency. When done by a practicing psychiatrist this is an impingement on Jessica’s free speech. You wrote this in response to a post that contained the following and only the following mentions of demons or auras:
1. During this time, I was intensely scrupulous; I believed that I was intrinsically evil, had destroyed significant parts of the world with my demonic powers, and was in a hell of my own creation. [after Jessica had left MIRI]
2. I heard that the paranoid person in question was concerned about a demon inside him, implanted by another person, trying to escape. [description of what someone else said]
3. The weirdest part of the events recounted is the concern about possibly-demonic mental subprocesses being implanted by other people. [description of Zoe’s post]
4. As weird as the situation got, with people being afraid of demonic subprocesses being implanted by other people, there were also psychotic breaks involving demonic subprocess narratives around MIRI and CFAR. [description of what other people said, and possibly an allusion to the facts described in the first quote, after she had left MIRI]
5. While in Leverage the possibility of subtle psychological influence between people was discussed relatively openly, around MIRI/CFAR it was discussed covertly, with people being told they were crazy for believing it might be possible. (I noted at the time that there might be a sense in which different people have “auras” in a way that is not less inherently rigorous than the way in which different people have “charisma”, and I feared this type of comment would cause people to say I was crazy.)
Only the last one is a description of a thing Jessica herself said while working at MIRI. Like Jessica when she worked at MIRI, I too believe that people experiencing psychotic breaks sometimes talk about demons. Like Jessica when she worked at MIRI, I too believe that auras are not obviously less real than charisma. Am I experiencing a psychiatric emergency?

Benquo 20 Dec 2021 5:42 UTC
5 points
in reply to: Aella’s comment on: Quis cancellat ipsos cancellores?
Elsewhere, you wrote:

I was the first person to publish the accusations of harm against brent on behalf of my friend who was his ex

If the mittenscautious Tumblr or Medium account was not run by you, in what sense was this true?

Benquo 18 Dec 2021 23:56 UTC
14 points
in reply to: tailcalled’s comment on: Occupational Infohazards
I’m just confused—if there’s agreement that MIRI isn’t particularly bad about this, then this seems to mostly preclude environmental attribution and suggest personal attribution?
I’ve read Moral Mazes and worked a few years in the corporate world at Fannie Mae. I’ve also talked a lot with Jessica and others in the MIRI cluster who had psychotic breaks. It seems to me like what happens to middle managers is in some important sense even worse than a psychotic break. Jessica, Zack, and Devi seem to be able to represent their perspectives now, to be able to engage with the hypothesis that some activity is in good faith, to consider symmetry considerations instead of reflexively siding with transgressors.
Ordinary statistical methods—and maybe empiricism more generally—cannot shed light on pervasive, systemic harms, when we lack the capacity to perform controlled experiments on many such systems. In such cases, we instead need rationalist methods, i.e. thinking carefully about mechanisms from first principles. We can also try to generalize efficiently from microcosms of the general phenomenon, e.g. generalizing from how people respond to unusually blatant abuse by individuals or institutions, to make inferences about the effects of pervasive abuse.
But corporate employers are not the only context people live in. My grandfather was an independent insurance broker for much of his career. I would expect someone working for a low-margin business in a competitive industry to sustain much less psychological damage, though I would also expect them to be paid less and maybe have a more strenuous job. I don’t think the guys out on the street a few blocks from my apartment selling fruit for cash face anything like what Jessica faced, and I’d be somewhat surprised if they ended up damaged the way the people in Moral Mazes seem damaged.

Benquo 12 Dec 2021 9:10 UTC
2 points
in reply to: philh’s comment on: The Rationalists of the 1950s (and before) also called themselves “Rationalists”
If I try to steelman the Rationalist-Empiricist divide:

Empiricists think that arguments justifying organized violence are nonsense so we ought to ignore them, do what we like instead, and argue about math and science.

Rationalists think that arguments justifying organized violence are sketchy so we should investigate them carefully as hypotheses for how mind organizes itself in the world.

Benquo 6 Dec 2021 20:15 UTC
31 points
in reply to: philh’s comment on: The Rationalists of the 1950s (and before) also called themselves “Rationalists”
This distinction was always fake. The Wikipedia page on Rationalism begins with portraits of Descartes, Spinoza, and Leibniz. Spinoza was a lens grinder who worked closely with astronomer-physicist Christiaan Huygens and wrote in his magnum opus, Ethics, that we only know about things in the world through our bodies interacting with them. It is unclear to me how it is possible for someone to be more committed to looking at the world. The Wikipedia page on Empiricism begins with portraits of Francis Bacon, John Locke, and David Hume. Hume’s An Enquiry Concerning Human Understanding includes the following, which implies that abstract mathematical reasoning is one of the two valid sources of knowledge, and refers to experimental reasoning:
If we take in our hand any volume; of divinity or school metaphysics, for instance; let us ask, Does it contain any abstract reasoning concerning quantity or number? No. Does it contain any experimental reasoning concerning matter of fact and existence? No. Commit it then to the flames: for it can contain nothing but sophistry and illusion.

Benquo 6 Dec 2021 18:18 UTC
6 points
in reply to: Said Achmiz’s comment on: Frame Control
A non-conscious intent? What on earth does this mean?
A brain can run computations optimizing for an outcome without running the additional computations needed to represent this optimization target to itself in explicit self-models available to reflective cognition.
Robin Hanson’s developed extensive, detailed sociological models that include this component. I think that the entire Overcoming Bias archives, not just Eliezer’s Sequences, ought to be canonical here, both because of their intellectual merit and because most of the Sequences were originally written on the Overcoming Bias blog in dialogue with Robin and his other co-bloggers there. A Theory of Identity seems particularly relevant here, and develops a directly relevant claim:
Our conscious minds are the public relations department of our larger minds, presenting and managing a story of ourselves to others.

Benquo 28 Nov 2021 14:08 UTC
2 points
on: Communication Requires Common Interests or Differential Signal Costs
The stronger the common interest in accurate signaling, the higher the opportunity cost of doing something else, which is sufficient for an economic differential signaling cost, though not necessarily an evolutionary one; in some cases where the expected value of honesty far exceeds the expected value of dishonesty we may still want to prevent dishonest behavior from being selected for too fast to reach the payoff.

Benquo 28 Nov 2021 13:30 UTC
1 point
in reply to: Rafael Harth’s comment on: Frame Control
This reply would be more interesting if it engaged with the last two paragraphs of my comment, in which I tried to develop a relevant causal hypothesis.

Benquo 28 Nov 2021 4:57 UTC
11 points
on: Almost everyone should be less afraid of lawsuits
an organization that always listens to lawyers is being more cautious than the optimum.
Even an organization that disobeys the lawyers sometimes but never regrets not having obeyed them is being more than optimally cautious.

Benquo 28 Nov 2021 4:25 UTC
49 points
on: Frame Control
This phenomenological account of frame control doesn’t provide a causal model precise enough for me to understand what additional question someone would be trying to answer when asking “is this person doing frame control?” aside from noticing which of the features of “frame control” they satisfy.
Some of the “red flags” seem like they could equally well point to someone fanatically committed to totally dominating others, or someone whose perspective responds to evidence but not social pressure, and many of the signs that someone is not a “frame controller” seem like the opposite. From the description of the first red flag:
They don’t laugh nervously, don’t give tiny signals that they are malleable and interested in conforming to your opinion or worldview.
These are signs of submission, as is the description of the first sign that someone isn’t “doing frame control”:
They give you power over them, like indications that they want your approval or unconditional support in areas you are superior to them. They signal to you that they are vulnerable to you.
This resembles a pattern I see across many contexts where the idea of listening to someone because they have something to say is replaced with the submissive behavior of believing someone out of loyalty. I don’t mean to say anything about your intent here, just to point out the general pattern, so we have a better chance of doing something else instead.
Second red flag:
A key component that makes frame control dangerous is when it’s linked to concrete consequences; maybe people really respect them, maybe they control resources, maybe they are the person throwing big events, maybe they gave you a new name, maybe they have the power to exclude you from your social group.
In general I would expect people whose beliefs are more responsive to valid evidence to have an easier time acting to produce the consequences they want. In environments that allocate credit straightforwardly I would also expect them to be more respected, entrusted with more resources, and have their concerns about others listened to more. In perverse environments I would expect the opposite.
The other red flags and opposite signs seem more ground-truth-oriented if I take them literally, as long as we don’t skip the step of investigating whether what the other person is saying is false or true. For instance, reframing “harm” as “beneficial” is helpful if I start out by mistakenly think that my discomfort confronting a criticism is a sign that the criticism is hurting me, but harmful if I’m being trained to treat the grinding-my-gears feeling of endorsing an incoherent narrative as a sign of healthy spiritual challenge.
If you can evaluate directly whether someone has more of a tendency to introduce errors or to correct them, then you can additionally judge whether they are introducing errors in a way calculated to make error-correction harder, to make themselves more central, and to make you more responsive to their commands. If they’re just giving commands and not making claims, then you can skip the first step and just see whether you think their commands are making you better off. In either case, it can help a lot to check whether you are behaving as though the other person is threatening you, whether they are behaving in a way optimized to produce that impression, and whether on reflection you believe that the threats are credible.
If you can’t evaluate that, then you’re in a more confusing situation. It can often be a good idea to switch contexts and see if things seem clear when outside that particular social field. If not, then you’re stuck, and at best you can decide which interactions seem to give you the best chance of achieving autonomy.
What links here?
- AnnaSalamon's comment on Frame Control by Aella (29 Nov 2021 19:44 UTC; 33 points)
- Ratios's comment on Frame Control by Aella (28 Nov 2021 10:58 UTC; 16 points)

Benquo 21 Nov 2021 1:30 UTC
3 points
in reply to: fuego’s comment on: Which booster shot to get and when?
This prison study is weird but appears to show that Moderna is MUCH more effective than the other vaccines. Applying Robin Hanson’s heuristic that the non-headline numbers in a study are less biased than the headline numbers, we should maybe treat this as more credible than official estimates of relative efficacy.

Benquo 21 Nov 2021 0:58 UTC
2 points
in reply to: Davis_Kingsley’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)
What do you see as the main sorts of interventions CFAR was organized around? I feel like this is a “different worlds” thing where I ought to be pretty curious what the whole scene looked like to you, what it seemed like people were up to, what the important activities were, & where progress was being made (or attempted).

Benquo 21 Nov 2021 0:54 UTC
2 points
in reply to: Kaj_Sotala’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)
I was relying on her self-reports, like https://www.lesswrong.com/posts/MnFqyPLqbiKL8nSR7/my-experience-at-and-around-miri-and-cfar-inspired-by-zoe#g9vLjj7rpGDH99adj

Benquo 21 Nov 2021 0:53 UTC
2 points
in reply to: mathenjoyer’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)

Specific claim: this is how to take over New York.

Didn’t work.

I think this needs to be broken up into 2 claims:

1 If we execute strategy X, we’ll take over New York. 2 We can use straightforward persuasion (e.g. appeals to reason, profit motive) to get an adequate set of people to implement strategy X.

2 has been falsified decisively. The plan to recruit candidates via appealing to people’s explicit incentives failed, there wasn’t a good alternative, and as a result there wasn’t a chance to test other parts of the plan (1).

That’s important info and worth learning from in a principled way. Definitely I won’t try that sort of thing again in the same way, and it seems like I should increase my credence both that plans requiring people to respond to economic incentives by taking initiative to play against type will fail, and that I personally might be able to profit a lot by taking initiative to play against type, or investing in people who seem like they’re already doing this, as long as I don’t have to count on other unknown people acting similarly in the future.

But I find the tendency to respond to novel multi-step plans that would require someone do take initiative by sitting back and waiting for the plan to fail, and then saying, “see? novel multi-step plans don’t work!” extremely annoying. I’ve been on both sides of that kind of transaction, but if we want anything to work out well we have to distinguish cases of “we / someone else decided not to try” as a different kind of failure from “we tried and it didn’t work out.”

Benquo 21 Nov 2021 0:36 UTC
0 points
in reply to: mathenjoyer’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)

Specific claim: the only nontrivial obstacle in front of us is not being evil

This is false. Object-level stuff is actually very hard.

This seems to be conflating the question of “is it possible to construct a difficult problem?” with the question of “what’s the rate-limiting problem?”. If you have a specific model for how to make things much better for many people by solving a hard technical problem before making substantial progress on human alignment, I’d very much like to hear the details. If I’m persuaded I’ll be interested in figuring out how to help.

So far this seems like evidence to the contrary, though, as it doesn’t look like you thought you could get help making things better for many people by explaining the opportunity.

Benquo 16 Nov 2021 3:04 UTC
16 points
on: Discussion with Eliezer Yudkowsky on AGI interventions
This reads like nothing new has happened in the discourse since 2011 except the development of interest in deep learning. Eliezer is still arguing with people exhibiting the same sorts of confusions he addressed in the Sequences. This seems like evidence that the “keep trying” behavior has long since hit diminishing returns relative to what I’d expect a “reorient” behavior to accomplish, e.g. more investigation into the causes of this kind of recalcitrance.

Benquo 31 Oct 2021 2:10 UTC
4 points
in reply to: EI’s comment on: Tell the Truth

Your failure starts with summarizing everything under one single label, here we are dealing with the word “liberalism.”

This reflects an empiricist view of language which is not compatible with analytic rigor. The kind of “liberalism” undermined by info about unequal capacities (a leftist commitment to economic leveling among citizens) is only associated with the word due to contingent facts about American coalitional politics. The natural cluster in political perspectives that “liberal” was invented to represent does not entail this position. Zack Davis’s Where to Draw the Boundaries? is relevant. If you don’t know the technical meaning of a term, this is a fact about you, not about the person using it.