I personally would not recommend financial support of MIRI, because I’m worried it will amplify net negative communications from him
Small note: Eliezer is largely backing off from direct comms and most of our comms in the next year will be less Eliezer’s-direct-words-being-promoted than in the past (as opposed to more). Obviously still lots of Eliezer thoughts and Eliezer perspectives and goals, but more filtered as opposed to less so. Just FYI.
It’s going to depend a lot on the social bubble/which group of friends. It’s not outrageous for the social circles I run in, which are pretty liberal/West Coast, but it would be outrageous for some bubbles I consider to be otherwise fun and fine and healthy.
Mainly it leans into the archetype of games like Truth or Dare, or Hot Seat, which are sort of canonically teenage party games and thus often trying to loosen those particular strictures.
Copying over a comment from the EA forum (and my response) because it speaks to something that was in some earlier drafts, that I expect to come up, and that is worth just going ahead and addressing imo.
IMO it would help to see a concrete list of MIRI’s outputs and budget for the last several years. My understanding is that MIRI has intentionally withheld most of its work from the public eye for fear of infohazards, which might be reasonable for soliciting funding from large private donors but seems like a poor strategy for raising substantial public money, both prudentially and epistemically.
If there are particular projects you think are too dangerous to describe, it would still help to give a sense of what the others were, a cost breakdown for those, anything you can say about the more dangerous ones (e.g. number of work hours that went into them, what class of project they were, whether they’re still live, any downstream effect you can point to, and so on).
My response:
(Speaking in my capacity as someone who currently works for MIRI)
I think the degree to which we withheld work from the public for fear of accelerating progress toward ASI might be a little overrepresented in the above. We adopted a stance of closed-by-default research years ago for that reason, but that’s not why e.g. we don’t publish concrete and exhaustive lists of outputs and budget.
We do publish some lists of some outputs, and we do publish some degree of budgetary breakdowns, in some years.
But mainly, we think of ourselves as asking for money from only one of the two kinds of donors. MIRI feels that it’s pretty important to maintain strategic and tactical flexibility, to be able to do a bunch of different weird things that we think each have a small chance of working out without exhaustive justification (or post-hoc litigation) of each one, and to avoid the trap of focusing only on clearly legible short chains of this—>that (as opposed to trying both legible and less-legible things).
(A colleague of mine once joked that “wages are for people who can demonstrate the value of their labor within a single hour; I can’t do that, which is why I’m on a salary.” A similar principle applies here.)
In the past, funding MIRI led to outputs like our alignment research publications and the 2020/2021 research push (that didn’t pan out). In the more recent past, funding MIRI has led to outputs like the work of our technical governance team, and the book (and its associated launch campaign and various public impacts).
That’s enough for some donors—”If I fund these people, my money will go into various experiments that are all aimed at ameliorating existential risk from ASI, with a lean toward the sorts of things that no one else is trying, which means high variance and lots of stuff that doesn’t pan out and the occasional home run.”
Other donors are looking to more clearly purchase a specific known product, and those donors should rightly send fewer of their dollars to MIRI, because MIRI has never been and does not intend to ever be quite so clear and concrete and locked-in.
(One might ask “okay, well, why post on the EA forum, which is overwhelmingly populated by the other kind of donor, who wants to track the measurable effectiveness of their dollars?” and the answer is “mostly for the small number who are interested in MIRI-like efforts anyway, and also for historical reasons since the EA and rationality and AI safety communities share so much history.” Definitely we do not feel entitled to anyone’s dollars, and the hesitations of any donor who doesn’t want to send their money toward MIRI-like efforts are valid.)
Most of the seven Extended Discussions under chapter 4 of Nate and Eliezer’s book’s supplementals are basically an expansion of this thesis (which I also agree with and think is true).
I have not yet been limiting screens, except insofar as, like, hanging out with Cadence from 6pm to sleep usually means no screens. But it seems like I’m obviously going to end up there, and am sort of just relinquishing my reluctance/figuring out a new flow that still allows me to do the things I need to get done each day.
Currently, I get some Third State in the mornings, actually, when Cadence has woken up and is puttering around but hasn’t come and fully woken me up yet.
This is great.
One constraint that Logan and I have both gotten a lot of benefit out of reinstating is one around artificial light; Logan is stricter than I but we’re both much more subject to natural darkness than the average American and afaict it’s doing a bunch of positive things. Better sleep, more connection with the “third state of consciousness” that is between waking and sleeping, we essentially never struggle to settle Cadence down for bed, etc.
In past eras of my life, I had “don’t drive less than two miles” as a soft rule and would instead walk any sub-two-mile distance, and I think this too was useful.
(This answer feels fragmented; sorry; am still on postsurgery medication and brain is at 40% capacity.)
Possibly of interest, although it sounds like you maybe already have all of these puzzle pieces.
Example: suppose someone says “I can imagine an atomic copy of ourselves which isn’t conscious, therefore consciousness is non-physical.” and I say “No, I can’t imagine that.”
Or the followup by Logan Strohl, even more directly on this
Just noting for the audience that the edits which Anna references in her reply to CronoDAS, as if they had substantively changed the meaning of my original comment, were to add:
The phrase “directly observed”
The parenthetical about having good epistemic hygiene with regards to people’s protestations to the contrary
The bit about agendas often not being made explicit
It did not originally specify undisclosed conflicts of interest in any way that the new version doesn’t. Both versions contained the same core (true) claim: that multiple of the staff members common to both CFAR!2017 and CFAR!2025 often had various (i.e. not only the AI stuff) agendas which would bump participant best interests to second, third, or even lower on the priority ladder.
I’ve also added, just now, a clarifying edit to a higher comment: “Some of these staff members are completely blind to some centrally important axes of care.” This seemed important to add, given that Anna is below making claims of having seen, modeled, and addressed the problems (a refrain I have heard from her, directly, in multiple epochs, and taken damage from naively trusting more than once). More (abstract, philosophical) detail on my views about this sort of dynamic here.
I claim to be as-aware and as-sensitive-to of all of these considerations as you are. I think I am being as specific as possible, given constraints (many of which I wish were not there; I have a preference for speaking more clearly than I can here).
I know of one parent that puts three dollars aside each time they violate the bodily sovereignty of their infant—taking something out of their mouth, or restricting where they can go
It’s me, by the way. Happy to identify myself.
(I have more agreement than disagreement with the authors on many points, here.)
I’ll note that both this and my top-level comment have a lot of agree-disagree votes, and that it would be wise for people looking in from the outside to ponder what it means for e.g. the top-level comment to have 18 people voting and to end up at −2.
(It might be tempting to sum it up as “ah, Duncan claimed that there’s something to be wary of here, and the hive-mind ultimately ended up in disagreement” but I think it’s more like “Duncan claimed there’s something to be wary of here, and close to half of the people agreed (but were drowned out by the somewhat more than half who disagreed).” Which is precisely what you would expect if there were some system or process that was consistently harmful to certain people, but not all or even most—a lot of people who passed through unscathed would be like “what do you mean? I was well-cared-for!” and might not pause to wonder about whether they were a black raven and what evidence their experience provides about claims of the existence of white ravens.)
I came to discover, over time, that the orientation I had toward participants (and subordinates, for that matter), and the care that I felt I owed the people under my supervision—
(and which I believed were standard and universal à la something like the hippocratic oath or the confidentiality standards of lawyers and therapists)
—were not, in fact, universal. I directly observed that certain staff members did not reliably have the best interests of participants at heart (whatever their protestations to the contrary), but instead had various agendas (which were often not made explicit) which meant that the best interests of the participants might sometimes be second, or third, or even lower than third on the priority list.
i.e. I believe that the past ones, in hindsight, were not only not adequately responsible and careful but were in a crucial way not even trying to be, and I do not have reason to believe that this problem will be any less in an era where people like myself and Kenzi and Julia Galef and Dan Keys are not present.
(There is a rebuttal that might be made that goes something like “ah, well, those staff members have owned up to that very problem and are explicitly striving to do the other thing, now,” but a) see the point above about not trusting people who have managed to fuck up X in multiple novel ways, and b) in the world where such a hypothetical rebuttal were in fact to be made, I wouldn’t personally put much weight on the self-report of people who are saying that they used to be something like deceptive/manipulative (to the detriment of others) but don’t worry, they aren’t doing that anymore.)
Why is it necessary? Do you think that you are the last potential wizard of Light in the world? Why must you be the one to try for greatness, when I have advised you that you are riskier than average? Let some other, safer candidate try!
These issues didn’t seem to be a problem for 95+% of participants. But I think many of my own friends and family members would feel differently about choosing to be vulnerable in the following two scenarios:
You are under the care of professionals who have your best interests at heart, and who have never abused their position of power to manipulate, deceive, or exploit people under their care
You are under the care of professionals who probably (95+% by raw numbers) have your best interests at heart, except in the rare (5-%) subset of cases where they think they can make some interesting use of you, which they may or may not be up front with you about, versus trying to twitch your strings to maneuver you into some position for the fulfillment of their own values, agnostic to yours
Thinking about thinking, tinkering with your mental and emotional algorithms, shaking up your worldview, adopting new perspectives and new strategies, spending a lot of time zeroing in and ruminating on your problems and goals and values and considering them in contact with other people and with suggestions about how to see them and think of them and change them. Setting aside your normal ways of doing things.
Becoming more mud, in other words.
This is already inherently vulnerable, but it gets moreso when you’re doing it in an isolated retreat context surrounded by other people for multiple days in which there is a clear status differential between the instructors and the participants.
There are ways to do this that are more responsible and careful, and there are ways to do this that are less responsible and careful. Separately, a person or group can have the intent to do such a thing responsibly and carefully, and this is not the same as being able to do this responsibly and carefully. Some of these staff members are completely blind to some centrally important axes of care.
(If you’ve seen a person or group try for X and fail repeatedly in multiple novel ways despite multiple rounds of figuring out what went wrong and fixing it in each specific case, it’s wise to be wary of their latest attempt at X. Sometimes people exhibit a curiously robust capacity to keep generating brand-new ways to get X wrong, and my desire to register a warning here is partially downstream of my belief that something like that is true, here.)
I think for reasons of not needlessly rending the social fabric, I don’t want to be more specific. I feel the need to register the warning, and I’m happy with people weighting or discounting it based on how much they trust my assessments generally, including the context that I worked at CFAR for three years and overlapped with much of the present staff.
Just noting for the audience that I would not recommend this, and would strongly discourage my friends and family members from going to it. There’s sort of nothing I can do about the fact that this is inherently rude, and clashes with the largely-positive tone of all the other discussion, but it feels rather important to represent this fact, especially as someone with slightly more context and grounded understanding than most.
(I have had zero interaction with CFAR since the end of my three years working there in 2019 and can make no confident predictions about the object-level experience, but I do not trust and indeed am substantially wary of the motives, methods, and competence of (some members of) the team creating the experience. I would not want anyone I love to put themselves in a vulnerable state under the care of (some of) these particular people, as I have justified reason to believe that duty-of-care will not be reliably discharged. I’ll note that when plans for something like this first reached my ears circa 2022*, I directly told at least one of the people I’m concerned about that I thought they should absolutely not participate in anything like this, and that I expected they would cause harm by doing so (because they had repeatedly caused harm in the past) and should leave the project to others.)
Note also that the CFAR handbook exists and is free to all.
*edited; previously said 2021; I went back and checked the email that I sent; it was in June of 2022 and was not responded to.
Yes, there are efforts; they are unfortunately controlled by the publisher and not the sort of thing we can outsource or influence. Renegade translations seem morally good to me, if people are moved to create them, provided that they actually try to do a good job.
Languages that the book is being translated into include (85% probability on any member of this list; I’m a bit brain-dead this weekend): Mandarin, Italian, Spanish, Russian, Korean, Vietnamese, Dutch, and Bulgarian. We’re working on translating the online supplementals into at least those first four.
(“Why such a strange list?” you might ask. Well, the list isn’t done; the publisher is still wrangling contracts in other nations/regions. Contracts come in when they come in. German, for instance, is highly likely to eventually get a translation—or at least, we’d be quite excited to see one, given Germany’s prominence in the EU. But again, out of our hands. We put most of our prioritization energy into making sure there would be a Chinese-language translation, as that seemed super obviously the most important non-English-speaking audience.)
A reply pretty near the top that also feels relevant to this overall point:
To be clear, it’s not because we agree with Buck’s model. It’s more that Eliezer has persistent health and stamina issues and others (Nate, Malo, etc.) need to step up and receive the torch.
(Also “less” doesn’t mean “zero”.)