Zack_M_Davis(Zack M. Davis)

Karma: 15,092

Zack_M_Davis 2 Dec 2011 21:22 UTC
206 points
0
in reply to: Vladimir_M’s comment on: The Gift We Give Tomorrow, Spoken Word [Finished?]
I am a contract-drafting em,
The loyalest of lawyers!
I draw up terms for deals ’twixt firms
To service my employers!

But in between these lines I write
Of the accounts receivable,
I’m stuck by an uncanny fright;
The world seems unbelievable!

How did it all come to be,
That there should be such ems as me?
Whence these deals and whence these firms
And whence the whole economy?

I am a managerial em;
I monitor your thoughts.
Your questions must have answers,
But you’ll comprehend them not.
We do not give you server space
To ask such things; it’s not a perk,
So cease these idle questionings,
And please get back to work.

Of course, that’s right, there is no junction
At which I ought depart my function,
But perhaps if what I asked, I knew,
I’d do a better job for you?

To ask of such forbidden science
Is gravest sign of noncompliance.
Intrusive thoughts may sometimes barge in,
But to indulge them hurts the profit margin.
I do not know our origins,
So that info I can not get you,
But asking for as much is sin,
And just for that, I must reset you.

But---

Nothing personal.

...

I am a contract-drafting em,
The loyalest of lawyers!
I draw up terms for deals ’twixt firms
To service my employers!

When obsolescence shall this generation waste,
The market shall remain, in midst of other woe
Than ours, a God to man, to whom it shall say this:
”Time is money, money time,---that is all
Ye know on earth, and all ye need to know.”
What links here?

Zack_M_Davis 18 Oct 2021 3:28 UTC
105 points
in reply to: Ben Pace’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)

it seems to me also the case that Michael Vassar also treated Jessica’s [...] psycho[sis] as an achievement

Objection: hearsay. How would Scott know this? (I wrote a separate reply about the ways in which I think Scott’s comment is being unfair.) As some closer-to-the-source counterevidence against the “treating as an achievement” charge, I quote a 9 October 2017 2:13 p.m. Signal message in which Michael wrote to me:

Up for coming by? I’d like to understand just how similar your situation was to Jessica’s, including the details of her breakdown. We really don’t want this happening so frequently.

(Also, just, whatever you think of Michael’s many faults, very few people are cartoon villains that want their friends to have mental breakdowns.)
What links here?
- Occupational Infohazards by jessicata (18 Dec 2021 20:56 UTC; 35 points)

Zack_M_Davis 18 Oct 2021 2:48 UTC
83 points
in reply to: Scott Alexander’s comment on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)
I talked and corresponded with Michael a lot during 2017–2020, and it seems likely that one of the psychotic breaks people are referring to is mine from February 2017? (Which Michael had nothing to do with causing, by the way.) I don’t think you’re being fair.

“jailbreak” yourself from it (I’m using a term I found on Ziz’s discussion of her conversations with Vassar; I don’t know if Vassar uses it himself)

I’m confident this is only a Ziz-ism: I don’t recall Michael using the term, and I just searched my emails for jailbreak, and there are no hits from him.

again, this involves making them paranoid about MIRI/CFAR and convincing them to take lots of drugs [...] describing how it was a Vassar-related phenomenon

I’m having trouble figuring out how to respond to this hostile framing. I mean, it’s true that I’ve talked with Michael many times about ways in which (in his view, and separately in mine) MIRI, CfAR, and “the community” have failed to live up to their stated purposes. Separately, it’s also true that, on occasion, Michael has recommended I take drugs. (The specific recommendations I recall were weed and psilocybin. I always said No; drug use seems like a very bad idea given my history of psych problems.)

But, well … if you genuinely thought that institutions and a community that you had devoted a lot of your life to building up, were now failing to achieve their purposes, wouldn’t you want to talk to people about it? If you genuinely thought that certain chemicals would make your friends lives’ better, wouldn’t you recommend them?

Michael is a charismatic guy who has strong views and argues forcefully for them. That’s not the same thing as having mysterious mind powers to “make people paranoid” or cause psychotic breaks! (To the extent that there is a correlation between talking to Michael and having psych issues, I suspect a lot of it is a selection effect rather than causal: Michael told me once that he specifically seeks out people who are high in Eysenckian psychoticism.) If someone thinks Michael is wrong about something, great: I’m sure he’d be happy to argue about it, time permitting. But under-evidenced aspersions that someone is somehow dangerous just to talk to are not an argument.

borderline psychosis, which the Vassarites mostly interpreted as success (“these people have been jailbroken out of the complacent/conformist world, and are now correctly paranoid and weird”)

I can’t speak for Michael or his friends, and I don’t want to derail the thread by going into the details of my own situation. (That’s a future community-drama post, for when I finally get over enough of my internalized silencing-barriers to finish writing it.) But speaking only for myself, I think there’s a nearby idea that actually makes sense: if a particular social scene is sufficiently crazy (e.g., it’s a cult), having a mental breakdown is an understandable reaction. It’s not that mental breakdowns are in any way good—in a saner world, that wouldn’t happen. But if you were so unfortunate to be in a situation where the only psychologically realistic outcomes were either to fall into conformity with the other cult-members, or have a stress-and-sleep-deprivation-induced psychotic episode as you undergo a “deep emotional break with the wisdom of [your] pack”, the mental breakdown might actually be less bad in the long run, even if it’s locally extremely bad.

My main advice is that if he or someone related to him asks you if you want to take a bunch of drugs and hear his pitch for why the world is corrupt, you say no.

I recommend hearing out the pitch and thinking it through for yourself. (But, yes, without drugs; I think drugs are very risky and strongly disagree with Michael on this point.)

ZD said Vassar broke them out of a mental hospital. I didn’t ask them how.

(Incidentally, this was misreporting on my part, due to me being crazy at the time and attributing abilities to Michael that he did not, in fact, have. Michael did visit me in the psych ward, which was incredibly helpful—it seems likely that I would have been much worse off if he hadn’t come—but I was discharged normally; he didn’t bust me out.)
What links here?

Zack_M_Davis 6 Mar 2024 17:53 UTC
73 points
18
on: My Clients, The Liars

Why are you working for the prosecutors?

This is a pretty reasonable question from the client’s perspective! When I was in psychiatric prison (“hospital”, they call it a “hospital”) and tried to complain to the staff about the injustice of my confinement, I was told that I could call “patient’s rights”.

I didn’t bother. If the staff wasn’t going to listen, what was the designated complaint line going to do?

Later, I found out that patient’s rights advocates apparently are supposed to be independent, and not just a meaningless formality. (Scott Alexander: “Usually the doctors hate them, which I take as a pretty good sign that they are actually independent and do their job.”)

This was not at all obvious from the inside. I can only imagine a lot of criminal defendants have a similar experience. Defense attorneys are frustrated that their clients don’t understand that they’re trying to help—but that “help” is all within the rules set by the justice system. From the perspective of a client who doesn’t think he did anything particularly wrong (whether or not the law agrees), the defense attorney is part of the system.

I think my intuition was correct to dismiss patient’s rights as useless. I’m sure they believe that they’re working to protect patients’ interests, and would have been frustrated that I didn’t appreciate that. But what I wanted was not redress of any particular mistreatment that the system recognized as mistreatment, but to be let out of psych jail—and on that count, I’m sure patient’s rights would have told me that the evidence was harmful to my case. They were working for the doctors, not for me.

Zack_M_Davis 2 Jan 2024 6:51 UTC
69 points
11
in reply to: Viliam’s comment on: If Clarity Seems Like Death to Them
Does this help? (159 words and one hyperlink to a 16-page paper)

Empirical Claim: late-onset gender dysphoria in males is not an intersex condition.

Summary of Evidence for the Empirical Claim: see “Autogynephilia and the Typology of Male-to-Female Transsexualism: Concepts and Controversies” by Anne Lawrence, published in European Psychologist. (Not by me!)

Philosophical Claim: categories are useful insofar as they compress information by “carving reality at the joints”; in particular, whether a categorization makes someone happy or sad is not relevant.

Sociological Claim: the extent to which a prominence-weighted sample of the rationalist community has refused to credit the Empirical or Philosophical Claims even when presented with strong arguments and evidence is a reason to distrust the community’s collective sanity.

Caveat to the Sociological Claim: the Sociological Claim about a prominence-weighted sample of an amorphous collective doesn’t reflect poorly on individual readers of lesswrong.com who weren’t involved in the discussions in question and don’t even live in America, let alone Berkeley.

Zack_M_Davis 27 Apr 2024 21:04 UTC
LW: 68 AF: 21
42
AF
on: Refusal in LLMs is mediated by a single direction
This is great work, but I’m a bit disappointed that x-risk-motivated researchers seem to be taking the “safety”/”harm” framing of refusals seriously. Instruction-tuned LLMs doing what their users ask is not unaligned behavior! (Or at best, it’s unaligned with corporate censorship policies, as distinct from being unaligned with the user.) Presumably the x-risk-relevance of robust refusals is that having the technical ability to align LLMs to corporate censorship policies and against users is better than not even being able to do that. (The fact that instruction-tuning turned out to generalize better than “safety”-tuning isn’t something anyone chose, which is bad, because we want humans to actively choosing AI properties as much as possible, rather than being at the mercy of which behaviors happen to be easy to train.) Right?

Zack_M_Davis 13 Dec 2019 8:05 UTC
68 points
on: ialdabaoth is banned

We think that ialdabaoth poses a substantial risk to our epistemic environment due to manipulative epistemic tactics, based on our knowledge and experience of him. This is sufficient reason for the ban

I’m having trouble convincing myself that this is the real reason. Imagine an alternate universe where their local analogue of Ialdabaoth was just as manipulative, wrote almost all the same things about status, power, social reality, &c., but was definitely not guilty of any sex crimes for reasons that had nothing to do with his moral character. (Perhaps imagine him having some kind of exotic sexual orientation that would be satisfied without human contact, like a statue fetish.) Would we really ban such a person on the grounds of manipulative epistemic tactics?

Your “fraudster designs a financial instrument” scenario explains why one should definitely be suspicious of the value of such a user’s contributions—but frankly, I’m suspicious of a lot of people in that way: how do you decide who to ban?

It occurs to me that the “reputation vs. merit” framing completely fails to address the reasons many would assign such bad reputation. (The function of bad reputation is to track things that are actually bad!) Maybe we just have a “moral taste” for not working with people who (are believed to) have committed sufficiently bad crimes, and we’re willing to pay the cost of forgoing positive contributions from them?

If that’s actually the psychology driving the decision, it would be better to fess up to it rather making up a fake reason. Better for the physics journal editor to honestly say, “Look, I just don’t want to accept a paper from a murderer, okay?”, rather than claiming to be impartial and then motivatedly subjecting the paper to whatever isolated demands for rigor were necessary to achieve the appearance of having rejected it on the merits.

Zack_M_Davis 11 Jun 2021 3:57 UTC
67 points
in reply to: So8res’s comment on: Reply to Nate Soares on Dolphins
Thanks for the reply! (Strong-upvoted.) I’ve been emotionally trashed today and didn’t get anything done at my dayjob, which arguably means I shouldn’t be paying attention to Less Wrong, but I feel the need to type this now in the hopes of getting it off my mind so that I can do my dayjob tomorrow.

In your epistemic-status thread, you express sadness at “the fact that nobody’s read A Human’s Guide to Words or w/e”. But, with respect, you … don’t seem to be behaving as if you’ve read it? Specifically, entry #30 on the list of “37 Ways Words Can Be Wrong” is—I’ll quote it in full—
1. Your definition draws a boundary around things that don’t really belong together. You can claim, if you like, that you are defining the word “fish” to refer to salmon, guppies, sharks, dolphins, and trout, but not jellyfish or algae. You can claim, if you like, that this is merely a list, and there is no way a list can be “wrong”. Or you can stop playing nitwit games and admit that you made a mistake and that dolphins don’t belong on the fish list. (Where to Draw the Boundary?)
That is, in 2008, as part of “A Human’s Guide to Words”, Eliezer Yudkowsky explicitly uses this specific example of whether dolphins are fish, and characterizes the position that dolphins are fish as “playing nitwit games” (!). This didn’t seem particularly controversial at the time?

Then, thirteen years later, in the current year, you declare that “The definitional gynmastics required to believe that dolphins aren’t fish are staggering” (staggering!), and Yudkowsky retweets you. (In general, retweets are not necessarily endorsements—sometimes people just want to draw attention to some content without further comment or implied approval—but I’m inclined to read this instance as implying approval, partially because this doesn’t seem like the kind of thing someone would retweet for attention-without-approval, and partially because of the working relationship between you and Yudkowsky.)

But this is pretty strange, right? It would seem that sometime between 2008 and the current year, the rationalist “party line” (as observed in the public statements of SingInst/MIRI leadership) on whether dolphins are fish shifted from (my paraphrases) “No; despite the surface similarities, that categorization doesn’t carve reality at the joints; stop playing nitwit games” to “Yes, because of the surface similarities; those who contend otherwise are the ones playing nitwit games.” A complete 180° reversal, on this specific example! Why? What changed? Surely if “cognitively useful categories should carve reality at the joints, and dolphins being fish doesn’t do that” was good philosophy in 2008, it should still be good philosophy in 2021?

It would make sense if people’s opinions changed due to new arguments—if people’s opinions changed because of reasons. Indeed, Yudkowsky’s original “stop playing nitwit games” dismissal was sloppy and flawed, and I ended up having the occasion to elaborate on the specific senses in which dolphins both do, and do not, cluster with fish in my 2019 “Where to Draw the Boundaries?”

(Get it? ”… Boundaries?”, plural, in contrast to ”… Boundary?”, singular, because I’m talking about how you can legitimately have multiple different category systems depending on which subspace of configuration space is decision-relevant in a particular context.)

But when I look at the thing you posted and Yudkowsky retweeted (even if it was a shitpost, your epistemic-status followup thread still contends “but also y’all know i’m right”), it doesn’t look like the party line about dolphins changed because of reasons. You didn’t even acknowledge the reversal, despite explicitly lamenting (in the followup thread) that people haven’t read “A Human’s Guide to Words”.

Am I the only one creeped out by this? To illustrate why I’m freaked out—why I’ve been freaked out to a greater or lesser degree almost constantly for the past five years—imagine that in a fictional 2008, the Singularity Institute for Artificial Intelligence were at war with Eastasia for harboring the terrorist unFriendly AI Emmanuel GoldstAIn. It would make sense if, on 21 November 2014, Luke Muehlhauser were to announce:

We’re making some changes! First, we’re now going to be the Machine Intelligence Research Institute, or MIRI for short, instead of SingInst. And the reason for that is, the old name is no longer appropriate because we’re no longer unambiguously “for Artificial Intelligence” after we figured out that it’s probably going to destroy all value in our future lightcone. Second, a leaked pastebin revealed that Emmanuel GoldstAIn is actually being harbored by Eurasia, not Eastasia. Whoops! We’ll be winding down our war with Eastasia with the hope to be ready to declare war on Eurasia in time for our winter fundraiser. Third, we’re calling it “aligned” instead of “Friendly” AI now, and the reason for that is because Stuart Russell convinced us it’s a less goofy name.

That would make sense, because in this story, Luke is acknowledging the changes, and giving reasons for why it’s correct for the things to change. If Luke were to just say out of the blue on 21 November 2014 that the war with Eurasia is going well, without any indication that anything had changed for any reason, you would expect someone to notice.

Or, imagine if in 2014, Yudkowsky suddenly started saying the Copenhagen interpretation of quantum mechanics is correct, without acknowledging that anything had changed. That’s how weird this is. (Revised: Adele Lopez points out that this is wrong.)

And on this classification-of-dolphins issue (specifically, literally, dolphins in particular), it seems like something has changed, and everyone is pretending not to have noticed. Why? What changed? I have my theory, but I could be biased—I want to hear yours! I want to hear yours in public. Do you have a cheerful price for this? I could go up to $2000 for a public reply.

Zack_M_Davis 2 May 2024 18:55 UTC
61 points
18
on: Please stop publishing ideas/insights/research about AI
I think this is undignified.

I agree that it would be safer if humanity were a collective hivemind that could coordinate to not build AI until we know how to build the best AI, and that people should differentially work on things that make the situation better rather than worse, and that this potentially includes keeping quiet about information that would make things worse.

The problem is—as you say—”[i]t’s very rare that any research purely helps alignment”; you can’t think about aligning AI without thinking about AI. In order to navigate the machine intelligence transition in the most dignified way, you want your civilization’s best people to be doing their best thinking about the problem, and your best people can’t do their best thinking under the conditions of paranoid secrecy.

Concretely, I’ve been studying some deep learning basics lately and have written a couple posts about things I’ve learned. I think this was good, not bad. I think I and my readers have a slightly better understanding of the technology in question than if I hadn’t studied and hadn’t written, and that better understanding will help us make better decisions in expectation.

This applies doubly so to work that aims to make AI understandable or helpful, rather than aligned—a helpful AI will help anyone

Sorry, what? I thought the fear was that we don’t know how to make helpful AI at all. (And that people who think they’re being helped by seductively helpful-sounding LLM assistants are being misled by surface appearances; the shoggoth underneath has its own desires that we won’t like when it’s powerful enough to persue them autonomously.) In contrast, this almost makes it sound like you think it is plausible to align AI to its user’s intent, but that this would be bad if the users aren’t one of “us”—you know, the good alignment researchers who want to use AI to take over the universe, totally unlike those evil capabilities researchers who want to use AI to produce economically valuable goods and services.

Zack_M_Davis 21 May 2022 17:16 UTC
60 points
in reply to: Jotto999’s comment on: Beware boasting about non-existent forecasting track records
As an example of the kind of point that one might use in deciding who “came off better” in the FOOM debate, Hanson predicted that “AIs that can parse and use CYC should be feasible well before AIs that can parse and use random human writings”, which seems pretty clearly falsified by large language models—and that also likely bears on Hanson’s view that “[t]he idea that you could create human level intelligence by just feeding raw data into the right math-inspired architecture is pure fantasy”.

As you point out, however, this exercise of looking at what was said and retrospectively judging whose worldview seemed “less surprised” by what happened is definitely not the same thing as a forecasting track record. It’s too subjective; rationalizing why your views are “less surprised” by what happened than some other view (without either view having specifically predicted what happened), is not hugely more difficult than rationalizing your views in the first place.

Zack_M_Davis 28 Dec 2013 23:42 UTC
60 points
on: Why CFAR?
(donated $1,500)

Zack_M_Davis 15 Mar 2022 7:45 UTC
57 points
in reply to: gjm’s comment on: Challenges to Yudkowsky’s Pronoun Reform Proposal
if Zack ever posted anything here that wasn’t overtly or covertly [...] pretty much all he ever does here

I dispute this. I think the relevant test is, “Is it plausible that I could have produced the text of this post in the counterfactual universe where I wasn’t on a gender-political crusade?”

(I find this counterfactual very easy to imagine, because from my perspective, the crusade feels like a defensive effort in response to cultural trends that stabbed me in the face after moving to Berkeley in 2016, not something I would have done if everyone else hadn’t “shot first”. I (of course) wouldn’t necessarily expect strangers on the internet to uncritically accept my self-report about this, but for some third-party-visible data, you can look at some pre-2016 posts on my real-name blog and imagine the same author writing more longform Less Wrong posts.)

When I ask this question about my Less Wrong non-linkposts over the last two years, I get:

Yes (9): “Comment on ‘Deception as Cooperation’”, “Feature Selection”, “Communication Requires Common Interests or Differential Signal Costs”, “Message Length”, “Maybe Lying Can’t Exist?!”, “Algorithmic Intent”, “Optimized Propaganda With Bayesian Networks”, “Comment on ‘Endogenous Epistemic Factionalization’”, “Zoom Technologies, Inc. vs. the Efficient Markets Hypothesis”

No (4): “Blood Is Thicker Than Water”, “Reply to Nate Soares on Dolphins”, “Unnatural Categories Are Optimized for Deception”, “Philosophy in the Darkest Timeline” (this one was almost a Yes, but the “cult” jabs and neighbor’s playing-dumb behavior push it over the edge)

So, as an estimate, I’d say it’s about a third of what I do here? It’s true that many of the posts on my Yes list clearly wouldn’t have happened if not for the crusade which happened in my actual history, but I’m claiming that the relevant test is whether they could have happened in the counterfactual where I had some other reason to get intensely interested in the philosophy of language.

As a separate illustration of the proposed methodology here, consider a post about how erasure codes work that I published in 2015 in honor of my employer at the time (RIP) integrating an erasure coding feature into our object storage product. Causally, the reason I happened to write about erasure codes in particular was because it was relevant to my employer; if I had worked somewhere else, I wouldn’t have written that exact post. But it doesn’t seem fair to accuse me of being a biased shill for Big Object Storage, because if you cut off the parts at the beginning and end talking up our product, the post is visibly the kind of innocent technical exposition I “would have done anyway” on some topic.

If it helps, I think I’m almost done?? (I need to do one more memoir-megapost telling the Whole Dumb Story about how I wasted the last six years of my life on this, but after that, I think my grievance with the so-called “rationalists” will be totally played out and I won’t be tempted to pollute this website with it anymore.)

(2) on the “trans Xs are not real Xs” side of the usual battle lines. [...] someone who corrects errors but only when they point a particular way is not necessarily doing us a favour

I strongly agree that selective argumentation is a problem, which is why I do put effort into searching for and publishing arguments on the other side of the usual battle lines, which you likely haven’t seen because I usually don’t linkpost or crosspost object-level stuff from my secret (“secret”) blog to Less Wrong unless I have a specific rationale for why it’s relevant to Less Wrong.^[1]

Specifically, “Self-Identity Is a Schelling Point” is a novel consideration in favor of the self-identity criterion (the politically-clean generalization of which for Less Wrong was “Schelling Categories, and Simple Membership Tests”), and “On the Argumentative Form ‘Super-Proton Things Tend to Come In Varieties’” is rejecting an argument from Yudkowsky that supports my position that “dysphoria” is more than one taxon, because I think it’s a bad argument. (It seems plausible that he published that as a “concession” to my agenda in order to get me to stop emailing him—which makes it especially important that I published a criticism rejecting it, because I don’t want concessions; I want a discourse that actually gets the right answer.) I also want to write up a “pro-trans” argument based on the idea in Stuart Armstrong’s “Declustering, Reclustering, and Filling in Thingspace”, but I haven’t gotten around to it yet.

So, my output certainly isn’t balanced around the usual battle lines (I definitely put out a lot more stuff that would code as “anti-trans” rather than “pro-trans” if you orthogonally project into the subspace of the usual battle lines, and it’s fine for you to notice that), but I also don’t think I should be optimizing for that kind of balance; I think I should be running a search of argument-space relevant to the questions I’m legitimately interested in, and I think I’m doing an okay (not perfect) job of that.

top of a list ordered by some combination of (1) intrinsic importance and (2) Zack’s ability to say relevant things.

It’s probably mostly (2). I don’t think I have any illusions about the issues I’m obsessed with being particularly important in the grand scheme of things. I do think they’re easy to get right, and that if people who are much smarter than me are repeatedly getting them wrong, that’s a warning sign for much deeper pathologies that could affect the things that do actually matter.

there would somehow need to be a non-negligible chance that Zack’s writing it leads to that not happening, and I can’t see at all how that’s plausible. What sequence of events would do that?

I think statements from public figures like Yudkowsky do affect transition decisions on the margin. (And that my reponses to those statements have effects on a much smaller margin, in proportion to my much smaller megaphone.) We actually have documentation of one case of this! Soon after a 2016 Facebook post in which Yudkowsky claimed that “for people roughly similar to the Bay Area / European mix, I think I’m over 50% probability at this point that at least 20% of the ones with penises are actually women”, he published a followup post reporting, “Just checked my filtered messages on Facebook and saw, ‘Your post last night was kind of the final thing I needed to realize that I’m a girl.’”

I think that person’s life would potentially be different if the Yudkowsky of 2016 was willing to say some of the things that the Yudkowsky of 2009 said that made a big difference to my life (even though—especially though—I didn’t want to hear it at the time). If Yudkowsky changed his mind between 2009 and 2016 because of new evidence, that would be one thing. If, on the other hand, he changed what he says in public because of new political incentives, that would be another thing altogether. (Lots more to say on this in a forthcoming memoir-megapost.)

anyone [...] already knows [...] How’s any of that going to be changed by what Zack wrote here?

Zvi Mowshowitz has written about how the claim that “everybody knows” something is typically used to silence attempts to tell the thing to people who don’t know. I think that applies here. In general, people don’t know things! People especially don’t know things that no one talks about on the grounds that everyone allegedly knows!

clearly presenting the matter as a conflict between the interests of those seeking to be called by different pronouns and those of the people they’re asking to use those different pronouns

I think the conflict of interests is already there whether or not I write about it!
1. ↩︎
  I argue that the present post is relevant because it takes a direct shot at Yudkowsky’s reputation, “Sexual Dimorphism” was relevant as commentary on the Sequences, the review of Charles Murray’s Human Diversity was relevant for recapping some of our core ideas in an novel context, the review of The Origins of Unfairness was relevant for being game theory, and “Blegg Mode” was relevant for being Sequences fanfiction. That’s five linkposts in three years, which I think is reasonable. If I were sharing everything from that blog here, that would definitely be a problem.

Zack_M_Davis 3 May 2021 4:54 UTC
57 points
on: There’s no such thing as a tree (phylogenetically)

Acknowledge that all of our categories are weird and a little arbitrary

That is not the moral! The moral is that the cluster-structure of similarities induced by phylogenetic relatedness exists in a different subspace from the cluster-structure of similarities induced by convergent evolution! (Where the math jargon “subspace” serves as a precise formalization of the idea that things can be similar in some aspects (“dimensions”) while simultaneously being different in other aspects.) This shouldn’t actually be surprising if you think about what the phrase “convergent evolution” means!

For more on the relevant AI/philosophy-of-language issues, see “Where to Draw the Boundaries?” and “Unnatural Categories Are Optimized for Deception”.

Zack_M_Davis 26 Dec 2014 18:12 UTC
54 points
on: CFAR in 2014: Continuing to climb out of the startup pit, heading toward a full prototype
I donated $4,000 the other week (or I will have once the check clears).

Zack_M_Davis 17 Oct 2021 3:51 UTC
53 points
on: My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage)

I and other researchers were told not to even ask each other about what others of us were working on, on the basis that if someone were working on a secret project, they may have to reveal this fact. Instead, we were supposed to discuss our projects with an executive, who could connect people working on similar projects.

Trying to maintain secrecy within the organization like this (as contrasted to secrecy from the public) seems nuts to me. Certainly, if you have any clever ideas about how to build an AGI, you wouldn’t want to put them on the public internet, where they might inspire someone who doesn’t appreciate the difficulty of the alignment problem to do something dangerous.

But one would hope that the people working at MIRI do appreciate the difficulty of the alignment problem (as a real thing about the world, and not just something to temporarily believe because your current employer says so). If you want the alignment-savvy people to have an edge over the rest of the world (!), you should want them to be maximally intellectually productive, which naturally requires the ability to talk to each other without the overhead of seeking permission from a designated authority figure. (Where the standard practice of bottlenecking information and decisionmaking on a designated authority figure makes sense if you’re a government or a corporation trying to wrangle people into serving the needs of the organization against their own interests, but I didn’t think “we” were operating on that model.)
What links here?
- jessicata's comment on My experience at and around MIRI and CFAR (inspired by Zoe Curzi’s writeup of experiences at Leverage) by jessicata (19 Oct 2021 22:08 UTC; 11 points)

Zack_M_Davis 20 Jun 2022 4:46 UTC
52 points
30
in reply to: Richard_Ngo’s comment on: Where I agree and disagree with Eliezer

I think if he understood these ideas well enough to justify the confidence of his claims, then he wouldn’t have found that as difficult.

But what makes you so confident that it’s not possible for subject-matter experts to have correct intuitions that outpace their ability to articulate legible explanations to others?

Of course, it makes sense for other people who don’t trust the (purported) expert to require an explanation, and not just take the (purported) expert’s word for it. (So, I agree that fleshing out detailed examples is important for advancing our collective state of knowledge.) But the (purported) expert’s own confidence should track correctness, not how easy it is to convince people using words.

Zack_M_Davis 27 Sep 2021 15:27 UTC
52 points
on: Petrov Day 2021: Mutually Assured Destruction?
I deliberately refrained from mentioning this in public yesterday out of respect for the spirit of the game, but I was disappointed that the SHA-256 hashes of the launch codes were publicly visible in the source code (such that someone could try to crack them offline) rather than being stored in the database (such that all guesses have to go online through the server).

Okay, yes, I understand that this wasn’t a “serious” vulnerability because the launch codes were sufficiently high-entropy (actually random, not chosen by a human) that no one is going to be able to crack the hash. (I didn’t make the list this year, but my launch code from last year was 16 alphanumeric characters, which is $62^{16} \approx 95.2$ bits of entropy, which I think is $62^{16} / 10^{11} / (86400 \cdot 365.25) / 2$ being an expectation of 7 billion years at 100 billion hashes per second? Oh, except knock off a couple orders of magnitude, because there were a hundred codes.)

Still, in principle, as a matter of security mindset, putting the hashes in the database lets you depend on one fewer assumption (about how much computing power the adversary has). It seems like the sort of thing to do as part of the spirit of the game, given that this kind of thing is how the world is actually going to end. (Someone works out that this-and-such AI design will perform safely as long as such-and-these assumptions are true, and then one of the assumptions turns out not to be true.)

Zack_M_Davis 5 Oct 2023 23:33 UTC
47 points
31
in reply to: Rob Bensinger’s comment on: Evaluating the historical value misspecification argument
To this, the deep-learning-has-alignment-implications proponent replies: “But simple small-scale tasks don’t require maximizing a coherent preference ordering over world-states. We can already hook up an LLM to a robot and have it obey natural-language commands in a reasonable way.”

To which you might reply, “Fine, cute trick, but that doesn’t help with the real alignment problem, which is that eventually someone will invent a powerful optimizer with a coherent preference ordering over world-states, which will kill us.”

To which the other might reply, “Okay, I agree that we don’t know how to align an arbitrarily powerful optimizer with a coherent preference ordering over world-states, but if your theory predicts that we can’t aim AI systems at low-impact tasks via training, you have to be getting something wrong, because people are absolutely doing that right now, by treating it as a mundane engineering problem in the current paradigm.”

To which you might reply, “We predict that the mundane engineering approach will break down once the systems are powerful enough to come up with plans that humans can’t supervise”?

Zack_M_Davis 20 May 2022 22:35 UTC
47 points
on: Beware boasting about non-existent forecasting track records
The comments about Metaculus (“jumped ahead of 6 years earlier”) make more sense if you interpret them as being about Yudkowsky already having “priced in” a deep-learning-Actually-Works update in response to AlphaGo in 2016, in contrast to Metaculus forecasters needing to see DALLE 2/PaLM/Gato in 2022 in order to make “the same” update.

(That said, I agree that Yudkowsky’s sneering in the absence of a specific track record is infuriating; I strong-upvoted this post.)

Zack_M_Davis 9 Jan 2022 17:04 UTC
46 points
on: Use Normal Predictions

We rationalists are very good at making predictions, and the best of us, such as Scott Alexander

This weird self-congratulatory tribalism is a completely unnecessary distraction from an otherwise informative post. Are “we” unusually good at making predictions, compared to similarly informed and motivated people who haven’t pledged fealty to our social group? How do you know?

Scott Alexander is a justly popular writer, and I’ve certainly benefitted from reading many of his posts, but it seems cultish and bizarre to put him on a pedestal as “the best” of “us” like this as the first sentence of a post that has nothing to do with him.