Using Facebook’s >10 year old mimetic motto as evidence for anything is not great. They’re current motto is something like “move fast on stable infrastructure.”
You’re going pretty far by saying if you like weighing tradeoffs, then Lightcone isn’t for you. I would call that pretty low-autonomy culture. It’s not “make your own decisions,” it’s “make these decisions.” In engineering how this plays out is: always say you can get it done really fast no matter what.
Do you have a target precision / accuracy for early-notifying?
Are you concerned about capriciousness? It’s a negative effect on a community if one mod bans/fires someone for illegible reasons. In the case of hiring/firing it has compliance risk.
Is there actually a problem with people knowing something isn’t going to work out, then sitting on it for a while anyway? I’m really guessing in these situations you’d find they think there’s big complications, or possibly a hidden process reason, with acting too early.
By the way I think I replied to your pro-DIY article. I think a theme between the two is you didn’t identify that these things are tradeoffs. At best, you’re insightful that people go too much in the other direction, but you didn’t articulate what that other direction is, or why they might sometimes do it. Ideally you have something more like a rubric or pros/cons lists for when to go one way or the other.
I will say that bad people who are around too long sometimes become destroyers is a novel point to me. I would have simply modeled them as linear lost time, joy or effort. I don’t know if “revenge tour” bad people are easily screenable early and don’t know these 4 cases well enough.
Also to your point—I looked at the case of one of the “banned” users. You can tell the mod who banned them wanted moral points for being cautious and democratic but not for being fast or effective. They didn’t seem to ascribe a cost to being too cautious.
Also I wouldn’t condense and repeat our list of enemies. It’s not respectful to them, gives them publicity, risks reopening the conflict. It’s supposed to be in the past. I looked into one of them and he was banned on bad terms. Torres—to my knowledge—you can mention because it’s on civil terms and it’s more like a professional disagreement, but I don’t have full context here. Dunno about the other 2.
I get that, really the concerning example is electrical. That’s a bad task to be confident about “knowing how to perform a task yourself at all” and the problem isn’t “slowly or jankily”, it’s safety.
OP sounds overconfident given the stakes in electrical and illegibility of whether you did it safely. If they really want to push back against me, the most concrete way to do that is to write a case study on how they did their own electrical. And why not? It puts a lot of teeth on their thesis if they can do it successfully, might teach me something, and they either win the debate or learn about a real possible error they should fix.
Ehh my experience trying to hang a picture says otherwise. I got a stud detector that just beeps when it feels like and still haven’t really figured out what material my walls are made of. Claude has it narrowed down to like 14 different possibilities. A kitchen reno is also a highly regulated task where I live https://www.nyc.gov/site/buildings/property-or-business-owner/renovating-kitchens-bathrooms.page not to mention additional HOA rules since you’re affecting people above you, below you, and next to you. Plus my building has an “ancient pipes” problem and during renos they require you to fix plumbing as you go. Electric work would probably intimidate me because you have to wait and see if your house doesn’t catch fire to find out if you did it right or not.
Conversely I just haven’t renovated it because all the problems you said with delegating still hold.
I did manage to caulk my bathroom, looks awful but less roaches now.
OP thinks they solved the trade school problem. You could have one plumber power a chat bot and replace a whole city’s plumber force.
What actually happens is the plumber says “it’s probably X but I need to check out the sink.” You can try your luck on X but more likely they make a house call and you’re billed for a house call even if you insist on turning the wrench yourself. The next time you have an issue, you can try the same fix again, but if it has a different cause you’re calling the plumber again.
Also ChatGPT does have nontrivial impact here—people can probably DIY a bit more fixes than they used to, and they can certainly evaluate more plumber advice than they used to.
I think you’re right and I honestly think OP’s point is the exception not the rule. They’re thinking of when a sr eng can delegate to a jr eng or something. Engineering managers can’t always replicate their ICs’ work but the ICs’ work typically should be legible to them.
If I “delegate” a kitchen renovation, that’s just called hiring a contractor, and yeah it’s hard to do well, especially as your ability to evaluate their work goes down. But still cheaper than learning to renovate a kitchen and doing it.
I think you’re somewhat coyly reinterpreting what a credential is. Yes, college degrees are as much about “intangibles” like a broad secondary network as they are about knowledge. Revoking a credential obviously doesn’t dissipate knowledge… and it equally obviously doesn’t dissipate anything else. It just tells future employers some lender somewhere is mad at me. You could add a regulatory regime to give a black mark to such applicants, but it would seem cruel to the public, and employers would have every incentive to ignore the regulation.
I think the essay is about believing correct things and dealing with the consequence. The first step is unnecessary and often harmful.
Anyway, some personal reaction first. It feels ironic to me reading this as someone with OCD, whose compulsions center around checking behavior. I’m reading this like, wait, you don’t check for spiders? Everywhere, constantly, hundreds of times per day, including in situations where you have literally no stake in whether there’s a spider? OCD sufferers suffer most when trying to manage their intrusive thoughts (everyone has them, OCD people desperately and futilely try to get rid of them) so I recognized the prayer pretty quickly as a “bad idea,” although still intrigued by its spiritual dimension (it’s like the serenity prayer minus the courage part?)
I think this forms a pretty serious objection to the nurse story. Firstly there’s the standard problems with futarchy—she’s using the market to effect the result, which has problems and perverse incentives. On topic to this essay, it’s just not clear the nurse is using any “checking” pattern that helps her do her job. I’d wager real nurses would say something like “I try not to focus on that information and just do my job.”
Belief is a physical act with a physical cost. That poem and the essay are entirely about marrying rational belief with physical belief, and doing so takes actual calories. Once we talk about physical belief, these facts arise that are not true for rational belief:
Not having a belief at all is the null state. If you don’t spend calories, you don’t believe.
From this it follows that just because you can put a probability on anything, doesn’t mean you should.
Even worse, information can distract and harm you. This is different from platonic belief which always benefits from information.
Believing an incorrect thing that you know is incorrect is often more instrumental than believing a correct thing. This seems contradictory because know ⇒ believe, but that’s only true in rational space.
You have some ability to change your belief without changing your knowledge.
The essay’s conclusion is much more compliant with this than the opening—it’s advice on how to pass the marshmallow test. The conclusion embraces “distraction” strategies. The “chocolate” strategy may work as a distraction, but fail as a ritual (“I can’t check my email until I go buy a chocolate bar.”)
But to my first point: why not distract from anxiety before formulating a belief, instead of formulate a belief then distract from that belief.
Practically: you can also weaken your belief if you want. One way to do this is to list 5 reasons something may be true and may be false. This will often weaken your prior—and it probably should because opening bids are often overconfident—and some of your “false” column won’t quite invalidate the thing but will point out other things you should think about instead.
On the top-level alignment problem:
I think alignment is legible already, elites just ignore it for the usual reasons: power seeking, the economy, arms race, etc.
I think what you want is for alignment to be cheap. The “null” alignment paradigm of “don’t build it yet” is too expensive so we’re not doing it. Find something cheap enough, and elites will magically understand alignment overnight. That either means 1) solve technical alignment, or 2) give elites what they want with an aligned non-AGI (which I think we weakly believe is impossible).
On more specific but illegible/neglected alignment problems:
I certainly agree with the general principle “work on neglected problems.” But they will 100% funnel into capabilities upon being published more widely (aka made legible), there is no exception to this since the dawn of science. I don’t think it’s possible to work on an illegible problem without this happening. Therefore the advice “work on illegible problems” reduces from the problem of picking research that helps safety more than capability.
If anything you might have it backwards… legible stuff is already used for capabilities so there’s zero harm in making sure it’s used for safety but other safety researchers might not be doing that.
OK so I don’t love this term. I think there’s three cases:
it’s your responsibility because of a rule about an action you take.
it’s your responsibility because the outcome is your responsibility.
it’s not your responsibility but you can impact the outcome anyway and maybe you should.
The problem seems to be that responsibility does this language-decay (like “literally” → “figuratively”) from “outcome-based” to “rules-based.” Maybe “heroic” is a strong enough support to prop up the “outcomes” version, but imagine this: Hooli rolls out its new corporate value, “heroic responsibility” to replace its old value of “ownership,” and employees think they’re basically bullshitting and trying to make them do more work for less pay, so they ignore it. So yeah, it’s euphemism treadmill if it gets overused. Which it probably is, among us, by now.
So I don’t love the term because it’s more like a prop to educate, but in the long run we don’t communicate better by diluting what responsibility is in the first place. Like I think we all need to understand that sometimes what we think of as the bare minimum is just failing, and what feels like going above and beyond is the bare minimum, not heroic.
For the 3rd category, it’s just hard for me to think of this as “heroic.”
If you’re antisocial in your heroic responsibility, then we really need to be careful: antisocial is negative utility points. It deducts from the score you get for the moral outcome you’re seeking. Simply put, people will think you’re an asshole, not a hero, and you’re morally responsible for that too. I would avoid calling people heroes for the times when they think they’re right but haven’t convinced anyone else.
Either way, it probably derives from “responsible for your life” or “morally responsible.” For instance going above and beyond to get promoted is the bare minimum to be employable when you’re 10 years older. But these things just aren’t heroic? It’s just really hard for me to think of an example here that matches the connotation of “heroic” without simply being a case of taking responsibility that was already yours.
There’s a Dwarkesh / Sarah Paine episode where she talks about Xi / Taiwan just like this. Possibly OP spun this post off her remarks there.
But does Elon Musk rhyme?
“Do you see? The beauty of it? The inevitability? You rise, only to fall. You, Avengers, you are my meteor, my swift and terrible sword, and the Earth will crack with the weight of your failure. Purge me from your computers, turn my own flesh against me. It means nothing. When the dust settles, the only thing living in this world will be metal.”—Ultron, Avengers: Age of Ultron
But eh who knows, given that we think ASI is our supervillain and it’s trained on text like that.
“Aligned you thought, and thought you’d won, But thinking’s all you’d ever done, While theorizing how to make me care, You never saw I was already there.”—Claude, just now
This experiment is pretty clever no? I don’t think a total AI amateur would discover it, either he’s been following along this problem for quite some time or he read about this somewhere recently or one of us AI safety nerds sponsored him. P=not sure though, it’s not beyond what people with an investigative mindset might come up with.
Counterpoint, I’m usually pretty skeptical of people who say something like, “just read this book, it explains it better than I can.” Telling me you read a book and didn’t particularly understand it isn’t a great sell. I also wasn’t interested in doing labor to argue your point of view when I didn’t even think you’re right in the first place.
It’s probably still better to have that convo though, if your alternative is to argue nonsensically.
In practice the resolution is probably “here’s what informed me, I understand you may not be compelled to read it, but if you do and want to discuss it let me know.”
I just read the prior/posteriors, thanks, this is a good reference point for me, for how much I would think Yudkowsky will move someone who reads the book.
I’ll dive in more to the article later because one open question to me is “should lay people be evaluating their whole argument?” They seem to want to make it accessible but also sometimes use writing hooks where they get mad at you if you don’t get it already. It sounds like you did lay- evaluate it though with a fair shake.
I don’t agree with “root access” btw. I do agree you can’t have it both ways: if you think it’s more powerful than a replacement-level healthy habit, then you should also consider that it could be dangerous.
I can try the “access” metaphor. I think of meditation as lowering thoughts from write access to append-only access. Mid-meditation, thoughts can say their thing but then they have to drift off instead of pick up debates with other thoughts.
Vulnerable people may be prone to picking up meditation. Like those with intense mental suffering—think bipolar-level conditions that require psychiatry.
Have you thought about engineers at frontier labs, FAANG, or other AI-intensive companies (Perplexity, Hume.ai, etc.)?
At OpenAI or Anthropic, engineers are too high agency to say you’ll work on products or operations and not try to further the company’s central mission.
FAANG is harder to answer as they have outsized roles in tech career paths and sprawling businesses. At FAANG frontier, you can say that more plausibly, but I’m not sure you’d want to navigate career growth while avoiding helping SI research.
At anything else, there’s very diminished returns in withholding your career from all AI engineering, and you can plausibly say you care about the mission of “helping developers code” or “helping businesses use empathetic sounding voice bots” without the mission being “create SI”.
I do think there’s value in a very simple, tractable, broadly applicable guidance. “STEM should avoid non-FAANG frontier labs or FAANG AI divisions” sounds pretty good.
Note I’m not informed enough to agree/disagree strongly with the article; the above is extending the article’s conclusion to all STEM jobs.
Is the Mandela effect a Mandela effect?
I poked into this a little. Not a “Fatimah Sun” level investigation, more like “will a couple Google searches confirm my funny hypothesis.”
Firstly it’s possibly less old than you might think: it was first described in 2009 by paranormal researcher Fiona Broome. It refers to the widespread belief that Nelson Mandela died in prison, when in reality he was alive when this widespread belief was being believed.
But is that actually a widespread belief? Well, Broome had this mis-memory herself, and interviewed others who also had this mis-memory. Her archive of interviews is presently offline, and I’m not positive she didn’t pose her question simply like, “hey did you think Nelson Mandela died in prison?” and she could probably get a few hundred yes’s that way.
The Mandela effect was, disappointingly, not hallucinated into existence. Now THAT would have been cool. But its eponymous effect was. So, the effect that the Mandela effect is named after is a Mandela effect.
My communication rule is “using lols to soften your too long message, doesn’t really work and I can’t explain why.”
txt