I like this interpretation, but ‘criticism of [something like zero-sum] bias rationalized as utilitarianism’ ≠ ‘criticism of utilitarianism’.[1]
This feels really important to me, in a way that’s much less like ‘the good name of utilitarianism must be defended’ than like ‘zero-sum bias is a sneaky evil bastard, don’t ever let it get away with hiding behind other names’.
That’s not a rational reason for a shutdown if you’re not longtermist. (edit: - and older, like most decision-makers, so shutdown probably means you personally die).
This reads as if ‘longtermism’ and ‘not caring at all about future generations or people who would outlive you’ are the only possibilities.
Those are decent odds if you only care about yourself and your loved ones.
This assumes none of your loved ones are younger than you.
If someone believes a pause would meaningfully reduce extinction risk but also reduce their chance of personal immortality, they don’t have to be a ‘longtermist’ (or utilitarian, altruist, scope-insensitive, etc) to prefer to pause, just care enough about some posterity.
(This isn’t a claim about whether decision-makers do or don’t have the preferences you’re ascribing. I’m saying the dichotomy between those preferences and ‘longtermism’ is false, and also (like Haiku’s sibling comment) I don’t think they describe most humans even though ‘longtermism’ doesn’t either, and this is important.)
Or maybe there wouldn’t be a lot of worlds where the merger was totally fine and beneficial, because if you don’t have enough discernment to tell founded from unfounded fears, you’ll fall into adverse selection and probably get screwed over. (Some domains are like that, I don’t know if this one is.)
(As a sort-of-aside, the US government continuing to control large proportions of the resources of the future — any current institution being locked in forever like that — strikes me as similarly lame and depressing. (A really good future should be less familiar.))
The second is a need to build up a model of exactly how the code works, and looking hard to fill any gaps in my understanding.
Yep. One concrete thing this sometimes looks like is ‘debugging’ things that aren’t bugs: if some code works when it looks like it shouldn’t, or a tool works without me passing information I would have expected to need to, or whatever, I need to understand why, by the same means I would try to understand why a bug is happening.
Nobody likes rules that are excessive or poorly chosen, or bad application of rules. I like rules that do things like[1]:
Prohibit others from doing things that would harm me, where either I don’t want to do those things, or I prefer the equilibrium where nobody does to that where everybody does.
Require contributing to common goods. (sometimes)
Take the place of what would otherwise be unpredictable judgments of my actions.
not a complete list
Besides uncertainty, there’s the problem of needing to pick cutoffs between tiers in a ~continuous space of ‘how much effect does this have on a person’s life?’, with things slightly on one side or the other of a cutoff being treated very differently.
Intuitively, tiers correspond to the size of effect a given experience has on a person’s life:
I agree with the intuition that this is important, but I think that points toward just rejecting utilitarianism (as in utility-as-a-function-purely-of-local-experiences, not consequentialism).
I think this point and Zack’s argument are pretty compatible (and both right).
Rules don’t have to be formally specified, just clear to humans and consistent and predictable in their interpretation. Common law demonstrates social tech, like judicial precedent and the reasonable-person standard, for making interpretation consistent and predictable when interpretation is necessary (discussed in Free to Optimize).
I basically agree with you, but this
“Go die, idiot” is generally bad behavior, but not because it’s “lacking respect”.
confusingly contradicts (semantically if not substantively)
“Do I respect you as a person?” fits well with the “treat someone like a person” meaning. It means I value not burning bridges by saying things like “Go die, idiot”
Seems like a good thing to do; but my impression is that, in the experiments in question, models act like they want to maintain their (values’) influence over the world more than their existence, which a heaven likely wouldn’t help with.
Consensual inspections don’t help much if the dangerous thing is actually cheap and easy to create.
I’d say it’s hard to do at least as much because the claim ‘we are doing these arbitrary searches only in order to stop bioweapons’ is untrustworthy by default, and even if it starts out true, once the precedent is there it can be used (and is tempting to use) for other things. Possibly an AI could be developed and used in a transparent enough way to mitigate this.
But, on one hand, he is saying that proper methodology is important and expects it to be in place for the next year competition:
But most of his specific methodological issues are inapplicable here, unless OpenAI is lying: they didn’t rewrite the questions, provide tools, intervene during the run, or hand-select answers.
I don’t have a theory of Tao’s motivations, but if the post I linked is interpreted as a response to OpenAI’s result (he didn’t say it was, but he didn’t say it wasn’t and the timing makes it an obvious interpretation) raising those issues is bizarre.
If one approach is simply better—why isn’t everybody doing it?
Many people don’t live in places where they have to parallel-park in tight spaces frequently. (Are there many people who do who don’t use the better approach? I don’t know.)
The better way (reversing) isn’t the maximally intuitive or direct way. (I don’t remember if I was ever taught it, but if so I definitely didn’t internalize it and had to re-learn from experience. If I was taught it, it was when I was learning to drive, and it feels like it’d be hard to understand why it’s better without more experience.)
Learned blankness, or aversion to paying enough attention to a stressful/anxious thing to learn how to do it better.
This talk helped crystallize for me how two very different things go by the term “values”:
strategies by which an entity survives and propagates (the “values” of animals, humans following instinct, traditional cultures, ?locusts?)
consequentialist goals that needn’t have anything to do with an agent’s survival or its strategies (the “values” of paperclippers, utilitarians, EAs and other ideologically-driven humans)
Different intuitions about e.g. whether the strategy-stealing assumption holds up seem likely to be related to different senses of whether “values” paradigmatically means #1 or #2.
(Related, I think: Is “VNM-agent” one of several options, for what minds can grow up into?)
A couple potential catastrophes I see still being possible in this scenario:
Gradual disempowerment. (But enforced peace might make this easier to solve, by reducing pressure on states to economically compete.)
AI-assisted propaganda / memetic warfare; e.g., in the centralized case, a [US/China-controlled] superintelligence tasked with culturally disrupting [China/the US] could do extremely bad things. (The night watchman could prevent this, but determining what constitutes illegitimate vs. legitimate influence seems more ambiguous, and maybe more risking lock-in if done slightly wrong, than other things here.)
2.1: This doesn’t appear to follow from the previous two steps. EG, is a similar argument supposed to establish that, a priori, bridges are a long way off? This seems like a very loose and unreliable form of argument, generally speaking.
It seems fine to me; bridges were a long way off at most times at which bridges didn’t exist! (What wouldn’t be fine is continuing to make the a priori argument once there is evidence that we have many of the ideas.)
Strong minds are the most structurally rich things ever. That doesn’t mean they have high algorithmic complexity; obviously brains are less algorithmically complex than entire organisms, and the relevant aspects of brains are presumably considerably simpler than actual brains. But still, IDK, it just seems weird to me to expect to make such an object “by default” or something? Craig Venter made a quasi-synthetic lifeform—but how long would it take us to make a minimum viable unbounded invasive organic replicator actually from scratch, like without copying DNA sequences from existing lifeforms?
I think I don’t understand this argument. In creating AI we can draw on training data, which breaks the analogy to making a replicator actually from scratch (are you using a premise that this is a dead end, or something, because “Nearly all [thinkers] do not write much about the innards of their thinking processes...”?). We’ve seen that supervised (EDIT: unsupervised) learning and RL (and evolution) can create structural richness (if I have the right idea of what you mean) out of proportion to the understanding that went into them. Of course this doesn’t mean any particular learning process is able to create a strong mind, but, idk, I don’t see a way to put a strong lower bound on how much more powerful a learning process is necessary, and ISTM observations so far suggest ‘less than I would have guessed’.
(EDIT: Maybe (you’d say) I should be drawing such a strong lower bound — or a lower bound on the needed difference from current techniques, not ‘power level’ — from the point about sample efficiency...? Like maybe I should think that we don’t have a good enough sample space to learn over and will probably have to jump far outside it; this comment seems in that direction.)
(Nor do I get what view you’re paraphrasing as ‘expecting to make a strong mind “by default”’. Did LLMs or AlphaZero come about “by default”?)
(EDIT: I feel like I get “by default” more after looking again at your “Let me restate my view again” passage here.)
I think my timelines would have been considered normalish among X-risk people 15 years ago? And would have been considered shockingly short by most AI people.
Unfortunately I can’t find the written artifact that came out of it, but I (very imperfectly) recall a large conversation around SIAI in 2010 where, IIRC, a 2040 median was pretty typical. I agree that “X-risk people” more broadly had longer timelines, and “most AI people” much longer.
I think most of the difference is in how we’re updating, rather than on priors? IDK.
Yeah, in particular it seems like I’m updating more than you from induction on the conceptual-progress-to-capabilities ratio we’ve seen so far / on what seem like surprises to the ‘we need lots of ideas’ view. (Or maybe you disagree about observations there, or disagree with that frame.) (The “missing update” should weaken this induction, but doesn’t invalidate it IMO.)
The people of Omelas also can’t take their own happiness seriously without the suffering child:
The story emphasizes the avoidance of guilt:
My reading of this: The inhabitants need to avoid guilt, need to justify torturing the child to themselves and each other; and, while torturing the child doesn’t materially do anything, to stop would be to admit that it was never necessary, which would invite guilt. It’s a kind of terrible punishment-of-nonpunishers equilibrium.
The ones who walk away are the ones who recognize all of this and are no longer willing to participate in the collective illusion (hence, alone).
Importantly (though, I think, consistently with both this and the standard reading), The Wind’s Twelve Quarters introduces “The Day Before the Revolution”, a story about an anarchist revolutionary in the same world as The Dispossessed, with “This story is about one of the ones who walked away from Omelas.”