Communications lead at MIRI. Unless otherwise indicated, my posts and comments here reflect my own views, and not necessarily my employer’s.
Rob Bensinger
Karma: 13,828
We have now received the first partial run that meets our quality bar. The run was submitted by LessWrong user Vanilla_cabs. Vanilla’s team is still expanding the run (and will probably fix some typos, etc. later), but I’m providing a copy of it here with Vanilla’s permission, to give others an example of the kind of thing we’re looking for:
https://docs.google.com/document/d/1Wsh8L—jtJ6y9ZB35mEbzVZ8lJN6UDd6oiF0_Bta8vM/edit
Vanilla’s run is currently 266 steps long. Per the Visible Thoughts Project FAQ, we’re willing to pay authors $20 / step for partial runs that meet our quality bar (up to at least the first 5,000 total steps we’re sent), so the partial run here will receive $5320 from the prize pool (though the final version will presumably be much longer and receive more; we expect a completed run to be about 1000 steps).
Vanilla_cabs is open to doing paid consultation for anyone who’s working on this project. So if you want feedback from someone who understands our quality bar and can demonstrably pass it, contact Vanilla_cabs via their LessWrong profile.
In case you missed it: we now have an FAQ for this project, last updated Jan. 7.
(Daniel Dennett’s book Darwin’s Dangerous Idea does a good job I think of imparting intuitions about the ‘Platonic inevitability’ of it.)
Possibly when Richard says “evolutionary theory” he means stuff like ‘all life on Earth has descended with modification from a common pool of ancestors’, not just ‘selection is a thing’? It’s also an empirical claim that any of the differences between real-world organisms in the same breeding population are heritable.
how do you get some substance into every human’s body within the same 1 second period? Aren’t a bunch of people e.g. in the middle of some national park, away from convenient air vents? Is the substance somehow everywhere in the atmosphere all at once?
I think the intended visualization is simply that you create a very small self-replicating machine, and have it replicate enough times in the atmosphere that every human-sized organism on the planet will on average contain many copies of it.
One of my co-workers at MIRI comments:
(further conjunctive detail for visualizer-plausibility: most of your replication time is in all the doublings before the last doubling, and in particular you can make a shitload in a pretty small space before launching it into the jet stream to disperse. the jet stream can be used to disperse stuff throughout the atmosphere (and it can use solar radiation, at least, to keep reproducing). it could in principle be powered and do minor amounts of steering.
example things the [AGI] who has no better plan than this paltry human-conceivable plan has to think about are “how does the time-cost of making sure [I hit the people] at the south pole base and [on] all the cruise liners and in all the nuclear submarines, trade off against the risk-cost of leaving that fragment of humanity alive”, etc.)
Regarding the idea of diamondoid nanotechnology, Drexler’s Nanosystems and http://www.molecularassembler.com/Nanofactory/index.htm talk about the general concept.
When I look at factory-farmed animals, I feel awful for them. So coming into this, I have some expectation that my eventual understanding of consciousness, animal cognition, and morality (C/A/M) will add up to normalcy (i.e. not net positive for many animals).
But:
‘It all adds up to normality’ doesn’t mean ‘you should assume your initial intuitions and snap judgments are correct even in cases where there’s no evolutionary or physical reason for the intuition/judgment to be correct’. It means ‘reductive explanations generally have to recapture the phenomenon somehow’. Here, the phenomenon is a feeling of your brain, and ‘that feeling is just anthropomorphism’ recaptures the phenomenon perfectly, regardless of whether animals are conscious, what their inner life is like (if they’re conscious), etc.
I agree with the claim ‘my gut reaction is that factory-farmed pigs suffer a lot’. I disagree with the claim ‘my gut reaction is that factory-farmed pigs would be better off not existing’. I think that’s a super different claim, and builds in a lot more theory and deliberative reasoning (though it may feel obvious once it’s been cached long enough).
I do think that that gut reaction is important information
I just disagree. I think it’s not important at all, except insofar as it helps us notice the hypothesis that life might be terrible, net-negative, etc. for chickens in factory farms.
E.g., a lot of people seem to think that chickens are obviously conscious, but that ants aren’t obviously conscious (or even that they’re obviously not conscious). This seems like an obviously silly position to me, unless the person has a very detailed, well-supported, predictive model of consciousness that makes that prediction. In this case, I think that going through the imaginative exercise of anthropomorphizing ants could be quite epistemically useful, to make it more salient that this really is a live possibility.
But no, I don’t think the imaginative exercise actually gives us Bayesian evidence about what’s going on inside ants’ brains — it’s purely ‘helping correct for a bias that made us bizarrely neglect a hypothesis a superintelligence would never neglect’; the way the exercise plays out in one’s head doesn’t covary with ant consciousness across possible worlds. And exactly the same is true for chickens.
Note that there might be other crucial factors in assessing whether ‘more factory farming’ or ‘less factory farming’ is good on net — e.g., the effect on wild animals, including indirect effects like ‘factory farming changes the global climate, which changes various ecosystems around the world, which increases/decreases the population of various species (or changes what their lives are like)’.
It then matters a lot how likely various wild animal species are to be moral patients, whether their lives tend to be ‘worse than death’ vs. ‘better than death’, etc.
And regarding:
The number would be much higher than 60% on strictly utilitarian grounds, but humans aren’t strict utilitarians and it makes sense for people working hard on improving animal lives to develop strong feelings about their own personal relationship to factory farming, or to want to self-signal their commitment in some fashion.
I do think that most of EA’s distinctive moral views are best understood as ‘moves in the direction of utilitarianism’ relative to the typical layperson’s moral intuitions. This is interesting because utilitarianism seems false as a general theory of human value (e.g., I don’t reflectively endorse being perfectly morally impartial between my family and a stranger). But utilitarianism seems to get one important core thing right, which is ‘when the stakes are sufficiently high and there aren’t complicating factors, you should definitely be impartial, consequentialist, scope-sensitive, etc. in your high-impact decisions’; the weird features of EA morality seem to mostly be about emulating impartial benevolent maximization in this specific way, without endorsing utilitarianism as a whole.
Like, an interest in human challenge trials is a very recognizably ‘EA-moral-orientation’ thing to do, even though it’s not a thing EAs have traditionally cared about — and that’s because it’s thinking seriously, quantitatively, and consistently about costs and benefits, it’s consequentialist, it’s impartially trying to improve welfare, etc.
There’s a general, very simple and unified thread running through all of these moral divergences AFAICT, and it’s something like ‘when choices are simultaneously low-effort enough and high-impact enough, and don’t involve severe obvious violations of ordinary interpersonal ethics like “don’t murder”, utilitarianism gets the right answer’. And I think this is because ‘impartially maximize welfare’ is itself a simple idea, and an incredibly crucial part of human morality.
I’d guess the most controversial part of this post will be the claim ‘it’s not incredibly obvious that factory-farmed animals (if conscious) have lives that are worse than nonexistence’?
But I don’t see why. It’s hard to be confident of any view on this, when we understand so little about consciousness, animal cognition, or morality. Combining three different mysteries doesn’t tend to create an environment for extreme confidence — rather, you end up even more uncertain in the combination than in each individual component.
And there are obvious (speciesist) reasons people would tend to put too much confidence in ‘factory-farmed animals have net-negative lives’.
E.g., when we imagine the Holocaust, we imagine relatively rich and diverse experiences, rather than reducing concentration camp victims to a very simple thing like ‘pain in the void’.
I would guess that humans’ nightmarish experience in concentration camps was usually better than nonexistence; and even if you suspect this is false, it seems easy to imagine how it could be true, because there’s a lot more to human experience than ‘pain, and beyond that pain, darkness’. It feels like a very open question in the human case.
But just because chickens lack some of the specific faculties humans have, doesn’t mean that (if conscious) chicken minds are ‘simple’, or simple in the particular ways people tend to assume. In particular, it’s far from obvious (and depends on contingent theories about consciousness and cognition) that you need human-style language or abstraction in order to have ‘rich’ experience that just has a lot of morally important stuff going on. A blank map doesn’t correspond to a blank territory; it corresponds to a thing we know very little about.
(For similar reasons, I think EAs in general worry far too little about whether chickens and other animals are utility monsters — this seems like a very live hypothesis to me, whether factory-farmed chickens have net-positive lives or net-negative ones.)
Animal welfare EA and personal dietary options
I haven’t done anything like a careful analysis, but at a guess, this shift has some promise for unifying the classical split between epistemic and instrumental rationality. Rationality becomes the art of seeking interaction with reality such that your anticipations keep synching up more and more exactly over time.
“Unifying epistemic and instrumental reality” doesn’t seem desirable to me — winning and world-mapping are different things. We have to choose between them sometimes, which is messy, but such is the nature of caring about more than one thing in life.
World-mapping is also a different thing from prediction-making, though they’re obviously related in that making your brain resemble the world can make your brain better at predicting future states of the world — just fast forward your ‘map’ and see what it says.
The two can come apart, e.g., if your map is wrong but coincidentally gives you the right answer in some particular case — like a clock that’s broken and always says it’s 10am, but you happen to check it at 10am. Then you’re making an accurate prediction on the basis of something other than having an accurate map underlying that prediction. But this isn’t the sort of thing to shoot for, or try to engineer; merely accurate predictiveness is a diminished version of world-mapping.
All of this is stuff that (in some sense) we know by experience, sure. But the most fundamental and general theory we use to make sense of truth/accuracy/reasoning needn’t be the earliest theory we can epistemically justify, or the most defensible one in the face of Cartesian doubts.
Earliness, foundationalness, and immunity-to-unrealistically-extreme-hypothetical-skepticism are all different things, and in practice the best way to end up with accurate and useful foundations (in my experience) is to ‘build them as you go’ and refine them based on all sorts of contingent and empirical beliefs we acquire, rather than to impose artificial earliness or un-contingent-ness constraints.
I don’t use microCOVID much. Two things I’d like from the site:
A simple, reasonable, user-friendly tool for non-rationalists I know who are more worried about COVID than me (e.g., family).
A tool I can use if a future strain arises that’s a lot more scary. Something fast and early, that updates regularly as new information comes in.
The latter goal seems more useful in general, and my sense is that microCOVID isn’t currently set up to do that kind of thing—the site currently says “Not yet updated for the Omicron variant”, over a month in.
For the latter goal, updating fast matters more than meticulously citing sources and documenting all your reasoning. I see less need for a ‘GiveWell of microCOVID’ (that carefully defends every claim), and more value in a sort of Bayesian approach where you take the individuals with the best forecasting track record on COVID-related things, ask for their take on all the uncertain parameters, and then let people pick their favorite forecaster (or favorite aggregation method) from a dropdown menu.
Reply by Holden Karnofsky: https://www.lesswrong.com/s/n945eovrA3oDueqtq/p/nNqXfnjiezYukiMJi
Buddhism is a huge part of Joshin’s life (which seems fine to note), but if there’s an implied argument ‘Buddhism is causally responsible for this style of discourse’, ‘all Buddhists tend to be like this’, etc., you’ll have to spell that out more.
Sounds right, yeah!
Firstly, I (partially?) agree that the current DL paradigm isn’t strongly alignable (in a robust, high certainty paradigm), we may or may not agree to what extent it is approximately/weakly alignable.
I don’t know what “strongly alignable”, “robust, high certainty paradigm”, or “approximately/weakly alignable” mean here. As I said in another comment:
There are two problems here:
Problem #1: Align limited task AGI to do some minimal act that ensures no one else can destroy the world with AGI.
Problem #2: Solve the full problem of using AGI to help us achieve an awesome future.
Problem #1 is the one I was talking about in the OP, and I think of it as the problem we need to solve on a deadline. Problem #2 is also indispensable (and a lot more philosophically fraught), but it’s something humanity can solve at its leisure once we’ve solved #1 and therefore aren’t at immediate risk of destroying ourselves.
If you have enough time to work on the problem, I think basically any practical goal can be achieved in CS, including robustly aligning deep nets. The question in my mind is not ‘what’s possible in principle, given arbitrarily large amounts of time?‘, but rather ‘what can we do in practice to actually end the acute risk period / ensure we don’t blow ourselves up in the immediate future?’.
(Where I’m imagining that you may have some number of years pre-AGI to steer toward relatively alignable approaches to AGI; and that once you get AGI, you have at most a few years to achieve some pivotal act that prevents AGI tech somewhere in the world from paperclipping the world.)
The weakly alignable baseline should be “marginally better than humans”.
I don’t understand this part. If we had AGI that were merely as aligned as a human, I think that would immediately eliminate nearly all of the world’s existential risk. (Similarly, I think fast-running high-fidelity human emulations are one of the more plausible techs humanity could use to save the world, since you could then do a lot of scarily impressive intellectual work quickly (including work on the alignment problem) without putting massive work into cognitive transparency, oversight, etc.)
I’m taking for granted that AGI won’t be anywhere near as aligned as a human until long after either the world has been destroyed, or a pivotal act has occurred. So I’m thinking in terms of ‘what’s the least difficult-to-align act humanity could attempt with an AGI?’.
Maybe you mean something different by “marginally better than humans”?
As DL methods are already a success story in partial brain reverse engineering (explicitly in deepmind’s case), there’s hope for reverse engineering the circuits underlying empathy/love/altruism/etc in humans—ie the approximate alignment solution that evolution found.
I think this is a purely Problem #2 sort of research direction (‘we have subjective centuries to really nail down the full alignment problem’), not a Problem #1 research direction (‘we have a few months to a few years to do this one very concrete AI-developing-a-new-physical-technology task really well’).
Logical induction, Löbian cooperation, reflection in HOL, and functional decision theory are all results where researchers have expressed surprise to MIRI that the results were achievable even in principle.
I think a common culprit is people misunderstanding Gödel’s theorems as blocking more things than they actually do. There’s also field-specific folklore — e.g., a lot of traditional academic decision theorists seem to have somehow acquired the belief that you can’t assign probabilities to your own actions, on pain of paradox.
I… think that makes more sense? Though Eliezer was saying the field’s progress overall was insufficient, not saying ‘decision theory good, ML bad’. He singled out eg Paul Christiano and Chris Olah as two of the field’s best researchers.
In any case, thanks for explaining!
I’d argue instead that MIRI bet heavily against connectivism/DL, and lost on that bet just as heavily.
I think this is straightforwardly true in two different ways:
Prior to the deep learning revolution, Eliezer didn’t predict that ANNs would be a big deal — he expected other, neither-GOFAI-nor-connectionist approaches to AI to be the ones that hit milestones like ‘solve Go’.
MIRI thinks the current DL paradigm isn’t alignable, so we made a bet on trying to come up with more alignable AI approaches (which we thought probably wouldn’t succeed, but considered high-enough-EV to be worth the attempt).
I don’t think this has anything to do with the OP, but I’m happy to talk about it in its own right. The most relevant thing would be if we lost a bet like ‘we predict deep learning will be too opaque to align’, but we still are just as pessimistic about humanity’s ability to align deep nets are ever, so if you think we’ve hugely underestimated the tractability of aligning deep nets, I’d need to hear more about why. What’s the path to achieving astronomically good outcomes, on the assumption that the first AGI systems are produced by 2021-style ML methods?
There are two problems here:
Problem #1: Align limited task AGI to do some minimal act that ensures no one else can destroy the world with AGI.
Problem #2: Solve the full problem of using AGI to help us achieve an awesome future.
Problem #1 is the one I was talking about in the OP, and I think of it as the problem we need to solve on a deadline. Problem #2 is also indispensable (and a lot more philosophically fraught), but it’s something humanity can solve at its leisure once we’ve solved #1 and therefore aren’t at immediate risk of destroying ourselves.
The rhetorical approach of the comment is also weird to me. ‘So you’ve never heard of CIRL?’ surely isn’t a hypothesis you’d give more weight to than ‘You think CIRL wasn’t a large advance’, ‘You think CIRL is MIRI-ish’, ‘You disagree with me about the size and importance of the alignment problem such that you think it should be a major civilizational effort’, ‘You think CIRL is cool but think we aren’t yet hitting diminishing returns on CIRL-sized insights and are therefore liable to come up with a lot more of them in the future’. etc. So I assume the question is rhetorical; but then it’s not clear to me what you believe about CIRL or what point you want to make with it.
(Ditto value learning, IRL, etc.)
I read Eliezer’s response as basically “Yes, in the following sense: I would certainly have learned very new and very exciting facts about intelligence...”
I prefer Eliezer’s response over just saying “yes”, because there’s ambiguity in what it means to be a “crux” here, and because “agentic” in Richard’s question is an unclear term.
I don’t know what you mean by “intelligence” or “an urge for IRL agenticness” here, but I think the basic argument for ‘sufficiently smart and general AI will behave as though it is consistently pursuing goals in the physical world’ is that sufficiently smart and general AI will (i) model the physical world, (ii) model chains of possible outcomes in the physical world, and (iii) be able to search for policies that make complex outcomes much more or less likely. If that’s not sufficient for “IRL agenticness”, then I’m not sure what would be sufficient or why it matters (for thinking about the core things that make AGI dangerous, or make it useful).
Talking about pivotal acts then clarifies what threshold of “sufficiently smart” actually matters for practical purposes. If there’s some threshold where AI becomes smart and general enough to be “in-real-life-agentic”, but this threshold is high above the level needed for pivotal acts, then we mostly don’t have to worry about “in-real-life agenticness”.
Here’s an explanation: https://arbital.com/p/pivotal/
What do you find confusing about it? Eliezer is saying that he’s not making a claim about what’s possible in principle, just about what’s likely to be reached by the first AGI developers. He then answers the question here (again, seems fine to me to supply a “Yes, in the following sense:”):
Expressing a thought in your own words can often be clearer than just saying “Yes” or “No”; e.g., it will make it more obvious whether you misunderstood the intended question.