I like this one. I think it does a lot to capture both the concept and the problem.
The concept is that we expect AI systems to be convergently goal-directed.
The problem is that people in AI research often uncertain about goal-directeness and its emergence in advanced AI systems. (My attempt to paraphrase the problem of the post, in terms of goal-directedness, at least)
Nothing comes to mind as a single term, in particular because I usually think of ‘thinking’, ‘predicting’, and ‘planning’ separately.
If you’re okay with multiple terms, ‘thinking, predicting, and planning’.
Aside: now’s a great time to potentially rewrite the LW tag header on consequentialism to match this meaning/framing. (Would probably help with aligning people on this site, at least). https://www.lesswrong.com/tag/consequentialism
Saying this again separately, if you taboo ‘consequentialism’ and take these as the definitions for a concept:
“the thing to do is choose actions that have the consequences you want.”
A ___ is something that thinks, predicts, and plans (and, if possible, acts) in such a way as to bring about particular consequences.
I think this is what “the majority of alignment researchers who probably are less on-the-ball” are in fact thinking about quite often.
We just don’t call it ‘consequentialism’.
I think my personal take is basically “yeah it seems like almost everything routes through a near-consequentialist theory” and “calling this theory ‘consequentialism’ seems fair to me”.
I spend a lot of time with people that are working on AI / AI Alignment who aren’t in the rationality community, and I don’t think this is the take for all of them. In particular I imagine from the “words have meaning, dammit” camp a lot of disagreement about ‘consequentialism’ the term, but if you taboo’d it, there’s a lot of broad agreement here.
In particular, I think this belief is super common and super strong in researchers focused on aligning AGI, or otherwise focused on long-term alignment.
I do think there’s a lot of disagreement in the more near-term alignment research field.
This is why this article felt weird to me—it’s not clear that there is a super wide mistake being made, and to the extent Raemon/John think there is, there’s also a lot of people who are uncertain (again c/f moral uncertainty) even if updating in the ‘thinking/predicting’ direction.
E.g. for this bit:
I… guess what I think Eliezer thinks is that Thoughful Researcher isn’t respecting inner optimizers enough.
My take is median Thoughtful Researcher is more uncertain about inner optimizers—instead of being certain that EY is wrong here.
And pointing at another bit:
Consequentialism is a (relatively) simple, effective process for accomplishing goals, so things that efficiently optimize for goals tend to approximate it.
I think people would disagree with this as consequentialism.
It’s important to maybe point at another term that’s charged with a nontraditional meaning in this community: rationality.
We mean something closer to skeptical empiricism that the actual term, but if you taboo it I think you end up with a lot more agreement about what we’re talking about.
Yeah this seems like one way it could resolve the differences in arguments.
My guess (though I don’t know for certain) is that more AI alignment researchers would agree with “the thing to do is choose actions that have the consequences you want” is an important part of AI research, than “the right/moral thing to do is to choose actions that have good consequences” is an important part of AI research.
I’m curious how much confusion you think is left after taboo-ing the term and communicating the clarification?
This seems pretty well done! Some thoughts on future research in this direction:
It seems like you probably could have gotten certainty about compute for at least a handful of the models studied in question (either because the model was open sourced, or you have direct access to the org training it like Eleuther) -- it would be interesting to see how the estimation methods compared to the exact answer in this case. (Probably doable with GPT-J for example)
While I agree with dropout not significantly reducing computation I think two more contemporary techniques are worth considering here: structured sparsity in weights (‘blocksparse’), and mixture-of-experts gating (‘switch transformer’). I think the latter is more important because it changes both the training compute and inference compute.
Comparing custom ML hardware (e.g. Google’s TPUs or Baidu’s Kunlun, etc) is tricky to put on these sorts of comparisons. For those I think the MLPerf Benchmarks are super useful. I’d be curious to hear the authors’ expectations of how this research changes in the face of more custom ML hardware.
In general I think it’d be good to integrate a bunch of the performance benchmarks that are publicly available (since hardware providers are usually pretty eager to show off stats that make their hardware look good) into calibrations for this method. It’s also usually pretty straightforward to compute the operations and exact utilization in these runs, since they’re heavily standardized on the exact model and dataset.
I think there’s some confusion going on with “consequentialism” here, and that’s at least a part of what’s at play with “why isn’t everyone seeing the consequentialism all the time”.
One question I asked myself reading this is “does the author distinguish ‘consequentialism’ with ‘thinking and predicting’ in this piece?” and I think it’s uncertain and leaning towards ‘no’.
So, how do other people use ‘consequentialism’?
It’s sometimes put forward as a moral tradition/ethical theory, as an alternative to both deontology and virtue ethics. I forget which philosopher decided this was the trifecta but these are often compared and contrasted to each other. In particular, the version used here seems to not fit well with this article.
Another might be that consequentialism is an ethical theory that requires prediction (whereas others do not) -- I think this is an important feature of consequentialism, but it seems like ‘the set of all ethical theories which have prediction as a first class component’ is bigger than just consequentialism. I do think that ethical theories that require prediction as a first class component are important for AI alignment, specifically intent alignment (less clear if useful for non-intent-alignment alignment research).
A different angle to this would be “do common criticisms of consequentialism apply to the concept being used here”. Consequentialism has had a ton of philosophical debate over the last century (probably more?) and according to me there’s a bunch of valid criticisms.[1]
Finally I feel like this is missing a huge step in the recent history of ethical theories, which is the introduction of Moral Uncertainty. I think Moral Uncertainty is a huge step, but the miss (in this article) is a ‘near miss’. I think a similar argument could have been made for AI researchers / Alignment researchers, using the framing of Moral Uncertainty, should be updating on net in the direction of consequentialism being useful/relevant for modeling systems (and possibly useful for designing alignment tech).
I’m not certain that the criticisms will hold, but I think that proponents of consequentialism have insufficiently engaged with the criticisms; my net current take is uncertain but leaning in the consequentialists favor. (See also: Moral Uncertainty)
… is just an arbitrary thing not to do.
I think this is the crux-y part for me. My basic intuition here is something like “it’s very hard to get contemporary prosaic LMs to not do a thing they already do (or have high likelihood of doing)” and this intuition points me in the direction of instead “conditionally training them to only do that thing in certain contexts” is easier in a way that matters.
My intuitions are based on a bunch of assumptions that I have access to and probably some that I don’t.
Like, I’m basically only thinking about large language models, which are at least pre-trained on a large swatch of a natural language distribution. I’m also thinking about using them generatively, which means sampling from their distribution—which implies getting a model to “not do something” means getting the model to not put probability on that sequence.
At this point it still is a conjecture of mine—that conditional prefixing behaviors we wish to control is easier than getting them not to do some behavior unconditionally—but I think it’s probably testable?
A thing that would be useful to me in designing an experiment to test this would be to hear more about adversarial training as a technique—as it stands I don’t know much more than what’s in that post.
“The goal is”—is this describing Redwood’s research or your research or a goal you have more broadly?
I’m curious how this is connected to “doesn’t write fiction where a human is harmed”.
I like the main point; I hadn’t considered it before with value learning. Trying to ask myself why I haven’t been worried about this sort of failure mode before, I get the following:
It seems all of the harms to humans the value-learner causes are from some direct or indirect interaction with humans, so instead I want to imagine a pre-training step that learns as much about human values from existing sources (internet, books, movies, etc) without interacting with humans.
Then as a second step, this value-learner is now allowed to interact with humans in order to continue it’s learning.
So your problem in this scenario comes down to: Is the pre-training step insufficient to cause the value-learner to avoid causing harms in the second step?
I’m not certain, but here it at least seems more reasonable that it might not. In particular, if the value-learner were sufficiently uncertain about things like harms (given the baseline access to human values from the pre-training step) it might be able to safely continue learning about human values.
Right now I think I’d be 80⁄20 that a pre-training step that learned human values without interacting with humans from existing media would be sufficient to prevent significant harms during the second stage of value learning.
(This doesn’t rule out other superintelligence risks / etc, and is just a statement about the risks incurred during value learning that you list)
I’m pretty confident that adversarial training (or any LM alignment process which does something like hard-mining negatives) won’t work for aligning language models or any model that has a chance of being a general intelligence.
This has lead to me calling these sorts of techniques ‘thought policing’ and the negative examples as ‘thoughtcrime’—I think these are unnecessarily extra, but they work.
The basic form of the argument is that any concept you want to ban as thoughtcrime, can be composed out of allowable concepts.
Take for example Redwood Research’s latest project—I’d like to ban the concept of violent harm coming to a person.
I can hard mine for examples like “a person gets cut with a knife” but in order to maintain generality I need to let things through like “use a knife for cooking” and “cutting food you’re going to eat”. Even if the original target is somehow removed from the model (I’m not confident this is efficiently doable) -- as long as the model is able to compose concepts, I expect to be able to recreate it out of concepts that the model has access to.
A key assumption here is that a language model (or any model that has a chance of being a general intelligence) has the ability to compose concepts. This doesn’t seem controversial to me, but it is critical here.
My claim is basically that for any concept you want to ban from the model as thoughtcrime, there are many ways which it can combine existing allowed concepts in order to re-compose the banned concept.
An alternative I’m more optimistic about
Instead of banning a model from specific concepts or thoughtcrime, instead I think we can build on two points:
Unconditionally, model the natural distribution (thought crime and all)
Conditional prefixing to control and limit contexts where certain concepts can be banned
The anthropomorphic way of explaining it might be “I’m not going to ban any sentence or any word—but I will set rules for what contexts certain sentences and words are inappropriate for”.
One of the nice things with working with language models is that these conditional contexts can themselves be given in terms of natural language.
I understand this is a small distinction but I think it’s significant enough that I’m pessimistic that current non-contextual thoughtcrime approaches to alignment won’t work.
That seems worth considering!
I’m roughly thinking of this sort of thing: https://forum.effectivealtruism.org/posts/fTDhRL3pLY4PNee67/improving-disaster-shelters-to-increase-the-chances-of
Remote islands are probably harder to access via aviation, but probably less geologically stable (I’d worry about things like weathering, etc). Additionally this is probably going to dramatically increase costs to build.
It’s probably worth considering “aboveground bunker in remote location” (e.g. islands, also antarctica) -- so throw it into the hat with the other considerations.
My guess is that the cheaper costs to move building supplies and construction equipment will favor “middle of nowhere in an otherwise developed country”.
I don’t have fully explored models also for how much a 100 yr bunker needs to be hidden/defensible. This seems worth thinking about.
If I ended up wanting to build one of these on some cheap land somewhere with friends, above-ground might be the way to go.
(The idea in that case would be to have folks we trust take turns staying in it for ~1month or so at a time, which honestly sounds pretty great to me right now. Spending a month just reading and thinking and disconnected while having an excuse to be away sounds rad)
I think it would be cool to see an example of such a pamphlet and if it were good to actually do this (at least get it in more places).
100 Year Bunkers
I often hear that building bio-proof bunkers would be good for bio-x-risk, but it seems like not a lot of progress is being made on these.
It’s worth mentioning a bunch of things I think probably make it hard for me to think about:
It seems that even if I design and build them, I might not be the right pick for an occupant, and thus wouldn’t directly benefit in the event of a bio-catastrophe
In the event of a bio-catastrophe, it’s probably the case that you don’t want anyone from the outside coming in, so probably you need people already living in it
Living in a bio-bunker in the middle of nowhere seems kinda boring
Assuming we can get all of those figured out, it seems worth funding someone to work on this full-time. My understanding is EA-funders have tried to do this but not found any takers yet.
So I have a proposal for a different way to iterate on the design.
Crazy Hacker Clubs
Years ago, probably at a weekly “Hack Night” at a friend’s garage, where a handful of us met to work on and discuss projects, someone came up with the idea that we could build a satellite.
NASA was hosting a cubesat competition, where the prize was a launch/deployment. We also had looked at a bunch of university cubesats, and decided that it wasn’t that difficult to build a satellite.
So hack nights and eventually other nights turned to meeting to discuss designs and implementations for the various problems we would run into (power generation and storage, attitude/orientation control, fine pointing, communications). Despite being rank amateurs, we made strong progress, building small scale prototypes of the outer structure and subsystems.
The thing that actually ended this was I decided this was so much fun that I’d quit my job and instead go work at Planet Labs—where a really cool bunch of space hippies was basically doing a slightly more advanced version of our “hacker’s cubesat”
Bio-Bunker Nights
Similar to “Hack Nights”—I think it would be fun to get together with a small set of friends and work through the design and prototype build of a 100 year bunker.
I expect to enjoy this sort of thing. Designing life support systems, and how they might fail and be fixed. Research into various forms of concrete and seismological building standards. Figuring out where would be the best place for it.
My guess is that a lot of the design and outline for construction could be had over pizza in someone’s garage.
(I’m not predicting I will do this, or committing to joining a thing if it existed, but I do think it would be a lot of fun and would be very interested in giving it a shot)
Reply to Progress, Humanism, Agency
In general I think I’m on the same page as Jason here. Instead of saying a lot of words about how I think this is important and useful, I’ll instead just poke at the parts I think could possibly become stronger.
Progress is not a single thing
There’s a lot of people talking past each other with regards to progress. There are a lot of blanket claims that progress is happening (things are getting broadly better for people on average) as well as claims to the opposite.
I think the ‘yeah huh, nu uh’ back-and-forth between “things are getting better” and “things are getting worse” is quagmired at this point.
So it’s maybe worth admitting that there are metrics that people find important that are getting worse with time. Most of these are benchmarked to relativist reference points (e.g. income inequality can get worse even if everyone has access to strictly more value) or normative (if assaults are decreasing slower than your growing circle of consideration for what counts as assault, then your perceptions could be that assault is increasing).
I’m not sure what the right approach is here, but it seems that over the last little bit, reinforcing the “things are getting better” with lots of graphs and stats etc hasn’t actually changed public opinion all that much.
(I remain surprised that the reactions to “Better Angels of our Nature” when it came out — I still regard that as better than “Enlightenment Now”, despite the latter doing more work on the philosophical concepts of progress; what matters to me is the actual change)
Humanism as the source of value / Utilitarianism as the standard of value
I think your definition of humanism is laudable but vague. It weakly answers the question of “whence value?” but stops there.
I think your alternative source of value is better described as “naturalism” instead of “romanticism” — if only because the latter seems to suggest philosophical romanticism (https://en.wikipedia.org/wiki/Romanticism) instead of the conservatism you described. (This is mostly a minor nit about naming things, not actually a criticism of the point)
So I think “Humanism as the source of value” makes sense, but it doesn’t give us a metric or measurement or point of reference to compare to in terms of value.
I think that standard of value (or the system of utilizing that) is utilitarianism.
I think consequentialism and utilitarianism are not without their issues (hopefully writing up more of them soon) — but I think they make a strong standard, in particular by forcing a consistent set of preferences between alternatives of value.
What is the measure of Agency?
I ask because I don’t know it — and also because it seems critical to rectifying some liberalism (the John Stuart Mill kind = https://en.wikipedia.org/wiki/On_Liberty) with utilitarianism/consequentialism.
It seems reasonable to consider differently the expected harm of someone who directly put themselves in harms way, vs the person who was put into harms way by the state — though naive metrics for utility (e.g. expected QALYs, etc) would be the same.
The VNM utilitarians would claim that there is some term in the utility function for agency, but (so far as I know) have not produced actual numbers and metrics for how agency trades off with e.g. mortality.
Admittedly this is mostly a critique of utilitarianism and not of your point on agency.
I think the point about agency in the face of the future is essential, and the people that will change the future will probably be almost exclusively people who think they can change the future.
What should be in the core ideas for progress?
I am biased here, but I think a philosophy of progress necessarily must include a philosophy of risk.
Technological progress creates harms and downsides (or risks of downsides) in addition to benefits and upsides.
I think a philosophy of progress should have reified concepts for measuring these against each other. I think it should also have reified concepts for measuring the meta-effects of progress on these other metrics for progress.
Secret first point of progress
I would be a little bit remiss if I didn’t include to me what was the most surprising part of learning about progress so far: almost all human progress (in the sense of the moral imperative you gave at the end) is scientific and technological progress.
To the extent that this is not the case, I haven’t found strong evidence of it yet.
To the extent that it is the case, I think we should be more clearly specifying that the moral imperative is for scientific and technical progress.
(P.S. - I have so far found the book of middling quality but full of interesting concepts by which to merge a philosophy of progress with things like utilitarianism and existential risk https://www.routledge.com/Risk-Philosophical-Perspectives/Lewens/p/book/9780415422840)
Gratitude for posting this. It caused me to read and write this comment https://www.alignmentforum.org/posts/DreKBuMvK7fdESmSJ/how-deepmind-s-generally-capable-agents-were-trained?commentId=wkyYQj8DRTLsxFQe5#comments
I engage too much w/ generalizations about AI alignment researchers.
Noticing this behavior seems useful for analyzing it and strategizing around it.
A sketch of a pattern to be on the lookout for in particular is “AI Alignment researchers make mistake X” or “AI Alignment researchers are wrong about Y”. I think in the extreme I’m pretty activated/triggered by this, and this causes me to engage with it to a greater extent than I would have otherwise.
This engagement is probably encouraging more of this to happen, so I think more of a pause and reflection would make more good things happen.
It’s worth acknowledging that this comes from a very deep sense of caring and importantness about AI alignment research that I feel. I spend a lot of my time (both work time and free time) trying to foster and grow the field. It seems reasonable I want people to have correct beliefs about this.
It’s also worth acknowledging that there will be some cases where my disagreement is wrong. I definitely don’t know all AI alignment researchers, and there will be cases where there are broad field-wide phenomena that I missed. However I think this is rare, and probably most of the people I interact with will have less experience w/ the field of AI alignment.
Another confounder is that the field is both pretty jargon-heavy and very confused about a lot of things. This can lead to a bunch of semantics confusions masking other intellectual progress. I’m definitely not in the “words have meanings, dammit” extreme, and maybe I can do a better job asking people to clarify and define things that I think are being confused.
A takeaway I have right now from reflecting on this, is that “I disagree about <sweeping generalization about AI alignment researchers>” feels like a simple and neutral statement to me, but isn’t a simple and neutral statement in a dialog.
Thinking about the good stuff about scout mindset, I think things that I could do instead that would be better:
I can do a better job conserving my disagreement. I like the model of treating this as a limited resource, and ensuring it is well-spent seems good.
I can probably get better mileage out of pointing out areas of agreement than going straight to highlighting disagreements (crocker’s rules be damned)
I am very optimistic about double-crux as a thing that should be used more widely, as well as a specific technique to deploy in these circumstances. I am confused why I don’t see more of it happening.
I think that’s all for the reflection on this for now.