I think it would be an easier challenge to align 100 small ones (since solutions would quite possibly transfer across).
I think it would be a bigger victory to align the one big one.
I’m not sure from the wording of your question whether I’m supposed to assume success.
To add to what Owain said:
I think you’re pointing to a real and harmful possible dynamic
However I’m generally a bit sceptical of arguments of the form “we shouldn’t try to fix problem X because then people will get complacent”
I think that the burden of proof lies squarely with the “don’t fix problem X” side, and that usually it’s good to fix the problem and then also give attention to the secondary problem that’s come up
I note that I don’t think of politicians and CEOs to be the primary audience of our paper
Rather I think in the next several years such people will naturally start having more of their attention drawn to AI falsehoods (as these become a real-world issue), and start looking for what to do about it
I think that at that point it would be good if the people they turn to are better informed about the possible dynamics and tradeoffs. I would like these people to have read work which builds upon what’s in our paper. It’s these further researchers (across a few fields) that I regard as the primary audience for our paper.
I don’t think I’m yet at “here’s regulation that I’d just like to see”, but I think it’s really valuable to try to have discussions about what kind of regulation would be good or bad. At some point there will likely be regulation in this space, and it would be great if that was based on as deep an understanding as possible about possible regulatory levers, and their direct and indirect effects, and ultimate desirability.
I do think it’s pretty plausible that regulation about AI and truthfulness could end up being quite positive. But I don’t know enough to identify in exactly what circumstances it should apply, and I think we need a bit more groundwork on building and recognising truthful AI systems first. I guess quite a bit of our paper is trying to open the conversation on that.
I think there’s also a capability component, distinct from “understanding/modeling the world”, about self-alignment or self-control—the ability to speak or act in accordance with good judgement, even when that conflicts with short-term drives.
In my ontology I guess this is about the heuristics which are actually invoked to decide what to do given a clash between abstract understanding of what would be good and short-term drives (i.e. it’s part of meta-level judgement). But I agree that there’s something helpful about having terminology to point to that part in particular. Maybe we could say that self-alignment and self-control are strategies for acting according to one’s better judgement?
Do you consider “good decision-making” and “good judgement” to be identical? I think there’s a value alignment component to good judgement that’s not as strongly implied by good decision-making.
I agree that there’s a useful distinction to be made here. I don’t think of it as fitting into “judgement” vs “decision-making” (and would regard those as pretty much the same), but rather about how “good” is interpreted/assessed. I was mostly using good to mean something like “globally good” (i.e. with something like your value alignment component), but there’s a version of “prudentially good judgement/decision-making” which would exclude this.
I’m open to suggestions for terminology to capture this!
I think the double decrease effect kicks in with uncertainty, but not with confident expectation of a smaller network.
I’m not sure I’ve fully followed, but I’m suspicious that you seem to be getting something for nothing in your shift from a type of uncertainty that we don’t know how to handle to a type we do.
It seems to me like you must be making an implicit assumption somewhere. My guess is that this is where you used to pair with . If you’d instead chosen as the matching then you’d have uncertainty between whether should be or . My guess is that generically this gives different recommendations from your approach.
Seems to me like there are a bunch of challenges. For example you need extra structure on your space to add things or tell what’s small; and you really want to keep track of long-term impact not just at the next time-step. Particularly the long-term one seems thorny (for low-impact in general, not just for this).
Nevertheless I think this idea looks promising enough to explore further, would also like to hear David’s reasons.
For #5, OK, there’s something to this. But:
It’s somewhat plausible that stabilising pivotal acts will be available before world-destroying ones;
Actually there’s been a supposition smuggled in already with “the first AI systems capable of performing pivotal acts”. Perhaps there will at no point be a system capable of a pivotal act. I’m not quite sure whether it’s appropriate to talk about the collection of systems that exist being together capable of pivotal acts if they will not act in concert. Perhaps we’ll have a collection of systems which if aligned would produce a win, or if acting together towards an unaligned goal would produce catastrophe. It’s unclear if they each have different unaligned goals that we necessarily get catastrophe (though it’s certainly not a comfortable scenario).
I like your framing for #1.
Thanks for the write-up, this is helpful for me (Owen).
My initial takes on the five steps of the argument as presented, in approximately decreasing order of how much I am on board:
Number 3 is a logical entailment, no quarrel here
Number 5 is framed as “therefore”, but adds the assumption that this will lead to catastrophe. I think this is quite likely if the systems in question are extremely powerful, but less likely if they are of modest power.
Number 4 splits my intuitions. I begin with some intuition that selection pressure would significantly constrain the goal (towards something reasonable in many cases), but the example of Solomonoff Induction was surprising to me and makes me more unsure. I feel inclined to defer intuitions on this to others who have considered it more.
Number 2 I don’t have a strong opinion on. I can tell myself stories which point in either direction, and neither feels compelling.
Number 1 is the step I feel most sceptical about. It seems to me likely that the first AIs which can perform pivotal acts will not perform fully general consequentialist reasoning. I expect that they will perform consequentialist reasoning within certain domains (e.g. AlphaGo in some sense reasons about consequences of moves, but has no conception of consequences in the physical world). This isn’t enough to alleviate concern: some such domains might be general enough that something misbehaving in them would cause large problems. But it is enough for me to think that paying attention to scope of domains is a promising angle.
This conclusion is way too strong. To just give one way: there’s a big space of possibilities where discovering the planning fallacy in fact makes you less susceptible to the planning fallacy, but not immune.
I don’t know who the intended audience for this is, but I think it’s worth flagging that it seemed extremely jargon-heavy to me. I expect this to be off-putting to at least some people you actually want to attract (if it were one of my first interactions with CFAR I would be less inclined to engage again). In several cases you link to explanations of the jargon. This helps, but doesn’t really solve the problem that you’re asking the reader to do a large amount of work.
Some examples from the first few paragraphs:
clear and unhidden
original seeing
original making
existential risk
informational content [non-standard use]
thinker/doer
know the right passwords
double crux
outreach efforts
I found this document kind of interesting, but it felt less like what I normally understand as a mission statement, and more like “Anna’s thoughts on CFAR’s identity”. I think there’s a place for the latter, but I’d be really interested in seeing (a concise version of) the former, too.
If I had to guess right now I’d expect it to say something like:
We want to develop a community with high epistemic standards and good rationality tools, at least part of which is devoted to reducing existential risk from AI.
… but I kind of expect you to think I have the emphasis there wrong in some way.
I like your (A)-(C), particularly (A). This seems important, and something that isn’t always found by default in the world at large.
Because it’s somewhat unusual, I think it’s helpful to give strong signals that this is important to you. For example I’d feel happy about it being a core part of the CFAR identity, appearing in even short statements of organisational mission. (I also think this can help organisation insiders to take it even more seriously.)
On (i), it seems clearly a bad idea for staff to pretend they have no viewpoints. And if the organisation has viewpoints, it’s a bad idea to hide them. I think there is a case for keeping organisational identity small—not taking views on things it doesn’t need views on. Among other things, this helps to make sure that it actually delivers on (A). But I thought the start of your post (points (1)-(4)) did a good job of explaining why there are in fact substantive benefits to having an organisational view on AI, and I’m more supportive of this than before. I still think it is worth trying to keep organisational identity relatively small, and I’m still not certain whether it would be better to have separate organisations.
This was helpful to me, thanks.
I think I’d still endorse a bit more of a push towards thinking in credences (where you’re at a threshold of that being a reasonable thing to do), but I’ll consider further.
Thanks. I’ll dwell more on these. Quick thoughts from a first read:
I generally liked the “further discussion” doc.
I do think it’s important to strongly signal the aspects of cause neutrality that you do intend to pursue (as well as pursuing them). These are unusual and important.
I found the mission statement generally opaque and extremely jargony. I think I could follow what you were saying, but in some cases this required a bit of work and in some cases I felt like it was perhaps only because I’d had conversations with you. (The FAQ at the top was relatively clear, but an odd thing to lead with.)
I was bemused by the fact that there didn’t appear to be a clear mission statement highlighted anywhere on the page!
ETA: Added some more in depth comments on the relevant comment threads: here on “further thoughts”, and here and here on the mission statement.
I’m relatively a fan of their approach (although I haven’t spent an enormous amount of time thinking about it). I like starting with problems which are concrete enough to really go at but which are microcosms for things we might eventually want.
I actually kind of think of truthfulness as sitting somewhere on the spectrum between the problem Redwood are working on right now and alignment. Many of the reasons I like truthfulness as medium-term problem to work on are similar to the reasons I like Redwood’s current work.