I’m a researcher at ACS working on understanding agency and optimisation, especially in the context of how ais work and how society is going to work once the ais are everywhere.
Raymond Douglas
Karma: 3,073
Pacing the Frontier: A Framework & Research Agenda
As someone who generally appreciates your takes, irrespective of how much you continue to post I’d be curious if there’s anything you think third parties could be doing to help with this dynamic. Weighing in on demon threads and attempting to sense make? Spending more attention on allocating votes in such threads?
Separately I wonder if there are UX fixes for this. Doubtless habryka et al have thought about this a bunch, but this tipping you into disengaging seems like a reason to slightly more seriously consider, idk, some win-win maneuver that lets certain comment threads decouple a bit from posts and go be their own thing, so that the author has less need and right to moderate?
Also, one out-of-the-box thing to at least consider is making an anonymous account.
And finally, n=1 but I do not think of you as having a reputation for hobbyhorsing and I do think of you as sticking up for something that I think is underrepresented and often relevant.
Big fan of this! I recently vibecoded a racing game where everyone’s phones are tilt-sensitive steering wheels. Also it’s possible to create QR codes for wifi, which reduces connection to just scanning two QR codes in a row.
FWIW, as a generally leftish post-agi-thinker:
I’m not totally sure what you mean by “the right wing analytical framework” or what it having issues implies
If you’re saying that there is one right-wing analytical framework, which richard has, and it uniquely has issues, such that it is undesirable to use it to reason about the future, and therefore it would not be useful for sequent to have him (or people like him) on board, I think I disagree with ~all the premises and implications
I do not think post-agi means rethinking ethics specifically towards the left; it seems like many of the major questions are more ‘right-coded’, especially around republicanism and liberty in the face of overwhelming power, or around the appropriate degree of social/cultural conservatism (in the technical sense)
I would be wary of comparing mainstream leftism to right wing conspiracy theories
(Again, as someone who is mostly a leftist) I wish people working on constitutions and character were more attentive to the pitfalls of leftism, and I expect this generalises to alignment solutions
[ed: we are discussing this more over private messages]
Sure, I don’t personally feel that strongly—mostly I was trying to intercept a potential miscommunication and registering how I initially parsed it. I think squidging those two claims into one statement is kind of hard. Maybe: “for all applications to get a comprehensive evaluation, or for every evaluator to look at every application”?
fwiw the natural way I parsed this sentence was as claiming that in SFF that each evaluator has to look at all applications, and indeed give them a comprehensive look. I now think you’re trying to say that in SFF each evaluator has to look at all applications, which is itself comprehensive, and you’re requiring neither that evaluators look at all applications, nor that any given application get a comprehensive treatment
point taken! I think I might be more sceptical than you about context provision, for the same general reason that argument mapping is hard.
I wonder about crazy-seeming conclusions. I think we maybe will need them? maybe a useful prompt is ‘how could AIFEC have helped in the pandemic?’(aforementioned reply from aforementioned private discussion)
Honestly right now we’re far enough from the frontier that I think it makes sense for people to just do whatever and see what works, in the same way that I think most junior safety researchers shouldn’t worry too much about infohazards or capability externalities. For example, field-wide, I want to push back against AIFEC being just the for-profit startup-y area, or being purely about strategies that scale with inference compute, but it seems premature to tell individuals to avoid whole areas.
Thanks for the detailed comments!
far, far more difficult, fora variety of reasons to make the general public be reasonable
Honestly, I’m not sure. Like, yeah, getting everyone to be reasonable seems hard, but aligning AI also seems really hard, and maybe ‘reasonable enough’ is easier than ‘aligned enough’? I mean, I think this hasn’t been tried partly because of severe capacity constraints; I think my optimal portfolio has a bit more in this bucket.
I agree that EA/rationality (especially EA) has a bad habit of relying on claiming morally-laden facts as true,
Right, but part of what I’m trying to get across here is that the framing counts for a lot, see eg yudkowsky on the drowning child. Like, even among the space of true things there’s a lot of degrees of freedom to exploit people’s intuitions.
I mostly do think robust democratic oversight and international coordination is far too intractable, and I think it’s more useful to focus on smaller sets of actors like the executive branch or individual members of congress.
Yeah, my take is: democratic oversight is a larger and more complex challenge, but also I really really don’t want to give up on it. I think people vary in how much they feel like “brief period of pseudo-dictatorship” is an acceptable part of the plan; definitely I’d take that over doom but I would also trade some doom-odds to avoid it.
this will require people to trust the AIs pretty largely
Yeah, I think one of the key challenges which unlocks a lot of coordination capacity is giving people compelling reasons to trust AIs at least in bounded domains, e.g. as genuinely neutral arbitrators.
unironically yes, I think some people have tried to differentially accelerate AI for healthcare with exactly this in mind
Epistemics and Coordination: It’s complicated!
On this very day, Matt Levine comments on EA and AI risk:
Now, of course, the way to make real money is by working at a frontier AI lab. This suggests that the optimal current form of effective altruism is:
Work at a frontier AI lab that is sprinting to build artificial superintelligence as quickly as possible.
Get paid like $100 million a year.
Live modestly and donate all of your earnings to a charity that is trying to stop frontier AI labs from building artificial superintelligence that might wipe out humanity.
I can see no flaws in this approach.
I think the thing being gestured at here is basically the type signature of ‘being aligned’. Deontological-aligned ~= “follows hard rules we endorse” (e.g. being corrigible), consequential-aligned ~= “is maximising a utility function we endorse”, virtue-aligned ~= “have a character we endorse”. I don’t think any normative/axiological payload is intended.
Some places other than LessWrong where I get useful/complementary AI takes:
Business/finance newsletters—mainly Byrne Hobart and Matt Levine. AI is enough of the market now that they both regularly do analyses of the economics, incentives, and politics. As well as keeping up with the news, I feel like this helps me pick up useful intuitions about potentially counterintuitive dynamics—e.g. why Amazon, despite being a near-monopoly on ecommerce, is pretty good to its customers, and what this might imply about AI. They’re also both excellent writers: wry, conversational, and highly information-dense.
Twitter—much more psychically draining and conflict-prone, hard for it to be nuanced, but it’s also useful to see what the conflicts look like and it feels more like the channels of power so it’s good to get a feel for how that works. I get the most mileage out of tracking (1) people who publish papers I like; (2) people whose twitter fights I find it productive to follow [e.g. habryka, ngo, kulveit]; (3) people who aren’t on LW much [e.g. janus, brundage, davidad]
Sentinel—pretty interesting just seeing how this differs from ‘normal news’: percentage estimates everywhere, generally focused on which things might blow up into catastrophes. I think the AI news is somewhat dominated by the other things I read these days but I both enjoy the other news and find it interesting to see which AI stuff makes the cut.
Zvi—one-stop shop, one man aggregator of all AI news, saves me having to read all of twitter or the model cards. I think this is just the best place for keeping on top of current events. Also I think Zvi is pretty good at pulling out general lessons (although not as much as Byrne or Matt), and very good at the occasional one-off special posts (e.g. his pieces on Claude welfare).
TRIP—a general politics podcast which again has been increasingly focused on AI lately as one of the hosts (Rory Stewart) has got more concerned. Most of the concentrated AI juice is in their “Leading” interview series. I found the interview with Jack Clark particularly insightful, because Rory came across as a lot more thoughtful about the risks than Jack.
Private group chats, slacks etc—Lots of important information is too sensitive to blast on the public internet where journalists will hunt it down. It is useful to find (or build) the infrastructure for more private discussion.
By contrast, places where I generally feel like I don’t get much:
Newsletters that are doing more like ‘pop news’, which often feel a bit propaganda-y and watered down to me
Places that invite contributors to write articles, where I think the average quality of the writing-on-AI is worse than I get from just following a few specific people and a few forums with upvote systems. The exceptions here are probably Asterisk and Works in Progress, although they only sometimes cover AI.
Governance substackers: not sure why, but I don’t feel like I’ve found any that really click for me. Maybe I’m just not the audience.
Dwarkesh, which I think is genuinely good, I just so rarely want to watch a two-hour video, which is what most of his recent AI stuff has been, and also I think I disagree with his takes on AI a bit. I do like a lot of his pure interview general interest stuff though. I do appreciate the occasional Zvi summary.
So much of twitter is a hot mess that’s trying to eat your brain
(Epistemic status: trying to get back up to speed on a massive email backlog after a wonderful weekend away.)
I like that framing. Somehow intuitively I feel like we’re further through the story. Maybe I feel less optimistic about how much good we’ll achieve by being reactive as opposed to proactive? I agree 2028 is going to be pretty hingey but I also expect a lot of it might be catch-up for things we’re already behind on, like, a lot of the challenge is serial work. And also I feel a bit like we’re losing leverage over time.
Structural Proxies
I somehow stumbled upon the 2018-2019 alignment review. Man, it’s really hard keeping perspective about how quickly the field is moving. The big signs of AI progress were RL for starcraft and dota, plus some GPT2 variants. The public debates were just starting around continuous vs discontinous takeoff. There are subsections on embedded agency and comprehensive AI services.
I’m not sure what I expected but it’s left me feeling a bit humbled.
I really like where this post ended up. I skimmed the start and then went back to read it properly once I realised what the actual subject was.
Rather, their mistake is in believing that death absolves them from their duty to their children.
Personally I’m inclined to agree. Unfortunately, I think one of the things that the x-risk community doesn’t grapple with enough is that others can have very different moral premises. Some people are fine incurring a higher chance of risk for the whole future if it increases the chance that their ailing grandparents can get singularity-level medical care. Some people have irreconcilable disagreements about what values should govern the future. I don’t know what you’re meant to do with that.
Sincere advocacy for AI successionism makes AI safety research and policy all the more urgent
Again, personally agree as written, but I would guess that some successionists would reply with something like “your so-called safety research is mostly an attempt to impose your own narrow values; what I do may seem incomprehensible to you but that is because it is real safety research.”
my impression is that this whole shortform has got a bit demonic and downvotes are being slung all over the place because two things are getting mushed up:
my read is that alex was remembering there being some take that some people (e.g. habryka) had, which was more nuanced than “it is hard to get AIs to learn / care about human values”, and he was basically trying to find out what that take was, by posing his recollection of it—specifically that it opens with something like “in some sense AIs don’t understand our values at all” and ends with “AIs being in control of the future would be bad”
I think some other people interpreted that as alex claiming that lots of people are foolishly going around on LW saying that claude doesn’t understand human values and that’s their crux on if alignment is hard, as opposed to getting it to care, and maybe claude is already aligned or something
these do not appear to mix well
Fwiw I get the impression that the fact that applicants don’t want to go through unnecessary hoops is itself a way that mentors pay a cost for putting up hoops.