LessWrong team member / moderator. I’ve been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I’ve been interested in improving my own epistemic standards and helping others to do so as well. as
Raemon
Karma: 68,090
Mmm. Yeah I am maybe sold the incentives don’t hold up here in the current formulation.
Somehow seems good to have human reviewers, while handling the fact that the realistically available reviewers are all busy people and need to get paid.
I guess you maybe know more about this area (you run an Alignment journal, is that right?). Do you have any other random bits of wisdom of how to think about human review that seems worth sharing?
I agree that you’d want to resolve good confusion at some point and sometimes bad confusion is a necessary byproduct of good exploration.
Part of my point is that good confusion is also a necessary byproduct of good exploration, for structurally similar reasons.
Confusion about your project and what to do next can (should?) also be something that is sometimes deep and longlasting because you are trying to solve a hard problem.
Mmnod.
Obviously use whatever internal ontology works for you, but, I would think in both cases you’re supposed to notice you’re in the state and try to resolve it. And while it’s good to actively steer into new situations where you’re confused about the world in new ways… I think in some sense (maybe not how you meant), but, it’s often good to steer into new projects on the edge of your ability where you will end up confused at times about what’s going on or what to do next.
I have felt both good confusion and bad confusion. Good confusion reflects real uncertainty in the world; bad confusion is where I don’t know what’s going on in the project.
I agree these are importantly different (and maybe requires different mental moves to deal with), but what makes one “good” vs “bad?”.
I think there are things that are bad to do re: speculating about capabilities and bottlenecks.
I think in this post I was pretty careful to only say things that are pretty obvious (especially by this point).
I mean first of all, CEV literally does whatever people turn out to think is a good idea after reflection, of which “preserve rights, or some functional equivalent” is a pretty central example of a sort of thing they might think was a good idea. If the majority of people care about rights, or a minority cares a lot and no one opposes it, it’d be part of the CEV deliberation process. If not, it won’t.
I can’t recall if the CEV document mentions rights, but I was responding more to this bit:
which allows everyone to provide meaningful input so that it allows for coalitions to veto directions they disapprove of far more easily than pushing towards whatever one person or coalition wants.
Which the 2004 CEV paper does specifically address:
Less influence is required to rule out narrow slices of our possible futures, or prevent things from happening; more influence is required to direct humanity into specific narrow slices of future, or make specific things happen. The narrower the slice that our CEV wants to avoid, the less consensus required, the more distant the future selves that might produce the warning. The narrower the slice of the future that our CEV wants to actively steer humanity into, the more consensus required.
Distance, muddle, and spread will more rapidly attenuate positive influence than negative influence; the initial dynamic for CEV should be conservative about saying “yes,” and listen carefully for “no.”
A different section which doesn’t exactly address a thing you said but feels relevant:
Q9. How does the dynamic force individual volitions to cohere? (Frequently Asked)
The dynamic doesn’t force anything. The engineering goal is to ask what humankind “wants,” or rather what we would decide if we knew more, thought faster, were more the people we wished we were, had grown up farther together, etc.
“There is nothing which humanity can be said to ‘want’ in this sense” is a possible answer to this question.
[...]
I would not object to an initial dynamic that contained a meta-rule along the lines of: “Extrapolate far enough that our medium-distance wishes cohere with each other and don’t interfere with long-distance vetoes. If that doesn’t work, try this different order of evaluation. If that doesn’t work then fail, because it looks like humankind doesn’t want anything.”
Definitely do not make any assumptions about current LLMs if you aren’t paying for the good stuff.
I’ve now posted them here:
Curious for more details on what you found interesting about that if you have them :)
Game Design Creativity evals
Strongly disagree; CEV (incorrectly?) asserts a coherent version of an individual’s desires that are close enough to fulfill their goals—and that such a volition is identifiable.
CEV is doesn’t assert this. CEV checks to what degree this is true, and returns “didn’t find any coherent values” if it doesn’t.
CEV is indeed a bet that such a thing will turn out to exist, to an agree that is useful to enough to try to help people in a way they want to be helped that is coherent.
it’s almost the opposite of a liberal democratic system that preserves rights even when most people dislike it, and which allows everyone to provide meaningful input so that it allows for coalitions to veto directions they disapprove of far more easily than pushing towards whatever one person or coalition wants.
I can’t tell whether the second half of this is about CEV, or “a liberal democratic system that preserves rights even when most people dislike it”.
To be clear I agree this wouldn’t be worth it even slightly with past models, and I’m bringing it up now because current models are like, on the edge where it starts to become an interesting question.
Relatedly, I think it would be more efficient for someone to do their own research than to “review” randomly generated papers.
I do agree there is something to this though, even when I imagine the models being pretty good.
Will chew on it.
One piece I was imagining that may or may not be relevant is: All posts have a one sentence summary and a 2-tweet-length summary, and the AIs (or humans) are meant to optimize those for being most of what a reviewer needs to figure out at a glance whether a thing is actually interesting.
I’m bringing this up because it feels like a piece of a unfinished puzzle on “how to get to the point where AIs actually help with #1 ‘draw reader attention to hypothesis’”.
Also it is kind of funny if “Nothing Went Foom” gets swept up into the genre “AI Doompop.”
Run It Again gave me chills and a visceral feeling of “this is the worst AIs will ever be at pulling at your heartstrings as you move to shut it down.”
I think “yeah don’t build it” is a pretty coherent answer.
But, why on Earth would you (or anyone, whatever their worldview, even ‘would personally prefer to hijack the universe’ people) prefer they do that than CEV, than “rewrite the rest of our minds.”
I don’t currently believe you actually believe this, as opposed to doing some performative trolling or something
Obviously the “I want to personally become god emperor” people are going to try to do that. But, why would any of them prefer someone else do that, than “do the things that everyone robustly agrees would be good to do?” (either the variant where you throw out the hijackers, or the variant where you don’t, I think both of those are clearly better from the perspective of most non-cosmopolitans who didn’t get to control it themselves))
And, to be clear, the CEV plan is not “exclude people who are not nice liberal pluralists”.
It’s “exclude people who would force everyone else to do what they want.” You can be an Islamic fundamentalist or whatever and still say “okay, I will submit my vote to the overall process like everyone else, because that is obviously better than going to war and gambling the future on whether the Christian or Techlibertarian fundamentalists win instead.”
(I have questions about whether there’s really a stable coherent world that is any good that doesn’t eventually get overwhelming superintelligence that risks being frighteningly alien. I seem where the ‘uplift humans in a meaningful way’ people are coming from and maybe that will work, but no one has gamed it out in as much detail as CEV that I’ve seen, and explicitly compared it on the merits)
What do you think should happen instead, if someone builds overwhelmingly superintelligent AI that is alignable to someone(s)?
Yeah I think the system will want voting of some kind, whether or not you do that particular thing with it.
I believe there is some existing theory work that looks something like “people start with 0 voting power, eigenkarma only flows from trusted sources, and it never does you any good to create multiple accounts.” (This is not currently true on LW, where we rely on the walled garden and moderator judgement to avoid the system breaking. For this hypothetical forum I think you’d want to actually build it robustly from the ground up)
I’m not sure if Raemon is thinking of a world where there are plenty of AIs hanging out on the internet with money, or instead a world where plenty of humans are running AIs that they give access to an account with $100.
Honestly I don’t know and seems to be prepared for either, but see my reply here for detail:
https://www.lesswrong.com/posts/KhmMXnR7s5HwXRLqB/an-alignment-forum-for-ais-or-verification-in-the-age-of?commentId=piFietuvW6bivRGKk
Nod. I’d imagine the forum mostly being for AIs that are run at the behest of researchers that are also trying to apply taste. It can also be for whatever rogue AIs, or AIs run by casual dilittente humans, who want to get their work more noticed than it’d be elsewhere.
The customers are—
people who are already spending thousands or majillions of dollars running AI swarms trying to do alignment research, for whom the additional $N00 for verification or $N0,000 for replication is not actually that big a part of the total budget and is just a reasonable operating expense for well funded researchers.
- independent researchers spending hundreds of dollars (like at least a $200/month Claude Max subscription) doing whatever you can easily do with that budget, for whom $20 is a larger but not that big a part of the total expenses.
- rando people/AIs for whom it’s not really practical or advisable to spend more than $1, who will probably mostly be submitting slop, interacting with other people in the underbelly submitting slop.
I’m confused by what you mean here. The original paper is pretty explicit about:
You say something about this like:
But, I’m not talking about them giving advice. The cat-bell as I understand it is “you are capable of looking at a brain snapshot of a current person, and, check directly “what does this person want?”, not asking “what would they want later?”. The process includes both extrapolating, and checking the extrapolation against the current person’s values. The legitimacy comes from the latter.
(the reason I consider “how to ever get advice from much-smarter-beings that isn’t manipulation” the hard part is like, the current person interpretation needs to include some amount of having them take in arguments or something
But I am separately confused about what sort of legitimacy you are even asking for here. CEV isn’t about telling people what values to have. It’s about deciding what to do in the world. Like, the thing I think we are supposed to be comparing CEV to is more like “American Democracy”, or other alternatives for translating “the will of the people” into “how the world is arranged for those people”, which is not at all about what those people end up valuing.
Is this distinction here or followup quote relevant to our failure-to-communicate here?