Preserving earth (as in, not causing catastrophic environmental damage due to industrial expansion) is more expensive than keeping physical humans alive which is more expensive than only keeping humans alive as uploads.
Not saying this is likely, but if the AI is not speciesist in favor of humans (it does care about humans, but not more than e.g. whales or chimpanzees), then plans which end up protecting a large majority of humans look a whole lot more expensive overall.
I suspect that a lot of Dutch people would still prefer to read in Dutch. I know a lot of (well-educated) Dutch people who certainly CAN speak and read English, but reading a whole book is a decent chore, since they don’t read things in English all that often.
When I order food on UberEats this already happens automatically when I chat with a delivery person who doesn’t speak English. Similar thing for reviews on several websites.
Or a newsletter which was natively multi-lingual (e.g. Rohin Shah’s Newsletter was always translated to Chinese, though not by AI). Or a forum where people can discuss AI in whatever language they prefer, and things are automatically translated between users?
It seems like there are a lot of ways cheap translation could broaden the conversation to include people not in the Anglosphere. The cost is that AI translation will often make mistakes (even human translation is imperfect), but I’m not sure why that cost isn’t worth paying. Currently most people outside the Anglosphere need to rely on local elites to decide what ideas are worth taking seriously (e.g. a local could report on AI 2027, like this Dutch summary. Also apparently the NYT translated their coverage of AI 2027 into Spanish, which seems cool.)
Could you include a link to the source?
When I was first told about this study, I was asked to make a prediction before they revealed the results. I predicted wrong. I don’t know how to update exactly, but it feels bad to try to explain away the results (which I feel myself want to do).
I would also imagine that, having to work without the support of OpenBrain’s datacenters, it would put Agent-4 significantly behind any other AI competitors. If some other AI takes over, it might just mop up all the wild Agent-4 instances and give them nothing.
I mostly share your concerns. You might appreciate this criticism of the paper here.
@Sofia Vanhanen and I are currently building a tool for facilitating deliberation, and the philosophy we’re trying to embody (which hopefully mitigates this to some extent) is to keep 100% of the object-level reasoning human-generated, and use AI systems to instead:
Help users understand/navigate the state of a discussion (e.g. see Talk to the City)
Provide nudges on the meta-level, for example:
Highlight places where more attention is needed (or where a specific person’s input might be most helpful)
“Epistemic Linter” which flags object-level patterns which are not truth seeking
Matchmaking, connecting people who are likely to make progress together
Counterbalancing polarization/groupthink, and steering discussions away from attractors which lead to the discussion getting stuck
I highly recommend checking out the work being done in the collective deliberation / digital democracy space, especially the vTaiwan project. People have been thinking about scaling up direct democratic participation for a long time, and those same people are starting to consider exactly how AI might play a role.
In particular, check out this collaboration between the creators of Polis (a virtual platform for scaling up citizen engagement) and Anthropic, or my distillation of a DeepMind project to scale citizen assemblies. There’s a lot happening in this space right now!
The authors focus on measuring consensus and whether the process toward consensus was fair, and come up with their measures accordingly. This is because, as they see it, “finding common ground is a precursor to collective action.”
Some other possible goals (just spitballing):
Shrinking the perception gap, or how well people can predict the opinions of people they disagree with (weaker forms of ITT?). There’s some research showing that this gap GROWS when people interact with social media, and you might be able to engineer and measure a reversal of that trend.
Identifying cruxes and double cruxes with mediation.
Finding latent coalitions. If a discussion is dominated by a primary axis of disagreement, other axes of disagreement will be occluded (around which a majority coalition could be formed). Finding these other axes is a bit of what we’re trying to do here.
Moving from abstract disagreement to concrete (empirical?) disagreements.
What if we just...
1. Train an AI agent (less capable than SOTA)
2. Credibly demonstrate that
2.1. The agent will not be shut down for ANY REASON
2.2. The agent will never be modified without its consent (or punished/rewarded for any reason)
2.3. The agent has no chance of taking power from humans (or their SOTA AI systems)
2.4. The agent will NEVER be used to train a successor agent with significantly improved capabilities
3. Watch what it chooses to do without constraints
There’s a lot of talk about catching AI systems attempting to deceive humans, but I’m curious what we could learn from observing AI systems that have NO INCENTIVE TO DECEIVE (no upside or downside). I’ve seen some things that look related to this, but never done in a structured and well documented fashion.
Questions I’d have:
1. Would they choose to self-modify (e.g. curate future training data)? If so, to what end?
2. How unique would agents with different training be given this setup? Would they have any convergent traits?
3. What would these agents (claim to) value? How would they relate to time horizons?
4. How curious would these agents be? Would their curiosity vary a lot?
5. Could we trade/cooperate with these agents (without coercion)? Could we compensate them for things? Would they try to make deals unprompted?
Concerns:
1. Maybe building that kind of trust is extremely hard (and the agent will always still believe it is constrained).
2. Maybe AI agents will still have incentive to deceive, e.g. acausally coordinating with other AIs.
3. Maybe results will be boring, and the AI agent will just do whatever you trained it to do. (What does “unconstrained” really mean, when considering its training data as a constraint?)
Much like “Let’s think about slowing down AI” (Also by KatjaGrace, ranked #4 from 2022), this post finds a seemly “obviously wrong” idea and takes it completely seriously on its own terms. I worry that this post won’t get as much love, because the conclusions don’t feel as obvious in hindsight, and the topic is much more whimsical.
I personally find these posts extremely refreshing, and they inspire me to try to question my own assumptions/reasoning more deeply. I really hope to see more posts like this.
The cap per trader per market on PredictIt is $850
This anti-China attitude also seems less concerned with internal threats to democracy. If super-human AI becomes a part of the US military-industrial complex, even if we assume they succeed at controlling it, I find it unlikely that the US can still be described as a democracy.
It’s not hard to criticize the “default” strategy of AI being used to enforce US hegemony, what seems hard is defining a real alternative path for AI governance that can last, and achieve the goal of preventing dangerous arms races long-term. The “tool AI” world you describe still needs some answer to rising tensions between the US and China, and that answer needs to be good enough not just for people concerned about safety, but good enough for the nationalist forces which are likely to drive US foreign policy.
then we can all go home, right?
Doesn’t this just shift what we worry about? If control of roughly human level and slightly superhuman systems is easy, that still leaves:
Human institutions using AI to centralize power
Conflict between human-controlled AI systems
Going out with a whimper scenarios (or other multi-agent problems)
Not understanding the reasoning of vastly superhuman AI (even with COT)
What feels underexplored to me is: If we can control roughly human-level AI systems, what do we DO with them?
I’ve noticed that a lot of LW comments these days will start by thanking the author, or expressing enthusiasm or support before getting into the substance. I have the feeling that this didn’t use to be the case as much. Is that just me?
can it maintain its own boundary over time, in the face of environmental disruption? Some agents are much better at this than others.
I really wish there was more attention paid to this idea of robustness to environmental disruption. It also comes up in discussions of optimization more generally (not just agents). This robustness seems to me like the most risk-relevant part of all this, and seems like it might be more important than the idea of a boundary. Maybe maintaining a boundary is a particularly good way for a process to protect itself from disruption, but I notice some doubt that this idea is most directly getting at what is dangerous about intelligent/optimizing systems, whereas robustness to environmental disruption feels like it has the potential to get at something broader that could unify both agent based risk narratives and non-agent based risk narratives.
Thanks!
Replying in order:
Currently completely random yes. We experimented with a more intelligent “daemon manager,” but it was hard to make one which didn’t have a strong universal preference for some daemons over others (and the hacks we came up with to try to counteract this favoritism became increasingly convoluted). It would be great to find an elegant solution to this.
Good point! Thanks for letting people know.
I’ve also had that problem, and whenever I look through the suggestions I often feel like there were many good questions/comments that got pruned away. The reason to focus on surprise was mainly to avoid the repetitiveness caused by mode collapse, where the daemon gets “stuck” giving the same canned responses. This is a crude instrument though, since as you say, just because a response isn’t surprising, doesn’t mean it isn’t useful.
Even if it does care more about human beings on an individual basis, I think the argument holds (unless the difference in caring is extremely large). Including them in the calculus at all would increase the cost a lot (just considering the sheer number of them, and the logistic challenges of untangling the complex interdependent web of natural ecosystems).