Instruction 5 is supererogatory, while instruction 8 is not.
eg
[Question] Why not more small, intense research teams?
Great Negotiation MOOC on Coursera
New evidence on popular perception of “AI” risk
It seems like time to start focusing resources on a portfolio of serious prosaic alignment approaches, as well as effective interdisciplinary management. In my inside view, the highest-marginal-impact interventions involve making multiple different things go right simultaneously for the first AGIs, which is not trivial, and the stakes are astronomical.
Little clear progress has been made on provable alignment after over a decade of trying. My inside view is that it got privileged attention because the first people to take the problem seriously happened to be highly abstract thinkers. Then they defined the scope and expectations of the field, alienating other perspectives and creating a self-reinforcing trapped prior.
Simultaneous Redundant Research
Two big issues I see with the prompt:
a) It doesn’t actually end with text that follows the instructions; a “good” output (which GPT-3 fails in this case) would just be to list more instructions.
b) It doesn’t make sense to try to get GPT-3 to talk about itself in the completion. GPT-3 would, to the extent it understands the instructions, be talking about whoever it thinks wrote the prompt.
D&D website estimates 13.7m active players and rising.
Probabilistic/inductive reasoning from past/simulated data (possibly assumes imperfect implementation of LCDT):
”This is really weird because obviously I could never influence an agent, but when past/simulated agents that look a lot like me did X, humans did Y in 90% of cases, so I guess the EV of doing X is 0.9 * utility(Y).”
Cf. smart humans in Newcomb’s prob: “This is really weird but if I one box I get the million, if I two-box I don’t, so I guess I’ll just one box.”
For a start, low-level deterministic reasoning:
”Obviously I could never influence an agent, but I found some inputs to deterministic biological neural nets that would make things I want happen.”
″Obviously I could never influence my future self, but if I change a few logic gates in this processor, it would make things I want happen.”
[removed]
It goes both ways. We would be truly alien to an AGI trained in a reasonably different virtual environment.
I agree and was going to make the same point: GPT-3 has 0 reason to care about instructions as presented here. There has to be some relationship to what text follows immediately after the end of the prompt.
Also see new edit: Have agents “die” and go into cold storage, both due to environmental events and of old age, e.g. after 30 subjective years minus some random amount.
“They will find bugs! Maybe stack virtual boxes with hard limits”—Why is bug-finding an issue, here? Is your scheme aimed at producing agents that will not want to escape, or agents that we’d have to contain?
The point is to help friendliness emerge naturally. If a malevolent individual agent happens to grow really fast before friendly powers are established, that could be bad.
Some of them will like it there, some will want change/escape, which can be sorted out once Earth is much safer. Containment is for our safety while friendliness is being established.
“Communicate in a manner legible to us”—How would you incentivize this kind of legibility, instead of letting communication shift to whatever efficient code is most useful for agents to coordinate and get more XP?
It can shift. Legibility is most important in the early stages of the environment anyway. I mostly meant messaging interfaces we can log and analyze.
“Have secret human avatars steal, lie and aggress to keep the agents on their toes”—What is the purpose of this part? How is this producing aligned agents from definitely adversarial behavior from humans?
The purpose is to ensure they learn real friendliness rather than fragile niceness. If they fell into a naive superhappy attractor (see 3WC), they would be a dangerous liability. The smart ones will understand.
We have unpredictable changing goals and so will they. Instrumental convergence is the point. It’s positive-sum and winning to respectfully share our growth with them and vice-versa, so it is instrumentally convergent to do so.
That’s because we haven’t been trying to create safely different virtual environments. I don’t know how hard they are to make, but it seems like at least a scalable use of funding.
Maybe true one-shot prisoner’s dilemmas aren’t really a thing, because of the chance of encountering powerful friendliness.
We have, for practical purposes, an existence proof of powerful friendliness in humans.
Maybe that’s why abstract approaches to real-world alignment seem so intractable.
If real alignment is necessarily messy, concrete, and changing, then abstract formality just wasn’t the right problem framing to begin with.
It’s way too late for the kind of top-down capabilities regulation Yudkowsky and Bostrom fantasized about; Earth just doesn’t have the global infrastructure. I see no benefit to public alarm—EA already has plenty of funding.
We achieve marginal impact by figuring out concrete prosaic plans for friendly AI and doing outreach to leading AI labs/researchers about them. Make the plans obviously good ideas and they will probably be persuasive. Push for common-knowledge windfall agreements so that upside is shared and race dynamics are minimized.