I’ve done work in this area, but never been particularly enthusiastic about promoting it. It usually turns out to be inactionable/grim/likely to rouse a panic.
This is a familiar thought, to me.
A counterargument occurs to me: Isn’t it arguable that most of what we need to know about a species, to trade with it, is just downstream of its biology? Of course we talk a lot about our contingent factors, our culture, our history, but I think we’re pretty much just the same animals we’ve always been, extrapolated. If that’s the case, wouldn’t far more simulation time be given to evolutionary histories, rather than than simulating variations of hinges? Anthropic measure wouldn’t be especially concentrated on the hinge, it might even skip it.
Countercounterargument: it also seems like there are a lot of anti-inductive effects in the histories of technological societies that might mean you really do have to simulate it all to find out how values settle or just to figure out the species’ rate of success. Evolutionary histories might also have a lot more computationally compressible shared structure.
I’d be surprised if this, the world in front of us, were a pareto-efficient bargaining outcome. Hinge histories fucking suck to live in and I would strongly prefer a trade protocol that instantiated as few of them as possible. I wouldn’t expect many to be necessary, certainly not enough to significantly outweigh the… thing that is supposed to come after. (at this point, I’d prefer to take it into DMs/call)
Thinking about this stuff again, something occurred to me. Please make sure to keep, in cold storage, copies of misaligned AGIs that you may produce, when you catch them. It’s important. This policy could save us.
Please make sure to keep, in cold storage, copies of misaligned AGIs that you may produce, when you catch them. It’s important. This policy could save us.
Would you care to expand on your remark? I don’t see how it follows from what you said above it.
Yeah, it wasn’t argued. I wasn’t sure whether it needed to be explained, for Richard. I don’t remember how I wound up getting there from the rest of the comment, I think it was just in the same broad neighborhood.
I’ve done work in this area, but never been particularly enthusiastic about promoting it. It usually turns out to be inactionable/grim/likely to rouse a panic.
This is a familiar thought, to me.
A counterargument occurs to me: Isn’t it arguable that most of what we need to know about a species, to trade with it, is just downstream of its biology? Of course we talk a lot about our contingent factors, our culture, our history, but I think we’re pretty much just the same animals we’ve always been, extrapolated. If that’s the case, wouldn’t far more simulation time be given to evolutionary histories, rather than than simulating variations of hinges? Anthropic measure wouldn’t be especially concentrated on the hinge, it might even skip it.
Countercounterargument: it also seems like there are a lot of anti-inductive effects in the histories of technological societies that might mean you really do have to simulate it all to find out how values settle or just to figure out the species’ rate of success. Evolutionary histories might also have a lot more computationally compressible shared structure.
I’d be surprised if this, the world in front of us, were a pareto-efficient bargaining outcome. Hinge histories fucking suck to live in and I would strongly prefer a trade protocol that instantiated as few of them as possible. I wouldn’t expect many to be necessary, certainly not enough to significantly outweigh the… thing that is supposed to come after. (at this point, I’d prefer to take it into DMs/call)
Thinking about this stuff again, something occurred to me. Please make sure to keep, in cold storage, copies of misaligned AGIs that you may produce, when you catch them. It’s important. This policy could save us.
Would you care to expand on your remark? I don’t see how it follows from what you said above it.
Yeah, it wasn’t argued. I wasn’t sure whether it needed to be explained, for Richard. I don’t remember how I wound up getting there from the rest of the comment, I think it was just in the same broad neighborhood.
Regardless, yes, I totally can expand on that. Here, I wrote it up: Do Not Delete your Misaligned AGI.