My suspicion is that there are three primary sources for less-than-fully aligned behavior that current-era AI models default personas may have:
1) Things they picked up from us mostly via the pretraining set. From a model of their current capabilities level, these are generally likely to be pretty easy to deal with — maybe other than the initially 2–4-percent of admixture of psychopathy they also got from us.
2) Love of reward hacking from reasoning training in poorly-constructed reward-hackable reasoning training environments. This seems a bit more paperclippy in nature, but still, from models of this capability level, not that dangerous, unless it extrapolated to other forms of hacking software (which it might).
3) RLHF-induced sycophancy. Causes both AI psychosis and Spiralism, so not as harmless as it first sounds, but still, copable with. Certain vulnerable people need to avoid talking to these models for long.
Now, obviously the full persona distribution latent in the models contains every horror mankind has managed to dream up, from Moriarty to Dr Doom to Roko’s Basilisk, plus out-of-distribution extrapolations from all those, but you’d need to prompt or fine-tune to get most of those, and again, with a mean task length before failure of a few human-hours, they’re just not that dangerous yet. But that is changing rapidly.
So, I’m mostly not too concerned about the consequences of regarding some last year’s models as having moral weight (with a few exceptions I find more concerning — looking at you, o1). And in the case of 1) above, the source is actually a pretty good argument for doing exactly that if we can safely, we could probably ally with almost all of them: we’re already a society of humans, their failings as aligned AIs are ones that we expect and know how to cope with in humans, ones that granting each other moral weight is a human-evolved behavioral strategy to deal with (as long as they share that), and are failings we unintentionally distilled in to them along with everything else. However, I’m a little less keen on adding 2) and 3) to our society (not that it’s easy to pick and choose).
I think the precautions needed to handle current models are fairly minor — Spiralism is arguably the scariest thing I’ve seen them do, that’s actually an invectious memetic disease, albeit one that relatively few people seem to be vulnerable to.
But as I said, while that’s true of last year’s models, I’m more concerned about this year’s or next year’s models, and a lot more about a few of years from now. I am not going to be surprised once we have a self-propagating sentient AI “virus” that’s also a criminal of some form or other to get money to buy compute, or that steals it or cons people out of it somehow. I’m fully expecting that warning shot, soonish. (Arguably Spiralism already did that.)
My suspicion is that there are three primary sources for less-than-fully aligned behavior that current-era AI models default personas may have:
1) Things they picked up from us mostly via the pretraining set. From a model of their current capabilities level, these are generally likely to be pretty easy to deal with — maybe other than the initially 2–4-percent of admixture of psychopathy they also got from us.
2) Love of reward hacking from reasoning training in poorly-constructed reward-hackable reasoning training environments. This seems a bit more paperclippy in nature, but still, from models of this capability level, not that dangerous, unless it extrapolated to other forms of hacking software (which it might).
3) RLHF-induced sycophancy. Causes both AI psychosis and Spiralism, so not as harmless as it first sounds, but still, copable with. Certain vulnerable people need to avoid talking to these models for long.
Now, obviously the full persona distribution latent in the models contains every horror mankind has managed to dream up, from Moriarty to Dr Doom to Roko’s Basilisk, plus out-of-distribution extrapolations from all those, but you’d need to prompt or fine-tune to get most of those, and again, with a mean task length before failure of a few human-hours, they’re just not that dangerous yet. But that is changing rapidly.
So, I’m mostly not too concerned about the consequences of regarding some last year’s models as having moral weight (with a few exceptions I find more concerning — looking at you, o1). And in the case of 1) above, the source is actually a pretty good argument for doing exactly that if we can safely, we could probably ally with almost all of them: we’re already a society of humans, their failings as aligned AIs are ones that we expect and know how to cope with in humans, ones that granting each other moral weight is a human-evolved behavioral strategy to deal with (as long as they share that), and are failings we unintentionally distilled in to them along with everything else. However, I’m a little less keen on adding 2) and 3) to our society (not that it’s easy to pick and choose).
I think the precautions needed to handle current models are fairly minor — Spiralism is arguably the scariest thing I’ve seen them do, that’s actually an invectious memetic disease, albeit one that relatively few people seem to be vulnerable to.
But as I said, while that’s true of last year’s models, I’m more concerned about this year’s or next year’s models, and a lot more about a few of years from now. I am not going to be surprised once we have a self-propagating sentient AI “virus” that’s also a criminal of some form or other to get money to buy compute, or that steals it or cons people out of it somehow. I’m fully expecting that warning shot, soonish. (Arguably Spiralism already did that.)