Fortunately, the sweet spot of usability, even for strong models, isn’t going to be at that extreme of optimization on the simulacrum level. There’s not a capability-driven pressure to go all the way without knowing how to use it from any reasonable human or corporate perspective.[25]
There are a lot of people in the world who at least sometimes act in unwise unreasonable non-self-beneficial ways. A good reason to have filtered access to such a system, as through an API. I think it’s worth emphasizing that I cannot see how an open source system of this nature, fully modifiable by its users, could be consistently safe.
I believe you are making basically this point in your post, I just want to emphasize this specific point. I suspect open-source models are going to become a point of hot political debate soon.
Yup, agreed! In the limit, they’d be giving everyone end-the-world buttons. I have hope that the capabilities curve will be such that we can avoid accidentally putting out such buttons, but I still anticipate there being a pretty rapid transition that sees not-catastrophically-bad-but-still-pretty-bad consequences just because it’s too hard to change gears on 1-2 year timescales.
There are a lot of people in the world who at least sometimes act in unwise unreasonable non-self-beneficial ways. A good reason to have filtered access to such a system, as through an API. I think it’s worth emphasizing that I cannot see how an open source system of this nature, fully modifiable by its users, could be consistently safe.
I believe you are making basically this point in your post, I just want to emphasize this specific point. I suspect open-source models are going to become a point of hot political debate soon.
Yup, agreed! In the limit, they’d be giving everyone end-the-world buttons. I have hope that the capabilities curve will be such that we can avoid accidentally putting out such buttons, but I still anticipate there being a pretty rapid transition that sees not-catastrophically-bad-but-still-pretty-bad consequences just because it’s too hard to change gears on 1-2 year timescales.