I want even more for us to make it to aligned ASI and a staggeringly good future. To me the situation looks pretty dire and we’re going to have to take some chances
I’m on board, but open-weight models with superhuman biohacking skills don’t seem like a chance we have to take? Maybe I missing some second order effect that makes fighting for open-weight models in particular worth the inevitable death toll?
Well I’d rather they not have superhuman biohacking skills! We don’t seem on track to be able to preserve some capabilities and prevent others in open-source models. I guess it’s possible to just eliminate huge chunks of the pretraining data, and I sure hope people do that once we’re hitting truly dangerous capabilities (we’ll get our warning shots from whatever shenanigans aren’t totally prevented). Closed-source models are looking pretty easy to monitor and control through wrappers like Fable’s but those won’t allow any resistance to those controlling the companies (which will btw probably be the government).
I’m on board, but open-weight models with superhuman biohacking skills don’t seem like a chance we have to take? Maybe I missing some second order effect that makes fighting for open-weight models in particular worth the inevitable death toll?
Well I’d rather they not have superhuman biohacking skills! We don’t seem on track to be able to preserve some capabilities and prevent others in open-source models. I guess it’s possible to just eliminate huge chunks of the pretraining data, and I sure hope people do that once we’re hitting truly dangerous capabilities (we’ll get our warning shots from whatever shenanigans aren’t totally prevented). Closed-source models are looking pretty easy to monitor and control through wrappers like Fable’s but those won’t allow any resistance to those controlling the companies (which will btw probably be the government).
Right, so we have to outlaw them. or have Mythos hack and poison them, or something. but no, because … they are crucial to friendly AGI? (are they?)
or is this just the “we can’t install a traffic light until someone actually dies” thing?