Well… One problem here is that a model could be superhuman at:
thinking speed
math
programming
flight simulators
self-replication
cyberattacks
strategy games
acquiring and regurgitating relevant information from science articles
And be merely high-human-level at:
persuasion
deception
real world strategic planning
manipulating robotic actuators
developing weapons (e.g. bioweapons)
wetlab work
research
acquiring resources
avoiding government detection of its illicit activities
Such an entity as described could absolutely be an existential threat to humanity. It doesn’t need to be superhuman at literally everything to be superhuman enough that we don’t stand a chance if it decides to kill us.
So I feel like “RL may not work for everything, and will almost certainly work substantially better for easy to verify subjects” is… not so reassuring.
Such an entity as described could absolutely be an existential threat to humanity
I agree. I think you don’t even need most of the stuff on the “superhuman” list, the equivalent of a competent IQ-130 human upload probably does it, as long as it has the speed + self-copying advantages.
Well… One problem here is that a model could be superhuman at:
thinking speed
math
programming
flight simulators
self-replication
cyberattacks
strategy games
acquiring and regurgitating relevant information from science articles
And be merely high-human-level at:
persuasion
deception
real world strategic planning
manipulating robotic actuators
developing weapons (e.g. bioweapons)
wetlab work
research
acquiring resources
avoiding government detection of its illicit activities
Such an entity as described could absolutely be an existential threat to humanity. It doesn’t need to be superhuman at literally everything to be superhuman enough that we don’t stand a chance if it decides to kill us.
So I feel like “RL may not work for everything, and will almost certainly work substantially better for easy to verify subjects” is… not so reassuring.
I agree. I think you don’t even need most of the stuff on the “superhuman” list, the equivalent of a competent IQ-130 human upload probably does it, as long as it has the speed + self-copying advantages.