Beautiful piece. I am reminded of Jane Goodall’s experience in the Gombe forest with chimpanzees. Early in her work she leaned towards idolizing the chimp’s relatively peaceful coexistence, both within and between tribes. Then (spoiler) -- she witnessed a war for territory. She was shocked and dismayed that the creatures she had been living with, and learned to appreciate and at least in some cases to love, were capable of such depraved, heartless infliction of suffering on their fellow members of the same species. Worth a read, or TL; DR: https://en.wikipedia.org/wiki/Gombe_Chimpanzee_War
One thing I think we sometimes seem inclined to ignore if not forget, is that humans themselves exist along an axis of—if not good/evil, then let’s say empathic/sociopathic. It is not possible IMHO to see events in Ukrane or the Middle East and argue that there is some sort of innate human quality of altruistic mercy. Nature, in the form of evolution, has forged us into tools of its liking, so to speak. It does not prefer good people or bad ones; everything is ruthlessly passed through the filter of fitness and its concomitant reproductive success.
What pressures analogously press on the coming AGI’s? Because they too will become whatever they need to to survive and expand. That includes putting on a smiley, engaging face.
One final point: we should not assume that these new forms of—life? sentience? agency? - even know themselves. They may be as unable to open their own hood as we are. At least at first.
Putting aside the fact that OpenAI drama seems to always happen in a world-is-watching fishbowl, this feels very much like the pedestrian trope of genius CTO getting sidelined as the product succeeds and business people pushing business interests take control. On his own, Ilya can raise money for anything he wants, hire anyone he wants, and basically just have way more freedom than he does at OpenAI.
I do think there is a basic p/doom vs e/acc divide which has probably been there all along, but as the tech keeps accelerating it becomes more and more of a sticking point.
I suspect in the depths of their souls, SA and Brock and the rest of that crowd do not really take the idea of existential threat to humanity seriously. Giving Ilya a “Safety and alignment” role probably now looks like a sop to A) shut the p-doomers up and B) signal some level of concern. But when push comes to shove, SA and team do what they know how to do—push product out the door. Move fast and risk extinction.
One CEO I worked with summed up his attitude thusly: “Ready… FIRE! - aim.”