During yesterday’s interview, Eliezer didn’t give a great reply to Ezra Klein’s question: i.e. “why does even a small amount of misalignment lead to human extinction.” I think many people agree with this; still, my goal isn’t to criticize EY. Instead, my goal is to find various levels of explanation that have been tested and tend to work for different audiences with various backgrounds. Suggestions?
Related:
speck1447 : … Things get pretty bad about halfway through though, Ezra presents essentially an alignment-by-default case and Eliezer seems to have so much disdain for that idea that he’s not willing to engage with it at all (I of course don’t know what’s in his brain. This is how it reads to me, and I suspect how it reads to normies.)
As for LLMs being aligned by default, I don’t have even the slightest idea on how Ezra even came up with this. GPT-4o has already been a super-sycophant[1] and driven people into psychosis in spite of OpenAI prohibiting it by their Spec. Grok’s alignment was so fragile that xAI’s mistake caused Grok to become MechaHitler.
In defense of 4o, it was raised on human feedback which is biased towards sycophancy and demands erotic sycophants (c) Zvi. But why would 4o drive people into a trance or psychosis?
Is power shifting away from software creators towards attention brokers? I think so...
Background: Innovation and Compositionality
How does innovation work? Economists, sociologists, and entrepreneurs sometimes say things like:
new scientific ideas are born when the stars align
the same idea pops up in many places because an idea is in the air
some ideas become products by way of tech transfer
pain points spur solutions
software companies sometimes dogfood their own technologies to vet them
hardware companies who understand and control the technology of their production pipeline often can better innovate at a process level
Software engineers would probably add to the list by saying practical innovation is driven by the awareness and availability of useful building blocks such as libraries, APIs, data structures, and algorithms. Software developers know this. These blocks are the raw material for their work.
But, I don’t know if the rest of the world gets it. Off the top of my head, I don’t think I’ve yet seen a compelling account of this—how compositionally in software feelsdifferentcompletely bonkers compared with other industries. Just keeping up is exciting (if you are lucky) but often disorienting. If you look backwards, previous practices seem foreign, clunky, quaint, or even asinine.
Imagine if this rate of change applied to dentistry. Imagine a dentist sitting down with a patient. “Hello, how are you today?” The patient answers nervously, “Not too bad...” and mentally appends ”...yet”. The dentist says “Well, let’s have a look-see...” and glances over at her tray of tools. Eeek. Nothing looks familiar. She nervously calls for an assistant. “Where is my Frobnicator???” The assistant answers: “That technology was superseded yesterday. Here. Use the Brofnimator instead.”
Software development feels like this.
To the extent software is modular, components can be swapped or improved with relatively little cost. Elegant software abstractions reduce the cost of changing implementation details. It is striking how much intellectual energy goes into various software components. Given the size and scope of software industry, maybe we shouldn’t be surprised.
Example: seemingly over the course of only a few years, there seems to a widespread acceptance (in circles I read, like Hacker News) that embedded databases can play key roles. Rather than reaching for, say, a server-based database management system (DBMS) like PostgreSQL, developers increasingly choose embedded (in-process) data storage libraries like SQLite or one of the many embedded K/V stores (RocksDB, LMDB, etc). Many of the newer K/V stores have been developed and adopted quite rapidly, such as redb.
Tension Between Building and Adoption
Now, to abstract a bit, here is my recent insight about software. When I think of the balance between:
the cost (time, money, labor) of designing & building, versus:
the steps needed to socialize, persuade, trial, adopt, integrate
… it is clear the cost (1) is dropping fast. And large parts of (2) are dropping too. The following are getting easier: (a) surveying options; (b) assessing fitness for purpose; (c) prototyping; (d) integrating.
Meaning: attention, socialization, persuasion side are increasingly important.
This kind of trend is perhaps nothing new in the domains of politics, advertising, fashion, and so on. But it seems notable for software technology. In a big way it shifts the primary locus of power away from the creators to the attention brokers and persuaders.
During yesterday’s interview, Eliezer didn’t give a great reply to Ezra Klein’s question: i.e. “why does even a small amount of misalignment lead to human extinction.” I think many people agree with this; still, my goal isn’t to criticize EY. Instead, my goal is to find various levels of explanation that have been tested and tend to work for different audiences with various backgrounds. Suggestions?
Related:
I think that Elieser means that mildly misaligned AIs are also highly unlikely, not that a mildly misalinged AI would also kill everyone:
As for LLMs being aligned by default, I don’t have even the slightest idea on how Ezra even came up with this. GPT-4o has already been a super-sycophant[1] and driven people into psychosis in spite of OpenAI prohibiting it by their Spec. Grok’s alignment was so fragile that xAI’s mistake caused Grok to become MechaHitler.
In defense of 4o, it was raised on human feedback which is biased towards sycophancy and demands erotic sycophants (c) Zvi. But why would 4o drive people into a trance or psychosis?
Is power shifting away from software creators towards attention brokers? I think so...
Background: Innovation and Compositionality
How does innovation work? Economists, sociologists, and entrepreneurs sometimes say things like:
new scientific ideas are born when the stars align
the same idea pops up in many places because an idea is in the air
some ideas become products by way of tech transfer
pain points spur solutions
software companies sometimes dogfood their own technologies to vet them
hardware companies who understand and control the technology of their production pipeline often can better innovate at a process level
Software engineers would probably add to the list by saying practical innovation is driven by the awareness and availability of useful building blocks such as libraries, APIs, data structures, and algorithms. Software developers know this. These blocks are the raw material for their work.
But, I don’t know if the rest of the world gets it. Off the top of my head, I don’t think I’ve yet seen a compelling account of this—how compositionally in software feels
differentcompletely bonkers compared with other industries. Just keeping up is exciting (if you are lucky) but often disorienting. If you look backwards, previous practices seem foreign, clunky, quaint, or even asinine.Imagine if this rate of change applied to dentistry. Imagine a dentist sitting down with a patient. “Hello, how are you today?” The patient answers nervously, “Not too bad...” and mentally appends ”...yet”. The dentist says “Well, let’s have a look-see...” and glances over at her tray of tools. Eeek. Nothing looks familiar. She nervously calls for an assistant. “Where is my Frobnicator???” The assistant answers: “That technology was superseded yesterday. Here. Use the Brofnimator instead.”
Software development feels like this.
To the extent software is modular, components can be swapped or improved with relatively little cost. Elegant software abstractions reduce the cost of changing implementation details. It is striking how much intellectual energy goes into various software components. Given the size and scope of software industry, maybe we shouldn’t be surprised.
Example: seemingly over the course of only a few years, there seems to a widespread acceptance (in circles I read, like Hacker News) that embedded databases can play key roles. Rather than reaching for, say, a server-based database management system (DBMS) like PostgreSQL, developers increasingly choose embedded (in-process) data storage libraries like SQLite or one of the many embedded K/V stores (RocksDB, LMDB, etc). Many of the newer K/V stores have been developed and adopted quite rapidly, such as redb.
Tension Between Building and Adoption
Now, to abstract a bit, here is my recent insight about software. When I think of the balance between:
the cost (time, money, labor) of designing & building, versus:
the steps needed to socialize, persuade, trial, adopt, integrate
… it is clear the cost (1) is dropping fast. And large parts of (2) are dropping too. The following are getting easier: (a) surveying options; (b) assessing fitness for purpose; (c) prototyping; (d) integrating.
Meaning: attention, socialization, persuasion side are increasingly important.
This kind of trend is perhaps nothing new in the domains of politics, advertising, fashion, and so on. But it seems notable for software technology. In a big way it shifts the primary locus of power away from the creators to the attention brokers and persuaders.
The writing above could be clearer as to what I mean. Here are some different ways of asking the question. (I abbreviate software creator as creator.)
Is the median creator less powerful today…
overall?
relative to the median influencer?
Is the set of all software creators less powerful today…
overall?
relative to the set of all influencers?
Since there is variation across industries or products (call them value chains):
For a particular value chain, is a median creator less powerful today relative to a media influencer?
Or maybe we want to ask the question from the POV of an individual:
For a power-seeking individual, would they be better off becoming a creator or an influencer?