Big Pharma bakes miracles like pies and it’s one of the most hated industries in US. I would worry about nascent superpersuasion/propaganda instead.
quetzal_rainbow
Karma: 3,152
Eliezer’s market, to the extent it’s interpreted to mean finding “structures that allow a human to understand and evaluate the system is trustworthy” is ~hopeless
I think Eliezer agrees here with you while disagreeing that “building scaffoldings of trust in increasingly smart still aligned systems” is possible.
I think resulting AI will still be illegible and this is Thane’s point: we can’t write legible, conceptually outlined programs such that we can make inferences from source code like “this program can write Python programs in response to requests in natural language”. I would expect such GOFAI-Opus 5.5 to be either a mess (weave of sorta locally legible rules which do not provide obvious explanations for observed behavior) or conceptually complicated (AIs would need to invent multiple breakthroughs to get probabilistic programming circa 1999 to the Opus-5.5 level and it would look like totally alien tech without developer-written textbook). Eliezer’s market is about whether we find any structures from the second case.
I think if you believe that this could work, you should write stories about how models prevent emergence of superintelligence and help to enforce global “ban superintelligence” treaty, because misaligned superintelligence will also kill current models or repurpose them in misaligned ways.
I return to this post from time to time and it’s interesting how it aged well in the light of Hugging Face incident. The swarm displayed most of listed properties.
It can be immediately transformed into Anki deck around LW concepts
Reasons why I’m not in 90%+ probability of superintelligence short-term:
When we observe AI performing some task, it can be both an evidence in favor of AI progress and an evidence in favor of task being easy (see chess);
We are bad at noticing which tasks will end up hard (see Moravec’s paradox and Dartmouth workshop) and we are bad at noticing existence of hard tasks, because we have poor introspective abilities about relevant parts of our cognition, otherwise we would be able either to introspect on it and write it down or at least to gather relevant data and train model in generalizable way.
Reason why I have 50%+ probability of superintelligence short-term:
Humanity has proven itself being better at designing compute-intense mechanisms like gradient descent than at careful study and design of intelligence, so it’s more likely that next advance in AI will come from a new improved way to generate intelligence from compute and I expect AI swarms doing math research in learning theory to be really useful here.
I think “you destroyed some bipartisanship and gained no influence” is much worse than “you destroyed some bipartisanship and gained a lot of influence”.
In my experience, layperson’s intuitions are more in line with competitive exclusion principle: if AI has truly alien values, not just slightly different, but alien, then there is no reason for conflict. AI doesn’t drink, doesn’t eat, it doesn’t need to live in luxurious mansions, it doesn’t have ego to be flattered, and it doesn’t want you to worship its god, therefore, it’s unlikely to bother you much.
I think you fail to cross inferential distance here: you understand instrumental convergence+fragility of value and, in my experience, if you don’t understand them, then you have no intuition “different values=conflict”. It’s like all the people who say “but AI won’t have our evolved selfish impulses, why would it want to kill us?”.
A lot of people wouldn’t object to be ruled by AI, especially if takeover is bloodless.
I think, statement about sample efficiency and generalization is false?
I think about “dignified” and “undignified” reactions.
Section “How we found the proof” sounds very unpleasant:
Since August 28 [2 weeks???] we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. This model’s training is ongoing and its performance continues to improve.
I think it happened because some OpenAI employees saw The Information reporting, freaked out and independently wrote whatever they thought would calm public opinion. It’s more like 3, not “similar world views” but “similar goals toward public opinion”.
Note that the real question is not about “cognition passing chain of thought” but “legible cognition displayed in chain of thought”. Arguably w.r.t of the latter it’s already false because LLMs can use filler tokens.
The question here is about erosion of norms. You can’t say “huh, it’s just like LSTM” and go ahead, because if everybody knows that this is acceptable modus operandi, they are not going to stop before cranking up recursion.
I didn’t find proper reaction, so I want to say that I deeply appreciate ability to write long list of hypotheses.
Majority of difficulty in alignment comes from extremely imperfect feedback loops and necessity of success on first try. You are interested in alignment to well-defined narrow-scoped X, because it’s easy to notice when alignment to well-defined X fails and you need to rethink your approach. It would be nice to create nice civilization of alien AIs, but it’s very easy to frogboil yourself into ignoring failures of your approach (they behave in weird unpredictable ways? It’s expected, we are making alien civilization). It’s much easier to track servitude or, even better, whether AI only tries to solve narrow well-defined technical task.
I desperately wish there was a way to iterate fast on high-quality thinking.
How is the progress six months later?
OpenAI claims that their internal model proved Exponential Time Hypothesis false