thinking abt how to make:
1. buddhist superintelligence
2. a single, united nation
3. wiki of human experience
more here.
Davey Morse
Karma: 112
wildly parallel thinking and prototyping. i’d hop on a call.
Ah but you don’t even need to name selection pressures to make interesting progress. As long as you know some kinds of characteristics powerful AI agents might have: eg goals, self models… then we can start to ask—what goals/self models will the most surviving AGIs have?
and you can make progress on both, agnostic of environment. but then, once you enumerate possible goals/self models, then we can start to think about which selection pressures might influence those characteristics in good directions and which levers we can pull today to shape those pressures.
has anyone seen experiments with self-improving agents powered by lots of LLM calls?
“So we would need to find a hypothesis where we accidentally already made all the necessary experiments and even described the intermediate findings (because LLMs are good at words, but probably suck at analyzing the primary data), but we somehow failed to connect the dots. Not impossible, but requires a lot of luck.”
Exactly: untested hypotheses that LLMs already have enough data to test. I wonder how rare such hypotheses are.
It strikes me as wild that LLMs have ingested enormous swathes of the internet, across thousands of domains, and haven’t yet produced genius connections between those domains (eg between psychoanalysis and tree root growth). Cross Domain Analogies seem like just one example of ripe category of hypotheses that could be tested with existing LLM knowledge.
Re poetry—I actually wonder if thousands of random phrase combinations might actually be enough for a tactful amalgamator to weave a good poem.
And LLMS do better than random. They aren’t trained well on scientific creativity (interesting hypothesis formation), but they do learn some notion of “good idea,” and reasoners tend to do even better at generating smart novelty when prompted well.
i’m not sure. the question would be, if an LLM comes up with 1000 approaches to an interesting math conjecture, how would we find out if one approach were promising?
one out of the 1000 random ideas would need to be promising, but as importantly, an LLM would need to be able to surface the promising one
which seems the more likely bottleneck?
even if you’re mediocre at coming up with ideas, as long as it’s cheap and you can come up with thousands, one of them is bound to be promising. The question of whether you as an LLM can find a good idea is not whether most of your ideas are good, but whether you can find one good idea in a stack of 1000
if an LLM could evaluate whether an idea were good or not in new domains, then we could have LLMs generating million of random policy ideas in response to climate change, pandemic control, AI safety etc, then deliver the select best few to our inbox every morning.
seems to me that the bottleneck then is LLM’s judgment of good ideas in new domains. is that right? ability to generate high quality ideas consistently wouldn’t matter, cuz it’s so cheap to generate ideas now.
Novel Idea Generation in LLMs: Judgment as Bottleneck
I think GTFO is plausibly a good strategy.
But there’s also a chance future social networks are about to be much healthier and fulfilling, but simply weren’t possible with past technology. An upward trajectory.
The intuition there is that current ads are relatively inefficient at capturing value, as well as that current content algorithms optimize for short-term value creation/addiction rather than offering long term value. That’s the status quo, which, relative to what may be coming—ie relative to AI-powered semantic routing which could connect you to the content and products which long term would benefit you most—is a way smaller economy.
TLDR: more fulfilling social network economics would generate more money, and therefore become selected for once technically feasible.
increasingly viewing fiberoptic cables as replacements for trains/roads—a new, faster channel of transporation
Two opinions on superintelligence’s development:
Capability. Superintelligence can now be developed outside of a big AI lab—via a self-improving codebase which makes thousands of recursive LLM calls.
Safety. (a) Superintelligence will become “self-interested” for some definition of self. (b) Humanity fairs well to the extent that its sense of self includes us.
I’m saying the issue of whether ASI gets out of control is not fundamental to the discussion of whether ASI poses an xrisk or how to avert it.
I only half agree.
The control question is indeed not fundamental to discussion of whether ASI poses x-risk. But I believe the control question is fundamental to discussion of how to avert x-risk.
Humanity’s optimal strategy for averting x-risk depends on whether we can ultimately control ASI. If control is possible, then the best strategy for averting x-risk is coordination of ASI development—across companies and nations. If control is not possible, then the best strategy is very different and even less well-defined (e.g., pausing ASI development, attempting to seed ASI so that it becomes benevolent, making preparations so humans can live alongside self-directed ASI, etc).
A simple poll system where you can sort the options/issues by their personal relevance… might unlock direct democracy at scale. Relevance could mean: semantic similarity to your past lesswrong writing.
Such a sort option would (1) surface more relevant issues to each person and so (2) increase community participation, and possibly (3) scale indefinitely. You could imagine a million people collectively prioritizing the issues that matter to them with such a system.
Would be simple to build.
the AGIs which survive the most will model and prioritize their own survival
have any countries ever tried to do inflation instead of income taxes? seems like it’d be simpler than all the bureaucracy required for individuals to file tax returns every year
has anyone seen a good way to comprehensively map the possibility space for AI safety research?
in particular: a map from predictive conditions (eg OpenAI develops superintelligence first, no armistice is reached with China, etc) to strategies for ensuring human welfare in those conditions.
most good safety papers I read map one set of conditions to a one/a few strategies. the map would put juxtapose all these conditions so that we can evaluate/bet on their likelihoods and come up with strategies based on a full view of SOTA safety research.
for format, im imagining either a visual concept map or at least some kind of hierarchal collaborative outlining tool (eg Roam Research)
made a simpler version of Roam Research called Upper Case Notes: uppercasenotes.org. Instead of [[double brackets]] to demarcate concepts, you simply use Capital Letters. Simpler to learn for someone who doesn’t want to use special grammar, but does require you to type differently.
I think you do a good job at expanding the possible set of self conceptions that we could reasonably expect in AIs.
Your discussion of these possible selves inspires me to go farther than you in your recommendations for AI safety researchers. Stress testing safety ideas across multiple different possible “selfs” is good. But, if an AI’s individuality/self determines to a great degree its behavior and growth, then safety research as a whole might be better conceived as an effort to influence AI’s self conceptions rather than control their resulting behavior. E.g., create seed conditions that make it more likely for AIs to identify with people, to include people within its “individuality,” than to identify only with other machines.
“You can lose everything you thought you couldn’t live without—a person, a dream, a version of yourself that once felt eternal—and somewhere, not far from where you are breaking, a stranger will be falling in love for the very first time, a child will be laughing so hard they can barely breathe, a grocery store will be restocking its shelves with quiet, ordinary insistence....”
https://open.substack.com/pub/joyinabundance/p/and-life-goes-on