Engineer working on next-gen satellite navigation at Xona Space Systems. I write about effective-altruist and longtermist topics at nukazaria.substack.com, or you can read about puzzle videogames and other things at jacksonw.xyz
Jackson Wagner
Future readers of this post might be interested this other lesswrong post about the current state of multiplex gene editing: https://www.lesswrong.com/posts/oSy5vHvwSfnjmC7Tf/multiplex-gene-editing-where-are-we-now
Future readers of this blog post may be interested in this book-review entry at ACX, which is much more suspicious/wary/pessimistic about prion disease generally:
They dispute the idea that having M/V or V/V genes reduces the odds of getting CJD / mad cow disease / etc.
They imply that Britain’s mad cow disease problem maybe never really went away, in the sense that “spontaneous” cases of CJD have quadrupled since the 80s, so it seems CJD is being passed around somehow?
https://www.astralcodexten.com/p/your-book-review-the-family-that
What kinds of space resources are like “mice & cheese”? I am picturing civilizations expanding to new star systems mostly for the matter and energy (turn asteroids & planets into a dyson swarm of orbiting solar panels and supercomputers on which to run trillions of emulated minds, plus constructing new probes to send onwards to new star systems).
re: the Three Body Problem books—I think the book series imagines that alien life is much, much more common (ie, many civilizations per galaxy) than Robin Hanson imagines in his Grabby Aliens hypothesis, such that there are often new, not-yet-technologically-mature civilizations popping up nearby each other, around the same time as each other. Versus an important part of the Grabby Aliens model is the idea that the evolution of complex life is actually spectacularly rare (which makes humans seem to have evolved extremely early relative to when you might expect, which is odd, but which is then explained by some anthropic reasoning related to the expanding grabby civilizations—all new civilizations arise “early”, because by the mid-game, everything has been colonized already). If you think that the evolution of complex life on other planets is actually a very common occurrence, then there is no particular reason to put much weight on the Grabby Aliens hypothesis.
In The Three Body Problem, Earth would be wise to keep quiet so that the Trisolarians don’t overheard our radio transmissions and try to come and take our nice temperate planet, with its nice regular pattern of seasons. But there is nothing Earth could do about an oncoming “grabby” civilization—the grabby civilization is already speeding towards Earth at near-lightspeed, and wants to colonize every solar system (inhabited and uninhabited, temperate planets with regular seasons or no, etc), since it doesn’t care about temperate continents, just raw matter that it can use to create dyson swarms. The grabby civilizations are already expanding as fast as possible in every direciton, coming for every star—so there is no point trying to “hide” from them.
Energy balance situation:
- the sun continually emits around 10^26 watts of light/heat/radiation/etc.
- per some relativity math at this forum comment, it takes around 10^18 joules to accelerate 1kg to 0.99c
- so, using just one second of the sun’s energy emissions, you could afford to accelerate around 10^8 kg (about the mass of very large cargo ships, and of the RMS Titanic) to 0.99c. Or if you spend 100 days’ worth of solar energy instead of one second, you could accelerate about 10^15 kg, the mass of Mt. Everest, to 0.99c.
- of course then you have to slow down on the other end, which will take a lot of energy, so the final size of the von neumann probe that you can deliver to the target solar system will have to be much smaller than the Titanic or Mt Everest or whatever.
- if you go slower, at 0.8c, you can launch 10x as much mass with the same energy (and you don’t have to slow down as much on the other end, so maybe your final probe is 100x bigger), but of course you arrive more slowly—if you’re travelling 10 light years, you show up 1.9 years later than the 0.99c probe. If you’re travelling 100 light years, you show up 19 years later.
- which can colonize the solar system and build a dyson swarm faster—a tiny probe that arrives as soon as possible, or a 100x larger probe that arrives with a couple years’ delay? this is an open question that depends on how fast your von neuman machine can construct solar panels, automated factories, etc. Carl Shulman in a recent 80K podcast figures that a fully-automated economy pushing up against physical limits, could double itself at least as quickly as once per year. So mabye the 0.99c probe would do better over the 100 light-year distance (arriving 19 years early gives time for 19 doublings!), but not for the 10 light-year distance (the 0.99c probe would only have doubled itself twice, to 4x its initial mass, by the time the 0.8c probe shows up with 100x as much mass)
- IMO, if you are trying to rapaciously grab the universe as fast as possible (for the ultimate purpose of maximizing paperclips or whatever), probably you don’t hop from nearby star to nearby star at efficient speeds like 0.8c, waiting to set up a whole new dyson sphere (which probably takes many years) at each stop. Rather, your already-completed dyson swarms are kept busy launching new probes all the time, targeting ever-more-distant stars. By the time a new dyson swarm gets finished, all the nearby stars have also been visited by probes, and are already constructing dyson swarms of their own. So you have to fire your probes not at the nearest stars, but at stars some distance further away. My intuition is that the optimal way to grab the most energy would end up favoring very fast expansion speeds, but I’m not sure. (Maybe the edge of your cosmic empire expands at 0.99c, and then you “mop up” some interior stars at more efficient speeds? But every second that you delay in capturing a star, that’s a whopping 10^26 joules of energy lost!)
Yes, it does have to be fast IMO, but I think fast expansion (at least among civilizations that decide to expand much at all) is very likely.
Of course the first few starships that a civilization sends to colonize the nearest stars will probably not be going anywhere near the speed of light. (Unless it really is a paperclips-style superintelligence, perhaps.) But within a million years or so, even with relatively slow-moving ships, you have colonized thousands of solar systems, built dyson swarms around every star, have a total population in the bajilions, and have probably developed about all the technology that it is physically possible to develop. So, at some point it’s plausible that you start going very close to the speed of light, because you’ll certainly have enough energy + technology to do so, and because it might be desirable for a variety of reasons:
- Maybe we are trying to maximize some maximizable utility function, be that paperclips or some more human notion, and want to minimize what Nick Bostrom calls “astronomical waste”.
- Maybe we fail to coordinate (via a strong central government or etc), and the race to colonize the galaxy becomes a free-for-all, rewarding the fastest and most rapacious settlers, a la Robin Hanson’s “Burning the cosmic commons”.
Per your own comment—if you only colonize at 0.8c so your ships can conserve energy, you are probably actually missing out on lots and lots of energy, since you will only be able to harvest resources from about half the volume that you could grab if you traveled at closer to lightspeed!
I think part of the “calculus” being run by the AI safety folks is as follows:
-
there are certainly both some dumb ways humanity could die (ie, AI-enabled bioweapon terrorism that could have easily been prevented by some RLHF + basic checks at protein synthesis companies), as well as some very tricky, advanced ways (AI takeover by a superintelligence with a very subtle form of misalignment, using lots of brilliant deception, etc)
-
It seems like the dumber ways are generally more obvious / visible to other people (like military generals or the median voter), wheras these people are skeptical of the trickier paths (ie, not taking the prospect of agentic, superintelligent AI seriously; figuring alignment will probably continue to be easy even as AI gets smarter, not believing that you could ever use AI to do useful AI research, etc).
-
The trickier paths also seem like we might need to get a longer head start on them, think about them more carefully, etc.
-
Therefore, I (one of the rare believers in things like “deceptive misalignment is likely” or “superintelligence is possible”) should work on the trickier paths; others (like the US military, or other government agencies, or whatever) will eventually recognize and patch the dumber paths.
-
re: your comments on Fermi paradox—if an alien super-civilization (or alien-killing AI) is expanding in all directions at close to the speed of light (which you might expect a superintelligence to do), then you mostly don’t see them coming until it’s nearly too late, since the civilization is expanding almost as fast as the light emitted by the civilization. So it might look like the universe is empty, even if there’s actually a couple of civilizations racing right towards you!
There is some interesting cosmological evidence that we are in fact living in a universe that will eventually be full of such civilizations; see the Robin Hanson idea of “Grabby Aliens”: https://www.youtube.com/watch?v=l3whaviTqqg
[spoilers for minor details of later chapters of the book] Isn’t the book at least a little self-aware about this plot hole? If I recall correctly, the book eventually reveals (rot13 from here on out)...
gung gur fcnpr cebtenz cyna jnf rffragvnyyl qrfvtarq nf n CE fghag gb qvfgenpg/zbgvingr uhznavgl, juvpu vaqrrq unq ab erny cebfcrpg bs jbexvat (gbb srj crbcyr, gbb uneq gb qbqtr zrgrbef, rirelguvat arrqf gb or 100% erhfrq, rgp, yvxr lbh fnl). Juvyr gur fcnpr cebtenz jnf unccravat, bgure yrff-choyvp rssbegf (eryngrq gb ahpyrne fhoznevarf, qvttvat haqretebhaq, rgp) ner vaqrrq batbvat, nygubhtu gur tbireazragf pregnvayl nera’g gelvat gb fnir uhaqerqf bs zvyyvbaf bs crbcyr (urapr gur qrfver sbe frperpl, V thrff).
Jung’f “fhecevfvat” nobhg gur obbx’f cybg, vf gung (fvapr Arny Fgrcurafba ernyyl jnagf gb jevgr n fgbel nobhg fcnpr, engure guna n fgbel nobhg haqretebhaq ohaxref), gur fcnpr cyna npghnyyl raqf hc fhpprrqvat ng nyy, naq vaqrrq va gur ybat eha raqf hc orvat zhpu zber vasyhragvny bire gur shgher guna gur inevbhf cerfhznoyl-zber-frafvoyr cynaf sbe haqretebhaq ershtrf.
In addition to the researchy implications for topics like deception and superpersuasion and so forth, I imagine that results like this (although, as you say, unsuprising in a technical sense) could have a huge impact on the public discussion of AI (paging @Holly_Elmore and @Joseph Miller?) -- the general public often seems to get very freaked out about privacy issues where others might learn their personal information, demographic characteristics, etc.
In fact, the way people react about privacy issues is so strong that it usually seems very overblown to me—but it also seems plausible that the fundamental /reason/ people are so sensitive about their personal information is precisely because they want to avoid being decieved or becoming easily manipulable / persuadable / exploitable! Maybe this fear turns out to be unrealistic when it comes to credit scores and online ad-targeting and TSA no-fly lists, but AI might be a genuinely much more problematic technology with much more potential for abuses here.
So, sure, there is a threshold effect in whether you get value from bike lanes on your complex journey from point A to point G. But other people throughout the city have different threshold effects:
Other people are starting and ending their trips from other points; some people are even starting and ending their trip entirely on Naito Parkway.
People have a variety of different tolerances for how much they are willing to bike in streets, as you mention.
Even people who don’t like biking in streets often have some flexibility. You say that you personally are flexible, “but for the sake of argument, let’s just assume that there is no flexibility”. But in real life, even many people who absolutely refuse to bike in traffic might be happy to walk their bike on the sidewalk (or whatever) for a single block, in order to connect two long stretches of beautiful bike path.
When you add together a million different possible journeys across thousands of different people, each with their own threshold effects, the sum of the utility provided by each new bike lane probably ends up looking much more like a smooth continuum of incremental benefits per each new bike lane that is added to a city’s network, with no killer threshold effects. This is very different from a bridge, where indeed half a bridge is not useful to any portion of the city’s residents.
Therefore, I don’t think “if you’re going to start building [a bike lane network], you better m ake sure you have plans to finish it”. Rather, I think adding random pieces of the network piecemeal (as random roads undergo construction work for other reasons, perhaps) is a totally reasonable thing for cities to do.
Another example of individual threshold effects adding up to continuous benefit from the provider’s perspective: suppose I only like listening to thrash-metal songs on Spotify. Whenever Spotify adds a song from a genre I don’t care for—pop, classical, doom-metal, whatever—it provides literally ZERO value to me! There’s a huge threshold effect, where I only care when spotify adds thrash-metal songs! But of course, since everyone has different preferences, the overall effect of adding songs from Spotify’s perspective is to provide an incremental benefit to the quality of their product each time they add a song. (Disclaimer: I am not actually a thrash-metal fanatic.)
Nice; Colorado recently passed a statewide law that finally does away with a similar “U+2” rule in my own town of Fort Collins (as well as other such rules in Boulder and elsewhere). To progress!
this is quality.
I don’t understand this post, because it seems to be parodying Anthropic’s Responsible Scaling Policies (ie, saying that the RSPs are not sufficient), but the analogy to nuclear power is confusing since IMO nuclear power has in fact been harmfully over-regulated, such that advocating for a “balanced, pragmatic approach to mitigating potential harms from nuclear power” does actually seem good, compared to the status quo where society hugely overreacted to the risks of nuclear power without properly taking a balanced view of the costs vs benefits.
Maybe you can imagine how confused I am, if we use another example of an area where I think there is a harmful attitude of regulating entirely with a view towards avoiding visible errors of commision, and completely ignoring errors of omission:Hi, we’re your friendly local pharma company. Many in our community have been talking about the need for “vaccine safety.”… We will conduct ongoing evaluations of whether our new covid vaccine might cause catastrophic harm (conservatively defined as >10,000 vaccine-side-effect-induced deaths).
We aren’t sure yet exactly whether the vaccine will have rare serious side effects, since of course we haven’t yet deployed the vaccine in the full population, and we’re rushing to deploy the vaccine quickly in order to save the lives of the thousands of people dying to covid every day. But fortunately, our current research suggests that our vaccine is unlikely to cause unacceptable harm. The frequency and severity of side effects seen so far in medical trials of the vaccine are far below our threshold of concern… the data suggest that we don’t need to adopt additional safety measures at present.
To me, vaccine safety and nuclear safety seem like the least helpful possible analogies to the AI situation, since the FDA and NRC regulatory agencies are both heavily infected with an “avoid deaths of commision at nearly any cost” attitude, which ignores tradeoffs and creates a massive “invisible graveyard” of excess deaths-of-omission. What we want from AI regulation isn’t an insanely one-sided focus that greatly exaggerates certain small harms. Rather, for AI it’s perfectly sufficient to take the responsible, normal, common-sensical approach of balancing costs and benefits. The problem is just that the costs might be extremely high, like a significant chance of causing human extinction!!
Another specific bit of confusion: when you mention that Chernobyl only killed 50 people, is this supposed to convey:
1. This sinister company is deliberately lowballing the Chernobyl deaths in order to justify continuing to ignore real risks, since a linear-no-threshold model suggests that Chernobyl might indeed have caused tens of thousands of excess cancer deaths around the world? (I am pretty pro- nuclear power, but nevertheless the linear-no-threshold model seems plausible to me personally.)
2. That Chernobyl really did kill only 50 people, and therefore the company is actually correct to note that nuclear accidents aren’t a big deal? (But then I’m super-confused about the overall message of the post...)
3. That Chernobyl really did kill only 50 people, but NEVERTHELESS we need stifling regulation on nuclear power plants in order to prevent other rare accidents that might kill 50 people tops? (This seems like extreme over-regulation of a beneficial technology, compared to the much larger number of people who die from the smoke of coal-fired power plants and other power sources.)
4. That Chernobyl really did kill only 50 people, but NEVERTHELESS we need stifling regulation, because future accidents might indeed kill over 10,000 people? (This seems like it would imply some kind of conversation about first-principles reasoning and tail risks and stuff, but this isn’t present in the post?)
I will definitely check out that youtube channel! I’m pretty interested in mechanism design and public-goods stuff, and I agree there are a lot of good ideas there. For instance, I am a huge fan of Georgism, so I definitely recognize that going all-in on the “libertarian individualist approach” is often not the right fit for the situation! Honestly, even though charter cities are somewhat an intrinsically libertarian concept, part of the reason I like the charter city idea is indeed the potential for experimenting with new ways to manage the commons and provide public goods—Telosa is explicitly georgist, for example, and even hyper-libertarian Prospera has some pretty interesting concepts around things like crime liability insurance, which in the USA is considered a pretty left-wing (or maybe “far-liberal”? idk...) idea for trying to reduce gun violence.
But yeah, a lot of common leftist critiques of society/capitalism/etc can feel kind of… shallow, or overly-formulaic, or confused about the incentives of a given situation, to me? So I’d like to get a better understanding of the best versions of the leftist worldview, in order to better appreciate what the common critiques are getting at.
Yup, there are definitely a lot of places (like 99+% of places, 99+% of the time!) which aren’t interested in a given reform—especially one as uniqely big and experimental as charter cities. This is why in our video we tried to focus on political tractability as one of the biggest difficulties—hopefully we don’t come across as saying that the world will instantly be tiled over with charter cities tomorrow! But some charter cities are happening sometimes in some places—in addition to the examples in the video, Zambia is pretty friendly towards the idea, and is supportive of the new-city project Nkwashi. (I think Charter Cities Institute considers Nkwashi to be their biggest current partnership?) Democracy was achieved, after all, even if it still hasn’t won a total victory even after 250+ years.
“What if we could redesign society from scratch? The promise of charter cities.” [Rational Animations video]
Thanks, this is exciting and inspiring stuff to learn about!
I guess another thing I’m wondering about, is how we could tell apart genes that impact a trait via their ongoing metabolic activities (maybe metabolic is not the right term… what I mean is that the gene is being expressed, creating proteins, etc, on an ongoing basis), versus genes that impact a trait via being important for early embryonic / childhood development, but which aren’t very relevant in adulthood. Genes related to intelligence, for instance, seem like they might show up with positive scores in a GWAS, but their function is confined to helping unfold the proper neuron connection structures during fetal development, and then they turn off, so editing them now won’t do anything. Versus other genes that affect, say, what kinds of cholesterol the body produces, seem more likely to have direct impact via their day-to-day operation (which could be changed using a CRISPR-like tool).
Do we have any way of distinguishing the one type of genes from the other? (Maybe we can just look at living tissue and examine what genes are expressed vs turned off? This sounds hard to do for the entire genome...) Or perhaps we have reason to believe something like “only 20% of genes are related to early development, 80% handle ongoing metabolism, so the GWAS --> gene therapy pipeline won’t be affected too badly by the dilution of editing useless early-development genes”?
Is there a plausible path towards gene therapies that edit dozens, hundreds, or thousands of different genes like this? I thought people were worried about off-target errors, etc? (Or at least problems like “you’ll have to take 1000 different customized doses of CRISPR therapy, which will be expensive”.) So my impression is that this kind of GWAS-inspired medicine would be most impactful with whole-genome synthesis? (Currently super-expensive?)
To be clear I agree with the main point this post is making about how we don’t need animal models, etc, to do medicine if we have something that we know works!
(this comment is kind of a “i didn’t have time to write you a short letter so I wrote you a long one” situation)
re: Infowar between great powers—the view that China+Russia+USA invest a lot of efforts into infowar, but mostly “defensively” / mostly trying to shape domestic opinion, makes sense. (After all, it must be easier to control the domestic media/information lansdscape!) I would tend to expect that doing domestically-focused infowar stuff at a massive scale would be harder for the USA to pull off (wouldn’t it be leaked? wouldn’t it be illegal somehow, or at least something that public opinion would consider a huge scandal?), but on the other hand I’d expect the USA to have superior infowar technology (subtler, more effective, etc). And logically it might also be harder to percieve effects of USA infowar techniques, since I live in the USA, immersed in its culture.Still, my overall view is that, although the great powers certainly expend substantial effort trying to shape culture, and have some success, they don’t appear to have any next-gen technology qualitatively different and superior to the rhetorical techniques deployed by ordinary successful politicians like Trump, social movements like EA or wokeism, advertising / PR agencies, media companies like the New York Times, etc. (In the way that, eg, engineering marvels like the SR-72 Blackbird were generations ahead of competitors’ capabilities.) So I think the overall cultural landscape is mostly anarchic—lots of different powers are trying to exert their own influence and none of them can really control or predict cultural changes in detail.
re: Social media companies’ RL algorithms are powerful but also “they probably couldn’t prevent algorithms from doing this if they tried due to goodharts law”. -- Yeah, I guess my take on this is that the overt attempts at propaganda (aimed at placating the NYT) seem very weak and clumsy. Meanwhile the underlying RL techniques seem potentially powerful, but poorly understood or not very steerable, since social media companies seem to be mostly optimizing for engagement (and not even always succeeding at that; here we are talking on LessWrong instead of tweeting / tiktoking), rather than deploying clever infowar superweapons. If they have such power, why couldn’t left-leaning sillicon valley prevent the election of Trump using subtle social-media-RL trickery?
(Although I admit that the reaction to the 2016 election could certainly be interpreted as sillicon valley suddenly realizing, “Holy shit, we should definitely try to develop social media infowar superweapons so we can maybe prevent this NEXT TIME.” But then the 2020 election was very close—not what I’d have expected if info-superweapons were working well!)
With Twitter in particular, we’ve had such a transparent look at its operations during the handover to Elon Musk, and it just seems like both sides of that transaction have been pretty amateurish and lacked any kind of deep understanding of how to influence culture. The whole fight seems to have been about where to tug one giant lever called “how harshly do we moderate the tweets of leftists vs rightists”. This lever is indeed influential on twitter culture, and thus culture generally—but the level of sophistication here just seems pathetic.Tiktok is maybe the one case where I’d be sympathetic to the idea that maybe a lot of what appears to be random insane trends/beliefs fueled by SGD algorithms and internet social dynamics, is actually the result of fairly fine-grained cultural influence by Chinese interests. I don’t think Tiktok is very world-changing right now (as we’d expect, it’s targeting the craziest and lowest-IQ people first), but it’s at least kinda world-changing, and maybe it’s the first warning sign of what will soon be a much bigger threat? (I don’t know much about the details of Tiktok the company, or the culture of its users, so it’s hard for me to judge how much fine-grained control China might or might not be exerting.)
Unrelated—I love the kind of sci-fi concept of “people panic but eventually go back to using social media and then they feel fine (SGD does this automatically in order to retain users)”. But of course I think that the vast majority of users are in the “aren’t panicking” / never-think-about-this-at-all category, and there are so few people in the “panic” category (panic specifically over subtle persuasion manipulation tech that isn’t just trying to maximize engagement but instead achieve some specific ideological outcome, I mean) that there would be no impact on the social-media algorithms. I think it is plausible that other effects like “try not to look SO clickbaity that users recognize the addictiveness and leave” do probably show up in algorithms via SGD.
More random thoughts about infowar campaigns that the USA might historically have wanted to infowar about:Anti-communism during the cold war, maybe continuing to a kind of generic pro-corporate / pro-growth attitude these days. (But lots of people were pro-communist back in the day, and remain anti-corporate/anti-growth today! And even the republican party is less and less pro-business… their basic model isn’t to mind-control everyone into becoming fiscal conservatives, but instead to gain power by exploiting the popularity of social conservativism and then use power to implement fiscal conservativism.)
Maybe I am taking a too-narrow view of infowar as “the ability to change peoples’ minds on individual issues”, when actually I should be considering strategies like “get people hyped up about social issues in order to gain power that you can use for economic issues” as a successful example of infowar? But even if I consider this infowar, then it reinforces my point that the most advanced stuff today all seems to be variations on normal smart political strategy and messaging, not some kind of brand-new AI-powered superweapon for changing people’s minds (or redirecting their focus or whatever) in a radically new way.
Since WW2, and maybe continuing to today, the West has tried to ideologically immunize itself against Nazi-ism. This includes a lot of trying to teach people to reject charismatic dictators, to embrace counterintuitive elements of liberalism like tolerance/diversity, and even to deny inconvenient facts like racial group differences for the sake of social harmony. In some ways this has gone so well that we’re getting problems from going too far in this direction (wokism), but in other ways it can often feel like liberalism is hanging on by a thread and people are still super-eager to embrace charismatic dictators, incite racial conflict, etc.
“Human brains are extremely predisposed to being hacked, governments would totally do this, and the AI safety community is unusually likely to be targeted.”
—yup, fully agree that the AI safety community faces a lot of peril navigating the whims of culture and trying to win battles in a bunch of diverse high-stakes environments (influencing superpower governments, huge corporations, etc) where they are up against a variety of elite actors with some very strong motivations. And that there is peril both in the difficulty of navigating the “conventional” human-persuasion-transformed social landscape of today’s world (already super-complex and difficult) and the potentially AI-persuasion-transformed world of tomorrow. I would note though, that these battles will (mostly?) play out in pretty elite spaces, wheras I’d expect the power of AI information superweapons to have the most powerful impact on the mass public. So, I’d expect to have at least some warning in the form of seeing the world go crazy (in a way that seems different from and greater than today’s anarchic internet-social-dynamics-driven craziness), before I myself went crazy. (Unless there is an AI-infowar-superweapon-specific hard-takeoff where we suddenly get very powerful persuasion tech but still don’t get the full ASI singularity??)
re: Dath Ilan—this really deserves a whole separate comment, but basically I am also a big fan of the concept of Dath Ilan, and I would love to hear your thoughts on how you would go about trying to “build Dath Ilan” IRL.What should an individual person, acting mostly alone, do to try and promote a more Dath-Ilani future? Try to practice & spread Lesswrong-style individual-level rationality, maybe (obviously Yudkowsky did this with Lesswrong and other efforts). Try to spread specific knowledge about the way society works and thereby build energy for / awareness of ways that society could be improved (inadequate equilibria kinda tries to do this? seems like there could be many approaches here). Personally I am also always eager to talk to people about specific institutional / political tweaks that could lead to a better, more Dath-Ilani world: georgism, approval voting, prediction markets, charter cities, etc. Of those, some would seem to build on themselves while others wouldn’t—what ideas seem like the optimal, highest-impact things to work on? (If the USA adopted georgist land-value taxes, we’d have better land-use policy and faster economic growth but culture/politics wouldn’t hugely change in a broadly Dath-Ilani direction; meanwhile prediction markets or new ways of voting might have snowballing effects where you get the direct improvement but also you make culture more rational & cooperative over time.)
What should a group of people ideally do? (Like, say, an EA-adjacent silicon valley billionaire funding a significant minority of the EA/rationalist movement to work on this problem together in a coordinated way.) My head immediately jumps to “obviously they should build a rationalist charter city”:
The city doesn’t need truly nation-level sovereign autonomy, the goal would just be to coordinate enough people to move somewhere together a la the Free State Project, gaining enough influence over local government to be able to run our own policy experiments with things like prediction markets, georgism, etc. (Unfortunately some things, like medical research, are federally regulated, but I think you could do a lot with just local government powers + creating a critical mass of rationalist culture.)
Instead of moving to a random small town and trying to take over, it might be helpful to choose some existing new-city project to partner with—like California Forever, Telosa, Prospera, whatever Zuzalu or Praxis turn into, or other charter cities that have amenable ideologies/goals. (This would also be very helpful if you don’t have enough people or money to create a reasonably-sized town all by yourself!)
The goal would be twofold: first, run a bunch of policy experiments and try to create Dath-Ilan-style institutions (where legal under federal law if you’re still in the USA, etc). And second, try to create a critical mass of rationalist / Dath Ilani culture that can grow and eventually influence… idk, lots of people, including eventually the leaders of other governments like Singapore or the UK or whatever. Although it’s up for debate whether “everyone move to a brand-new city somewhere else” is really a better plan for cultural influence than “everyone move to the bay area”, which has been pretty successful at influencing culture in a rationalist direction IMO! (Maybe the rationalist charter city should therefore be in Europe or at least on the East Coast or something, so that we mostly draw rationalists from areas other than the Bay Area. Or maybe this is an argument for really preferring California Forever as an ally, over and above any other new-city project, since that’s still in the Bay Area. Or for just trying to take over Bay Area government somehow.)
...but maybe a rationalist charter city is not the only or best way that a coordinated group of people could try to build Dath Ilan?
(Copies from EA Forum for the benefit of lesswrongers following the discussion here)
Definitely agree that empathy and other social feelings provide indirect evidence for self-awareness (ie, “modeling stuff about yourself” in your brain) in a way that optimism/pessimism or pain-avoidance doesn’t. (Although wouldn’t a sophisticated-enough RL circuit, interacting with other RL circuits in some kind of virtual evolutionary landscape, also develop social emotions like loyalty, empathy, etc? Even tiny mammals like mice/rats display sophisticated social behaviors...)
I tend to assume that some kind of panpsychism is true, so you don’t need extra “circuitry for experience” in order to turn visual-information-processing into an experience of vision. What would such extra circuitry even do, if not the visual information processing itself? (Seems like maybe you are a believer in what Daniel Dennet calls the “fallacy of the second transduction”?)
Consequently, I think it’s likely that even simple “RL algorithms” might have a very limited, very shallow, non-self-aware kinds of experience: an image-classifier is doing visual-information-processing, so it probably also produces isolated “experiences of vision”! But of course it would not have any awareness of itself as being a thing-that-sees, nor would those isolated experiences of vision be necessarily tied together into a coherent visual field, etc.
So, I tend to think that fish and other primitive creatures probably have “qualia”, including something like a subjective experience of suffering, but that they probably lack any sophisticated self-awareness / self-model, so it’s kind of just “suffering happening nowhere” or “an experience of suffering not connected to anything else”—the fish doesn’t know it’s a fish, doesn’t know that it’s suffering, etc, the fish is just generating some simple qualia that don’t really refer to anything or tie into a larger system. Whether you call such a disconnected & shallow experience “real qualia” or “real suffering” is a question of definitions.
I think this personal view of mine is fairly similar to Eliezer’s from the Sequences: there are no “zombies” (among humans or animals), there is no “second transduction” from neuron activity into a mythical medium-of-consciousness (no “extra circuitry for experience” needed), rather the information-processing itself somehow directly produces (or is equivalent to, or etc) the qualia. So, animals and even simpler systems probably have qualia in some sense. But since animals aren’t self-aware (and/or have less self-awareness than humans), their qualia don’t matter (and/or matter less than humans’ qualia).
...Anyways, I think our core disagreement is that you seem to be equating “has a self-model” with “has qualia”, versus I think maybe qualia can and do exist even in very simple systems that lack a self-model. But I still think that having a self-model is morally important (atomic units of “suffering” that are just floating in some kind of void, unconnected to a complex experience of selfhood, seem of questionable moral relevance to me), so we end up having similar opinions about how it’s probably fine to eat fish.
I guess what I am objecting to is that you are acting like these philosophical problems of qualia / consciousness / etc are solved and other people are making an obvious mistake. I agree that I see a lot of people being confused and making mistakes, but I don’t think the problems are solved!
Socialism / communism is about equally abstract as Georgism, and it certainly inspired a lot of people to fight! Similarly, Republican campaigns to lower corporate tax rates, cut regulations, reduce entitlement spending, etc, are pretty abstract (and often actively unpopular when people do understand them!), but have achieved some notable victories over the years. Georgism is similar to YIMBYism, which has lots of victories these days, even though YIMBYism also suffers from being more abstract than conspiracy theories with obvious villains about people “hoarding” vacant housing or chinese investors bidding up prices or whatever. Finally, Georgism itself was extremely popular once, so it clearly has the potential!! Overall, I don’t think being abstract is fatal for a mass movement.
But I also don’t think that we need to have some kind of epic Georgist popular revolution in order to get Georgist policies—we can do it just by making small incremental technocratic reforms to local property tax laws—getting local governments to use tools like ValueBase (developed by Georgist Lars Doucet) to do their property value assessments, getting reforms in a few places and then hopefully seeing success and pointing to that success to build more momentum elsewhere, etc.
As Lars Doucet tells it, the main problem with historical Georgism wasn’t unpopularity (it was extremely popular then!), but just the technical infeasibility of assessing land value separate from the value of the buildings on the land. But nowadays we have machine learning tools, GIS mapping systems, satellite imagery, successful home-value-estimation companies like Zillow and Redfin, etc. So nowadays we can finally implement Georgism on a technical level, which wasn’t possible in the 1890s. For more on this, see the final part of Lars’s epic series of georgism posts on Astral Codex Ten: https://www.astralcodexten.com/p/does-georgism-work-part-3-can-unimproved?utm_source=url