One of the main reasons to rebrand is that putting “Berkeley” there makes a lot of people think it’s affiliated with the university, which it’s not.
TsviBT
Karma: 14,569
Interesting, thanks! (It might not work because freedom from samsara in particular seems wrong.)
“Berkeley” is accurate, but it’s not super distinctive and also to some people it sounds like a claim of affiliation with the university. “Genomics” doesn’t make that much sense, since we don’t do genomics. “Project” kinda has the wrong connotations.
Interesting, thanks. Maybe too long but I’ll consider it. I would have just put “Reprogenetics”.
I’m considering rebranding the Berkeley Genomics Project. Curious if anyone has takes on what the core name should be, rather than “Berkeley”. (There will also be other words in the full name.) Some candidates are below; even many of these are unsatisfactory (e.g. many of them sound weird or are hard to pronounce), so this is a broader list. Just wondering if anyone has any opinions or good ideas. My current leaning is “Ephodion”.
[[The rest of this is AI generated, candidates picked from a bunch of searches; not ordered.]]
Rebrand shortlist — thirteen names
-
Aleurone
Pronunciation: AL-yuh-rone, or uh-LOOR-own
Meaning / origin: Greek áleuron, “wheat flour”. The aleurone is the living outer layer of a cereal grain’s food store. When the seed germinates, the embryo signals to it and it produces the enzymes that turn the stored starch and protein into sugars and amino acids the seedling can use.
Connection with reprogenetics: The layer that turns a stored inheritance into usable food, on cue, when the new life is ready for it. A seed’s provision made available at the moment of growth.
-
Crescat
Pronunciation: KRES-kat
Meaning / origin: Latin crescere, “to grow, to increase” (the root of crescent and accrue; Ceres shares the same root). Crescat is the third-person subjunctive: “let it grow”, “may it grow”. Best known from the University of Chicago’s motto Crescat scientia; vita excolatur, “Let knowledge grow from more to more; and so be human life enriched” (from Tennyson).
Connection with reprogenetics: A wish spoken over something young: may it grow. Applies equally to the child, to the science, and to the life the motto’s second half is about.
-
Fuseki
Pronunciation: foo-SEH-kee
Meaning / origin: Japanese 布石, “the spreading of stones”: the opening phase of Go, when stones are placed loosely across the whole board to sketch out frameworks and influence before any local fighting. Everyday Japanese uses 布石を打つ, “to lay a fuseki”, for laying the groundwork for something.
Connection with reprogenetics: The early moves that give a long game its shape and its possibilities. A trajectory-setting word: a fuseki is a sketch of potential, not a finished position, and the game is played out over the hundreds of moves that follow.
-
Scutellum
Pronunciation: skew-TEL-um
Meaning / origin: Latin, “little shield”, a modern diminutive of scutum. In botany the scutellum is the single seed-leaf of a grass embryo (wheat, maize, rice): a shield-shaped organ lying against the food store, which takes up the digested reserves and passes them to the growing embryo.
Connection with reprogenetics: The organ through which a seed’s inheritance actually reaches the new plant: it draws the provision in and hands it on, and it is shaped like a shield. Aleurone’s partner.
-
Ephodion
Pronunciation: eh-FOH-dee-on
Meaning / origin: Greek ἐφόδιον, from epí + hodós (“road”): the supplies or money given to someone setting out on a journey. Used figuratively from antiquity for what a person carries through life; Diogenes Laertius (5.21) records Aristotle’s saying κάλλιστον ἐφόδιον τῷ γήρᾳ τὴν παιδείαν, “education is the finest ephodion for old age” (Greek checked against the Wikisource text, 2026-09-19).
Connection with reprogenetics: Provisions for a journey: what parents pack for a child who will travel much further than they can see. A gift given at the start and used all the way along.
-
Clinamen
Pronunciation: kly-NAY-men, or kli-NAH-men
Meaning / origin: Latin, from clinare, “to lean, to incline”. Lucretius’ word (De rerum natura II) for the smallest possible swerve of an atom falling through the void, at no fixed time or place. Without it nothing would ever meet, nothing new would form, and there could be no free will. Later borrowed by pataphysics and by Harold Bloom for creative misreading.
Connection with reprogenetics: The small early swerve from which everything new follows, and the classical argument that novelty and freedom enter the world through exactly such swerves. A word about openness rather than a plan.
-
Vitellus
Pronunciation: vih-TEL-us
Meaning / origin: Latin, “yolk” (also “little calf”, diminutive of vitulus). Source of vitelline and vitellogenin, the yolk protein that egg-laying mothers synthesize and ship into the egg (humans keep only pseudogenes of it); and of the yolk sac, which gives a human embryo its first nourishment and its first blood cells, and beside which the primordial germ cells first appear.
Connection with reprogenetics: The provision laid down in the egg before there is an embryo to use it: the first gift, and, in the yolk sac, the place where the next generation’s germ cells begin.
-
Chreod
Pronunciation: KREE-ohd
Meaning / origin: Coined by C. H. Waddington (1957, spelled creode) from Greek chrē, “it is necessary”, and hodós, “path”: a “necessary path”; chreod is René Thom’s later respelling, and chreode also circulates, so the word has three spellings. In his picture of development as a landscape of branching valleys, a chreod is one valley: a course that development follows and returns to after small disturbances.
Connection with reprogenetics: A channelled course of development, named as such. The channels are stable and forgiving of small pushes, and a small change early on can open a different valley: development as a landscape with many paths.
-
Carmenta
Pronunciation: kar-MEN-tuh
Meaning / origin: Roman goddess (also Carmentis), named from carmen, “song, prophecy, incantation” (root of charm). Goddess of childbirth and of foresight; mother of Evander; credited with turning the Greek letters into the Latin alphabet. Her festival gained a second day after Rome’s married women, denied their carriages, went on a childbearing strike until the Senate gave way; the story is set in the time of Camillus (c. 395 BC). In Plutarch’s telling (Roman Questions 56) they refused to conceive; in Ovid’s (Fasti 1) they ended the pregnancies they were carrying. Her attendants Prorsa and Postverta face forward and backward.
Connection with reprogenetics: Birth, foresight, and letters together in one figure: the goddess who gave Rome its alphabet also watched over its births. And a story of women winning back a right by withholding childbearing, told as her feast’s origin (in Ovid’s version, an abortion strike, which a hostile reader will find).
-
Balbiani
Pronunciation: bal-bee-AH-nee
Meaning / origin: Édouard-Gérard Balbiani (1823–1899), French embryologist. In 1864 he described the Balbiani body: a cluster of mitochondria, RNA and germ-plasm material that gathers in the young egg cell of many animals, from insects to mammals including humans, and is thought to select the healthy mitochondria the future embryo will inherit and, in many species, to assemble the material from which the next generation’s germ cells form.
Connection with reprogenetics: The egg’s own preparation for the child it will become, made long before fertilization. The most exact instance in biology of provisioning the next generation.
-
Cistron
Pronunciation: SIS-tron
Meaning / origin: Coined by Seymour Benzer in 1957 from his cis-trans complementation test (Latin cis “on this side”, trans “across”) plus the suffix -on (his siblings for it were muton and recon; codon and operon came later). It names the unit of genetic function: the stretch of DNA that has to be whole for one thing to work. Benzer’s achievement was mapping the fine structure inside a gene.
Connection with reprogenetics: The gene as a working unit, defined by a clean experiment: the level at which reprogenetics acts. Cistron also carries the sound of cistern, a reservoir filled to be drawn on later.
-
Stemma
Pronunciation: STEM-uh
Meaning / origin: Greek stémma, a wreath or fillet: a band of wool, or a garland of leaves and flowers, wound round a branch, a head, or an altar as a mark of honor. In Latin stemma came to mean a pedigree, from the Roman custom of displaying ancestor portraits in the atrium joined by painted lines (Pliny’s description); it still means a pedigree, and in textual criticism the tree of manuscripts descended from one original. English stem (the stalk of a plant; the trunk of a family tree) is a Germanic word with no connection to it; the plant sense is a surface echo only.
Connection with reprogenetics: A family tree; the wreath in the Greek gives it a little greenery, and the English stem (stem cells, the stem of a lineage) sits on its surface, but the word’s own meaning is a pedigree of ancestors. Note: that makes it face backward, and the Roman sense is aristocratic lineage display: Juvenal’s eighth satire opens Stemmata quid faciunt?, “what good are pedigrees?”, an attack on pride of birth, the association that killed Scion.
-
Yesod
Pronunciation: yeh-SOHD
Meaning / origin: Hebrew יְסוֹד, “foundation, base, element” (root y-s-d, to found; in modern Hebrew the chemical elements are yesodot). In Kabbalah the ninth sefirah, Foundation: the channel through which everything above flows into the world, tied to the covenant, to generation, and to the verse “the righteous one is the foundation of the world” (Proverbs 10:25).
Connection with reprogenetics: A foundation in the sense of what a life is built on and a channel through which things are handed to the next generation.
Rejected earlier
Rejected on taste
Posterity
Relevance: the future generations the work is for.
Rejected: “limp and generic”.
Kindred
Relevance: family, kin, shared descent.
Rejected: “kinda confusing and generic; could mean a daycare”.
Exemplar
Relevance: a model or pattern to be followed.
Rejected: “too abstract, just a word; kinda elitist”.
Antikythera
Relevance: the ancient mechanism, an emblem of early sophisticated technology.
Rejected: “off topic”.
Coppice
Relevance: woodland cut to the base so it regrows from the stump, generation after generation.
Rejected: “sounds weird; weird to be cutting to the base”.
Orrery
Relevance: a working model of the planets, a system understood well enough to be built.
Rejected: “hard to say, sounds weird”.
Metis
Relevance: Greek for practical cunning; also the Titaness of wisdom swallowed by Zeus.
Rejected: “sounds weird, off-topic”; also collides with the Métis Nation.
Selvedge
Relevance: the self-finished edge of a woven cloth that keeps it from unravelling.
Rejected: on taste.
Usufruct
Relevance: the right to use and enjoy something without owning it, a stewardship idea.
Rejected: on taste.
Vaṃśa
Relevance: Sanskrit for lineage or dynasty (literally “bamboo”, growing joint by joint).
Rejected: on taste; also unsayable for most audiences.
Sylva
Relevance: Latin for woodland; Evelyn’s Sylva on planting trees for future generations.
Rejected: on taste.
Beryl, Corundum, Garnet, Apatite
Relevance: gemstones and minerals as a “stately proper noun” category.
Rejected: the whole stones-and-gems category retired on taste.
Kikashi
Relevance: a Go move that pays off later.
Rejected: the “forcing move” connotation; replaced by Fuseki and Atsumi.
Set aside as “generic” or “meh” (not rejected; still ranked 35–38 in the consolidated table)
Primordium
Relevance: the earliest rudiment of an organ; primordial germ cells are IVG’s starting material.
Verdict (2026-09-19): “a bit like stemma but more so, i.e. kinda generic”.
Amborella
Relevance: sole surviving sister to all flowering plants.
Verdict (2026-09-19): “meh. sounds cool but i don’t get the thematic connection very well”; also faces backward.
Quorum
Relevance: bacteria act only once enough are present; the number needed to decide.
Verdict (2026-09-19): “meh”.
Dropped on the 2026-09-19 verdicts
Clerestory
Relevance: high windows that let light into a great hall; “clear story”; transparency and safety.
Rejected: pronunciation problem; the story is strained.
Cotyledon
Relevance: the seed leaf that feeds the seedling until it can feed itself; placental lobes have the same name.
Rejected: hard to tell how to pronounce.
Klados
Relevance: Greek “branch, young shoot”, the root of clade.
Rejected: sounds like “making a new clade”, i.e. superbabies.
Argonauta
Relevance: the paper nautilus secretes a shell as a cradle for her eggs, then the young leave it.
Rejected: sounds like Argonauts, which is irrelevant.
Portolan
Relevance: medieval coastal charts built by pooling sailors’ observations.
Rejected: “we’re not mapping the child’s future”.
Stylobate
Relevance: the stepped platform a Greek temple stands on; a foundation without the word.
Rejected: too involved to say for what it buys over “plinth”.
Tiller
Relevance: a boat’s steering handle and a grass’s new shoot.
Rejected: “not about steering”.
Liked, but died on occupancy
Meristem
Relevance: the plant’s growing tip, cells that stay undifferentiated and keep producing the whole plant; the best “growing point” story there was.
Rejected: meristem.pro, an autism-transition nonprofit in Fair Oaks, CA.
Lineage
Relevance: descent itself.
Rejected: Lineage Cell Therapeutics, a public pluripotent-cell company.
Orchard
Relevance: planted, tended, fruiting over generations.
Rejected: Orchard Therapeutics (gene therapy).
Ginkgo
Relevance: the living fossil tree, survivor and continuity.
Rejected: Ginkgo Bioworks.
Methuselah
Relevance: longevity; the oldest tree is a bristlecone pine named for him.
Rejected: Methuselah Foundation.
Meridian
Relevance: a reference line for navigation; noon, the high point.
Rejected: Meridian Bioscience.
Silene
Relevance: the campion genus; 32,000-year-old Silene stenophylla plants were regenerated from fruit tissue found in Siberian permafrost (the seeds themselves did not germinate; Yashina et al. 2012).
Rejected: Silene Biotech (cell preservation, Seattle) and a Catalan nature-spirituality nonprofit.
Lucina
Relevance: Roman goddess of childbirth (“she who brings into the light”).
Rejected: many bio and fertility uses already.
Biophore
Relevance: Weismann’s hypothetical unit of heredity, “life-bearer”.
Rejected: Biophore India Pharmaceuticals (active, with a US office); also pre-gene vocabulary.
Incunabula
Relevance: “cradle books”, the infancy of printing; the infancy of a technology.
Rejected: incunabula.org hosts the Ong’s Hat conspiracy mythos about renegade scientists’ forbidden experiments, with a character named Kallikak.
Zircon
Relevance: the oldest known mineral grains, carrying a record from the earliest Earth.
Rejected: Zircon Corporation, the world’s largest stud-finder maker, nearby in Northern California.
Shifra
Relevance: one of the Hebrew midwives of Exodus 1 who defied Pharaoh.
Rejected: several Jewish pregnancy-support organizations use it; and the Exodus 1 cluster adopts the disability-rights critique’s own metaphor.
Yetzirah
Relevance: Hebrew “formation”; Sefer Yetzirah, the book of creation by letters.
Rejected: a Jewish poetry nonprofit.
Shalshelet
Relevance: Hebrew “chain”, the chain of transmission (and a rare cantillation mark).
Rejected: a yeshiva and a music foundation.
Sampo
Relevance: the Kalevala’s mill of plenty, a made thing that produces abundance.
Rejected: Sampo Group (Finnish finance).
Laima
Relevance: Baltic goddess of fate and childbirth.
Rejected: Laima, the Latvian chocolate brand.
Erewhon
Relevance: Butler’s utopia, “nowhere” reversed; a book about machines and heredity.
Rejected: the Los Angeles grocery chain.
Capella
Relevance: a bright star; “little goat”; a chapel.
Rejected: Capella University.
Tikkun
Relevance: Hebrew “repair”, tikkun olam, mending the world.
Rejected: Tikkun magazine.
Tikvah
Relevance: Hebrew “hope”.
Rejected: the Tikvah Fund.
Dorot
Relevance: Hebrew “generations”.
Rejected: the Dorot Foundation.
Trillium
Relevance: a three-petalled spring woodland flower; a seed dispersed by ants.
Rejected: the Ontario Trillium Foundation.
Pomona
Relevance: Roman goddess of fruit trees and orchards.
Rejected: Pomona College.
Delos
Relevance: the island where Apollo and Artemis were born; a floating island made fixed.
Rejected: the Westworld corporation.
Demeter
Relevance: goddess of grain and fertility of the earth.
Rejected: the biodynamic-farming certification body.
Frontier / Reproductive Frontiers
Relevance: the summit’s own name, two years of recognition.
Rejected: the summit already owns it; undecided whether “frontier” is a parent theme; also a PCRS meeting collided with it.
Killed by a second meaning
Scion
Relevance: a grafted shoot; a descendant.
Rejected: in English it means the heir of a wealthy family; grafting is somatic and non-heritable; Toyota.
Rootstock
Relevance: the root system a graft grows on.
Rejected: “stock” is how racists talk about people.
Toledot
Relevance: Hebrew “generations”, the genealogies of Genesis.
Rejected: Toledot Yeshu, the medieval anti-Christian polemic, bad in exactly the religious-bioethics rooms that matter; unstable spelling.
Etymon
Relevance: the original form of a word, a root.
Rejected: etymos means “true”, so it asserts the original is the authentic form, the naturalness premise the institute rebuts; faces backward.
Matrix
Relevance: Latin for womb, and a typefounder’s mold; the best etymology found.
Rejected: ruined by the film.
Palingenesis
Relevance: rebirth, regeneration.
Rejected: Griffin’s definition of fascism is “palingenetic ultranationalism”.
Eunomia
Relevance: Greek “good order”, one of the Horae.
Rejected: too close to eugenics.
Hygieia
Relevance: goddess of health.
Rejected: “racial hygiene”.
Moriah
Relevance: the mountain of the binding of Isaac; a father and son.
Rejected: child sacrifice.
Leto
Relevance: mother of Apollo and Artemis.
Rejected: Dune’s Leto and its breeding program.
Golem
Relevance: a made being, Jewish tradition.
Rejected: a cautionary tale about creation going wrong.
Entelechy
Relevance: Aristotle’s realization of potential.
Rejected: Driesch’s vitalism; also a live US medical-services trademark (checked 2026-09-19).
Epigenesis
Relevance: the doctrine that the embryo develops progressively rather than being preformed, historically the anti-determinist side.
Rejected: heard as “epigenetics”; a defunct pharma and a health nonprofit hold the exact name.
Ephemeris
Relevance: a table of where the planets will be, prediction over time.
Rejected: the root means “lasting a day”; astrology.
Warp
Relevance: the lengthwise threads set up first on the loom, through which everything is woven.
Rejected: “warped”.
Archetype
Relevance: the original pattern.
Rejected: Jung.
Colophon
Relevance: the note at the end of a book naming how it was made.
Rejected: it means endings.
Silphium
Relevance: the ancient plant whose seed shape, on one theory, became the heart symbol.
Rejected: it was a contraceptive, and it is extinct.
Encephalartos woodii
Relevance: a cycad known from a single wild male, kept alive only by cuttings.
Rejected: it cannot reproduce.
Columbine
Relevance: a flower; “dove-like”.
Rejected: the school shooting.
Vanilla
Relevance: an orchid whose pollination had to be done by hand, worked out by a twelve-year-old.
Rejected: “vanilla” means plain.
Kolyma
Relevance: the river and region where the 32,000-year-old Silene fruits were dug from permafrost (see Silene).
Rejected: the gulag.
-
It means enabling people to greatly increase their own general intelligence or the general intelligence of their future children. See https://www.lesswrong.com/posts/jTiSWHKAtnyA723LE/overview-of-strong-human-intelligence-amplification-methods
When I have a bit more time I’ll work out what the pilot would look like in more detail. Broadly, something like, the participants do research in some area related to the goal of HIA, and then turn that into articles or something.
See: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#Timelines And: https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense
Since then I’ve probably updated longerward based on a bit more hope in a slowdown, and shorterward from (logical and empirical) updates regarding neuro research (see https://www.lesswrong.com/posts/MnroTdcCCZoFSXEHy/a-case-that-whole-brain-emulation-research-is-net-harmful-by). Maybe they roughly cancel out, or rather, concentrate mass in 20-40 years a little bit more, relative to before? Overall it just feels like “mostly no one has any idea basically”.
(To be clear, it makes sense to be freaked out by gippity behavior, and we definitely should have a global ban on AI R&D yesterday.)
I think I’m interested, though significantly less interested than in a defense from someone who actually holds the view. It sounds like you could maybe occupy some AGI-soon hypotheses well enough, so could be good. I’m super busy at the moment—maybe we can try in a month or two? Or if you like you could start the dialogue post thing, and then we can slow-cook it (like, a “correspondence game”)?
(For the most part I have stopped arguing about this because approximately no one seemed willing to defend the confident short timelines view in a public debate, with the notable exception of Abram, despite me putting this challenge to people a number of times (not sure how many). But for the record, AFAICT everyone with confident short timelines is overconfident, i.e. their evidence doesn’t match their degree of confidence. This continues to seem to me to have practical implications for resource allocation, leading to mistakenly underinvesting in long-term interventions.)
Effective long-horizon continual learning is likely the last remaining step to superintelligence. It could be a harder problem than a Millennium Prize problem; I consider that unlikely.
Why do you think this? Cf. https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions
Is there anyone who helps operate an AI safety / AI existential derisking talent pipeline, and who might be interested in adding a small pilot human intelligence amplification track to that pipeline? Do reach out to me and/or to @kman.
(We may write a more comprehensive proposal, e.g. outlining programs of research for adult HIA (and separately for reprogenetic HIA, which may be less feasible due to perceived controversialness) that participants could work on, and explaining how we might go about mentorship. But that might come months from now, so I just wanted to float the idea if anyone happens to see this and may want to run with it.)
So funding isn’t really a bottleneck. But talent is!
Up for a call to discuss human intelligence amplification? (I applied but I am worried that the application reviewer may make hasty judgements.)
Your notion of “morally bad actor” does not seem to require/imply that the person “know what they’re doing is bad”, because you could convert someone who doesn’t know this into being a friend,
“Morally bad actor” means something like “systematically resists accurate moral correction”. So first you try accurate moral correction, such as explaining what’s wrong and why it’s wrong. Then, as that fails, you try other stuff, such as social hostility. (In parallel, the whole time, you of course try other other stuff, such as global regulation.)
This seems to be what Liron and Holly are in part trying to do?
Sure. Unclear on the gricean implicature here—do you think I’m arguing against persuasion or applying social pressure? I’m not. Much of both of their behavior seems good to me. I might object to some of it, though at the moment I don’t feel I have a strong view on that given what I’ve seen. I’m saying that
You should orient to morally bad actors differently from people who actually just haven’t been made aware of the problem with their behavior.
It’s bad on the margin to be socially hostile to someone who is actually not yet morally bad by my definition; it’s better to first explain, strictly before being hostile.
As a separate issue, for dealing with morally bad actors, it’s often better to first convince the audience that their actions are bad. I gave some reasons in my upthread comment reply to Liron. (But I do not have a strong opinion about specifics, so this feels like a pretty inconsequential discussion, but IDK.)
Because of 2. and 3., you should err a bit on the side of not being socially hostile.
In short:
For the notion of “morally bad actor” I’m trying to use here, I would endorse that. Instead I’d just say this AI is a “Threat” or something. To me, the point of thinking “morally bad actor” is to think something like:
This actor is supposed to be a friend. They have the capacity to be a friend. Normally we (friends) don’t coordinate to attack this actor, because they are a friend or potential friend. But right now they are harmfully being hostile. So we should defend against that harm while trying to convert them into being a friend. For now we should mostly act as though they are still not totally beyond the pale of potential friendliness / reasonableness.
Or if we use a somewhat more general notion, then I endorse that but only in cases where the AI wasn’t plausibly a member of some negotiated agreement among various agents including humanity.
More elaboration:
I’m using some sort of tripartite classification. (Though I’m making this up / backfilling this right now, so it may be incoherent and/or inconsistent with my past statements.) It classifies agents thusly:
A. “Friend”: a morally good actor; someone who tends over time to act more and more in some good balance of the best interests of themselves, their friends, their community, all of humanity, and the future of humanity; AND who is human, and participates in the collectivity of humanity. (I will leave this undefined at the moment, sorry; I’ve started writing about this here but it’s not necessarily in a LW-clear format.) B. “Adversary”: a morally bad actor; someone who could be a Friend, but they are not, because they are going against the friendly tendency. C. “Threat”: someone who couldn’t be a Friend. (I called this “Alien” at first, but that is not a great terminology; actual aliens could totally be our friends in the general sense of fun peaceful peers, or even in my specific sense of participating in or “converting to” humanity.)
A reason I want to make this distinction is:
With a Threat (who is an actual threat, rather than just being a rock), you just try to kill it, basically. You’re not “in relationship” with it.
With an Adversary, you do whatever you need to do to defend yourself from their harm. But you also try to convert them to being a Friend. You also rally people in a different way.
For one thing, you have to get them to understand that the Adversary (morally bad actor) is acting badly enough that they need to be opposed and punished; normally someone who is an Adversary would look like a Friend (because they have the potential to be a Friend), so you have to overcome that burden of proof.
For another thing, what’s at stake is not just defending against a threat, but also upholding the norm. If you don’t eradicate a virus, ok, that sucks, but you haven’t opened the floodgates for a bunch of other viruses to be like “Ah, it’s actually morally acceptable to infect and kill people.”. If you don’t enforce social norms by thwarting and punishing bad actors, that does happen.
So for example, a virus is a Threat, but not a morally bad actor (Adversary). To answer your question, this AI wasn’t something over which we were feasibly exerting any force due its potential for participating with us in being human. So it’s just a Threat, to be prevented or killed before it exists if possible.
There’s a whole separate thing, where you replace ”...someone who tends over time to act [in accordance with humanity]” with ”...someone who tends over time to act [as part of a negotiated agreement among various agents including humanity]”. Then you get
A’. Negotiating partner B’. Potential negotiating partner, but currently in a conflict C’. Not a potential negotiating partner
From this perspective, the AI might be a morally bad actor because it was supposed to negotiate some omniversal acausal agreement, or feel the weight of the threat of the extent to which it is in a simulation and being watch and punished etc., or something.
It seems like these are all aimed at worlds where there’s still a bunch of frontier AI capabilities research? That in itself seems ok, but why is there nothing of the form “try to get a global ban on this research”??
(Also, separate plug:)
Unsure if we disagree, but maybe? Specifically, I’m saying something rather unsophisticated, which is like:
In general, when people try to do expected utility calculations, they feel uncomfortable using large positive or negative utilities.
This includes as a special case that irreversible outcomes don’t get counted properly. Normal intuitions account for this better than someone trying to use expected value calculations but with this middling-number bias.
Expected values are good, but you can use them to trample other intuitions people have which are better, such as:
Don’t do something that’s “bold”, which is to say, something that’s sticking your neck out in a way other people don’t want you to and they aren’t doing—or to say it in a more rationalist-friendly way, beware the unilateralist’s curse.
Don’t do something that might cause a whole bunch of harm.
Don’t
I guess “expected values” should be packaged with “BTW, some people are systematically biased to put middling numbers on things (probabilities, utilities); if you’re such a person, you can’t trust your expected value calculations to not be way way off the true utility; maybe try tapping into these other intuitions, and then rationalizing (or if you prefer, “reexplaining”) them in terms of expected value.”.
Even if that turns out not to be possible, my second most likely class of near-term takeover stories involve superhuman planning + superhuman persuasion. My guess is that neither necessarily involves conceptual insights or progress at the level you’re imagining (though surely conceptual insights would make it easier).
I think you would get the same pattern. Massive quantities of relatively low-hanging persuasion fruit, and then the hard stuff is out of reach for a while until there’s more fundamental AI capabilities increases. Humans aren’t remotely secure, but many of us do learn eventually. AI won’t follow along well with that learning (for the time being).
If we get some kind of initial ban on superintelligence, one of the high priority and difficult but important implementation elements will be banning neuroscience research that directly pertains to understanding or computer-emulating neural learning or cortical functioning. That might be a rough additional fight, IDK.
As a layman, Terry Tao’s Sept. 3 thread makes it not seem that implausible that a modern AI could solve it.
One of the main reasons to rebrand is that putting “Berkeley” there makes a lot of people think it’s affiliated with the university, which it’s not.
(I kinda agree with your other points, but still don’t know what to call it.)