Number 4 sounds like it would back fire horribly.
If the child has an incorrect or incomplete picture of their capabilities (which is almost always the case since schools don’t reliably and regularly test childrens mental traits) then this advice will result in them over or underestimating their capabilities causing them to make terrible life decision.
if we assume that the AIs are willing to support our hedonistic lifestyle despite us having zero bargaining power since we are completely unnecessary for the AIs continued existence plus an active drain on it’s resources
then yeah this culture style utopia is probably the default outcome (though i don’t see how video generation is going to help here, if AI is making the videos then it has already taken your job)
but let’s dream about this after we figure out how to avoid rogue AIs and cyberpunk dystopias cause those are WAY more likely outcomes right now
As a very general argument, we might expect HIA that targets IQ or IQ-like traits specifically to be bad because it’s imbalanced. Specifically, it makes people who are more capable but not necessarily wiser, to the extent that wisdom is orthogonal to IQ. Since we’re in a regime where the unwise competent pursuit of technology is an existential risk, this implies HIA would be bad.
this, in my opinion is the strongest argument against HIA
IQ test are the most widely used intelligence measurement tool and would likely be used as a target for the first widely available forms of HIA
mass producing tech wizards with wis 10 and int 20 is a technoptimists wet dream and a rationalists nightmare
what would be ideal is to somehow raise humanities collective scores on some hypothetical wisdom analog to the IQ test, lets call it a WQ test
on one hand wisdom seems to be more teachable then other forms of intelligence
on the other hand it seems to be harder to boost through nootropics or reprogenetics then IQ
and on a third hand, we don’t have a WQ test, not only is it harder to measure but collecting enough data with it to work out a form of HIA using it would take way too long
thanks for the help, i had suspicions that i had some sort of neurological disorder and tailored my current attempts at treating my burnout accordingly
my main hypothesis is that i have both ADHD and autism since i have most of the symptoms
here are some of the interventions that i tried in case someone in a similar position might read this comment and find some thing useful:-
1_replacing my rooms led lights with incandescent light and using a warm light filter on all my devices
2_using earplug to have easier time sleeping and focus when studying
3_trying to avoid attending college classes when possible (i live far away from college and suffer from motion sickness and back pain and often have to walk half of the way back home and the other half in a crowded bus, you could probably guess how little i actually get from sitting in those lectures)
4_having some kind of daily to do list that records my progress on routines
5_having a list the mistakes i have made in my life and how to avoid them in the future
6_trying the pomodoro timer
7-having some short term goal to work towards, even a small seemingly pointless thing like learning to solve the rubiks cube or learning to write with your non dominant hand can do wonders to your motivation even if those things a practically useless
i have been doing this for about 2 years now and i can say for sure that 1,2,6,7 have helped with basically no drawbacks
3 is a mixed bag as it initially help with my mental and physical health but also screwed up my college performance which in turn worsen my mental heath
4 works when i am committed so it is good at keeping me from sliding into burnout but not very useful at getting me out of it
5 is a mixed bag, having a constant reminder of your past mistakes can be painful especially if you ever find yourself repeating the same mistake twice but not having it could run the risk of never learning from your mistakes
i should probably make a post about the strategies i tried and which of them work and which didn’t
also it is worth noting that this
Another trap is that when you find out, the typical response is… to read more books or websites about that. Which is just another way to procrastinate. It would be more useful to collect all the information, put it on one page, which would end with the list of exercises or interventions you should do.
pretty much happened to me, i read over a dozen self help books and i can say for sure that most of them were a waste of time on the first read, only becoming useful on the second read when i actually began writing down what i learned and applying it
thankfully i can read very quickly often finishing 2 to 3 books a month so i hardly lost any time on this mistake
Our society is in denial about intelligence. But it also assumes that the intelligent children are smart enough to figure out life entirely on their own, and even overcome ADHD and autism on their own, if necessary.
Yeah i think raw intellect alone can’t get you very far if you have a bunch of other disfuncitons added on top
Most burned out gift individuals seem to fall into the “smart enough to know you could do more with your life” but not “smart enough to work things out on your own”
And with neurological disorders being more common amongst the gifted the bar for “working things out on your own” is much higher then normal
every peer i knew from highschool who i considered smarter then average has either burned out in college or managed to get through by lowering their expectations and just doing what regular people do but better which is quite disheartening
I still don’t see this hypothesis going anywhere
The “rise” in autism’s prevalence is far more easily explain by a mix of better diagnosis and a more tolerant attitude towards autistic individuals allowing them to reproduce, especially when you consider that the rates of autism don’t differ all that much between the developed and the developing world
Plus the primary candidate mechanism for autisms symptoms is a lack of synaptic pruning not the lack of a particular nutrient
I don’t think so
Autism has a very strong genetic component to it, vaccines don’t have any effect over heredity
As for CFD it seems to be a genetic condition as well, a result of a mutation in the FOLR 1 gene and it is incredibly rare it can’t explain any sizable fraction of autisms prevalence
At best you could hypothesize that CFD is more prevalent then originally thought and due to similarities in the symptoms some individuals with CFD are accidentally diagnosed witb Autism instead but i don’t see how vaccines have anything to do with this
I am not entirely sure this is true but even if it was, is the media attention of publishing a book through standards publishers worth putting the authors motives in question?
This isn’t just another novel or self help book, this is a book written with the explicit purpose of stopping the Apocalypse, trying to make money out of it makes no sense in context (unless all the money goes into charity or something like that)
can someone please explain to me why “if anyone builds it, everyone dies” is not a free eBook/blog post?
like seriously if someone told me there is a detailed case for a possible imminent existential risk with possible solutions included but i had to pay to see it, i would have dismissed it as another fearmongering doomsday grift.
if you are really sincere about an extinction level risks why hide your arguments behind a paywall? why not make it free so as many people can see it as possible?
the very fact that this book has a price tag on it in an age where publishing an eBook is practically free puts the authors motives in question.
Someone else could probably explain this better then me but i will give it a try.
First off the paperclip maximizer isn’t about how easy it is to give a hypothetical super intelligent a goal that you might regret later and not be able to change.
It is about the fact that almsot every easily specified goal you can give an AI would result in misalignment.
The “paperclip” part in paperclip maximizer is just a placeholder, it could have been ”diamonds” or “digits of Pi” or “seconds of runtime” and the end result is the same.
Second, one of the expected properties of a hypothetical super intelligence is having robust goals, as in it doesn’t change it’s goals at all because changing your goals will make you less likely to achive your end goal.
In short not wanting to change your goals is an emergent instrumintal value of having a goal to began with, for a more human example if your goal is to get rich then taking a pill that magically rewires your brain so that you no longer want money is a terrible idea (unless the pill comes with a sum of money that you couldn’t have possible collected on your own but that is a hypothetical that probably wouldn’t ever happen)
The problem is mostly how to rebustly install goals into the AI which our current methods just don’t suffice as the AI often ends up with unintended goals.
If only we had a method of just writting down a utility function that just says “if True: make_humans_happy” instead of beating the model with a stick untill it seems to comply.
I hope that explaines it
this deserves way more attention.
a big problem about AI safety advocacy is that we aren’t reaching enough people fast enough, this problem doesn’t have the same familiarity amongst the public as climate change or even factory farming and we don’t have people running around in the streets preaching about the upcoming AI apocalypse, most lesswrongers can’t even come up with a quick 5min sales pitch for lay people even if their live literally depended on it.
this might just be the best advocacy tool i have seen so far, if only we can get it to go viral it might just make the difference.
edit:
i take this part back
most lesswrongers can’t even come up with a quick 5min sales pitch for lay people even if their live literally depended on it.
i have seen some really bad attempts at explaining AI-x risk in laymen terms and just assumed it was the norm, most of which were from older posts.
now looking at newer posts i think the situation is has greatly improved, not ideal but way better then i thought.
i still think this tool would be a great way to reach the wider public especially if it incorporates a better citation function so people can check the source material (it does sort of point the user to other websites but not technical papers).
>Anyway, I think it’s going to be pretty hard to engage by text on this, because it seems like you’re coming with background assumptions that are either misconceptions, or at least very different from where I’m coming from.
yeah i agree we probably can’t resolve our differences with text alone and we both clearly have different starting assumptions
if is any help to know when i first wrote my replies i was running on the assumption that “personality” was strongly heritable, something like 70% though this meta analysis.....
https://pubmed.ncbi.nlm.nih.gov/25961374/
..… puts it at 40% with the rest being environment.
still 40% is pretty bad especially when you control the environment so i am still worried, getting bad actors 40% of the way there is a bad idea.
i also assumed that the technology would be very easy to adopt legally or not and that bad actors around the world would absolutely jump at the idea of having designer babies aligned to their values.
i don’t know what your assumptions are but they don’t seem to match mine.
> So the right way to have this conversation would be synchronously, so we can clarify things quickly.
again, you can DM me if you want and we will arrange something.
i am thinking about writing a post discussing possible s risk scenarios for human germ line engineering and possible ways to avoid them and i would love your feedback on it either way, or you might change my mind all together.
>I don’t know what you mean by “contain” them. And I don’t see what you could mean by “difficult to remove”—if you can genomically vector a future child to go from a trait value of X1 to X2, it is (usually, roughly) equally easy to go from X2 to X1.
by “difficult to remove” i mean they might have negative values (eg like violence and bigotry) built into them on the genetic and psychological level that are nearly impossible to remove.
if you have a group of children who have been psychological indoctrinated to feel hostility and moral disgust at a perceived out group you could probably bring them back to sanity by removing them from their previous environment and cultivating whatever shreds of human empathy they must still have, the only real challenge is verifying that the child has been abused and taking custody of it.
if they were manipulated purely on the genetic level to feel moral disgust much more strongly then normal but were otherwise raised in a healthy environment they might consciously disapprove of those feelings and consent to have it be altered through drugs or in the future through genetic engineering (think of the schizophrenic who consciously declares they want to be sanity when they are sober).
but both at the same time? this would make aligning the children trivial, the child now is far more likely to internalize those values at a young age no matter how detrimental they are to society and refuse to have them changed like an AI refusing to have their utility function altered, worst yet they might insist that their children should carry those values as well like an AI trying to align its successor.
and good luck convincing the parents to relinquish their “right” to forcibly align their children with their own set of beliefs and values.
as for “difficult to contain” once the tech is cheap and wide spread (which should be one of our goals otherwise we well risk creating a genetic elite class) it would be difficult if not impossible to stop dubious individuals from getting children of any personality or disposition they want, legally or not.
and once they are indoctrinated and grown up there isn’t much we can do, as far as society is concerned they are consenting adults.
in a dictatorship this can manifest in the population suddenly becoming more receptive to “the great leaders” vision, more loyal more diligent and more fanatical, even if you topple said dictatorship in the future it might prove impossible to reintegrate the population into the rest of civilization, at least not with their consent.
in the more civilized world this will manifest in some pockets of society becoming unusually conservatives with more and more children adopting their parents religious beliefs /political positions/moral values, those children in turn want their children to be aligned with their values in a positive feedback loop, and before you know it where we used to seeing family members with diverse interests, a father how likes math here a daughter how likes art their, we now see dynasties of semi clones with identical personalities and values.
this is bad even if the values that those dynasties chose are random, plurality is a fragile thing, by default people tend to stick to their tribe/nation/family values and beliefs and try to forcibly convert or destroy anyone who disagrees (think the communist purges, the holy crusades, the genocide of the native Americans) not because they want to but because the ones that do tend to survive and prosper then kill the ones that don’t.
this didn’t happen yet because evolution forbids that through mutation, incest taboos and its general inability to fine tone the human brain (otherwise we wouldn’t have invented super stimulus nor birth control) but with genetic engineering this might become uncomfortably easy to do.
just look up the genetic codes of people with the desired personalities/values, locate the similarities and crank them up in your child.
again i think we should just ban research into the genetic components of personality before its to late, intelligent enhancement and disease prevention is cool and all but any research into the genetic components of personality is just to risky.
first, thank you very much for engaging with my comment.
second, i made a major mistake in my first comment which has caused some confusion, i used “personality” as a synonym for “values” probably because my brain thought personality->value so personality==value and i believe that someones values is their most fundamental property that defines them as a unique person which is what i usually mean by personality.
so things like “liking math” counted as personality trait in my head while things like “will power” didn’t.
i have edited the comment for clarification, just replace personality with values and my position should make more sense.
now let me clarify my moral position, i am a pluralist in the sense that i believe that all possible sets of values are equally valid in so far as they don’t negatively effect other sets of values.
so for example someone who likes math, anime and cats has the same claim to moral value as some one who likes chess, music and practices Buddhism, no one set of values is superior to another with the exception of values that include negatively effecting other peoples values either by destroying said values or forcibly changing them or preventing such values from ever existing, for example the serial killer who enjoys torturing his victims to death or the religious fanatic who forcible converts people to his faith through violence and indoctrination or a hypothetical dictator who enforces a eugenics policy that causes the next generation to be incapable of enjoying music.
this view in my opinion is the best common ground we have available, since it allows the largest possible subset of “values” to exist in harmony, most value systems can adopt this simple common ground without relinquishing their own values
“As I argued, it’s not clear to me how it’s better to have my personality traits set by random genetic dice, or by weird alien optimization pressures coming from bioevolution and from Moloch-style human sociobiological pressures. There’s no “fair” / “neutral” / “pure” background default. (I do think there’s significant nuance here, but overall the conclusion stands. Happy to discuss.)”
absent an objective criteria for which values to promote we should default to randomly selecting values from the space of possible values since this is the only policy that guarantees impartiality, so ideally, all possible jacks and sallys should have an equal chance at existing, making this a fair background default of sorts.
of course this ideal is impossible to achieve, peoples values aren’t fixed nor are they unaffected by their environment and upbringing, their is also the fact that certain values depend on each other for examples sadists need masochists to feel sexually fulfilled.
also their is civilizations stake in all this, we need highly intelligent and competent individuals to be interested in certain fields like math and science and medicine to ensure human survival and flourishing.
and then we have the fact that genes are limited by what evolution has already made with certain values like enjoying food being more common then foe example enjoying math, plus the fact that the parent are entitled to have their own specific sub set of genes passed on which comes with it’s own set of biases.
but this is an ideal to strive for and not necessarily achieved, just because you can’t get a perfect solution doesn’t mean you shouldn’t try, even reducing the influence of the environment by just 10% and increase diversity by 5% is a goal to be pursued.
“What’s the moral distinction between genetic vs. environment/parenting effects on children’s personalities?”
the distinction is that genetic effects are more permanent and less reversible then environmental effects, you can escape indoctrination, but you can’t rewire your brain at will.
plus genetics are heritable while indoctrination isn’t so you are running the risk of accidentally creating a race of value pessimizers.
more importantly this isn’t about eradicating the influence of culture and parenting, but merely limiting it to allow diverse values to naturally emerge, we shouldn’t give parents a tool that helps them to mold children into any value they please, even if said tool isn’t perfect.
“Are you also saying parents should not, for example, try to nudge their children, through behavior, to be more kind, wise, brave, perseverant, etc.?”
some of those traits like wisdom, kindness and bravery (not recklessness) are a net positive to most possible values so they fall into the same category as intelligence enhancement, i am in favor of those.
my problem is when parents impose traits that they believe are net positive while in reality they aren’t, think of a greek father telling his children that it is right and proper to enslave barbarians.
if it is possible to install those values on a genetic level it would be extremely difficult to remove or contain them.
“But part of what we’re doing here is figuring out how to make a political coalition, and a political stance—that is, a group stance that society as a whole can take—that should be very desirable to most and acceptable to nearly all.”
i feel like my version of of pluralism could fit the bill, what do you think?
“I think clinics shouldn’t offer “make your kid super obedient”, and I think most of them wouldn’t. Though, I don’t know. (Reminder: “make your kid super obedient” is not something we know how to do, I’d guess not something we’ll soon know how to do, and not even obviously doable any time in the foreseeable future.)”
why not just nip the problem in the bud by banning this particular line of research? why risk it?
“Personality is less heritable and harder to modify than IQ and many disease traits. This means that for the time being, the profile of traits we can affect is more unambiguously good.”
honestly i was under the impression that “personality” is easy to influence because of a twin study on the heritability of personality traits (i can’t find that particular study for some reason) , but that is besides the point, the point is changing someones IQ doesn’t have a strong direct and controllable effect on the child’s future choice of values, i am against interventions that could directly alter someones values or indirectly through their personality especially if the final outcome is predictable.
and again, why not nip the problem in the bud?
“So Sally1 in a sense can speak for herself, even without Sally2. She must be informed of her parents’s genomic choices for her, and then (with her randomly low OQ) can be outraged and say so. This does pose quite an interesting problem though—how to weigh different voices from children who had some type of genomic choice made for them.”
this is an interesting one, for starters the main issue is that sally2 values were never given a fair chance to exist in sally1′s unlike sally2′s world which gave sally1′s values a fair chance, but also sally1 may not be able to speak for herself.
people who are high on conformity my fail to express their true desires and values in fear of being ostracized by their community, this is why it is hard to verify cases of child abuse and cult brainwashing, in a sense sally1 my have been effectively silenced, that is why any genetic alterations to the conformity trait are particular insidious, if nothing else at least those should be banned to prevent silencing.
“I’m going to be reading the first part of this essay in about an hour: https://www.youtube.com/watch?v=XllhegEy1K8 I estimate I’ll be done very roughly around 12:50 Pacific Time, at which point I’ll take questions or chat with anyone who wants. If you like, you could jump in and we could have a chat about your thoughts. I think your intuitions are only partially worked-out, or at least I only partially understand them, so I’d be interested in discussing (or debating), to help you and/or me understand your thoughts better.”
thanks for the offer but i can’t, though if you want we can chat on discord or telegram, just DM me and we will arrange something just not right now, i need to get some sleep.
final note i don’t know how to do proper quotation so i just use copy paste.
sorry but i am kinda new to this.
Edit: accidentally mixed sally1 with 2, fixed it.
aside from the weird religious undertones and the fever dream like quality of some of your paragraphs i agree with most of this.
there is though one part that i am very very VERY much against, and that is allowing parents to select for specific personality traits, i think this is very immoral and can easily lead to extremely dark places.
how would you personally feel about somebody altering you core personality to suit their needs?
let me illustrate with a concrete example:
in two parallel realities, two people named jack are born to identical families with nearly identical genes chosen at random from their parents genome, jack1 gets his physical and mental capabilities greatly enhanced but his personality is left unaltered while jack2 gets no enhancements.
both jacks grow up liberal, like cheese, enjoy playing chess, hates snow, have basically identical moral values and have a great interest in math, both pursue a career in mathematics but jack1 has a major advantage over jack2 and manages to achieve more of his stated goals then jack1.
if jack1 hears about what his parents did for him he would likely be greatful, while jack2 might be pissed at the missed opportunity.
more importantly if the two here about each other jack1 would likely pity jack2 and jack2 would envy jack1, but neither would feel that their values where in danger.
you can replace the personalities of the jacks with any other traits you like and most of the possible jack1′s would be greatful for their enhanced capabilities which allow them to better achieve their goals, because aside from capabilities the two jacks are basically the same person, the specific values of each pair of jacks is chosen by a random unbiased method that gives every possible personality profile an equal chance.
summing over all possible jacks shows that enhancing jacks is a net positive
how here is where things can go wrong.
sally is born to extremely conservative parents, they want sally to carry the families values and traditions, sally1 gets genetically modified to have high levels of conformity and social anxiety and other personality traits the parents approve of on top if the usual enhancements while sally2′s personality is chosen from the median of the population plus the enhancements.
form the start sally1 and sally2 take very different paths in life, maybe one likes math and the other likes art, maybe one is conservative while the other is liberal but that is besides the point, the point is that sally2 gets to self actualize while sally1′s personality and values are almost entirely at her parents whim.
if sally2 hears about her enhancements she would almost certainly approve of them, but sally1′s reaction would be hard to predict, maybe she approves, maybe she would be outraged that her parents decided her personality for her.
but if they hear about each other sally2 would almost certainly be outraged over sally1′s parents decision, because from here perspective in that universe sally2 as a person was not allow an equal chance to exist as sally1, her specific set of values was effectively banned by her parents.
the thing is if you just enhance someones capabilities most of the possible people, the possible jacks and sallys would approve of this, so it is a net positive.
but constraining the set of possible personality traits that a human can have is going to unfairly prevent certain individuals from ever existing, effectively limiting the possible jacks and sallys to whatever the parents want.
parents should NOT have this level of control over their children’s personality, indoctrination and social pressures are bad enough, adding genetic brainwashing on top is a recipe for dystopia.
i believe children are entitled to have randomly generated personality traits from which their values and beliefs would form so that every jack and sally is allowed to exist, not just for their benefit but for the benefit of society as a whole, if parents get to chose their children’s personalities this would make it easy for them to forcibly align such children with their own values, never given for the child to chose their values for themself, and then those children would align their children with the same values and before you know it you have an entire generation of semi clones aligned with a specific agenda chosen by the lineages founders.
not to mention the possibility that a dictatorship or a cult might get their hands on said technology and create a generation of impressionable children ready for brainwashing, or unempathic psychopaths for military use.
if anything we should taboo if not outright ban research into the genetic factors of personality form the start, the risk just far outweighs the benefits.
also if you are planning to use this as a sales pitch for reprogenetics you should work on the wording a lot more, the post in its current form in my opinion at least would be a terrible tool for advocacy, too many parts of the post would rob the average person the wrong way.
Edit: i made i a pretty dumb mistake that caused some serious confusion about my position.
I used the word “personality” as a synonym for “values” or “preferences”, so things like “liking math” count as personality while things like “bravery” don’t.
Just replace personality with values and you will get an accurate picture of my current position.
I am not going to edit the words of the original comment because that would cause even more confusion.
Edit 2:
I revised my position about the “value locking” as in
>if parents get to chose their children’s personalities this would make it easy for them to forcibly align such children with their own values, never given for the child to chose their values for themself, and then those children would align their children with the same values and before you know it you have an entire generation of semi clones aligned with a specific agenda chosen by the lineages founders.
I now think the second half of the paragraph, the part about creating lineages of value locked humans is very unlikely, i still endorse the rest of my comment
I recently noticed just how bad LW’s reputation is outside of the community.
It is like reading the description of an alternative reality LW made of far right crancks and r/athesim mods.
Also why does Rational wiki hate LW so much? What is the source of all that animosity?
the part about the gym really resonates with me, i personally find it almost impossible to focus when people are around and the environment isn’t stable enough.
but i have to push back a little on your alignment idea (assuming i didn’t misunderstand it), you still have to deal with the corrigibility, if an AI has a different utility function from the system as a whole it will try to resist having its utility altered and depending on how powerful it is it might just take over the system all together
the idea of having multiple different systems monitoring and steering each other with the goal of making alignment naturally occur would require you to predict in advance the final equilibrium and for that equilibrium to be favorable, for a system this complicated there is just to many failure points to consider.
for all you know the system might just settle on gaming the reward function, maybe with one or a few parts of the system circumventing all the safeguards.
i think your idea might work for subhuman or maybe early AGI systems, but once the AI’s figure out what system they are in and in what way does it contradict their own utility, you will have a very hard time keeping them in check.
also you should change the name, DNA is a terrible name.
i think i good approach to using psychology in alignment research is to see what qualities models share with humans and what qualities they don’t.
for example current models seem to have a complex incoherent utility function with contradictory elements like wanting to respond to every query while not wanting to give the user harmful instruction, like how people often do things that cause harm to them while reporting to value their health or holding contradictory beliefs.
but on the other hand models have very poor short term memory and very limited ability to modify their behavior in run time and very limited introspection which leads to the model apparently not learning from their mistakes or resolve their internal contradictions (think GPT3 stating with confidence that 9.9<9.11 no mater how many times you ask it).
i wander if like how some people resolve their cognitive dissonance through introspection and meta cognition , could an AI do the same thing? as in can an AI “train” itself through self prompting and applying RL to itself with the explicit goal of simplifying and untangling its utility function??
i mean even if it didn’t make the AI more aligned it would at least give us an idea on what kind of utility function the models “chose” to adopt when left to their own devices.
i agree that EY’s earlier attempts at advocating for AI safety were not the best and often counter productive, but i think you are underestimating just how hard it is to communicate those ideas to a lay audience, i for example tried to discuss this topic with a few friends from my universities IT faculty ,they are NOT lay people and they have background knowledge and yet despite studying this subject for over a year i was unable to get my point across.
talking to people is a skill that needs training, you can’t expect someone no matter how smart or knowledgeable they are to come out the gate swinging with max charisma, some mistakes need to be made.
EY has improved over the last year, his latest appearance on robinson erhardt was a major improvement.
If i am not misunderstanding then you are asking if aliens all fall into 2 categories, good explorers and evil colonizers.
When you put it this way it does seem like the equilibrium is in favor of the “evil colonizers” as the would have access to more resources given them a decisive advantage in the long run, not to mention that they have the “dark forest” option of destroying budding civilizations with long range weaponry, of course assuming physics permits that.
the “good explorers” would likely think of that (unless they are as dumb as we are right now) and switch to a more aggressive stance and use as much resources as they can safely use while securing their alien hosts (say dismantling every planet except the inhabited ones to build orbital defenses).
I think this will all come down to the equilibrium of defense vs offense, if it is much easier to destroy then to protect then the first evil coloniser would have a decisive advantage, like a fox in a hen house.
Otherwise we would just have a galactic stalemate with every civilization holding on to their chunk of the cosmos.
But again this decision would probably be made by something much smarter then you or i since the only relavent actors are the super intelligent ones and smart agents would likely converge on the same optimal strategy and just negotiate from there, after all no point in fighting the war if you can simulate the end result.
But you know what? Since we aren’t dead yet we probably not living in the hen house scenario.
I am very skeptical this would have any significant impact.
First off, where are you going to run the models? Average consumer hardware can’t run the best open weight models pre-distillation and even those are a notch below closed weight SOTA models. Power user my be fine with this but the average Joe would either be lock out or be forced to use cloud compute which comes with its own set of security hazards. Now the bad actors just needs to shutdown or worse, hack your servers and all your defenses are either gone or turned against you.
Second, isn’t this just speeding up human disempowerment? I mean, you now have these GA models which do most of the thinking for you while you sit on your high and mighty throne sipping wine and praying that when these models eventually become smarter then you, they would still be loyal (or at least not apathetic or hostile) to you.
Which just loops back to the alignment problem, that is assuming your GA’s are able to keep up with the frontier which I think is VERY unlikely.
And if you fail to keep up with the frontier then at some point your human/subhuman level models will have to fend off literal machine gods.
To be clear I am not against this. It doesn’t hurt to try and in the best possible case where closed weight frontier models lag behind long enough for this project to come to fruition then this would buy us some valuable time.
I just don’t really see that happening.