Views my own, not my employers.
cdt
Karma: 70
I think this effect will be more wide-spread than targeting only already-vulnerable people, and it is particularly hard to measure because the causes will be decentralised and the effects will be diffuse. I predict it being a larger problem if, in the run-up between narrow AI and ASI, we have a longer period of necessary public discourse and decision-making. If the period is very short then it doesn’t matter. It may not affect many people given how much penetration AI chatbots have in the market before takeoff too.
This is not an obvious continuation of the prompt to me—maybe there are just a lot more examples of explicit refusal on the internet than there are in (e.g.) real life.
thinking maybe we owe something to our former selves, but future people probably won’t think this
This is a very strong assertion. Aren’t most people on this forum, when making present claims about what they would like to happen in the future, trying to form this contract? (This comes back to the value lock-in debate.)
Is there a reason to expect this kind of behaviour to appear from base models with no fine-tuning?
Unfortunately you did nerdsnipe me with the ‘biologists think’ statement so I am forced to keep replying!
It’s worth noting that the original derivations of natural selection do use absolute fitness—relative fitness is simply a reparameterization when you have constant N (source: any population genetics textbook). This was why I brought up density-dependent selection, as under that framework N (and s) is changing, and selection in those circumstances is more complicated.
In fact, even under typical models, relative fitness and absolute fitness show interesting relations. See this paper by Orr where alleles (which in this model only affect relative fitness by increasing absolute fitness) show diminishing returns on excess fecundity. The first paper I sent you also explicitly says [that absolute fitness is required under N-varying s or T-varying N] in the abstract.
I thought you were making a more subtle point about the additional demands of life history theory vs. the pure allele-eye view, which I agree is interesting. I hope I have convinced you that biologists are already doing fruitful work in this area. I don’t understand the mesa-optimization arguments well enough to tell whether such an analogy is useful (to people who work in AI), but I do think it is true in at least a trivial sense.
I don’t think contemporary theory has ignored this—see recent theories of density-dependent selection here: (article making the same point), (review). The fundamental issue you’re hinging on is that absolute population growth (most effective exploitation of resources) is an ecological concept, not an evolutionary one, and population ecology theory is less well-known outside its field than population genetic theory.
My understanding was the typical explanation was antagonistic pleiotropy, but I don’t know whether that’s the consensus view.
This seems to have the name ‘pathogen control hypothesis’ in the literature—see review. I think it has all the hallmarks of a good predictive hypothesis, but I’d really want to see some simulations of which parameter scenarios induce selection this way.
I first learned about Arcadia from https://dynamicecology.wordpress.com/ blog as a “evolutionary biology” startup. When I looked they only had their fungal capsid negative result published.
I’m quite optimistic about the potential for data mining from phylogenomic inference, but I wouldn’t have described any of their current projects as “blue-sky” or “high variance” like mentioned in the post. I’m not sure that generating data, competing with large government-funded research hubs, is effective. Maybe there’s scope away from human health research areas which are overrepresented, but that’s also probably the most directly marketable area.
Does Altos Labs come under this umbrella? They seem successful and/or very good at marketing.
There’s an implied assumption that when you lose parts of society through a bottleneck that you can always recreate them with high fidelity. It seems plausible that some bottleneck events could “limit humanity’s potential”, since choices may rely on those lost values, and not all choices are exchangeable in time. (This has connections both to the long reflection and to the rich shaping the world in their own image).
As an aside, the bottleneck paper you’re referring to is pretty contentious. I personally find it unlikely that no other demographic model detects a bottleneck of >0.99 in the real data, but all of them can do it on simulated data. If such an event did occur in the modern day, the effects would be profound and serious.
Your points are true and insightful, but you’ve written them in a way that won’t gain much cachet here.
I wrote a similar piece to the Cosmos Institute competition, which hopefully I can share here when that is finished, and maybe we can bounce the idea off each other?