Talking to people in person, I’ve tried to push back on exchanging p(doom)’s on the basis of “it’s not sensible to talk about a probability that depends on one’s own actions”. And then in reply I got hit with “you really think that you personally can influence the chance that the world ends by more than a couple percent?” And I have to admit that’s a pretty good reply. I think the best one can do in response is gesture in the general direction of logical correlations between my actions and those of many other people.
Yeah. I think this is a CDT fallacy. We can influence the chance that the world ends quite a lot. “We” is made up of a bunch of “me”. Therefore, yes, I do think I can make a difference (as part of my decision group).
I would suggest asking if they vote, but a lot of people vote for non-FDT reasons (eg high expected impact even if you only have a 1/million chance of changing things).
Almost all physically instantiated algorithms aren’t logically correlated to any other physically instantiated algorithms. Plausibly logical correlation is also very rare among all algorithms period.
If you think of something like a phylogenetic tree of physical instantiations of algorithms, descending with modification, then surely algorithms on the same tree are somewhat logically correlated, and generally, the closer they are on this tree, the more logically correlated they are. It seems clear to me that a vast majority of physically instantiated algorithms[1] originate in this way, be it Darwinian-evolved or designed by intelligent creators (who are algorithms themselves[2]).
You also have the possibility of the analogue of horizontal gene transfer via communication between algorithms, which strengthens the effect.
At least those, which it makes sense to model as algorithms, i.e., modeling them as such allows you to derive some valuable implications running along the seams of reality or something.
Interesting! Maybe this changes my mind. For designed algorithms this seems more applicable than for evolved/trained ones, since in my vague current model logical correlation drops off quickly with difference in the causal structure on an algorithm. Perhaps for two non-selected algorithms: with every bit of different source code the logical correlation drops off exponentially quickly. (This doesn’t apply to algorithms that try to be logically correlated, but that is relatively rare, still).
And for the effect to matter enough the utility gain has to outweigh the exponential drop-off.
Perhaps for two non-selected algorithms: with every bit of different source code the logical correlation drops off exponentially quickly.
This doesn’t feel to me like a right way to think about this, but it may take me some time to legibilize this. Vaguely, you probably want to think about diffs on some computational abstraction layer, rather than bits.
[ETA: ok, I guess I tried legibilizing it, but I may do a better job at some later time]
(Possible that in order to have a more productive/tractable conversation about this, we’d first like to specify some desiderata/context for the idea of a logical correlation.)
One (very tentative) intuition pump: If a mutation really massively decays a lot of logical correlation (with the prior copy of the algorithm, i.e., of some computation in the organism as ~specified/conditioned by the genome), then it also means that other parts of the system that assumed this base to be as it was and not as it is, will be “surprised”, and the system fails to cohere, so the mutation is maladaptive, and thus disfavored by the selection pressure, ceteris paribus.
Something like: logical correlation is partly induced/preserved by correlated selection pressures.
(Is the “algorithm” for bonding the mother and the child in humans logically correlated with its dolphin homologue?)
[Speculation, lower confidence:]
Somewhat but not entirely separately, this might be pointing to something like “manipulability”/”wieldability”/”steerable flexibility” of the system. If you think about a generic Turing machine, you probably have an image of a weird and chaotic soup, where a random bit of the program screws up the program. But genomes are not like that. Swapping C→G has a more structured and (at least locally) predictable impact on the resulting organism (change of the amino acid[1]). And there are additional meta-adaptations, like heat shock proteins, which, in addition to making the organism better able to withstand increased temperatures, also amortize against many effects of mutations that would be much more deadly/detrimental in their absence.[2] And you also have more protected/conserved regions (like the ribosome, etc) where mutations are most likely very bad, so they don’t occur; evolution has a somewhat well-tuned bias over possible mutations.[3] Edits to high-level programming languages are even more “wieldable”.
If this is ~right, then it suggests one more reason to expect that “natural”/”empirical” instantiations of algorithms don’t decay logical correlations that easily.
From Peter Godfrey-Smith’s Darwinian Populations and Natural Selection:
Heat-shock proteins assist a cell in the production of a normal protein from a gene despite adverse temperature conditions. So they are a buffer against environmental variation. It turns out, however, that they also act as a buffer variation, selection, origins against the effects of some kinds of genetic variation. When normal heat-shock proteins are absent, in model organisms such as fruit flies or mustard weed, a range of normally invisible genetic variation is unmasked in the form of deviant phenotypes (Rutherford 2000). There are mutations that do not have dramatic consequences in the presence of heat-shock proteins, that do have dramatic consequences without them. So if we take a total fly genotype that can produce a normal-looking fly in the absence of heat-shock proteins, this genotype is surrounded by many slight variants that cannot produce a normal-looking fly in those conditions. Using the landscape metaphor, when heat-shock proteins are absent the genotypes that produce a normal fly are surrounded by many deep holes. When heat-shock proteins are present, the holes are smoothed over.
There was an episode of the Big Biology podcast where some research provided evidence that, in at least some, most likely not very unrepresentative model organisms, most mutations (I presume not in the non-coding regions) are not detrimental, but neutral, contrary to the prevailing assumptions. But that’s just one study, so beware.
I honestly think it’s even worse than this, conditional on the AI doom cases being even remotely real, because they’re substantially correlated with AI being a lot less legible and visble to the public because of internal deployment mattering way more, and it’s likely that conditional on AI progress accelerating in the next few years as much as certain people think, the number of people who get to control the probabilities of AI doom go down a lot, even in less extreme cases like no nationalization/heavy handed government involvement.
This is perhaps not an optimistic view, but it is freeing, as it means basically no-one should care about whether we all die from AI if AI doom has a significant probability (because no control/useful actions + negatives of rumination mean it’s -EV to care about AI doom.)
Talking to people in person, I’ve tried to push back on exchanging p(doom)’s on the basis of “it’s not sensible to talk about a probability that depends on one’s own actions”. And then in reply I got hit with “you really think that you personally can influence the chance that the world ends by more than a couple percent?” And I have to admit that’s a pretty good reply. I think the best one can do in response is gesture in the general direction of logical correlations between my actions and those of many other people.
Yeah. I think this is a CDT fallacy. We can influence the chance that the world ends quite a lot. “We” is made up of a bunch of “me”. Therefore, yes, I do think I can make a difference (as part of my decision group).
I would suggest asking if they vote, but a lot of people vote for non-FDT reasons (eg high expected impact even if you only have a 1/million chance of changing things).
Almost all physically instantiated algorithms aren’t logically correlated to any other physically instantiated algorithms. Plausibly logical correlation is also very rare among all algorithms period.
If you think of something like a phylogenetic tree of physical instantiations of algorithms, descending with modification, then surely algorithms on the same tree are somewhat logically correlated, and generally, the closer they are on this tree, the more logically correlated they are. It seems clear to me that a vast majority of physically instantiated algorithms[1] originate in this way, be it Darwinian-evolved or designed by intelligent creators (who are algorithms themselves[2]).
You also have the possibility of the analogue of horizontal gene transfer via communication between algorithms, which strengthens the effect.
At least those, which it makes sense to model as algorithms, i.e., modeling them as such allows you to derive some valuable implications running along the seams of reality or something.
Albeit in this case it’s less of a descent with modification thing and more of a being spawned by a very similar process thing.
Interesting! Maybe this changes my mind. For designed algorithms this seems more applicable than for evolved/trained ones, since in my vague current model logical correlation drops off quickly with difference in the causal structure on an algorithm. Perhaps for two non-selected algorithms: with every bit of different source code the logical correlation drops off exponentially quickly. (This doesn’t apply to algorithms that try to be logically correlated, but that is relatively rare, still).
And for the effect to matter enough the utility gain has to outweigh the exponential drop-off.
Are you talking about this?
This doesn’t feel to me like a right way to think about this, but it may take me some time to legibilize this. Vaguely, you probably want to think about diffs on some computational abstraction layer, rather than bits.
[ETA: ok, I guess I tried legibilizing it, but I may do a better job at some later time]
(Possible that in order to have a more productive/tractable conversation about this, we’d first like to specify some desiderata/context for the idea of a logical correlation.)
One (very tentative) intuition pump: If a mutation really massively decays a lot of logical correlation (with the prior copy of the algorithm, i.e., of some computation in the organism as ~specified/conditioned by the genome), then it also means that other parts of the system that assumed this base to be as it was and not as it is, will be “surprised”, and the system fails to cohere, so the mutation is maladaptive, and thus disfavored by the selection pressure, ceteris paribus.
Something like: logical correlation is partly induced/preserved by correlated selection pressures.
(Is the “algorithm” for bonding the mother and the child in humans logically correlated with its dolphin homologue?)
[Speculation, lower confidence:]
Somewhat but not entirely separately, this might be pointing to something like “manipulability”/”wieldability”/”steerable flexibility” of the system. If you think about a generic Turing machine, you probably have an image of a weird and chaotic soup, where a random bit of the program screws up the program. But genomes are not like that. Swapping C→G has a more structured and (at least locally) predictable impact on the resulting organism (change of the amino acid[1]). And there are additional meta-adaptations, like heat shock proteins, which, in addition to making the organism better able to withstand increased temperatures, also amortize against many effects of mutations that would be much more deadly/detrimental in their absence.[2] And you also have more protected/conserved regions (like the ribosome, etc) where mutations are most likely very bad, so they don’t occur; evolution has a somewhat well-tuned bias over possible mutations.[3] Edits to high-level programming languages are even more “wieldable”.
If this is ~right, then it suggests one more reason to expect that “natural”/”empirical” instantiations of algorithms don’t decay logical correlations that easily.
Or, perhaps even more likely (for eukaryotes), nothing, because much/most of the eukaryotic genome is non-functional DNA.
From Peter Godfrey-Smith’s Darwinian Populations and Natural Selection:
There was an episode of the Big Biology podcast where some research provided evidence that, in at least some, most likely not very unrepresentative model organisms, most mutations (I presume not in the non-coding regions) are not detrimental, but neutral, contrary to the prevailing assumptions. But that’s just one study, so beware.
I honestly think it’s even worse than this, conditional on the AI doom cases being even remotely real, because they’re substantially correlated with AI being a lot less legible and visble to the public because of internal deployment mattering way more, and it’s likely that conditional on AI progress accelerating in the next few years as much as certain people think, the number of people who get to control the probabilities of AI doom go down a lot, even in less extreme cases like no nationalization/heavy handed government involvement.
This is perhaps not an optimistic view, but it is freeing, as it means basically no-one should care about whether we all die from AI if AI doom has a significant probability (because no control/useful actions + negatives of rumination mean it’s -EV to care about AI doom.)