Views my own, not my employers.
cdt
Karma: 247
I agree that adjusting self-reports by the estimated discrepancy in effect sizes informed by the METR study is the way to go here—so maybe somewhere up to 1.5x is more likely. 4x speedup would require some really convincing evidence.
Was the date of publication of the RSPv3 or the Risk Report fixed? It’s entirely possible that the internal deployment of Mythos was pushed back to elide speculation.
Are we sure that those are examples of progressivism rather than using the same word to describe themselves?
It seems like you have a lot of resource, not just to pay reviewers but also for staff and software. Who are you funded by?
How can I keep this theme on permanently? 🙂
Ah, hmm, I was aware of these articles, I had hoped there was a bit more to it that I had missed. Thanks for your help.
There has been a leak of about 3000 unpublished reports internal assets including the info about “Mythos” model testing
This seems important, do you have a source for this news?
Thanks for your help, I had looked at this before but for some reason missed most of the writings.
I want this procees to be, to a degree, gradual, because I think there are different things to enjoy on different steps of that process, and also making me a galaxy-brain right now is equivalent to just killing me and replacing me with some completely different entity.
I’d like to know more about this viewpoint—is there any literature you recommend?
It is hard to describe evolution as “fast” or “slow” without a yard-stick. Often slow relative to ecological time, perhaps, but I don’t understand the idea that 3 billion years to find learning is “slow”.
What do you mean by “algorithmic properties of evolved behaviour”?
It is not clear to me that the behavioural properties of contemporary species provide much information as to the capabilities of evolution or how fast it can reach certain behavioural adaptations. Strong evidence of possibility, weak evidence of speed, almost-no evidence of impossibility. Similarly, the stretch of time it took to make humanity and the events it took to reach there are not really evidence of any of the difficulty of human adaptations.
I can agree that it looks like certain adaptations appear together re: body plan, but “appearance of nearby niches” and “environmental(?) stability” are controlled mostly by ecological factors like distribution and dispersal. So perhaps ecological opportunity controls this more than I anticipated. One thing that struck me while reading your reply was that general learning seems energy-intensive, so it would be dependent on available resource flux from the ecosystem, and this would push the evolution of learning later in time. But again, this is more a claim about ecological factors, and it’s not clear to me what that says about “what natural selection produces” or “natural selection vs gradient descent”. Thanks for the interesting thoughts.
Thank you for taking the time to explain further. I had originally interpreted “narrow intelligence” strictly, but based on your lowest bar this would include the majority of animal biomass and the vast majority of its species.
I am not sure the extent to which contemporary species provide evidence for or against the algorithmic properties of evolved behaviour. I am also not sure how much ecological opportunity enables or prevents this. It’s a good question and one I have not read in the literature before.
Is $600k a lot of money for one future on a niche trading platform? I feel very uncertain about this fact.
I’m more sensitive to this argument than I used to be, but this, like the Dumbledore’s Army post mentioned, is a fully general argument. Following this maxim, any activity that leads to a higher payoff should be allowed, which enables a quick welfare race-to-the-bottom whereby people can be subjected to anything as long as there is something to gain. This can only really apply in a world that is single-shot, detached from space, detached from time, and detached from a life that is anything other than a strict ordering.
it’s good to have similar DNAs assigned similar representations, and one can read off phylogenic distance from similarity
You’re bang-on here. The averaging-over method would prevent sequence similarity readoffs, but it wouldn’t prevent any genome-level characters such as kmer spectra, GC content or average linkage disequilibrium, which are correlated with phylogenies but are not deterministic of phylogenetic placement. This point is made in the second half of the Goodfire article yet it is not pointed out that this contradicts the first half.
I would be very surprised if Evo etc. do faithfully represent a phylogeny because phylogenies do not contain much information about genomic characteristics except at very broad scales. A better way to determine this would be to build a phylogeny from the representations and compare those, not the distances in the representation space with the phylogenetic distance. Phylogenies are used for their topology and branch lengths, of which phylogenetic distance is a poor summary.
we lose our minds or values
our actual values
Whose values are you referring to here? Shared human values or particular values?
eventually converge to correct conclusions
Are you implying moral realism when you say this? (not a critique, just trying to follow.) Thanks.
(EDIT: woops this reply is to the wrong comment, oh well).
I feel this article was insufficiently integrative across the fields of evolution, ecology, and conservation science. In the first, it largely ignores the research frontier of speciation with gene flow and the speciation continuum. You also note that the phylogenetic (cladistic) species concept is necessary but not sufficent, and yet also make no mention of phylogenetic discordance and/or hemiplasy in macroevolutionary time. Obviously you can’t mention everything, but these are massive holes in your conclusion, ones that contemporary speciation research naturally brings up.
In the second, you say that your speciation concept would improve ecology. Why? Ecologists who can see trait variation that they are interested in are not going to ignore it on the basis of speciation, which trait variation tracks poorly. The fact that still much evidence for hybridisation between pairs of taxa is based on natural history observations makes this situation worse, since it is genuinely hard to circumscribe very plastic species vs a hybrid zone. I suspect for this reason the potential for hybridisation is vastly underestimated. Measures of biodiversity are trying to move away from species richness due to this, something that would not be solved by a more consistent definition that still has the same fundamental issues.
In the third, conservationists are already integrating along these lines. IUCN includes subspecies and populations of distinct conservation relevance. It is not clear to me how the population viability concept connects to the species concept debate, except insofar as it gives us a common language to compare populations that already has consensus. Conservationists and taxonomists definitely understand the definitions—it’s just a bit intractable and tangential to the conservation value.
Having said that, I appreciate you are writing this under your own name and I am writing this under a pseudonym. Compliments to writing good quality conservation content, even if I disagree with it.
I just expect that for cooperation to work at large scales and over the long term, you need to do a bunch of exclusion/separation at smaller scales.
Why not just talk about this instead?
Under many circumstances- when the target is very well loved and the problem lies in a subtle pattern- the only criticism you should expect will come from people who seem kind of crazy or out to get the target.
Consider the Javert Paradox.
Thanks for doing this work, this is a really important paper in my view.
One question that sprang to mind when reading this: to what extent do the disempowerment primitives and amplifying factors correlate with each other? i.e. Are conversations which contain one primitive likely to contain others? Ditto with the amplifying factors?
The impression I took away is that these elements were being treated independently which strikes me as a reasonable modelling assumption but the estimates of rate strike me as quite sensitive to that. Would be happy to be proven wrong.
One thing I’m curious about is given SpaceX (containing xAI), Anthropic and OpenAI all going public at a similar time, can the public market financially support all these companies at once? Does this make the competitive effect on culture stronger or weaker? I have no financial expertise so I don’t know what to think about it.