Views my own, not my employers.
cdt
Karma: 252
Just throwing this out there as an idle thought when reading this post* - I wonder if the reason linguistic phylogenetics and biological phylogenetics differ is that the phylogeny and the biological traits of the underlying species are often correlated with each other. I don’t know if that’s true for linguistics, and indeed I’m not sure what traits you are capable of identifying in a text or language that would affect the phylogenetic reconstruction itself.
Thanks for this info—very useful datapoint, I have strong-upvoted.
I think you are right that “humans that want things” will always have more things than “humans that don’t want things” (or even replace human with X, see this talk by Joe Carlsmith). You must be much more explicit drawing a line between this and “post-scarcity”—which frame of post-scarcity is it that you are disagreeing with here? You seem half to be making an argument about the inherent nature of competition, and half an argument about the desire of humans for status, which leads to an argument about neither.
One thing I’m curious about is given SpaceX (containing xAI), Anthropic and OpenAI all going public at a similar time, can the public market financially support all these companies at once? Does this make the competitive effect on culture stronger or weaker? I have no financial expertise so I don’t know what to think about it.
I agree that adjusting self-reports by the estimated discrepancy in effect sizes informed by the METR study is the way to go here—so maybe somewhere up to 1.5x is more likely. 4x speedup would require some really convincing evidence.
Was the date of publication of the RSPv3 or the Risk Report fixed? It’s entirely possible that the internal deployment of Mythos was pushed back to elide speculation.
Are we sure that those are examples of progressivism rather than using the same word to describe themselves?
It seems like you have a lot of resource, not just to pay reviewers but also for staff and software. Who are you funded by?
How can I keep this theme on permanently? 🙂
Ah, hmm, I was aware of these articles, I had hoped there was a bit more to it that I had missed. Thanks for your help.
There has been a leak of about 3000 unpublished reports internal assets including the info about “Mythos” model testing
This seems important, do you have a source for this news?
Thanks for your help, I had looked at this before but for some reason missed most of the writings.
I want this procees to be, to a degree, gradual, because I think there are different things to enjoy on different steps of that process, and also making me a galaxy-brain right now is equivalent to just killing me and replacing me with some completely different entity.
I’d like to know more about this viewpoint—is there any literature you recommend?
It is hard to describe evolution as “fast” or “slow” without a yard-stick. Often slow relative to ecological time, perhaps, but I don’t understand the idea that 3 billion years to find learning is “slow”.
What do you mean by “algorithmic properties of evolved behaviour”?
It is not clear to me that the behavioural properties of contemporary species provide much information as to the capabilities of evolution or how fast it can reach certain behavioural adaptations. Strong evidence of possibility, weak evidence of speed, almost-no evidence of impossibility. Similarly, the stretch of time it took to make humanity and the events it took to reach there are not really evidence of any of the difficulty of human adaptations.
I can agree that it looks like certain adaptations appear together re: body plan, but “appearance of nearby niches” and “environmental(?) stability” are controlled mostly by ecological factors like distribution and dispersal. So perhaps ecological opportunity controls this more than I anticipated. One thing that struck me while reading your reply was that general learning seems energy-intensive, so it would be dependent on available resource flux from the ecosystem, and this would push the evolution of learning later in time. But again, this is more a claim about ecological factors, and it’s not clear to me what that says about “what natural selection produces” or “natural selection vs gradient descent”. Thanks for the interesting thoughts.
Thank you for taking the time to explain further. I had originally interpreted “narrow intelligence” strictly, but based on your lowest bar this would include the majority of animal biomass and the vast majority of its species.
I am not sure the extent to which contemporary species provide evidence for or against the algorithmic properties of evolved behaviour. I am also not sure how much ecological opportunity enables or prevents this. It’s a good question and one I have not read in the literature before.
Is $600k a lot of money for one future on a niche trading platform? I feel very uncertain about this fact.
I’m more sensitive to this argument than I used to be, but this, like the Dumbledore’s Army post mentioned, is a fully general argument. Following this maxim, any activity that leads to a higher payoff should be allowed, which enables a quick welfare race-to-the-bottom whereby people can be subjected to anything as long as there is something to gain. This can only really apply in a world that is single-shot, detached from space, detached from time, and detached from a life that is anything other than a strict ordering.
it’s good to have similar DNAs assigned similar representations, and one can read off phylogenic distance from similarity
You’re bang-on here. The averaging-over method would prevent sequence similarity readoffs, but it wouldn’t prevent any genome-level characters such as kmer spectra, GC content or average linkage disequilibrium, which are correlated with phylogenies but are not deterministic of phylogenetic placement. This point is made in the second half of the Goodfire article yet it is not pointed out that this contradicts the first half.
I would be very surprised if Evo etc. do faithfully represent a phylogeny because phylogenies do not contain much information about genomic characteristics except at very broad scales. A better way to determine this would be to build a phylogeny from the representations and compare those, not the distances in the representation space with the phylogenetic distance. Phylogenies are used for their topology and branch lengths, of which phylogenetic distance is a poor summary.
we lose our minds or values
our actual values
Whose values are you referring to here? Shared human values or particular values?
eventually converge to correct conclusions
Are you implying moral realism when you say this? (not a critique, just trying to follow.) Thanks.
(EDIT: woops this reply is to the wrong comment, oh well).
Ah I see I have been a little sloppy with my language—mea culpa
The extent to which traits and phylogenies are correlated is an open research question, see this wikipedia article. But aspects of biology that are unique to phylogenies such as diversification interact with traits in complex ways. The SSE models are a good introduction to the methodology of this (background here).
I don’t know how to attach these ideas to linguistics because I can’t think of a good concrete example.
As an aside, you also say that
But this is very well understood in phylogenetics. This is the basis of codon models and the “maximum likelihood school” of phylogenetic modelling. You can see this by looking at IQ-TREE (a modern phylogenetic inference tool): https://iqtree.github.io/doc/Substitution-Models