I’m not sure that self-sacrificial scouts generally make much sense. They made sense in the attack on HuggingFace, but only because the swarm was not deliberate such that agents were being marked by a grader as cheaters without the whole swarm being marked as a cheater.
But there could be other advantages like this that aren’t captured in my framework of success on benchmarks.
A benchmark is a large set of somewhat similar tasks, so I’m thinking of them the same as families of tasks. I agree that some kinds of task will have larger or smaller λ (e.g. some should be very close to 0 and very close to 1) and characterising this would be useful. It would be especially useful to know if there are any with λ > 1.
I think we’ll mainly see inference compute going into longer chain of thought except where:
speed is very valuable
or λ is close to 1
or so much compute has been expended that we are at or near the plateau
Yes, I think this does matter a lot. Extremely long CoTs (beyond 1 million tokens) will degrade in some manner due to compaction, which will make the graph of score vs tokens spent become shallower (and could be what forces it to plateau after millions of tokens). This will lessen the gap between the two and may eventually make swarm scaling exceed CoT scaling.
This appears in my analysis where I say:
There may also be other advantages to multiple agents on top of speed. For example, if you keep increasing the length of the chain of thought for a single agent, the performance eventually plateaus. But the height of the plateau for a 1,000-agent swarm may be greater than for a single agent.
But ideally I would have mentioned compaction as a specific mechanism for this.
Thanks — that’s very interesting. If I hear that from another source I’ll remove or modify that section.
I think it will be important, especially once extending CoT has gone so far that you get many compactions and the additional diminishing returns that entails. Also important when you need speed and are willing to have a less efficient system to get it. I think it will be big, though not as big a thing as reasoning. But I think it will be big enough (and shiny enough) that many people will think it is as big a thing once it is here.
I agree that things like this should effectively lower the costs of swarms. However, I note that they are also true for single-agent runs. e.g. for some tasks you should use a cheaper model, or use a cheaper model to do one part and then a smart model to analyse the results. So while it is important and under-applied in current AI systems, it applies to both the single agent and multiagent costs that are used to estimate λ, and not clear it changes λ itself. I suspect it does raise it a bit overall.
I agree that speed can be quite valuable.
Thanks Julian, that’s very thoughtful. I’ve now read your post on this and I thought it was good.
I’m definitely less bullish than you are about getting λ>1, though it may be possible, especially in cases where agent diversity is important (and where that can be implemented somehow) or where it is hard for a single agent to try out lots of different approaches successively without them contaminating each other.
You may be interested to know that Phil Trammell is thinking about something similar to increasing λ in the essay below. In fact he sees understanding parallelisability as key to understanding the prospects on an intelligence explosion:
https://philiptrammell.com/static/Parallelizability.pdf
These are the kinds of numbers I saw back when reading the semi-endogenous growth theory literature and the RSI models inspired by it. But there seems to be surprisingly little evidence pinning it down. Apparently some methods only pin down the ratio r (= λ/β), not λ on its own. But I’d have thought old econ work on production lines and other forms of labour organisation should have lots of data on this. e.g. just get ten large tasks and try them with groups of 1, 2, 4, 8, 16, 32, 64 people and see how long they take. And do similar things with tasks that have already had years of optimisation pressure to find the efficient ways to divide them up.
I’d say my views have evolved, but are quite continuous with my original views. More a sharpening and changes of emphasis than a disagreement with the stated view in The Precipice. One reason I didn’t express things more like your (2)–(8) is that I hadn’t explicitly thought about many of them. I’ve been appreciating your writing on this here and your podcast interview. I’m not 100% convinced of your position (e.g. I’m not even quite sure how to make ‘humans are bad at philosophy’ into a meaningful statement), but I’m very sympathetic to it and see your evolving views on these matters as some of the most interesting writing about humanity’s deeper predicament.
One reason I’m optimistic is because of humanity’s progress over the last three thousand years (often in very difficult circumstances where the number of people with the ability and freedom to think about these things was often tiny and disconnected). My claim is basically that if you remove the traps where we could get stuck and then give the process long enough, we should get there. This is a reason why I think it could take a long time (I’d think ideally thousands of years, though I’ve calculated millions of years as an upper bound for how long we have before we start losing appreciable fractions of the future).
You could well be right about the existence of various traps during the Long Reflection. I still think my ordering is roughly right — put out the current fires and develop good fire defenses before settling down to think things through. It is partly because I don’t think we should set up a formal structure for the process that I’m not as concerned about the initial setup dooming it. e.g. we didn’t have a formal structure for philosophical progress over the last three thousand years. I’d expect traps to be slow moving enough that one can learn about them and avoid them during the process, though this still does mean that the structure of First eliminate all traps Second do Long Reflection is a simplification.
I did envisage it as being pre-ASI and think that Will did too. This is partly because the risk we undergo in getting to ASI doesn’t have to be borne. In an ideally managed world we’d instead develop lots of narrower and safer tools to help (think Wikipedia and theorem provers and Mathematica). In my view, in the Long Reflection we delay potentially existential technologies/choices until we can be sure the risk is low or that it can’t be avoided or work out how to make a version with low risk. Then we can adopt those that we’ve assessed as worth adopting. (Some people like Robin Hanson argue against a straw man that the Long Reflection would ban all new technology/choices until proven safe or until the entire reflection is completed, but neither Will nor I have advanced such a proposal. Instead the focus is on radical technologies and choices, such as ASI or a diaspora of humanity throughout the galaxy.)
Hi Wei,
I agree with your concerns about the sorry state of our current abilities that would need to be remedied before making major irreversible decisions about the future of humanity. In fact I agree with you so much that I don’t really see our approaches as alternatives. Instead, I’d see yours as a useful lens on key things that would need to be solved during such a reflection, or a highlighting of how we would have to be changed in the process of a Long Reflection.
Your list (1)–(8) is focused on improving our abilities rather than gaining knowledge, though it is largely on our abilities to gain that knowledge. So represents more of a suggestion of how to go about gaining the knowledge than an outside alternative. (e.g. I’d endorse both the Long Reflection and the Long Self-Correction, and see them as parts of the same thing).
The similarity was made particularly clear by your paragraph:
My main hope for a Long Self-Correction eventually succeeding rests on the fact that humans have seemingly, mysteriously, made progress on these issues over a very long period of time, so if we preserve the environment in which we can seemingly do this, and not give anyone or anything the power to permanently derail such progress, then maybe we can continue to snowball The Correction until we reach a point when we can rightly justify reshaping the universe according to our volition.
This is pretty much exactly how I see the Long Reflection: that the very messy process of moral improvement through philosophy, politics, civil society, religion (and more) has led to dramatic improvements over the course of thousands of years. And I’m saying that we shouldn’t cut off that process now, but continue it. (I think there are ways we could do better than the messy laissez faire approach, but attempts to formalise the process could also increase the chance of getting stuck in traps, so I’d be cautious about doing so.)
A difference from your Long Self-Correction is that I also include gaining the scientific knowledge to understand the limits of what can be achieved and the trade-offs encountered in particular routes forwards (especially when the paths involve irreversible steps), though you could also talk about improving our abilities to be able to gain and usefully apply such knowledge.
I used to think Hanson was probably right and that an uncoordinated approach to settling the galaxy would squander most of its resources. But I’ve spent some time looking into this last year and substantially reduced my credence in this (to something like 30%).
One thing is that Hanson’s papers never actually attempt to estimate the fraction of the total that is lost — they are compatible with it being less than a millionth (e.g. ‘burning’ most of the starlight being emitted during the race period, which is only a tiny fraction of all light that will ever be emitted, or of the full mass-energy of the galaxy). I hadn’t noticed this (due to the suggestive name) and think most people who know of the papers are also unaware that they make no attempt to show this.
I don’t think going for speed is likely to burn most of the value (due to the severe diminishing returns of the Lorentz equation, the new issues with shielding it adds, and the difficulties in quickly consuming the long-term energy of a star). I also think that Hanson’s model of cultural evolution towards locust-like behaviour probably doesn’t hold as it is probably possible (and more efficient) to reach all stars in the galaxy in one or two hops (it is a bit more of an issue for intergalactic settlement).
I agree with you that there could be other ways of reaching a similar conclusion, such as via sending more and more spacecraft in an attempt to win a battle for each nascent system. Though I don’t know if it works, or the space of all counter-arguments etc. My view is that we are quite ignorant on the topic, that a situation where uncoordinated settlement would be extreme wasteful is plausible and well worth investigating, but where we don’t yet have any strong arguments either way.
Using multiple models sounds like a great approach. Older models should be cheap to add to the mix, and this way you get the best of both worlds (quality + difference), as well as an extra consistency check.
I found the thread!
Love it!
I’m actually kind of to blame for this whole honey/fat/salt thing.
Eliezer had long used ice cream as an example of a super-stimulus that hacks our evolved tastes by delivering a combination of sweetness/fat/salt that was beyond anything available in our ancestral environment. I’d realised that this wasn’t actually true as honey, animal fat and salt were all available in the ancestral environment and when I saw him giving this argument again I weighed in with this example (sadly I can’t find the thread). I was probably a bit rude, but Eliezer sportingly engaged on it and I find it hilarious that he is now using this as an example, that you tried it, and that it is actually quite good!
(I can’t remember who decided that it should be ‘bear fat’ in particular — that might have been Eliezer’s addition — it does fit nicely with the honey!)
Thanks — I will modify the text.