Previously “Lanrian” on here. Research analyst at Open Philanthropy. Views are my own.
Lukas Finnveden
Karma: 2,463
In the PDF version of the handbook, this section recommends these further resources on focusing:
Eugene Gendlin’s book Focusing is a good primer on the technique. We
particularly recommend the audiobook (76 min), as many find it easier to
try the technique while listening to the audiobook with eyes closed.
Gendlin, Eugene (1982). Focusing. Second edition, Bantam Books.
The Focusing Institute used to have an overview of the research on Focusing
on their website. Archived at:
https://web.archive.org/web/20190703145137/https://focusing.org/research-basis
Physical bottlenecks, compute bottlenecks, etc.
Compute would also be reduced within a couple of years, though, as workers at TSMC, NVIDIA, ASML and their suppliers all became much slower and less effective. (Ege does in fact think that explosive growth is likely once AIs are broadly automating human work! So he does think that more, smarter, faster labor can eventually speed up tech progress; and presumably would also expect slower humans to slow down tech progress.)
So I think the counterfactual you want to consider is one where only people doing AI R&D in particular are slowed down & made dumber. That gets at the disagreement about the importance of AI R&D, specifically, and how much labor vs. compute is contributing there.
For that question, I’m less confident about what Ege and the other mechanize people would think.
(They might say something like: “We’re only asserting that labor and compute are complementary. That means it’s totally possible that slowing down humans would slow progress a lot, but that speeding up humans wouldn’t increase the speed by a lot.” But that just raises the question of why we should think our current labor<>compute ratio is so close to the edge of where further labor speed-ups stop helping. Maybe the answer there is that they think parallel work is really good, so in the world where people were 50x slower, the AI companies would just hire 100x more people and not be too much worse off. Though I think that would massively blow up their spending on labor relative to capital, and so maybe it’d make it a weird coincidence that their current spending on labor and capital is so close to 50⁄50.)
Re your response to “Ege doesn’t expect AIs to be much smarter or faster than humans”: I’m mostly sympathetic. I see various places where I could speculate about what Ege’s objections might be. But I’m not sure how productive it is for me to try to speculate about his exact views when I don’t really buy them myself. I guess I just think that the argument you presented in this comment is somewhat complex, and I’d predict higher probability that people object (or haven’t thought about) some part of this argument then that they bite the crazy “universal human slow-down wouldn’t matter” bullet.
FWIW, that’s not the impression I get from the post / I would bet that Ege doesn’t “bite the bullet” on those claims. (If I’m understanding the claims right, it seems like it’d be super crazy to bit the bullet? If you don’t think human speed impacts the rate of technological progress, then what does? Literal calendar time? What would be the mechanism for that?)
The post does refer to how much compute AIs need to match human workers, in several places. If AIs were way smarter or faster, I think that would translate into better compute efficiency. So the impression I get from the post is just that Ege doesn’t expect AIs to be much smarter or faster than humans at the time when they first automate remote work. (And the post doesn’t talk much about what happens afterwards.)
Example claims from the post:
My expectation is that these systems will initially either be on par with or worse than the human brain at turning compute into economic value at scale, and I also don’t expect them to be much faster than humans at performing most relevant work tasks.
...
Given that AI models still remain less sample efficient than humans, these two points lead me to believe that for AI models to automate all remote work, they will initially need at least as much inference compute as the humans who currently do these remote work tasks are using.
...
These are certainly reasons to expect AI workers to become more productive than humans per FLOP spent in the long run, perhaps after most of the economy has already been automated. However, in the short run the picture looks quite different: while these advantages already exist today, they are not resulting in AI systems being far more productive than humans on a revenue generated per FLOP spent basis.
AI-enabled coups: a small group could use AI to seize power
SB1047 was mentioned separately so I assumed it was something else. Might be the other ones, thanks for the links.
lobbied against mandatory RSPs
What is this referring to?
Thanks. It still seems to me like the problem recurs. The application of Occam’s razor to questions like “will the Sun rise tomorrow?” seems more solid than e.g. random intuitions I have about how to weigh up various considerations. But the latter do still seem like a very weak version of the former. (E.g. both do rely on my intuitions; and in both cases, the domain have something in common with cases where my intuitions have worked well before, and something not-in-common.) And so it’s unclear to me what non-arbitrary standards I can use to decide whether I should let both, neither, or just the latter be “outweighed by a principle of suspending judgment”.
To be clear: The “domain” thing was just meant to be a vague gesture of the sort of thing you might want to do. (I was trying to include my impression of what eg bracketed choice is trying to do.) I definitely agree that the gesture was vague enough to also include some options that I’d think are unreasonable.
Also, my sense is that many people are making decisions based on similar intuitions as the ones you have (albeit with much less of a formal argument for how this can be represented or why it’s reasonable). In particular, my impression is that people who are are uncompelled by longtermism (despite being compelled by some type of scope-sensitive consequentialism) are often driven by an aversion to very non-robust EV-estimates.
If I were to write the case for this in my own words, it might be something like:
There are many different normative criteria we should give some weight to.
One of them is “maximizing EV according to moral theory A”.
But maximizing EV is an intuitively less appealing normative criteria when (i) it’s super unclear and non-robust what credences we ought to put on certain propositions, and (ii) the recommended decision is very different depending on what our exact credences on those propositions are.
So in such cases, as a matter of ethics, you might have the intuition that you should give less weight to “maximize EV according to moral theory A” and more weight to e.g.:
Deontic criteria that don’t use EV.
EV-maximizing according to moral theory B (where B’s recommendations are less sensitive to the propositions that are difficult to put robust credences on).
EV-maximizing within a more narrow “domain”, ignoring the effects outside of that “domain”. (Where the effects within that “domain” are less sensitive to the propositions that are difficult to put robust credences on).
I like this formulation because it seems pretty arbitrary to me where you draw the boundary between a credence that you include in your representor vs. not. (Like: What degree of justification is enough? We’ll always have the problem of induction to provide some degree of arbitrariness.) But if we put this squarely in the domain of ethics, I’m less fuzzed about this, because I’m already sympathetic to being pretty anti-realist about ethics, and there being some degree of arbitrariness in choosing what you care about. (And I certainly feel some intuitive aversion to making choices based on very non-robust credences, and it feels interesting to interpret that as an ~ethical intuition.)
Just to confirm, this means that the thing I put in quotes would probably end up being dynamically inconsistent? In order to avoid that, I need to put in an additional step of also ruling out plans that would be dominated from some constant prior perspective? (It’s a good point that these won’t be dominated from my current perspective.)
One upshot of this is that you can follow an explicitly non-(precise-)Bayesian decision procedure and still avoid dominated strategies. For example, you might explicitly specify beliefs using imprecise probabilities and make decisions using the “Dynamic Strong Maximality” rule, and still be immune to sure losses. Basically, Dynamic Strong Maximality tells you which plans are permissible given your imprecise credences, and you just pick one. And you could do this “picking” using additional substantive principles. Maybe you want to use another rule for decision-making with imprecise credences (e.g., maximin expected utility or minimax regret). Or maybe you want to account for your moral uncertainty (e.g., picking the plan that respects more deontological constraints).
Let’s say Alice have imprecise credences. Let’s say Alice follows the algorithm: “At each time-step t, I will use ‘Dynamic Strong Maximality’ to find all plans that aren’t dominated. I will pick between them using [some criteria]. Then I will take the action that plan recommends.” (And then at the next timestep t+1, you re-do everything I just said in the quotes.)
If Alice does this, does she ended up being dynamically inconsistent? (Vulnerable to dutch-books etc.)
(Maybe it varies depending on the criteria. I’m interested if you have a hunch for what the answer will be for the sort of criteria you listed: maximin expected utility, minimax regret, picking the plan that respects more deontological constraints.)
I.e., I’m interested in: If you want to use dynamic strong maximality to avoid dominated strategies, does that require you to either have the ability to commit to a plan or the inclination to consistently pick your plan from some prior epistemic perspective. (Like an “updateless” agent might.) Or do you automatically avoid dominated strategies even if you’re constantly recomputing your plan?
if the trend toward long periods of internal-only deployment continues
Have we seen such a trend so far? I would have thought the trend to date was neutral or towards shorter period of internal-only deployment.
Tbc, not really objecting to your list of reasons why this might change in the future. One thing I’d add to it is that even if calendar-time deployment delays don’t change, the gap in capabilities inside vs. outside AI companies could increase a lot if AI speeds up the pace of AI progress.
ETA: Dario Amodei says “Sonnet’s training was conducted 9-12 months ago”. He doesn’t really clarify whether he’s talking the “old” or “new” 3.5. Old and new sonnet were released in mid-June and mid-October, so 7 and 3 months ago respectively. Combining the 3 vs. 7 months options with the 9-12 months range imply 2, 5, 6, or 9 months of keeping it internal. I think for GPT-4, pretraining ended in August and it was released in March, so that’s 7 months from pre-training to release. So that’s probably on the slower side of Claude possibilities if Dario was talking about pre-training ending 9-12 months ago. But probably faster than Claude if Dario was talking about post-training finishing that early.
Taking it all together, i think you should put more probability on the software-only singluarity, mostly because of capability improvements being much more significant than you assume.
I’m confused — I thought you put significantly less probability on software-only singularity than Ryan does? (Like half?) Maybe you were using a different bound for the number of OOMs of improvement?
In practice, we’ll be able to get slightly better returns by spending some of our resources investing in speed-specific improvements and in improving productivity rather than in reducing cost. I don’t currently have a principled way to estimate this (though I expect something roughly principled can be found by looking at trading off inference compute and training compute), but maybe I think this improves the returns to around .
Interesting comparison point: Tom thought this would give a way larger boost in his old software-only singularity appendix.
When considering an “efficiency only singularity”, some different estimates gets him r~=1; r~=1.5; r~=1.6. (Where r is defined so that “for each x% increase in cumulative R&D inputs, the output metric will increase by r*x”. The condition for increasing returns is r>1.)
Whereas when including capability improvements:
I said I was 50-50 on an efficiency only singularity happening, at least temporarily. Based on these additional considerations I’m now at more like ~85% on a software only singularity. And I’d guess that initially r = ~3 (though I still think values as low as 0.5 or as high as 6 as plausible). There seem to be many strong ~independent reasons to think capability improvements would be a really huge deal compared to pure efficiency problems, and this is borne out by toy models of the dynamic.
Though note that later in the appendix he adjusts down from 85% to 65% due to some further considerations. Also, last I heard, Tom was more like 25% on software singularity. (ETA: Or maybe not? See other comments in this thread.)
Based on some guesses and some poll questions, my sense is that capabilities researchers would operate about 2.5x slower if they had 10x less compute (after adaptation)
Can you say roughly who the people surveyed were? (And if this was their raw guess or if you’ve modified it.)
I saw some polls from Daniel previously where I wasn’t sold that they were surveying people working on the most important capability improvements, so wondering if these are better.
Also, somewhat minor, but: I’m slightly concerned that surveys will overweight areas where labor is more useful relative to compute (because those areas should have disproportionately many humans working on them) and therefore be somewhat biased in the direction of labor being important.
Hm — what are the “plausible interventions” that would stop China from having >25% probability of takeover if no other country could build powerful AI? Seems like you either need to count a delay as successful prevention, or you need to have a pretty low bar for “plausible”, because it seems extremely difficult/costly to prevent China from developing powerful AI in the long run. (Where they can develop their own supply chains, put manufacturing and data centers underground, etc.)
Is there some reason for why current AI isn’t TCAI by your definition?
(I’d guess that the best way to rescue your notion it is to stipulate that the TCAIs must have >25% probability of taking over themselves. Possibly with assistance from humans, possibly by manipulating other humans who think they’re being assisted by the AIs — but ultimately the original TCAIs should be holding the power in order for it to count. That would clearly exclude current systems. But I don’t think that’s how you meant it.)
Nice scenario!
I’m confused about the ending. In particular:
I don’t get why it’s important for humans to understand the world, if they can align AIs to be fully helpful to them. Is it that:
When you refer to “the technology to control the AIs’ goals [which] arrived in time”, you’re only referring to the ability to give simple / easily measurable goals, and not more complex ones? (Such as “help me understand the pros and cons of different ways to ask ‘what would I prefer if I understood the situation better?’, and then do that” or even “please optimize for getting me lots of option-value, that I can then exercise once I understand what I want”.)
...or that humans for some reasons choose to abstain from (or are prevented from) using AIs with those types of goals?
...or that this isn’t actually about the limitations of humans, but instead a fact about the complexity of the world relative to the smartest agents in it? I.e., even if you replaced all the humans with the most superintelligent AIs that exist at the time — those AIs would still be stuck in this multipolar dilemma, not understand the world well enough to escape it, and have just as little bending power as humans.