Michaël Trazzi
Note: I updated the parent comment to take into account interest rates.
In general, the way to mitigate trust would be to use an escrow, though when betting on doom-ish scenarios there would be little benefits in having $1000 in escrow if I “win”.
For anyone reading this who also thinks that it would need to be >$2000 to be worth it, I am happy to give $2985 at the end of 2032, aka an additional 10% to the average annual return of the S&P 500 (ie 1.1 * (1.105^10 * 1000)), if that sounds less risky than the SPY ETF bet.
For anyone of those (supposedly) > 50% respondents claiming a < 10% probability, I am happy to take 1:10 odds $1000 bet for:
“by the end of 2032, fewer than a million humans are alive on the surface of the earth, primarily as a result of AI systems not doing/optimizing what the people deploying them wanted/intended”
Where, similar to Bryan Caplan’s bet with Yudwkosky, I get paid like $1000 now, and at the end of 2032 I give them back, adding 100 dollars.(Given inflation and interest, this seems like a bad deal for the one giving the money now, though I find it hard to predict 10y inflation and I do not want to have extra pressure to invest those $1000 for 10y. If someone has another deal in mind that would sound more interesting, do let me know here or by DM).To make the bet fair, the size of the bet would be the equivalent of the value in 2032 of $1000 worth in SPY ETF bought today (400.09 at May 16 close). And to mitigate the issue of not being around to receive the money, I would receive a payment of $1000 now. If I lose I give back whatever $1000 of SPY ETF from today is worth in 2032, adding 10% to that value.
Thanks for the survey. Few nitpicks:
- the survey you mention is ~1y old (May 3-May 26 2021). I would expect those researchers to have updated from the scaling laws trend continuing with Chinchilla, PaLM, Gato, etc. (Metaculus at least did update significantly, though one could argue that people taking the survey at CHAI, FHI, DeepMind etc. would be less surprised by the recent progress.)- I would prefer the question to mention “1M humans alive on the surface on the earth” to avoid people surviving inside “mine shafts” or on Mars/the Moon (similar to the Bryan Caplan / Yudkowsky bet).
I can’t see this path leading to high existential risk in the next decade or so.
Here is my write-up for a reference class of paths that could lead to high existential risk this decade. I think such paths are not hard to come up with and I am happy to pay a bounty of $100 for someone else to sit for one hour and come up with another story for another reference class (you can send me a DM).
Even if the Tool AIs are not dangerous by itself, they will foster productivity. (You say it yourself: “These specialized models will be oriented towards augmenting human productivity”). There is already a many more people working in AI than in the 2010s, and those people are much more productive. This trend will accelerate, because AI benefits compound (eg. using Copilot to write the next Copilot) and the more ML applications automate the economy, the more investments in AI we will observe.
from the lesswrong docs
An Artificial general intelligence, or AGI, is a machine capable of behaving intelligently over many domains. The term can be taken as a contrast to narrow AI, systems that do things that would be considered intelligent if a human were doing them, but that lack the sort of general, flexible learning ability that would let them tackle entirely new domains. Though modern computers have drastically more ability to calculate than humans, this does not mean that they are generally intelligent, as they have little ability to invent new problem-solving techniques, and their abilities are targeted in narrow domains.
If we consider only the first sentence, then yes. The rest of the paragraph points to something like “being able to generalize to new domains”. Not sure if Gato counts. (NB: this is just a LW tag, not a full-fledged definition.)
Ethan Caballero on Private Scaling Progress
the two first are about data, and as far as I know compilers do not use machine learning on data.
third one could technically apply to compilers, though I think in ML there is a feedback loop “impressive performance → investments in scaling → more research”, but you cannot just throw more compute to increase compiler performance (and results are less in the mainstream, less of a public PR thing)
Well, I agree that if two worlds I had in mind were 1) foom without real AI progress beforehand 2) continuous progress, then seeing more continuous progress from increased investments should indeed update me towards 2).
The key parameter here is substitutability between capital and labor. In what sense is Human Labor the bottleneck, or is Capital the bottleneck. From the different growth trajectories and substitutability equations you can infer different growth trajectories. (For a paper / video on this see the last paragraph here).
The world in which dalle-2 happens and people start using Github Copilot looks to me like a world where human labour is substitutable by AI labour, which right now is essentially being part of Github Copilot open beta, but in the future might look like capital (paying the product or investing in building the technology yourself). My intuition right now is that big companies are more bottlenecked by ML talent than by capital (cf. the “are we in ai overhang” post explaining how much more capital could Google invest in AI).
Thanks for the pointer. Any specific section / sub-section I should look into?
I agree that we are already in this regime. In the section “AI Helping Humans with AI” I tried to make it more precise at what threshold we would see substantial change in how humans interact with AI to build more advanced AI systems. Essentially, it will be when most people would use those tools most of their time (like on a daily basis) and they would observe some substantial gains of productivity (like using some oracle to make a lot of progress on a problem they are stuck on, or Copilot auto-completing a lot of their lines of code without having to manually edit.) The intuition for a threshold is “most people would need to use”.
Re diminishing returns: see my other comment. In summary, if you just consider one team building AIHHAI, they would get more data and research as input from the outside world, and they would get increases in productivity from using more capable AIHHAIs. Diminishing returns could happen if: 1) scaling laws for coding AI do not hold anymore 2) we are not able to gather coding data (or do other tricks like data augmentation) at a pace high enough 3) investments for some reasons do not follow 4) there are some hardware bottlenecks in building larger and larger infrastructures. For now I have only seen evidence for 2) and this seems something that can be solved via transfer learning or new ML research.
Better modeling of those different interactions between AI labor and AI capability tech are definitely needed. For some high-level picture that mostly thinks about substitutability between capital and labor, applying to AI, I would recommend this paper (or video and slides). The equation that is the closest to self-improving {H,AI} would be this one.
Some arguments for why that might be the case:
-- the more useful it is, the more people use it, the more telemetry data the model has access to
-- while scaling laws do not exhibit diminishing returns from scaling, most of the development time would be on things like infrastructure, data collection and training, rather than aiming for additional performance
-- the higher the performance, the more people get interested in the field and the more research there is publicly accessible to improve performance by just implementing what is in the litterature (Note: this argument does not apply for reasons why one company could just make a lot of progress without ever sharing any of their progress.)
Why Copilot Accelerates Timelines
When logging to the AF using github I get an Internal Server Error (callback)
Great quotes. Posting podcast excerpts is underappreciated. Happy to read more of them.
The straightforward argument goes like this:
1. an human-level AGI would be running on hardware making human constraints in memory or speed mostly go away by ~10 orders of magnitude
2. if you could store 10 orders of magnitude more information and read 10 orders of magnitude faster, and if you were able to copy your own code somewhere else, and the kind of AI research and code generation tools available online were good enough to have created you, wouldn’t you be able to FOOM?
Code generation is the example I use to convince all of my software or AI friends of the likelihood of AI risk.
most of them have heard of it or even tried it (eg. via Github Copilot)
they all recognize at directly useful for their work
it’s easier to understand labour automation when it applies to your own job
fast takeoff folks believe that we will only need a minimal seed AI that is capable of rewriting its source code, and recursively self-improving into superintelligence
Speaking only for myself, the minimal seed AI is a strawman of why I believe in “fast takeoff”. In the list of benchmarks you mentioned in your bet, I think APPS is one of the most important.
I think the “self-improving” part will come from the system “AI Researchers + code synthesis model” with a direct feedback loop (modulo enough hardware), cf. here. That’s the self-improving superintelligence.
I haven’t look deeply at what the % on the ML benchmarks actually mean. On the one hand it would be a bit weird to me if in 2030 we still have not made enough progress on them, given the current rate. On the other hand, I trust the authors in that it should be AGI-ish to pass those benchmarks, and then I don’t want to bet money on something far into the future if money might not matter as much then. (Also, without considering money mattering less or the fact that the money might not be delivered in 2030 etc., I think anyone taking the 2026 bet should take the 2030 bet since if you’re 50⁄50 in 2026 you’re probably 75:25 with 4 extra years).
The more rational thing should then to take the bet for 2026 when money still matters, though apart from the ML benchmarks there is this dishwashing thing where the conditions of the bet are super tough and I don’t imagine anyone doing all the reliability tests filming a dishwasher etc. in 3.5y. And then for Tesla I feel the same about those big errors every 100k miles. Like, 1) why only Tesla? 2) wouldn’tmost humans would make risky blunders on such long distances? 3) would anyone really do all those tests on a Tesla?
I’ll have another look at the ML benchmarks, but on the mean time it seems that we should do other odds because of Tesla + dishwasher.
I found the concept of flailing and becoming what works useful.
I think the world will be saved by a diverse group of people. Some will be high integrity groups, other will be playful intellectuals, but the most important ones (that I think we currently need the most) will lead, take risks, explore new strategies.
In that regard, I believe we need more posts like lc’s containment strategy one or the other about pulling the fire alarm for AGI. Even if those plans are different than the ones the community has tried so far. Integrity alone will not save the world. A more diverse portfolio might.