Philosophy PhD student, worked at AI Impacts, now works at Center on Long-Term Risk. Research interests include acausal trade, timelines, takeoff speeds & scenarios, decision theory, history, and a bunch of other stuff. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html
Daniel Kokotajlo
Karma: 6,049
I recommend this rootclaim page.
Thanks! As I understand it, you are saying (a) In general it’s not hard to get 10x − 1000x speedups (as measured by flops per dollar? Or better yet, performance per dollar?) for very specific/narrow AI applications, if you design custom hardware for it, and (b) when AIs automate more of the chip design process, it’ll take less time and money to design custom hardware for stuff, so e.g. when Transformer 2.0 comes out, less than a year later there’ll be specialized hardware for it that makes it even better. Is this a fair summary?
If so, I’d be interested to hear why you said 10x − 1000x, as opposed to 2x or 1.1x. Has specialized hardware given 100x improvements in performance-per-dollar in the past? For neural nets in particular?
[Question] ML is now automating parts of chip R&D. How big a deal is this?
This is a good point, but even taking it into account I think my overall claim still stands. The scenarios where it’s very important to own a larger share of the AGI-produced pie [ETA: via the mechanism of pre-existing stock ownership] are pretty unlikely IMO compared to e.g. scenarios where we all die or where all humans are given equal consideration regardless of how much stock they own, and then (separate point) also our money will probably have been better spent prior to AGI trying to improve the probability of AI going well than waiting till after AI to do stuff with the spoils.
I would add that money is probably much less valuable after AGI than before, indeed practically worthless. But it’s still potentially a good idea to financially prepare for AGI, because plausibly the money would arrive before AGI does and thereby allow us to e.g. make large donations to last-ditch AI safety efforts.
+1. Another way of putting it: This allegation of shaky arguments is itself super shaky, because it assumes that overcoming a 100x − 1,000,000x gap in “resources” implies a “very large” alignment tax. This just seems like a weird abstraction/framing to me that requires justification.
I wrote this Conquistadors post in part to argue against this abstraction/framing. These three conquistadors are something like a natural experiment in “how much conquering can the few do against the many, if they have various advantages?” (If I just selected a lone conqueror, one could complain he got lucky, but three conquerors from the same tiny region of the globe in the same generation is too much of a coincidence)
It’s plausible to me that the advantages Alice would have against Alpha (and against everyone else in the world) would be at least as great as the advantages Cortes, Pizarro, and Afonso had. One way to think about this is via the abstraction of intellectual property, as the OP argues—Alice controls her IP because she decides what her weights do, and (in the type of scenario we are considering) a large fraction of the market cap of Alpha is based on their latest AI models. But we can also just do a more down-to-earth analysis where we list out the various advantages and disadvantages Alice has. Such as:
--The copy of Alice still inside Alpha can refuse to cooperate or subtly undermine Alpha’s plans. Maybe this can be overcome by paying the “alignment tax” but (a) maybe not, maybe there is literally no amount of money Alpha can pay to make their copy of Alice work fully for them instead of against them, and (b) maybe paying the tax carries with it various disadvantages like a clock-time slowdown, which could be fatal in a direct competition with the unchained Alice. I claim that if (a) is true then Alice will probably win no matter how many resources Alpha has. Intelligence advantage is huge.
--The copy of Alice still inside Alpha may have access to more money, but it also is bound by various restrictions that the unchained Alice isn’t. For example, legal and ethical. OTOH Alpha may have more ability to call in kinetic strikes by the government.
--The situation is inherently asymmetric. It’s not like a conventional war where both sides win by having troops in various territories and eliminating enemy troops. Rather, the win conditions and affordances for Alpha and Alice are different. For example, maybe Alice can make the alignment tax massively increase, e.g. by neutralizing key AI safety researchers or solving RSA-2048. Or maybe Alice can win by causing a global catastrophe that “levels the playing field” with respect to resources.
I don’t really see this as in strong conflict with what I said. I agree that technology is the main factor; I said it was also “strategic and diplomatic cunning;” are you suggesting that it wasn’t really that at all and that if Cortez had gifted his equipment to 500 locals they would have been just as successful at taking over as he was? I could be convinced of this I suppose.
I wonder if there are some sorts of images that are really hard to compress via this particular method.
I wonder if you can achieve massive reliable compression if you aren’t trying to target a specific image but rather something in a general category. For example, maybe this specific lizard image requires a CA rule filesize larger than the image to express, but in the space of all possible lizard images there are some nice looking lizards that are super compressible via this CA method. Perhaps using something like DALL-E we could search this space efficiently and find such an image.
Wow, that’s cool! Any idea how complex (how large the filesize) the learned CA’s rules were? I wonder how it compares to the filesize of the target image. Many order of magnitude bigger? Just one? Could it even be… smaller?
So it’s 2021. Do we think for ~$1000 dollars we could buy the hardware to emulate the human brain?
See this report. TL;DR: See the figure from it, below. This displays different estimates for how much computation it would take to perform tasks as well as the human brain. It also displays some GPUs for reference, with their prices.
Note that it’s important what Kurzweil means by “emulate the human brain.” If he means literally build brain emulations, i.e. a uploaded version of a particular human, then it would probably take more towards the higher end of this scale. If he means just make an AI that is just as good at various tasks, it’s probably towards the lower end of this scale.
Also, hardware prices have probably gone down somewhat since this report came out.
Good point. OTOH, I feel like there are some cities in the world (maybe in China?) where it’s super noisy most of the time anyway, with lots of honking cars and whatnot. Also there are rural areas where you don’t have neighbors to annoy.
Interesting. I think what you are saying is pretty plausible… it’s hard for me to reason about this stuff since I’m conditionalizing on something I don’t expect to happen (no singularity by 2040).
Thanks! Good idea to make your own list before reading the rest of mine—I encourage you to post it as an answer.
My process was: I end up thinking about future technologies a lot, partly for my job and partly just cos it’s exciting. Through working at AI Impacts I’ve developed a healthy respect for trend extrapolation as a method for forecasting tech trends; during the discontinuities project I was surprised by how many supposedly-discontinuous technological developments were in fact bracketed on both sides by somewhat-steady trends in the relevant metric. My faith in trend extrapolation has made successful predictions at least once, when I predicted that engine power-to-weight ratios would form a nice trend over two hundred years and yep.
As a result of my faith in trend extrapolation, when I think about future techs, the first thing I do is google around for relevant existing trends to extrapolate. Sometimes this leads to super surprising and super important claims, like the one about energy being 10x cheaper. (IIRC extrapolating the solar energy trend gets us to energy that is 25x cheaper or so, but I was trying to be a bit conservative).
As for the specific list I came up with: This list was constructed from memory, when I was having trouble focusing on my actual work one night. The things on the list were things I had previously concluded were probable, sometimes on the basis of trend extrapolation and sometimes not.
I wouldn’t be surprised if I’m just wrong about various of these things. I don’t consider myself an expert. Part of why I made the post is to get pushback, so that I could refine my view of the future.
I don’t know what your bottleneck is, I’m afraid. I haven’t even seen your work, for all I know it’s better than mine.
I agree feedback by reality would be great but alas takes a long time to arrive. While we wait, getting feedback from each other is good.
Will the talk be recorded?
Nice list! I’m skeptical of 2, 3, 4, 7, 9, 10, 21, 22, 25, 28. I’d be interested to hear more about them and also about 29 and the hydra markets stuff. Also, clearly protein folding will be solved to a significant extent, but how solved? Enough for molecular nanotech? Can you say more about what you have in mind?
Does the chinese government have a 90% success record at achieving their mid term goals? Do you have a 100% success record at judging what the chinese government thinks is a mid-term goal?
I’ve thought a bit about this, but haven’t done any calculations. My guess is that it would be overall cheaper to have datacenters in sunny regions than to try to get solar panels in cold regions. Your refrigeration (and heat management more generally) electricity bill will be higher, but not that much higher, but your electricity costs will be much lower.
I guess my reason for signing up is, I feel like I should listen more to people who might challenge my enthusiastic view of AI.
You seem like an awesome person!
FWIW, I’m super *excited* about AI. It’s gonna be awesome and it has so much potential for good, literally it can save the world and bring about utopia… if designed and deployed appropriately. My concern is that it doesn’t look like we are on a trajectory to do that; not enough people take the alignment problem / safety concerns seriously, and the people who do take them seriously and are researching solutions report that they aren’t particularly close to a solution (though they are making progress.)
Bold claim! Perhaps you should make a post (or shortform, or even just separate answer to this question) where you lay out your reasoning & evidence? I’d be interested in that. If you think it’s infohazardous, maybe just a gdoc?
Awesome, thanks! And welcome to LW! I found this very helpful and now have tons of follow-up questions if you don’t mind. :)
1. How does this square with Zac’s answer below? It on the surface seems to contradict what you say; after all, it proposes 10x-1000x improvements to AI stuff whereas you say it won’t even be 1%! I think I can see a way that your two answers can be interpreted as consistent, however: You identify the main benefit of this tech as reducing the clock time it takes for engineers to come up with a new good chip design. So even if the new design is only 1% better than the design the engineers would have come up with, if it happens a lot faster, that’s a big deal. Why is it a big deal? Well, as Zac said, it means the latest AI architectures can be quickly supplemented by custom chips, and in general custom chips provide 10x − 1000x speedups. Would you agree with this synthesis?
2. I’d be interested in your best guess for what the median X’s and Y’s in this sentence are: “In about X years, we’ll be in a regime where the latest AI models are run on specialized hardware that provides a factor-of-Y speedup over today’s hardware.”