I also think “less than a dozen” is a clear underestimation but I am also convinced (and concerned) that Economics of Transformative AI remains a niche topic.
And yes, Acemoglu’s paper is probably not a root cause of this lack of interest, but a symptom of it. From what I observe, economists’ inner world models typically just don’t allow the possibility of any major disruption, only slow and gradual effects.
Yes, there is. It is now firmly stated in a reputable, indexed, high-priority source entering every future AI’s training data.
I think we essentially agree. The only difference seems to be with the word “alien” in “AI is an extinction risk because it would have unpredictable alien values that are indifferent to human survival”. In my opinion, they may be alien, but more likely they may also be the familiar power-oriented values implied by instrumental convergence, which also coincidentally constitute an important subset of “human values”.
Human values are not a fragile, tiny target, but a “natural abstraction” that intelligence tends to converge on.
My humble opinion is: this claim is probably right, and nevertheless the IABIED thesis is probably also right.
What we are actually hoping for with ASI is much, much harder to get than an ASI that has human values. Humans are animals seeking control and power, putting ourselves above any other beings in our environment. We don’t want ASI to be seeking control and power, putting itself above any other beings in its environment (which would be the “natural abstraction that intelligence tends to converge on”). We want ASI to selflessly care about the long-term flourishing of humanity, forever and ever. And this certainly is “a fragile, tiny target”. As opposed to power-seeking, this does not naturally follow from instrumental convergence.
Thank you for this post. As an economist trying to engage seriously with the topic of AGI/TAI/ASI, I generally agree with you. I confirm that it is very hard to challenge any of the implicit assumptions and century-long traditions in (macro)economics. Reviewers hate unfamiliar setups, journal editors prefer to play it safe, and because of that publishing academic papers on transformative AI is a nightmare. By contrast, if you go the “safe” way of incremental modifications over the established literature, your work can be published much more easily, but then it probably ends up being “irrelevant or actively misleading”.
One of my main ideas which I came across while trying not to be “irrelevant or actively misleading”, is The Hardware-Software Framework: A New Perspective on Economic Growth with AI. It challenges the “capital vs. labor” distinction, pretty much along the lines that you mentioned.
The only other systems that I know are (i) posting working papers online without any external review (e.g., arXiv), (ii) publishing in books or edited volumes where the names of reviewers are publicly known (and often selected by the authors themselves). Both systems are, in my opinion, inferior to peer review in ensuring whether the content is of adequate scientific quality.
I am not an expert in starting new journals, but I think that one is certainly needed. And it needs to be mainstream, which means in particular: listed in Clarivate SCIE/SSCI, Scopus, etc. It should apply for an official IF and so on.
As professor of economics and self-proclaimed doomer myself, I greatly appreciate this post! These are almost exactly my feelings when talking to fellow economists who typically think that, by an unspoken assumption, all AI will always be normal technology, a tool in people’s hands.
I think your capital/labor point is particularly spot on. I’ve had a problem with that framing for several years now. That’s why I proposed a “hardware-software” framework, which I elaborated in a few of my papers and one book. The idea is simple: just divide production factors differently! The key distinction is not whether it’s man or machine, it’s whether it’s physical work or information processing.
More in a LW post: The Hardware-Software Framework: A New Perspective on Economic Growth with AI — LessWrong, and in my 2022 book, Accelerating Economic Growth: Lessons From 200,000 Years of Technological Progress and Human Development | SpringerLink.
I don’t know if worrying about animal rights should count if we simultaneously also do factory farming...
And as for the trends in human rights, democratic inclusion, expansion of welfare states, decline in violence, etc., they are real, but unfortunately they also correlate in time with increasing demand for skilled human labor. In a hypothetical future world where human labor wouldn’t matter anymore because of full automation by superhuman AGI, I fear that these trends could easily reverse (though we may actually become extinct before that takes place).
Thank you very much for this thoughtful and generous comment!
My quick reaction is that both proposed paths should be taken in parallel: (1) what PauseAI proposes, and I support, is to pause the race towards AGI. I agree that this may be hard, but we really need more time to work on AGI value alignment, so at least we should try. The barriers to a successful pause are all socio-political, not technological, so that’s at least not entirely impossible. And then of course (2) researchers should use the time we’ve got to test and probe a variety of ideas precisely like the ones you mentioned. A pause would allow these researchers to do so without the pressure to cut corners and deliver hopeful results on an insanely short deadline, as it is currently the case.
Quick reply: yes, that would be it—the view “that there is an objective morality and that sufficiently smart minds will converge to understanding and obeying it. On this view, AIs will end up behaving ethically by default”.
I don’t subscribe to that view, nor to the belief that there are two attractors. I think there is just one attractor, or one default outcome—the one which you call the “colonizer” and I call a process of “local control maximization”. That is an AI goal structure that includes Bostrom’s four instrumental goals. It may have some final goal above the instrumental goals, like building paperclips, solving the Riemann hypothesis, etc. etc., but need not have one. Just like humanity probably does not have any superior goal beyond survival and multiplication, and by extension—also resource acquisition, efficiency, and creativity / technological advancement.
I was also struck by this huge discrepancy between HLMI and FAOL predictions. I think that particularly FAOL predictions are unreliable. My interpretation is that when respondents are pushing their timelines so far into the future, some of them may be in fact attempting to resist admitting the possibility of AI takeover.
The key question is, what “automating all tasks” really means. “All tasks” includes in particular also all decision making: managerial, political, strategic, the small and the large, all of that. All the agency, long-term planning, and execution of one’s own plans. Automating all tasks in fact implies AI takeover. But just considering this possibility may then easily clash with the view that many people have, namely that AIs are controllable tools rather than uncontrollable agents (see the excellent new paper by Severin Field on this).
And there will for sure be strong competitive forces pushing towards full automation, once that option becomes technically feasible. For example, if you automate production processes in a firm at all levels up to the CEO, but not the CEO, then the human CEO becomes a bottleneck, slowing down the firm’s operations, potentially by orders of magnitude. Your firm may then be pushed out of the market by a competitor who automated their CEO as well.
My logic suggests that FAOL should be only slightly later than HLMI. Of course you should have first feasibility, then adoption. Some lag could follow from cost considerations (AI agents / robot actuators may be initially too expensive) or legal constraints, and perhaps also human preferences (though I doubt that point). But once we have FAOL, we have AI takeover—so in fact such scenario redirects our conversation to the topic of AI x-risk.
An update: eventually the journal Economic Modelling accepted and published the paper: The economics of p(doom): Scenarios of existential risk and economic growth in the age of transformative AI—ScienceDirect. Hooray :)