Dr. David Denkenberger co-founded and is a director at the Alliance to Feed the Earth in Disasters (ALLFED.info) and donates half his income to it. He received his B.S. from Penn State in Engineering Science, his masters from Princeton in Mechanical and Aerospace Engineering, and his Ph.D. from the University of Colorado at Boulder in the Building Systems Program. His dissertation was on an expanded microchannel heat exchanger, which he patented. He is an associate professor at the University of Canterbury in mechanical engineering. He received the National Merit Scholarship, the Barry Goldwater Scholarship, the National Science Foundation Graduate Research Fellowship, is a Penn State distinguished alumnus, and is a registered professional engineer. He has authored or co-authored 156 publications (>5600 citations, >60,000 downloads, h-index = 38, most prolific author in the existential/global catastrophic risk field), including the book Feeding Everyone no Matter What: Managing Food Security after Global Catastrophe. His food work has been featured in over 25 countries, over 300 articles, including Science, Vox, Business Insider, Wikipedia, Deutchlandfunk (German Public Radio online), Discovery Channel Online News, Gizmodo, Phys.org, and Science Daily. He has given interviews on 80,000 Hours podcast (here and here) and Estonian Public Radio, Radio New Zealand, WGBH Radio, Boston, and WCAI Radio on Cape Cod, USA. He has given over 80 external presentations, including ones on food at Harvard University, MIT, Princeton University, University of Cambridge, University of Oxford, Cornell University, University of California Los Angeles, Lawrence Berkeley National Lab, Sandia National Labs, Los Alamos National Lab, Imperial College, Australian National University and University College London.
denkenberger
Why the downvotes? One can think that overall Yudkowsky or Hanson was more overall right in 2008, but how can you disagree with Hanson being more right about the time to react to the rise of AI?
In practice, things have been much more gradual (as Christiano called correctly almost a decade ago).
And Hanson almost two decades ago.
Basis point means 0.01 percentage points, e.g. moving from p(existential risk) of 10% to 9.99%. Yes, it is small, but once you start spending many billions of dollars, it becomes larger. AI safety interventions are not my area of expertise, but resilience interventions to pandemics/nuclear war that may be downstream of AI are. I think these interventions are reliably positive, though the effectiveness would be reduced somewhat by second order effects. In this paper, I estimated that the value of resilience to nuclear winter might be reduced by 1 part in 100 because of moral hazard:
“Moral hazard would be if awareness of a food backup plan makes nuclear war more likely or more
intense. It is unlikely that, in the heat of the moment, the decision to go to nuclear war (whether
accidental, inadvertent, or intentional) would give much consideration to the nontarget countries.
However, awareness of a backup plan could result in increased arsenals relative to business as
usual, as awareness of the threat of nuclear winter likely contributed to the reduction in arsenals
[74]. Mikhail Gorbachev stated that a reason for reducing the nuclear arsenal of the USSR was the
studies predicting nuclear winter and therefore destruction outside of the target countries [75]. One
can look at how much nuclear arsenals changed while the Cold War was still in effect (after the
Cold War, reduced tensions were probably the main reason for reduction in stockpiles). This was
~20% [76]. The perceived consequences of nuclear war changed from hundreds of millions of
dead to billions of dead, so roughly an order of magnitude. The reduction in damage from reducing
the number of warheads by 20% is significantly lower than 20% because of marginal nuclear
weapons targeting lower population and fuel loading density areas. Therefore, the reduction in
impact might have been around 10%. Therefore, with an increase in damage with the perception
of nuclear winter of approximately 1000% and a reduction in the damage potential due to a smaller
arsenal of 10%, the elasticity would be roughly 0.01. Therefore, the moral hazard term of loss in
net effectiveness of the interventions would be 1%.”
I believe that working on frontier AI capabilities is usually wrong, just like it it would usually be wrong to work on mining dirty coal or locking people up without due process or bundling subprime mortgages or marketing tobacco to kids. If you go out and get a job like that and your children aren’t starving, then in my opinion, that’s pretty strong evidence that you have bad character. It’s always hard to see into other people’s hearts, and I think we should have a very high bar for accusing others of having bad character, but for me, working on frontier AI capabilities in 2026 typically clears that bar.
What about if they were donating to effective charities? 80k had an article that someone in finance (which could include bundling subprime mortgages, which was a significant reason why 80k got a lot of press soon after the financial crisis) donating half of their salary to the Against Malaria Foundation likely caused orders of magnitude more good than harm. This was controversial, and perhaps only consequentialists would argue that it is a good thing to do if the benefit were just slightly higher than the harms. However, the majority of people in Oceania, Americas, Europe, and Asia surveyed would kill 1 person to save 5 lives. And if the benefit were two orders larger than the harm, I think it would be the vast majority of people thinking it was a good thing to do.
As for AI, donating to existential risk reduction might have a cost effectiveness of ~~$300 M per basis point reduction in X risk. Say the capabilities researcher donates 10% of total compensation package of $5 M over 5 years—that’s $0.5 M, or ~~0.0017 basis point of X risk. As for the harm, let’s say a capabilities researcher is 1⁄10,000 responsible for the risk, but is 90% replaceable, so 1⁄100,000 of the risk. And let’s say the existential risk is 10% over the next 5 years (if you think it is higher, then the cost effectiveness of the charity would be higher). This is a harm of ~~0.01 basis points of X risk, which is an order of magnitude greater than the benefit of the charity. Of course there are orders of magnitude of uncertainty in some of these numbers, so the conclusion could be flipped.
Then there is the question of weighing the benefits of AI development. Depending on one’s estimate of X risk from non-AI causes, I think that many people would say that at 1% P(doom), AI development could be net beneficial from a long-term perspective (of the few EA respondents, 1% was their median acceptable P(doom)). And as some economists and Bostrom have pointed out, a much higher X risk could be justified if you have a short term perspective.
So overall whether the AI capabilities researchers are causing net harm really depends on P(doom) and how far one looks into the future, but maybe not so much on their charity?
Here are the articles I was thinking of with AI summaries (that I endorse):
Filippa Lentzos, “Will splashy philanthropy cause the biosecurity field to focus on the wrong risks?” (Bulletin of the Atomic Scientists, April 2019). She argues Open Phil’s large grants were pulling the biosecurity field’s limited expert capacity toward catastrophic, “extremely unlikely” scenarios and away from more probable risks (natural disease, lab accidents, negligence) — noting Open Phil’s single grants ($16M, $12M, $3.5M) dwarfed the entire Biological Weapons Convention Implementation Support Unit’s annual budget (~$1M), risking “drowning out the diversity of perspectives” in the field.
“A Case Against Focusing on Tail-End Nuclear War Risks” (EA Forum, from a 2022 CERI fellow). Argues that prioritizing worst-case scenarios (nuclear winter, civilizational collapse) over preventing any nuclear use at all is a mistake, on three grounds: even a single detonation could escalate catastrophically, there’s essentially no empirical base to assign likelihoods to tail outcomes, and narrowly emphasizing the worst cases risks eroding the normative taboo against any nuclear use. Advocates instead for preventing deployment of any scale, undifferentiated.
Christian Ruhl (Founders Pledge), “Philanthropy to the Right of Boom” (EA Forum). Documents a roughly 30-to-1 funding gap between “left of boom” (prevention) and “right of boom” (escalation management, resilience, post-war response) nuclear philanthropy, argues the neglect reflects bias rather than evidence of ineffectiveness (Cold War-era stigma, PR concerns, differing moral frameworks — he catalogs eight explanations), and makes the case that hedging with right-of-boom investment is rational since prevention can fail via accident or miscalculation.
I think it’s great that you are learning from past examples of this, e.g., the animal welfare movement. Another example is EA’s focus on global catastrophic biological risk and their interaction with existing professionals in public health. A further example is EA’s focus on the most severe nuclear scenarios and right of boom (e.g. escalation and resilience) versus existing professionals focusing mostly on non-state actors/prevention.
A minimal solar power system could pay back its embodied energy in days
That probably couldn’t be maintained for the entire expansion to Dyson Swarm, but you might be on track to get there in about a year.
Yes, SFF is the longest EA application I’ve done, but shorter than I think any non-EA applications I’ve done (excluding pre-proposals and internal university money).
I ran this by a PV expert and he said there would be a lot of R&D to make this happen, but he thought the overall concept of making short-lived low embodied energy PV made sense.
The delivered column counts the output as the electricity produced, which is the one most relevant for our purposes. The primary column shows the more common LCA primary energy convention (Frischknecht et al. (2020)), which measures the thermal energy required to produce that electricity.
If the manufacture of the PV required mostly electricity, the delivered and primary would be more like a factor of 3 different. So they are finding that most of the energy required is high temperature heat, which electricity doesn’t have very much advantage in?
The table below shows the mass of each potentially-constraining element required to build the system at four deployment scales, against world reserves and resources.
Some of the units in the table didn’t render correctly.
Pumped hydro and compressed-air storage at suitable geology can come in below 1 MJ per MJ, but viable reservoirs and salt caverns are scale-limited well short of what a self-replicating economy needs.
Compressed air storage can use regular saline aquifers, like we do for storing natural gas, so I don’t think they will be scale limited.
I think this is very valuable to decompose manual jobs/tasks to see what we could already do with current robots with intelligent control. I think we could have big improvements in that control before AGI.
The number of robots needed to automate manufacturing is not large; in a rapidly growing economy the robot capital stock is small, and making robots is not a significant constraint relative to producing energy and raw materials.
US manufacturing labor is ~13 million, but one robot would displace more than one worker because manufacturing typically runs multiple shifts, so maybe 4 million robots. So that’s about 2 years delay in your model. But by then, we would have built up more manufacturing, so maybe a little longer delay, so that seems significant. But if you repurpose auto manufacturing to making robots, then you could build robots a lot faster.
Thanks—I’ll read those. And did you investigate gold and silver? In my quick analysis, I found that if we had to get as much gold and silver as we produce now from common rock, it would cost more than the entire economy. Of course we would conserve as the price increased, but gold is fairly important in electronics.
I’m glad to see this rigorous analysis of potential growth rates.
I realize you only had OECD data, but how do you think China’s growth rate would compare?
4 hours of battery storage is not nearly enough to create reliable electricity from PV.
Aluminum-wound motors and transformers are roughly 1.3x larger and heavier for the same output, acceptable for most industrial uses.
I agree on the larger volume, but as you note about power conductors, they are typically lighter weight.
A 10x economy is not possible under any assumptions about oil resources.
Unless you turn natural gas or coal into oil, which has already been commercially deployed, but it would need a lot of scale up. Also, some of the capital that consumed oil is no longer in service. And oil is used less for capital formation (smelting, electricity production) than other fossil fuels.
If the supply curve shifts right, quantity supplied increases.
I was saying for the case if managers could still only manage the same number of equivalent people. I agree that they would figure out how to manage more, but do you think they could each manage 1000 equivalent junior AI researchers?
If we already have an autonomous AI researcher, there’s a good chance the researcher takes off to ASI while policy-makers are still trying to figure out what’s doing on. Originally I said it was harder to enforce but really I think the concern is that it makes AI development go a lot faster.
Ok—faster makes more sense. But we have seen with Fable that policy-makers can act fast if they want to. Labs would probably want to use the compute to go for ASI, but they could still make a lot of money if customers paid for the compute (though of course they would need to reserve some compute to support the human AI safety researchers).
I don’t know how employment would play out. If there were an automated Junior AI researcher, then senior AI researchers could plausibly manage 10 of them each (like now) and they may lay off the Junior AI researchers if they can’t manage the automated Junior AI researchers. That would save the AI companies money, but would not really get them more research done. The total potential is millions of automated researchers, so maybe it would be possible for one human to manage 1000 automated researchers? Anyway, it seems to me that at some point, there would be mass layoffs, in which case the workers could argue that they could pivot to safety.
As for whether it is much harder to enforce if there are automated AI researchers, I don’t know. It seems like regardless, they could prohibit training of new more capable models, but I guess the labs might argue they need to train a new safer model? But that blurriness would be the case regardless of whether the researchers are human or AI. Are you saying that it’s harder to enforce a pause if progress is faster or cheaper?
There are now ‘closed loop systems’ that allow data centers to only draw water once.
But they use more energy because they don’t get the evaporative cooling to reduce the strain on the chiller (or potentially obviate the chiller).
But perhaps instead of shutting down, an AI company could reallocate 100% of its budget on some combination of safety research + global coordination to make AI development safer, and do just those things until it runs out of money. Think of how much more safety work a they could do if they dedicated all their resources to the problem!
A lot of people think we should pause at AGI, but one problem is that AI capabilities are spiky, so we will probably get powerful and dangerous capabilities even before it is human level at all tasks. So I was thinking what might actually be politically/economically feasible is pausing further development once they get an automated AI researcher, because all those AI capability researchers are not going to be happy about being laid off. So we could turn them into AI safety researchers!
So what is a mercury to lead problem?
I think that The Game: Penetrating the Secret Society of Pickup Artists claims 7 hours on average of interaction. But in the olden days, you were only supposed to date every week, so that could easily turn into weeks depending on the person.
No mention of mirror bacteria?