That might be a crux here, because my view is that hardware improvements are much harder to do effectively, especially in secret around the human level, due to Landauer’s Principle essentially bounding efficiency of small scale energy usage close to that of the brain (20 Watts.) Combine this with 2-3 orders of magnitude worse efficiency than the brain and basically any evolutionary object compared to human objects, and the fact it’s easier to get better software than hardware due to the virtual/real life distinction, and this is a crux for me.
I’m not sure how this is a crux. Hardware improvements are irrelevant to what either of us were saying.
I’m saying that there is little risk difference between an AGI reprogramming itself to have better software, and programming some other computer with better software.
That might be a crux here, because my view is that hardware improvements are much harder to do effectively, especially in secret around the human level, due to Landauer’s Principle essentially bounding efficiency of small scale energy usage close to that of the brain (20 Watts.) Combine this with 2-3 orders of magnitude worse efficiency than the brain and basically any evolutionary object compared to human objects, and the fact it’s easier to get better software than hardware due to the virtual/real life distinction, and this is a crux for me.
I’m not sure how this is a crux. Hardware improvements are irrelevant to what either of us were saying.
I’m saying that there is little risk difference between an AGI reprogramming itself to have better software, and programming some other computer with better software.