First, he needs to explain why any efficiency constraints can’t be overcome by just throwing a lot of material and energy resources into building and powering inefficient or as-efficient-as-human-brains GPUs. If energy is not a taut constraint for AGI, and it’s also expected to be an increasing fraction of costs over time, then that sounds like an argument that we can overcome any efficiency limits with increasing expenditures to achieve superhuman performance.
If Jake claims to disagree with the claim that ai can starkly surpass humans [now disproven—he has made more explicit that it can], I’d roll my eyes at him. He is doing a significant amount of work based on the premise that this ai can surpass humans. His claims about safety must therefore not rely on ai being limited in capability; if his claims had relied on ai being naturally capability bounded I’d have rolled to disbelieve [edit: his claims do not rely on it]. I don’t think his claims rely on it, as I currently think his views on safety are damn close to simply being a lower resolution version of mine held overconfidently [this is intended to be a pointer to stalking both our profiles]; it’s possible he actually disagrees with my views, but so far my impression is he has some really good overall ideas but hasn’t thought in detail about how to mitigate the problems I see. But I have almost always agreed with him about the rest of the points he explicitly spells out in OP, with some exceptions where he had to talk me into his view and I eventually became convinced. (I really doubted the energy cost of the brain being near optimal for energy budget and temperature target. I later came to realize it being near optimal is fundamental to why it works at all.)
from what he’s told me and what I’ve seen him say, my impression is he hasn’t looked quite as closely at safety as I have, and to be clear, I don’t think either of us are proper experts on co-protective systems alignment or open source game theory or any of that fancy high end alignment stuff; I worked with him first while I was initially studying machine learning 2015-2016, then we worked together on a research project which then pivoted to building vast.ai. I’ve since moved on to more studying, but given assumption of our otherwise mostly shared background assumptions with varying levels of skill (read: he’s still much more skilled on some core fundamentals and I’ve settled into just being a nerd who likes to read interesting papers), I think our views are still mostly shared to the degree our knowledge overlaps.
@ Jake, re: safety, I just wish you had the kind of mind that was habitually allergic to C++’s safety issues and desperate for the safety of rustlang, exactly bounded approximation is great. Of course, we’ve had that discussion many times, he’s quite the template wizard, with all the good and bad that comes with that.
(open source game theory is a kind of template magic)
Respectfully, it’s hard for me to follow your comment because of the amount of times you say things like “If Jake claims to disagree with this,” “based on the premise that this is false,” “must therefore not rely on it or be false,” and “I don’t think they rely on it.” The double negatives plus pointing to things with the word “this” and “it” makes me lose confidence in my ability to track your line of thinking. If you could speak in the positive and replace your “pointer terms” like “this” and “it” with the concrete claims you’re referring to, that would help a lot!
Understandable, I edited in clearer references—did that resolve all the issues? I’m not sure in return that I parsed all your issues parsing :) I appreciate the specific request!
It helps! There are still some double negatives (“His claims about safety must therefore not rely on ai not surpassing humans, or be false” could be reworded to “his claims about safety can only be true if they allow for AI surpassing humans,” for example), and I, not being a superintelligence, would find that easier to parse :)
The “pointers” bit is mostly fixed by you replacing the word “this” with the phrase “the claim that ai can starkly surpass humans.” Thank you for the edits!
If Jake claims to disagree with the claim that ai can starkly surpass humans [now disproven—he has made more explicit that it can], I’d roll my eyes at him. He is doing a significant amount of work based on the premise that this ai can surpass humans. His claims about safety must therefore not rely on ai being limited in capability; if his claims had relied on ai being naturally capability bounded I’d have rolled to disbelieve [edit: his claims do not rely on it]. I don’t think his claims rely on it, as I currently think his views on safety are damn close to simply being a lower resolution version of mine held overconfidently [this is intended to be a pointer to stalking both our profiles]; it’s possible he actually disagrees with my views, but so far my impression is he has some really good overall ideas but hasn’t thought in detail about how to mitigate the problems I see. But I have almost always agreed with him about the rest of the points he explicitly spells out in OP, with some exceptions where he had to talk me into his view and I eventually became convinced. (I really doubted the energy cost of the brain being near optimal for energy budget and temperature target. I later came to realize it being near optimal is fundamental to why it works at all.)
from what he’s told me and what I’ve seen him say, my impression is he hasn’t looked quite as closely at safety as I have, and to be clear, I don’t think either of us are proper experts on co-protective systems alignment or open source game theory or any of that fancy high end alignment stuff; I worked with him first while I was initially studying machine learning 2015-2016, then we worked together on a research project which then pivoted to building vast.ai. I’ve since moved on to more studying, but given assumption of our otherwise mostly shared background assumptions with varying levels of skill (read: he’s still much more skilled on some core fundamentals and I’ve settled into just being a nerd who likes to read interesting papers), I think our views are still mostly shared to the degree our knowledge overlaps.
@ Jake, re: safety, I just wish you had the kind of mind that was habitually allergic to C++’s safety issues and desperate for the safety of rustlang, exactly bounded approximation is great. Of course, we’ve had that discussion many times, he’s quite the template wizard, with all the good and bad that comes with that.
(open source game theory is a kind of template magic)
Respectfully, it’s hard for me to follow your comment because of the amount of times you say things like “If Jake claims to disagree with this,” “based on the premise that this is false,” “must therefore not rely on it or be false,” and “I don’t think they rely on it.” The double negatives plus pointing to things with the word “this” and “it” makes me lose confidence in my ability to track your line of thinking. If you could speak in the positive and replace your “pointer terms” like “this” and “it” with the concrete claims you’re referring to, that would help a lot!
Understandable, I edited in clearer references—did that resolve all the issues? I’m not sure in return that I parsed all your issues parsing :) I appreciate the specific request!
It helps! There are still some double negatives (“His claims about safety must therefore not rely on ai not surpassing humans, or be false” could be reworded to “his claims about safety can only be true if they allow for AI surpassing humans,” for example), and I, not being a superintelligence, would find that easier to parse :)
The “pointers” bit is mostly fixed by you replacing the word “this” with the phrase “the claim that ai can starkly surpass humans.” Thank you for the edits!