“Me: As it happens, the threat model I’m working on is not LLMs, but rather “brain-like” Artificial General Intelligence (AGI), which (from a safety perspective) is more-or-less a type of actor-critic model-based reinforcement learning (RL) agent. LLMs are profoundly different from what I’m working on. Saying that LLMs will be similar to RL-agent AGI because “both are AI” is like saying that LLMs will be similar to the A* search algorithm because “both are AI”, or that a frogfish will be similar to a human because “both are animals”. They can still be wildly different in every way that matters.”
How would you respond to the critique that this basically amounts to saying “I’m not interested in or saying anything about things that exist, but this thing that AI X-Risk types made up because it’s the most worrying hypothetical to talk about sure is a worrying hypothetical.”
[glibly phrased but meant in a spirit of genuine curiousity because I still don’t really understand why people care about AI risk but lack any interest in stuff that actually exists right now]
Thanks for linking me. But no, I don’t really find that post illuminating with respect to the question that I have here? I see that you are making a case for why you find it reasonable to be concerned with what you are concerned with, but unless I’m missing something it still feels like at every turn the reasoning is from how you have chosen to define entirely hypothetical things that you’re pretty sure somebody is going to invent some day.
“Me: As it happens, the threat model I’m working on is not LLMs, but rather “brain-like” Artificial General Intelligence (AGI), which (from a safety perspective) is more-or-less a type of actor-critic model-based reinforcement learning (RL) agent. LLMs are profoundly different from what I’m working on. Saying that LLMs will be similar to RL-agent AGI because “both are AI” is like saying that LLMs will be similar to the A* search algorithm because “both are AI”, or that a frogfish will be similar to a human because “both are animals”. They can still be wildly different in every way that matters.”
How would you respond to the critique that this basically amounts to saying “I’m not interested in or saying anything about things that exist, but this thing that AI X-Risk types made up because it’s the most worrying hypothetical to talk about sure is a worrying hypothetical.”
[glibly phrased but meant in a spirit of genuine curiousity because I still don’t really understand why people care about AI risk but lack any interest in stuff that actually exists right now]
Sure—check out Intro Series §3.9 for my response. Does that help?
Thanks for linking me. But no, I don’t really find that post illuminating with respect to the question that I have here? I see that you are making a case for why you find it reasonable to be concerned with what you are concerned with, but unless I’m missing something it still feels like at every turn the reasoning is from how you have chosen to define entirely hypothetical things that you’re pretty sure somebody is going to invent some day.