Feedback welcomed: www.admonymous.co/zeshen
I sometimes write my thoughts here: airisks.substack.com
zeshen
Karma: 409
Yeah, at least “literally everyone dies” has a concrete ending even though it doesn’t have concrete intermediate concrete steps. Gradual disempowerment seem less concrete on both the ending and the intermediate steps, so it becomes even less action-relevant.
On thinking about AI risks concretely
…But I’m not sure that actual existing efforts towards delaying AGI are helping.
But perhaps actual existing efforts to hype up LLMs are helping? I am sympathetic to François Chollet’s position:
OpenAI basically set back progress towards AGI by quite a few years probably like five to 10 years for two reasons. They caused this complete closing down of Frontier research publishing but also they triggered this initial burst of hype around LLMs and now LLMs have sucked the oxygen out of the room.
This happened all the time at my line of work. Forecasts become targets and you become responsible for meeting them. So whenever I was asked to provide a forecast, I will either i) ask as many questions as I need to know the exact purpose of the request, and produce a forecast that meets exactly that intent, or ii) pick a forecast and provide it, but first list down all the assumptions and caveats behind the forecast that I can possibly think of. With time, I’d also get a sense of who I need to be extra careful with when providing any forecasts because of all sorts of ways that might backfire.
Agreed. I’m also pleasantly surprised that your take isn’t heavily downvoted.
There’ll be discussions about how these systems will eventually become dangerous, and safety-concerned groups might even set up testing protocols (“safety evals”).
My impression is that safety evals were deemed irrelevant because a powerful enough AGI, being deceptively aligned, would pass all of them anyway. We didn’t expect the first general-ish AIs to be so dumb, like how GPT-4 was being so blatant and explicit about lying to the TaskRabbit worker.
Scott Alexander talked about explicit honesty (unfortunately paywalled) in contrast with radical honesty. In short, explicit honesty is being completely honest when asked, and radical honesty is being completely honest even without being asked. From what I understand from your post, it feels like deep honesty is about being completely honest about information you perceive to be relevant to the receiver, regardless of whether the information is explicitly being requested.
Scott also links to some cases where radical honesty did not work out well, like this, this, and this. I suspect deep honesty may lead to similar risks, as you have already pointed out.
And with regards to:
“what is kind, true, and useful?”
I think they would form a 3-circle venn diagram. Things that are within the intersection of all three circles would be a no-brainer. But the tricky bits are the things that are either true but not kind/useful, or kind/useful but not true. And I understood this post as a suggestion to venture more into the former.
Can’t people decide simply not to build AGI/ASI?
Yeah, many people, like the majority of users on this forum, have decided to not build AGI. On the other hand, other people have decided to build AGI and are working hard towards it.
Side note: LessWrong has a feature to post posts as Questions, you might want to use it for questions in the future.
Definitely. Also, my incorrect and exaggerated model of the community is likely based on the minority who have a tendency of expressing those comments publicly, against people who might even genuinely deserve those comments.
I agree with RL agents being misaligned by default, even more so for the non-imitation-learned ones. I mean, even LLMs trained on human-generated data are misaligned by default, regardless of what definition of ‘alignment’ is being used. But even with misalignment by default, I’m just less convinced that their capabilities would grow fast enough to be able to cause an existential catastrophe in the near-term, if we use LLM capability improvement trends as a reference.
Thanks for this post. This is generally how I feel as well, but my (exaggerated) model of the AI aligment community would immediately attack me by saying “if you don’t find AI scary, you either don’t understand the arguments on AI safety or you don’t know how advanced AI has gotten”. In my opinion, a few years ago we were concerned about recursively self improving AIs, and that seemed genuinely plausible and scary. But somehow, they didn’t really happen (or haven’t happened yet) despite people trying all sorts of ways to make it happen. And instead of a intelligence explosion, what we got was an extremely predictable improvement trend which was a function of only two things i.e. data + compute. This made me qualitatively update my p(doom) downwards, and I was genuinely surprised that many people went the other way instead, updating upwards as LLMs got better.
I’ve gotten push-back from almost everyone I’ve spoken with about this
I had also expected this reaction, and I always thought I was the only one who thinks we have basically achieved AGI since ~GPT-3. But looking at the upvotes on this post I wonder if this is a much more common view.
My first impression was also that axis lines are a matter of aesthetics. But then I browsed The Economist’s visual styleguide and realized they also do something similar, i.e. omit the y-axis line (in fact, they omit the y-axis line on basically all their line / scatter plots, but almost always maintain the gridlines).
Here’s also an article they ran about their errors in data visualization, albeit probably fairly introductory for the median LW reader.
I’m pretty sure you have come across this already, but just in case you haven’t:
Strong upvoted. I was a participant of AISC8 in the team that went on to launch AI Standards Lab, which I think counterfactually would not be launched if not for AISC.
Non-loss of control AGI-related catastrophes are out of control too
Why is this question getting downvoted?
This seems to be another one of those instances where I wish there was a dual-voting system to posts. I would’ve liked to strong disagree with the contents of the post without discouraging well-intentioned people from posting here.
Thanks for writing this up! I also want to register that I agree with all of this, maybe except for the part where AIs can’t tell novel funny jokes—I expect this to be relatively easy. But of coursre it depends on the definition of ‘novel’.
I struggled to do this exercise myself because when I looked at AI as a normal technology I felt like I basically agree with most of their thinking, but it was also hard to find concrete differences between their predictions and AI2027 at least in the near term. For example, for things like “LLMs are broadly acknowledged to be plateauing”, it’s probably going to be concurrently both true and false in a way that’s hard to resolve—a lot of people may complain that it’s plateauing but the benchmark scores and the usage stats could show otherwise.