Software engineer from Ireland who’s interested in EA and AI safety research.
Stephen McAleese
Karma: 264
GPT-4 Predictions
Estimating the Current and Future Number of AI Safety Researchers
Summary of “AGI Ruin: A List of Lethalities”
Retrospective on ‘GPT-4 Predictions’ After the Release of GPT-4
AGI as a Black Swan Event
How Do AI Timelines Affect Existential Risk?
I’m not sure about software engineering as a whole but can I see AI making programming obsolete.
it will move up to the next level of abstraction and continue from there
My worry is that the next level of abstraction above Python is plain english and that anyone will be able to write programs just by asking “Write an app that does X” except they’ll ask the AI that instead of asking a freelance developer.
The historical trend has been that programming becomes easier. But maybe programming will become so easy that everyone can do programming and programmers won’t be needed anymore.
A historical analogy is search which used to be a skilled job that was done by librarians and involved creating logical queries using keywords (e.g. ‘house’ AND ‘car’). Now natural language language search makes it possible for anyone to use Google and we don’t need librarians for search anymore.
The same could happen to programming. Like librarians for search, it seems like programmers are a middleman between the user requesting a feature and the finished software. Historically programming computers has been too difficult for average people but that might not be true for long.
Since this seems to be Carn’s first post on LessWrong, I think some of the other readers should have been more lenient and not downvoted the post or explained why they downvoted the post.
I would only downvote a post if it was obviously bad, flawed, very poorly written, or a troll post.
This post contains lots of interesting ideas and seems like a good first post.
The original post “Reward is not the optimization target” has 216 upvotes and this one has 0. While the original post was written better, I’m skeptical of the main idea and it’s good to see a post countering it so I’m upvoting this post.
You were right. I forgot the 1B parameter model row so the table was shifted by an order of magnitude. I updated the table so it should be correct now. Thanks for spotting the mistake.
One major reason why there is so much AI content on LessWrong is that very few people are allowed to post on the Alignment Forum.
I analyzed some recent AI posts on LessWrong and found that only about 15% of the authors were also members of the Alignment Forum. I’m personally very interested in AI but I post all of my AI content on LessWrong and not the Alignment Forum because I’m not a member.
Anecdotally, I several people working full-time on AI safety who are still not members of the Alignment Forum and consequently post all their work on LessWrong.
My recommendation is to increase the number of people who are allowed to post on the Alignment Forum because the bar seems too high. And instead of having just a single class of members, there could be more members but different grades of members.
There are other reasons why AI has become more popular relative to rationality. Rationality isn’t really a field that progresses as fast as AI and consequently, writing on topics such as cognitive biases is already covered in The Sequences.
On the other hand, breakthroughs are made every week in the field of AI which prompts people to write about it.
I’ve seen some of the screenshots of Bing Chat. It seems impressive and possibly more capable than ChatGPT but I’m not sure. Here’s what Microsoft has said about Bing Chat:
“We’re excited to announce the new Bing is running on a new, next-generation OpenAI large language model that is more powerful than ChatGPT and customized specifically for search. It takes key learnings and advancements from ChatGPT and GPT-3.5 – and it is even faster, more accurate and more capable.”
If the model is more powerful than GPT-3.5 then maybe it’s GPT-4 but “more powerful” is too vague and phrase to come up with any clear conclusions. I don’t think I have enough information at this point to make strong claims about it so I think we’ll have to wait and see.
I think the word ‘taunt’ anthropomorphizes Bing Chat a bit too much where, according to Google, taunt is defined as “a remark made in order to anger, wound, or provoke someone”.
While I don’t think Bing Chat has the same anger and retributive instincts as humans, it could in theory simulate them given that it presumably contains angry messages in its training dataset and uses its chat history chat to predict and generate future messages.
I like how this post lists several arguments impartially instead of picking one and arguing that it’s the best option.
I recently analyzed the past 6 months of LessWrong posts about found that about 25% were related to AI.
Wow, this is an incredible achievement given how AI safety is still a relatively small field. For example, this post by 80,000 hours said that $10 - $50 million was spent globally on AI safety in 2020 according to The Precipice. Therefore this grant is roughly equivalent to an entire year of global AI safety funding!
The limbic system that controls motivations such as the sex drive is much older than the relatively new neocortex that’s responsible for human intelligence.
My guess is that the limbic system evolved by trial and error over millions of years. If this is what happened, maybe we should seek out iterative methods for aligning AI systems such as iteratively testing and developing the motivations of sub-AGI systems.
But as Eliezer Yudkowsky says in his AGI Ruin post, you can’t iteratively develop an AGI that’s operating at dangerous levels of capability if each mistake kills you. Therefore we might need to extrapolate the motivations of sub-AGI systems to superhuman systems or solve the problem in advance using a theoretical approach.
Thanks for the links!
Which posts? Would you mind sharing some links? I couldn’t find many posts related to black swans.
Apart from the first section summarizing black swans, everything here is my personal opinion.
Great post. I also fear that it may not be socially acceptable for AI researchers to talk about the long-term effects of AI despite the fact that, because of exponential progress, most of the impact of AI will probably occur in the long term.
I think it’s important that AI safety and considerations related to AGI become mainstream in the field of AI because it could be dangerous if the people building AGI are not safety-conscious.
I want a world where the people building AGI are also safety researchers rather than one where the AI researchers aren’t thinking about safety and the safety people are shouting over the wall and asking them to build safe AI.
This idea reminds me of how software development and operations were combined into the DevOps role in software companies.