I think this would be a good argument against Said Achmiz’s suggested response, but I feel the text doesn’t completely support it, e.g. the Epistemologist says “such schemes often go through two phases” and “many schemes like that start with a flawed person”, suggesting that such schemes are known to him.
Simon Fischer
In Section 5 we discuss why expect oversight and control of powerful AIs to be difficult.
Another typo, probably missing a “we”.
The soft optimization post took 24 person-weeks (assuming 4 people half-time for 12 weeks) plus some of Jeremy’s time.
Team member here. I think this is a significant overestimate, I’d guess at 12-15 person-weeks. If it’s relevant I can ask all former team members how much time they spent; it was around 10h per week for me. Given that we were beginners and spent a lot of time learning about the topic, I feel we were doing fine and learnt a lot.
Working on this part-time was difficult for me and the fact that people are not working on these things full-time in the camp should be considered when judging research output.
Missile attacks are not piracy, though, right?
It’s good that you learned a few things from these incidents, but I’m sceptical of the (different) claim implied by the headline that Peter Zeihan was meaningfully correct here. If you interpret “directions” imprecisely enough, it’s not hard to be sometimes directionally correct.
I know this answer doesn’t qualify, but very likely the best you can currently do is: Don’t do it. Don’t train the model.
(I downvoted your comment because it’s just complaining about downvotes to unrelated comments/posts and not meaningfully engaging with the topic at hand)
“Powerful AIs Are Black Boxes” seems like a message worth sending out
Everybody knows what (computer) scientists and engineers mean by “black box”, of course.
I guess it’s hard to keep “they are experimenting with / building huge amounts of tanks” and “they are conducting combined arms exercises” secret from France and Russia, so they would have a lot of advance warning and could then also develop tanks.
But if you have lot more than a layman’s understanding of tank design / combined arms doctrine, you could still come out ahead in this.
“6. f6” should be “6. h3″.
Microsoft is the sort of corporate bureaucracy where dynamic orgs/founders/researchers go to die. My median expectation is that whatever former OpenAI group ends up there will be far less productive than they were at OpenAI.
I’m a bit sceptical of that. You gave some reasonable arguments, but all of this should be known to Sam Altman, and he still chose to accept Microsoft’s offer instead of founding his own org (I’m assuming he would easily able to raise a lot of money). So, given that “how productive are the former OpenAI folks at Microsoft?” is the crux of the argument, it seems that recent events are good news iff Sam Altman made a big mistake with that decision.
I’m confused by this statement. Are you assuming that AGI will definitely be built after the research time is over, using the most-plausible-sounding solution?
Or do you believe that you understand NOW that a wide variety of approaches to alignment, including most of those that can be thought of by a community of non-upgraded alignment researchers (CNUAR) in a hundred years, will kill everyone and that in a hundred years the CNUAR will not understand this?
If so, is this because you think you personally know better or do you predict the CNUAR will predictably update in the wrong direction? Would it matter if you got to choose the composition of the CNUAR?
Another big source of potential volunteers: People who are going to be dead soon anyway. I’d probably volunteer if I knew that I’m dying from cancer in a few weeks anyway.
Typo: This should be .
after 17… dxc6 or 17. c6
This should probably be “after 17… cxd6 or 17… c6”.
I suspect Wave refers to this company: https://www.wave.com/en/ (they are connected to EA)
Planecrash is a glowfic co-written by Yudkowsky: https://glowficwiki.noblejury.com/books/planecrash
Seconding the recommendation of the rest in motion post, it has helped me with a maybe-similar feeling.
AISC team report: Soft-optimization, Bayes and Goodhart
I don’t believe these “practical” problems (“can’t try long enough”) generalize enough to support your much more general initial statement. This doesn’t feel like a true rejection to me, but maybe I’m misunderstanding your point.
I think I mostly agree with this, but from my perspective it hints that you’re framing the problem slightly wrong. Roughly, the problem with the outsourcing-approaches is our inability to specify/verify solutions to the alignment problem, not that specifying is not in general easier than solving yourself.
(Because of the difficulty of specifying the alignment problem, I restricted myself to speculating about pivotal acts in the post linked above.)
Ah, I think there was a misunderstanding. I (and maybe also quetzal_rainbow?) thought that in the inverted world also no “apparently-very-lucrative deals” that turn out to be scams are known, whereas you made a distinction between those kind of deals and Ponzi schemes in particular.
I think my interpretation is more in the spirit of the inversion, otherwise the Epistemologist should really have answered as you suggested, and the whole premise of the discussion (people seem to have trouble understanding what the Spokesperson is doing) is broken.