That’s a good point, I’ll write up a brief explanation/disclaimer and put it in as a footnote.
HunterJay
Plausible A.I. Takeoff Scenario Short Story
Thanks!
I really enjoy’d this writeup! I’d probably even go a little bit on the pessimistic (optimistic?) side, and bet that almost all of this technology would be possible with only a few years of development from today—though I suppose it might be 20 if development doesn’t start/ramp up in earnest.
I took the original sentence to mean something like “we use things external to the brain to compute things too”, which is clearly true. Writing stuff down to work through a problem is clearly doing some computation outside of the brain, for example. The confusion comes from where you draw the line—if I’m just wiggling my fingers without holding a pen, does that still count as computing stuff outside the brain? Do you count the spinal cord as part of the brain? What about the peripheral nervous system? What about information that’s computed by the outside environment and presented to my eyes? I think it’s kind of an arbitrary line, but reading this charitably their statement can still be correct, I think.
(No response from me on the rest of your points, just wanted to back the author up a bit on this one.)
I am curious if this has changed over the past 6 years since you posted this comment. Do you get the feeling that high profile researchers have shifted even further towards Xrisk concern, or if they continue with the same views as in 2016? Thanks!
I like this story pitch! It seems pretty compelling to me, and a clever way to show the difficulty and stakes of alignment. Good luck!
Minimum Viable Alignment
Will check them out, thank you.
Aye, I agree it is not a solution to avoiding power seeking, only that there may be a slightly easier target to hit if we can relax as many constraints on alignment as possible.
Thanks Chris, but I think you linked to the wrong thing there, I can’t see your post in the last 3 years of your history either!
In the spirit of doing low status things with high potential, I am working on a site to allow commissioning of fringe erotica and am looking to hire a second web developer.
The idea is to build a place where people with niche interests can post bounties for specific stories. In my time moonlighting as an erotic author, I’ve noticed a lack of good sites to do freelance erotic writing work. I think the reason for this is that most people think porn is icky, so despite there being a huge market for extremely niche content, the platforms currently available are pretty abysmal. This is our opportunity.
We’re currently in beta and can pay a junior-level wage, with senior-level equity. If you’re a web developer who wants to join a fully remote startup, please reach out.
As with my other startups, I began this project with the goal of generating wealth to put towards alignment research.
Side note—France isn’t a great example for your point here “France, for example, is a very old, well-established and liberal democracy.” because the Fifth Republic was only established in 1958. It’s also notable for giving the president much stronger executive powers compared with the Fourth Republic!
My AI Predictions 2023 − 2026
Thanks, fixed. I did mean 3.5 to 4, not 3 to 4.
I broadly agree. I think AI tools are already speeding up development today, and on reflection I don’t actually think AI being more capable than humans at modeling the natural world would be a discontinuous point on the ramp up to superintelligence, actually.
It would be a point where AI gets much harder to predict, though, which is probably why it was on my mind when I was trying to come up with predictions.
I am extremely worried about safety, but I don’t know as much about it as I do about what’s on the edge of consumer / engineering trends, so I think my predictions here would be not useful to share right now! The main way it relates to my guesses here is if regulation successfully slows down frontier development within a few years (which I would support).
I’m doing the ARENA course async online at the moment, and possibly moving into alignment research in the next year or two, so hoping to be able to chat more intelligently on alignment soonish.
Agree on lower depth models being possible, a few other possibilities:
-
Smaller models with lower latency could be used, possibly distilled down from larger ones.
-
Compute improvements might make it practical onboard (like with Tesla’s self-driving hardware inside the chest of their andriod).
-
New architectures could work on more than one time scale—kind of like humans do. E.g. when we walk, not all of the processing is done in the brain. Your spinal cord can handle a tonne of it autonomously. (Will find source tomorrow).
-
LLM-type models could do the parts that can accept higher latency, leaving lower level processes to handle themselves. Imagine for a household cleaning robot that a LLM based agent puts out high level thoughts like “Scan the room for dirty clothes. … Fold them. … Put them in the third draw”, and existing low level stuff actually carried out the instructions. That’s an exaggerated example, but you get the idea, it doesn’t have to replace the PID controller!
-
I am honestly very surprised it became a front page post too! It totally is just speculation.
I tried to be super clear that these were just babbled guesses, and I was mainly just telling people to try to do same, rather than trusting my starting point here.
The other thing that surprised me is that there haven’t been too many comments saying “this part is off”, or “you missed trend X!”. I was kind of hoping for that!
Typo corrected, thanks for that.
I agree, it’s more likely for the first AGI to begin on a supercomputer at a well-funding institution. If you like, you can imagine that this AGI is not the first, but simply the first not effectively boxed. Maybe its programmer simply implemented a leaked algorithm that was developed and previously run by a large project, but changed the goal and tweaked the safeties.
In any case, it’s a story, not a prediction, and I’d defend it as plausible in that context. Any story has a thousand assumptions and events that, in sequence, reduce the probability to infinitesimal. I’m just trying to give a sense of what a takeoff could be like when there is a large hardware overhang and no safety—both of which have only a small-ish chance of occurring. That in mind, do you have an alternative suggestion for the title?