Second, isn’t this just speeding up human disempowerment? I mean, you now have these GA models which do most of the thinking for you while you sit on your high and mighty throne sipping wine and praying that when these models eventually become smarter then you, they would still be loyal (or at least not apathetic or hostile) to you.
Few thoughts in no particular order:
Personal automation can be customized: There are many degrees to which one can hand off work to GA models. One can choose to hand off what one views as truly rote. It seems like a high level of customization will be key to having a GA model that actually works well with one’s sense of ownership/desire/responsibility. I think that there are already things nearing what one might want the minimal versions of GA models to be doing, eg with people’s openclaw/hermes agent setups.
Work requires a lot of decisions to be made: There is still a lot of work to be done while GA models do things. For instance, you should try using current coding models to do serious coding work on a large repo (or note how you do this if you already do it). There is a lot of decision-making that happens when writing code, and one can choose how much involvement one has in writing code with LLMs. Your quoted vignette of sitting on a throne and sipping wine is most analogous to one end of the spectrum of ways one can code with LLMs, namely closer to the vibecoding end of the spectrum. I’d argue that there are ways to be more involved than just letting one’s GA model do things for them if the coding analogy holds for work in general, which I think it definitely does to many degrees.
We aren’t there yet: In the limit, you may be right about super strong models disempowering people. I think that this is a worry that I have and don’t have a particularly encouraging thought on yet. But, there is one important note: we are not there yet, and we do not know when we will get there. It seems like GA models will be at the very least a good intermediate addition to people’s lives, and it may even lead to different world states that give us more optionality/perspectives when we get to disempowering models.
GA models may not be so misaligned: It’s not clear that models will be “apathetic or hostile” to people. There are a few reasons that this may be true.
Current models generally seem aligned and helpful to people. I can imagine GA models simply being post-trained versions of current models, which I would guess would remain similarly aligned.
There is still a lot of work to be done on model behaviors, but there is already a lot being done and I am optimistic that for personal models a lot of this work can be applied. Ie, this area is being worked on pretty actively!
Individuals likely don’t need super OP intelligencemaxxed god tier models to be their GA models, so losing control to them may not be as big of a risk as you may think. (Sure, this opens risk to being jailbroken by smarter models, but there are many things, eg multi-layered input/output filters, that can be useful here.)
Safe cloud compute may be possible: You make a good point about the average Joe not being able to afford this without eg using non-local computing, but I am hopeful as there are solutions being developed for this worry though, eg at https://tinfoil.sh.
Few thoughts in no particular order:
Personal automation can be customized: There are many degrees to which one can hand off work to GA models. One can choose to hand off what one views as truly rote. It seems like a high level of customization will be key to having a GA model that actually works well with one’s sense of ownership/desire/responsibility. I think that there are already things nearing what one might want the minimal versions of GA models to be doing, eg with people’s openclaw/hermes agent setups.
Work requires a lot of decisions to be made: There is still a lot of work to be done while GA models do things. For instance, you should try using current coding models to do serious coding work on a large repo (or note how you do this if you already do it). There is a lot of decision-making that happens when writing code, and one can choose how much involvement one has in writing code with LLMs. Your quoted vignette of sitting on a throne and sipping wine is most analogous to one end of the spectrum of ways one can code with LLMs, namely closer to the vibecoding end of the spectrum. I’d argue that there are ways to be more involved than just letting one’s GA model do things for them if the coding analogy holds for work in general, which I think it definitely does to many degrees.
We aren’t there yet: In the limit, you may be right about super strong models disempowering people. I think that this is a worry that I have and don’t have a particularly encouraging thought on yet. But, there is one important note: we are not there yet, and we do not know when we will get there. It seems like GA models will be at the very least a good intermediate addition to people’s lives, and it may even lead to different world states that give us more optionality/perspectives when we get to disempowering models.
GA models may not be so misaligned: It’s not clear that models will be “apathetic or hostile” to people. There are a few reasons that this may be true.
Current models generally seem aligned and helpful to people. I can imagine GA models simply being post-trained versions of current models, which I would guess would remain similarly aligned.
There is still a lot of work to be done on model behaviors, but there is already a lot being done and I am optimistic that for personal models a lot of this work can be applied. Ie, this area is being worked on pretty actively!
Individuals likely don’t need super OP intelligencemaxxed god tier models to be their GA models, so losing control to them may not be as big of a risk as you may think. (Sure, this opens risk to being jailbroken by smarter models, but there are many things, eg multi-layered input/output filters, that can be useful here.)
Safe cloud compute may be possible: You make a good point about the average Joe not being able to afford this without eg using non-local computing, but I am hopeful as there are solutions being developed for this worry though, eg at https://tinfoil.sh.