Yep, I agree that there’s a significant chance/risk that alternative AI approaches that aren’t as safe as LMAs are developed, and are more effective than LMAs when run in a standalone manner. I think that SCAs can still be useful in those scenarios though, definitely from a safety perspective, and less clear from a performance perspective.
For example, those models could still do itemized, sandboxed, and heavily reviewed bits of cognition inside an architecture, even though that’s not necessary for them to achieve what the architecture working towards. Also, this is when we start getting into more advanced safety features, like building symbolic/neuro-symbolic white box reasoning systems that are interpretable, for the purpose of either controlling cognition or validating the cognition of black box models (Davidad’s proposal involves the latter).
I implied the whole spectrum of “LLM alignment”, which I think is better to count as a single “avenue of research” because critiques and feedback in “LMA production time” could as well be applied during pre-training and fine-tuning phases of training (constitutional AI style).
If I’m understanding correctly, is your point here that you view LLM alignment and LMA alignment as the same? If so, this might be a matter of semantics, but I disagree; I feel like the distinction is similar to ensuring that the people that comprise the government is good (the LLMs in an LMA) versus trying to design a good governmental system itself (e.g. dictatorship, democracy, futarchy, separation of powers, etc.). The two areas are certainly related, and a failure in one can mean a failure in another, but the two areas can involve some very separate and non-associated considerations.
It’s only reasonable for large AGI labs to ban LMAs completely on top of their APIs (as Connor Leahy suggests)
Could you point me to where Connor Leahy suggests this? Is it in his podcast?
or research their safety themselves (as they already started to do, to a degree, with ARC’s evals of GPT-4, for instance)
To my understanding, the closest ARC Evals gets to LMA-related research is by equipping LLMs with tools to do tasks (similar to ChatGPT plugins), as specified here. I think one of the defining features of an LMA is self-delegation, which doesn’t appear to be happening here. The closest they might’ve gotten was a basic prompt chain.
I’m mostly pointing these things out because I agree with Ape in the coat and Seth Herd. I don’t think there’s any actual LMA-specific work going on in this space (beyond some preliminary efforts, including my own), and I think there should be. I am pretty confident that LMA-specific work could be a very large research area, and many areas within it would not otherwise be covered with LLM-specific work.
Do you have a source for “Large labs (OpenAI and Anthropic, at least) are pouring at least tens of millions of dollars into this avenue of research?” I think a lot of the current work pertains to LMA alignment, like RLHF, but isn’t LMA alignment per say (I’d make a distinction between aligning the black box models that compose the LMA versus the LMA itself).
Have you seen Seth Herd’s work and the work it references (particularly natural language alignment)? Drexler also has an updated proposal called Open Agencies, which seems to be an updated version of his original CAIS research. It seems like Davidad is working on a complex implementation of open agencies. I will likely work on a significantly simpler implementation. I don’t think any of these designs explicitly propose capping LLMs though, given that they’re non-agentic, transient, etc. by design and thus seem far less risky than agentic models. The proposals mostly focus on avoiding riskier models that are agentic, persistent, etc.
Have you read Eric Drexler’s work on open agencies and applying open agencies to present-day LLMs? Open agencies seem like progress towards a safer design for current and future cognitive architectures. Drexler’s design touches on some of the aspects you mention in the post, like:
The system can be coded to both check itself against its goals, and invite human inspection if it judges that it is considering plans or actions that may either violate its ethical goals, change its goals, or remove it from human control.
My experience on Upwork is actually the same as yours! In our tests of the platform, it appears to be very difficult to find jobs due to the intense competition. I was unpleasantly surprised at first when I saw how difficult it was to earn money on Upwork as a new user. However, that was the whole point of the initial tests we did, so we expanded and have still been expanding the program to encompass other forms of virtual work that pay reliably and still have room to grow. Upwork will be a minor or non-existent part of our program.
If my program was just on Upwork, then I would be inclined to side with your analysis. Thankfully, it’s not.
I think I understand the point: hypothetically, this program would take work away from people more in need, possibly even making the world worse off because of that. But if I magically made half of the virtual workforce disappear, then the half of the people that were removed would be really poor and the other half would be twice as rich. But is that creating more good? No, because the richer half would not need the money as much as the poorer half. If I added more people who were earning less money before being added then I am creating a net good, and that’s what I am trying to do. I don’t think the impact of helping several dozen people (just at first!) get out of poverty is insignificant, and since the program could be expanded if our tests indicate it works effectively, I think it could be considered high impact it terms of the number of people it could help and how much it could change their lives.
Well the total pool of work available for everyone is imperceptibly decreased in the short run, not aversely affecting anyone to any significant degree, while giving more of the poor who really need the money work opportunities… Is employing several dozen more people a small net good? I guess it’s a matter of opinion.
We are continuing our search for similar projects, thank you for your suggestion. I hope that we have not missed any pitfalls, but like Strangeattractor wrote, we are indeed doing tests of the concept in various stages of development, and this project is kind of a pilot in and of itself, so hopefully we can catch anything we might have missed.
’In a charitable way” meaning good for the people. Just because there are for-profit companies out there doing this doesn’t mean they are doing what is best for the people, they are distributing wealth, but also keeping a lot of it for themselves. A charitable venture would give most of the profits to the people involved, and this project also involves providing many things to people like internet and computer access, training, opportunities, something a lot of freelancers have to acquire for themselves in developing countries. It is very difficult for a would-be-freelancer to find access to all of the technology, one-on-one help, etc, hence the value of this project. While there are virtual employment companies, there are no companies helping freelancers get started, which is unique and fills a need.
Thank you, that is one of the markets we are looking to branch out to.
I did not throw every detail into the video and fundraiser/my post in LessWrong, that is correct… I do think I described the jist of it. I explicitly stated that funds will go towards providing computer and internet access, training given by staff, and opportunities that staff have to find. As implied, expenses will go towards computer acquisition, internet, and helping staff implement various facets of the project. I could have explained each and every detail, but it would be too long for the target audience to read. The campaign is not noticeably more vague than other related Indiegogo fundraisers I have seen.
I especially do not think dishonesty should be assumed. It’s just common sense, but to try and put it in words, the effort put into the campaign and video, the numerous people involved, the fact that I’m a high school student putting my reputation on the line, the fact that we are a “verified nonprofit” shown by Indiegogo after confirming our 501(c)(3) status… It would be a very unlikely and elaborate scam, especially for the very low amount of money that this is likely to earn.
For the record, this project is operating under close scrutiny by the faculty sponsor mentioned in the video, by the nonprofit sponsor we have mentioned in the bio of Silicon Rainforest, by our adult volunteers, by our business partners, etc. If I wanted to do this as a scam, I would try to sell miraculously affordable virtual employment services, take the money, and run ;)
MTurk employs a lot of people in developed countries. I have read they are starting to reject Indian based workers because of poor work quality. I can find employment for people who can provide a similarly high standard of work relative to workers in more developed countries, but who need the income more. Member participants would otherwise have had difficulties joining, say, MTurk because of a lack of computers, internet access, proper guidance, training… I don’t think there are any companies helping freelancers find work because it’s not very profitable, and yet there is a great need to reach people who are not working to their potential.
I’m having trouble seeing how a for-profit corporation would create more good and be a more effective structure in this case. A non-profit organization can operate without income tax and attract donations which can be tax-deductible to donors. A for-profit organization could get investment capital, but I think it’s highly unlikely I would be able to find any interested investors, and it otherwise performs worse compared to a non-profit with the same business model.
The way I see it, making the project a nonprofit allows it to better compete with for-profit companies because of tax-advantages. It can also get donations. A for-profit corporation has the advantage of attracting investments from people hoping to make a profit, but I am quite sure that I would not be able to attract large sums of investment capital. That pretty much gives starting this program as a nonprofit the only logical choice.
Regarding your point about re-compensation, I don’t think I cannot extract the value, it will just be difficult to pay myself an extraordinarily large sum of money all at once, in the hundreds of thousands of dollars. If that ever did become a reality, then hypothetically I could create a for-profit branch of the organization that could partner up with the nonprofit branch in managing core revenue generating operations, thus allowing me to siphon income out of the nonprofit.
Belizeans would probably be competing with wealthier people for work because their high level of English mastery allows them to compete for more advanced positions. The websites I mentioned have many workers from more developed countries. For example, half of MTurk’s users are from the United States.
Many people in developing countries do not have access to the technology needed to participate in virtual employment, so we will provide computer and internet access. We will be doing marketing in a way, yes, although it is guidance and training as well. In the future, we will move on from guiding people through using third party systems to directly selling virtual employment services, which should be much more profitable.
Thank you for your suggestions. I have in fact surveyed people and organizations in Belize. The general consensus is that there are a lot of people who are unemployed or working for very low wages, and getting higher paying employment would improve their standard of living. You mentioned a small scale pilot, we have actually run many such pilots, which is how we found that it would be possible to help people earn around $3 USD an hour. We are currently working on remote testing of our program before actually sending staff to Belize.