LLMs can act as glue that makes AI’s more G?
Matt Goldenberg(Matt Goldenberg)
Karma: 4,245
I think what we’re seeing here is that LLMs can act as glue to put together these modules in surprising ways, and make them more general. You see that here and with Saycan. And I do think that Chapman’s point becomes less tenable with them in the picture.
I didn’t say it should update your beliefs (edit: I did literally say this lol but it’s not what I meant!) I said it should update the beliefs of people who have a specific prevailing attitude.
I don’t think that Cicero is a general agent made by gluing together superhuman narrow agents! It’s not clear that any of its components are super human in a meaningful sense
I don’t either! I think it should update your beliefs that that’s possible though.
Yes, I got down to the Nash Bargaining part which is a bit harder and got confused again, but this helped as a very simple math intuition for why to Kelly Bet, if not how to calculate it in most real world betting situation.
I think people who are freaking out about Cicero moreso than foundational model scaling/prompting progress are wrong; this is not much of an update on AI capabilities
I think there’s a standard argument that goes “You can’t just copy paste a bunch of systems that are superhuman in their respective domains and get a more general agent out.” (e.g. here’s David Chapman saying something like this: https://mobile.twitter.com/Meaningness/status/1563913716969508864)
If you have that belief, I imagine this paper should update you more towards AI capabilities. It is indeed possible to duct tape a bunch of different machine learning models together and get out something impressive. If you didn’t believe this, it should update you on the idea that AGI could come from several small new techniques duct taped together to handle each other’s weakness.
You, however, do not know if it is a fair coin, and are offering me a fair bet. I only have 100 dollars to my name, and I am can bet as much as I want (up to 100 dollars) in either direction at even odds.
If I bet 100 dollars on heads, heads-me gets 200 dollars, and tails-me gets nothing. If I bet 100 dollars on tails, tails-me gets 200 dollars, and heads me gets nothing. If I bet nothing, both versions of me get 100 dollars.
However, every dollar in the hands of heads-me is worth 1.5 times as much as a dollar in the hands of tails-me, since heads-me exists 1.5 times as much. (I am ignoring here any diminishing returns in my value of money.)
Thus, to maximize value I should bet 100 dollars on heads. However, maybe it is better to think of tails-me as the rightful owner of 40 percent of my resources. When I bet 100 dollars on heads, I am seizing money from tails-me for the greater good, since heads-me has the (proportionally greater) existence necessary to better take advantage of it.
Alternatively, I could say that since 60 percent of me is heads-me, heads me should only control 60 dollars, which can be bet on heads. Tails me should control 40 dollars, which can be bet on tails. These two bets partially cancel each other out, and the net result is that I bet 20 dollars on heads.
If you are especially fast at maximizing expected logarithms, you might see where this is going.
Wow I have been looking for an intuitive explanation of Kelly Betting for years, and this is the first one that really hit from an intuitive mathematical perspective.
I think there is a name for the core of the approaches which works, which is “parts work.”
The ICF framework seems to add some things on top of the basic parts work idea that make it similar to IFS. For instance, the process of unblending at the beginning is basically the same as what IFS calls “getting into self”. In contrast, there are many effective parts work frameworks that do the work from a blended state, such as voice dialogue. It imports the assumption from IFS that there is some “neutral self” that can be reached by continually unblending, and that this self can moderate between parts.
In addition, IFS and ICF both seem to emphasize “conversation” as a primary modality, whereas other parts work modalities (e.g. Somatic Experiencing) emphasize other modalities when working with parts, such as somatic, metaphorical, or primal. Again, there’s an assumption here about what parts are and how they should be worked with, around the primacy of particular ways of thinking and relating which is heavily (if unconsciously) influenced by the prevalance of IFS and its’ way of working.
It seems like while ICF is trying to describe a general framework, it is quite influenced by the assumptions of IFS/IDC and imports some of their quirks, even while getting rid of others.
Hmmm, the moral uncertainty here is actually very interesting to think about.
as a concrete example, if your two best strategies to save the world are:
one whose crux is a theorem being true, which you expect is about 70% likely to be true
one whose crux is a person figuring out a required clever idea, which you expect is about 70% likely to happen
So taking that at face value, there are two separate options.
In one of them there’s a 30% chance you’re dooming ALL worlds to failure, and a 70% chance that ALL worlds have success. It’s more totalistic, which as you say means there’s a 30% chance no one survives—but on the other hand, there’s something noble about”if we get this right, we all succeed together”, “if we get this wrong, we all go down together.
In another, you’re dooming 30% of worlds to failure and giving 70% of them success. Sure there’s now the possiblity that some worlds win—but you’re also implicitly saying you’re fine with the collatoral damage of the 30% of worlds that simply get unlucky.
It seems to me one could make the case for either being the “more moral” course of action. One could imagine a trolley problem that mimicked these dynamics and different people choosing different options.
Yes, but without defining winning it’s not clear that it had anything to do with EA
I don’t think it’s all that clear, there are reasonable interpretations where he’s saying that he simply cared about winning as in making money, and it’s possible he was trying to unambiguously lean into that narrative but did so poorly.
He lives in an apartment with multiple roommates. Pretty obvious explanation when there’s multiple different egg cartons and JUST egg in there.
I am really curious about the disagree votes here, do people think this is not an empirical work demonstrating mesa-optimization?
In case you missed it, here’s previous empirical work demonstrating mesa-optimization: https://arxiv.org/abs/2105.14111
But it could make AGIs less dangerous by causing them to make exploitable mistakes, or fail to learn facts or techniques that would make them too powerful.
There is in fact class of AI safety proposals that try to make the AI unaware of certain things, such as not being aware that there is a shutdown button for it.
One of the issues with these types of proposals is that as the AI gets smarter/more powerful, it has to come up with increasingly crazy hypotheses about the world to ignore the facts that all the evidence is pointing towards (such as the fact that there’s a conspiracy, or an all powerful being, or something doing this). This could, in the long term, cause it to be very dangerous in it’s unpredictability.
One answer is the concept of “mesa-optimizers”—that is, if a machine learning algorithm is trained to answer questions well, it’s likely that in order to do that, it will build an internal optimizer that’s optimizing for something else other than answering questions—and that thing will have the same dangers as a non tool/oracle AI. Here’s the AI safety forum tag page: https://www.alignmentforum.org/tag/mesa-optimization
To further muddy things, apple cider in the US is unfiltered and unpastereuzied, whereas apple juice is typically filtered and pasteurized.
Afaict that page says it happened twice—William Shyne gave out candy coated laxative pills, and Ronald O’Bryce gave out poison-laced pixie stix
What do you think the failure rate has to get down to for this to be more worth it than the status quo?
I give some arguments for self-coercion in this post:
https://www.lesswrong.com/posts/HDHqvf9vMqSXEBCLh/would-most-people-benefit-from-being-less-coercive-to