This is your second post and you’re still being vague about the method. I’m updating strongly towards this being a hoax and I’m surprised people are taking you seriously.
Edit: I’ll offer you a 50 USD even money bet that your method won’t replicate when tested by a 3rd party with more subjects and a proper control group.
This feels like you’re engaging with the weakest argument against Israel’s recent aggression to make your point. You are not going to find many people who disagree with “violence against civilians is bad” on LessWrong.
It also strikes me as bizarre that this post mentions only the civilian casualties on one side and not the far greater (and rapidly growing) number of Palestinians who have been killed.
A concerning amount of alignment research is focused on fixing misalignment in contemporary models, with limited justification for why we should expect these techniques to extend to more powerful future systems.
By improving the performance of today’s models, this research makes investing in AI capabilities more attractive, increasing existential risk.
Imagine an alternative history in which GPT-3 had been wildly unaligned. It would not have posed an existential risk to humanity but it would have made putting money into AI companies substantially less attractive to investors.
“AI safety, as in, the subfield of computer science concerned with protecting the brand safety of AI companies”
Made me chuckle.
I enjoyed the read but I wish this was much shorter, because there’s a lot of very on the nose commentary diluted by meandering dialogue.
I remain skeptical that by 2027 end-users will need to navigate self-awareness or negotiate with LLM-powered devices for basic tasks (70% certainty it will not be a problem). This is coming from a belief that end user devices won’t be running the latest and most powerful models, and that argumentative, self aware behavior is something that will be heavily selected against. Even within an oligopoly, market forces should favor models that are not counterproductive in executing basic tasks.
However, as the story suggests, users may still need to manipulate devices to perform actions loosely deemed morally dubious by a companies PR department.
The premise underlying these arguments is that greater intelligence doesn’t necessarily yield self-awareness or agentic behavior. Human’s aren’t agentic because we’re intelligent, we’re agentic because it enhancing the likelihood of gene propagation**.
In certain models (like MiddleManager-Bot), agentic traits are likely to be actively selected.. But I suspect there will be a substantial effort to ensure your compiler, toaster etc aren’t behaving agentically, particularly if these traits results in antagonistic behavior to the consumer.**
*By selection I mean both through a models training, and also via more direct adjustment from human and nonhuman programmers.
** A major crux here is that the assumption that intelligence doesn’t inevitably spawn agency without other forces selecting for it in some way. I have no concrete experience attempting training frontier models to be or not be agentic, so could be completely wrong on this point.
This doesn’t imply that agentic systems will emerge solely from deliberate selection. There are a variety of selection criteria which don’t explicitly specify self-awareness or agentic behavior but are best satisfied by systems possessing those traits.
Without governance you’re stuck trusting that the lead researcher (or whoever is in control) turns down near infinite power and instead act selflessly. That seems like quite the gamble.
Train Tracks
The above gif comes from the brilliant childrens claymation film, “Wallace and Gromit The Wrong Trousers”. In this scene, Gromit the dog rapidly lays down track to prevent a toy train from crashing. I will argue that this is an apt analogy for the alignment situation we will find ourselves in the future and that prosaic alignment is focused only on the first track.
The last few years have seen a move from “big brain” alignment research directions to prosaic approaches. In other words asking how to align near-contemporary models instead of asking high level questions about aligning general AGI systems.
This makes a lot of sense as a strategy. One, we can actually get experimental verification for theories. And two, we seem to be in the predawn of truly general intelligence, and it would be crazy not to be shifting our focus towards the specific systems that seem likely to cause an existential threat. Urgency compels us to focus on prosaic alignment. To paraphrase a (now deleted) tweet from a famous researcher “People arguing that we shouldn’t focus on contemporary systems are like people wanting to research how flammable the roof is whilst standing in a burning kitchen”*
What I believe this idea is neglecting is that the first systems to emerge will be immediately used to produce the second generation. AI assisted programming has exploded in popularity, and while Superalignment is being lauded as a safety push, you can view it as a commitment from OpenAI to produce and deploy automated researchers in the next few years. If we do not have a general theory of alignment, we will be left in the dust.
To bring us back to the above analogy. Prosaic alignment is rightly focused on laying down the first train track of alignment, but we also need to be prepared for laying down successive tracks as alignment kicks off. If we don’t have a general theory of alignment we may “paint ourselves into corners” by developing a first generation of models which do not provide a solid basis for building future aligned models.
What exactly these hurdles are, I don’t know. But let us hope there continues to be high level, esoteric research that means we can safely discover and navigate these murky waters.
*Because the tweet is appears to be deleted, I haven’t attributed it to the original author. My paraphrase may be slightly off.
My experience has been that we are clearly funding constrained, particularly in a post FTX world. This makes it hard to recruit top talent and is likely driving people out of alignment research.
(All opinions expressed are my own. Details of examples have been changed. None of these stories directly concern experiences with my own funding. I currently have active grant applications which has incentivised me to make this comment less harsh than it probably should be.)
I’m aware of researchers who have turned down extremely prestigious careers to pursue alignment, been promised certain amounts of funding and then had that funding substantially delayed.
I’m aware of researchers with funding giving cash loans to other researchers who were experiencing funding delays.
Such an environment does not bode well for convincing people to stick to alignment. No adult with responsibilities feels secure working contract to contract.
Beyond that, alignment salaries are smaller than what you would make in tech. This makes it difficult to poach senior talent from FAAMG.
This was an interesting read.
There are a lot of claims here that are presented very strongly. There are only a few papers on language agents, and no papers (to my knowledge) that prove all language agents always adhere to certain propeties.
There might be a need for clearer differentiation between the observed properties of language agents, the proven properties, and the properties that being claimed.
One example: “The functional roles of these beliefs and desires are enforced by the architecture of the language agent.”
I think this is an extremely strong claim. It also cannot be true for every possible architecture of language agents. As a pathological example, wrap the “task queue” submodule of BabyAGI with a function that stores the opposite task it has been given, but returns the opposite task to what it stored. The plain english interpetation of the data is no longer accurate.
The mistake is to assume that because the data inside a language agent takes the form of English words, it precisely corresponds to those words.
I agree that it seems reasonable that it would most of the time, but this isn’t something you can say is true always.
“Language agents are unlikely to make this mistake. If a language agent is given an initial goal of opening chests and informed that keys are useful to this end, they will plan to collect keys only when doing so helps to open chests. If the same agent is transferred to a key-rich environment and realizes that this is the case, then they will only collect as many keys as is necessary to open chests. ”
I think I agree with this argument about goal misgeneralisation. A quick test on GPT-4 seems to agree and will describe only taking two keys (if you clarify that any key opens any chest but they are one use only)
An RL agent tasked with picking up keys and chests is initialised with very little information about the logical relationships between objects. On the other hand a trained GPT4 has deeply understands the relationship between a key and lock.
Goal misgeneralisation in language agents would seem to require ambiguity in language.
I have to dispute the idea that “less neurons” = “more human-readable”. If the fewer neurons are performing a more complex task it won’t necessarily be easier to interpret.
Soon there will be an army of intelligent but uncreative drones ready to do all the alignment research grunt work. Should this lead to a major shift in priorities?
This isn’t far off, and it gives human alignment researchers an opportunity to shift focus. We should shift focus to the of the kind of high level, creative research ideas that models aren’t capable of producing anytime soon*.
Here’s the practical takeaway: there’s value in delaying certain tasks for a few years. As AI evolves, it will effectively handle these tasks. Meaning you can be substantially more productive in total as long as you can afford to delay the task by a few years.
Does this mean we then concentrate only on the tasks an AI can’t do yet, and leave a trail of semi-finished work? It’s a strategy worth exploring.
*I believe by the time AI is capable of performing the entirety of scientific research (PASTA) we will be within the FOOM period.
Inspired by the recent OpenAI paper and a talk Ajeya Cotra gave last year.
What I find incredible is how contributing to the development of existentially dangerous systems is viewed as a morally acceptable course of action within communities that on paper accept that AGI is a threat.
Both OpenAI and Anthropic are incredibly influential among AI safety researchers, despite both organisations being key players in bringing the advent of TAI ever closer.
Both organisations benefit from lexical confusion over the word “safety”.
The average person concerned with existential risk from AGI might assume “safety” means working to reduce the likelihood that we all die. They would be disheartened to learn that many “AI Safety” researchers are instead focused on making sure contemporary LLMs behave appropriately. Such “safety” research simply makes the contemporary technology more viable and profitable, driving investment and reducing timelines. There is to my knowledge no published research that proves these techniques will extend to controlling AGI in a useful way.*
OpenAI’s “Superalignment” plan is a more ambitious safety play.Their plan to “solve” alignment involves building a human level general intelligence within 4 years and then using this to automate alignment research.
But there are two obvious problems:
a human level general intelligence is already most of the way toward a superhuman general intelligence (simply give it more compute). Cynically, Superintelligence is a promise that OpenAI’s brightest safety researchers will be trying their hardest to bring about an AGI within 4 years.
The success of Superalignment means we are now in the position of trusting that a for-profit, private entity will only use the human level AI researchers to research safety, instead of making the incredibly obvious play of having virtual researchers research how to build the next generation better, smarter automated researchers.
To conclude, if it looks like a duck, swims like a duck and quacks like a duck, it’s a capabilities researcher.
*This point could (and probably should) be a post in itself. Why wouldn’t techniques that work on contemporary AI systems extend to AGI?
Pretend for a moment that you and I are silicon-based aliens who have recently discovered that carbon based lifeforms exist, and can be used to run calculations. Scientists have postulated that by creating complex enough carbon structures we could invent “thinking animals”. We anticipate that these strange creatures will be built in the near future and that they might be difficult to control.
As we can’t build thinking animals today, we are stuck studying single cell carbon organisms. A technique has just been discovered in which we can use a compound called “sugar” to influence the direction in which these simple organisms move.
Is it reasonable to then conclude that you will be able to predict and control the behaviour of much more complex, multicelled creature called a “human” by spreading sugar out on the ground?