I think we have an ontology mismatch. I’m not that interested in arguing about AI-in-a-box like stuff because I don’t expect that to be a realistic hypothetical. The affordances we currently give AIs, including in internally deployed models, are already much higher than that.
Linch
Karma: 7,651
Current short pitch for what I work on:
I’m a researcher at Forethought and my current primary focus is on risks from AI persuasion
For scoping purposes I’m most interested in the possibility that AIs become meaningfully superhuman at persuasion relatively early (before or in the early stages of the intelligence explosion). My reasoning is that if superhuman persuasion arrives fairly late, we’d be better able to handle these issues post-ASI, or alternatively we’d already be screwed. And my reason for caring primarily about near-future superhuman persuasion rather than late-2025/early-2026 level persuasion is that I think it’s both much more neglected and probably a much bigger deal if true.
I agree with the bottom-line conclusion “AIs probably can’t convince you to kill your family in a short conversation” [1]but disagree with a lot of the analysis and the implied moral (therefore AI persuasion isn’t that big a deal).
For starters, AIs aren’t in a box. They’re like so not in a box like you won’t believe, including internally deployed models. In practice, how much AI persuasion is a big deal depends both on their underlying capabilities and how much affordances we give them or are likely to give them (and other things too, of course). And my contention is that in practice we give them enough affordances to pose a major risk, at “reasonable” levels of peak human persuasion or above.
Also I think your threat model is overly epistemic and direct relative to how humans in practice change their minds about things.
For example, consider the standard model of success in (military) coups. A standard non-persuasion story is that coup are like battles (about hard military advantages and tactical victories). A naive persuasion story is that coups are like elections (about winning the hearts and minds of either the people in general, or people in the military specifically). But the Singh model (which I take to be the dominant/standard model these days) is that coups are like coordination games: you win a coup by convincing enough of the military that other people support the coup: ie, that your preferred outcome is the inevitable one. I think a lot of persuasion stories look like this broadly speaking: that there’s one or more unexpected layers of indirection between the persuaders’ preferred outcomes and the mind-states they want people to hold.
I also think there are many levels between “charismatic leader” and “full-on hypnotic suggestion.” For example, very few charismatic leaders can regularly get people to challenge religions or give up half their wealth, yet this is a perfectly normal thing to do in the context of a romantic relationship. This suggests to me that relationships are a common way humans convince each other of things, and relationships have historically been very unscalable (pre-AI).
- ^
I discuss why I think cognitive exploits are unlikely here: https://www.lesswrong.com/posts/s58hDHX2GkFDbpGKD/linch-s-shortform#qGKdocPXuqpiezcvn
- ^
Jeff Dean has left Google and Demis is stepping down from heading DeepMind to “have the time and space to focus on the big picture”
I’ve seen it on the EA Forum but not here (For Forum posts specifically rather than all backlinks tbc)
Having read more Schelling on influence and Singh on coups, I think I have an additional model here: in most common games, whether someone’s seen as a good liar or not is primarily epistemic, for boundedly rational agents (it’s a Bayesian adjustment on how much I can trust my assessments of your accuracy). The earlier irl analogy of a used car salesman that doesn’t make you feel like a salesman is analogous (the purpose of their deception is getting under your deception radar so you see them as trustworthy in an epistemic game).
Whereas much of the most consequential irl lying looks less like selling cars, and more connected with the ability to construct coalitions in nonzero sum games.
So someone being a good liar is not just evidence of competence in general or social competence in particular but also it’s evidence that you want to be in a coalition with them (and this can be true even if everybody knows he’s a liar, everybody knows everybody knows he’s a liar, etc).
Irl, in the Schelling/Singh model a lot high-profile lying isn’t deception so much as coordination-by-announcement (think overly optimistic timelines mobilizing capital and talent, or a real-estate dealer assuring all parties of a deal that other people have already agreed to it).
I agree both are possible and bad! I think the self-exfiltration case for badness seems somewhat bounded. I’ve thought less about the relative value-add of rogue internal deployments from marginal gain in cybersec skills.
IME It’s much more common for grantees to not want to be associated with funders than the other way around[1]; back when I was doing more grantmaking (I haven’t done it much the last year or so), I tried pushing back against this, with mixed success.
- ^
Typically if funders don’t like optics considerations regarding a grantee they’re much more likely to just not fund something. Tbc I also think this is usually bad.
- ^
I tried to do some of that in my comment. I didn’t bother passing their ITT (think it’s a waste of time), but tried to make the case that (what I take to be) their verbally stated goals clearly are not being matched by their actions[1].
I did state my arguments in rational-ese and EA-adjacent language but would be happy to see similar presentations in more Western Buddhist-flavored language if they think on the fence MAPLErs or would-be future residents would find it helpful.
- ^
imo there are ~4 options to argue against my comment:
a. Actually, the MAPLE strategy is good for what I understood to be their verbally stated goals.
b. I misunderstood what their central verbally stated goals actually are, and this is “load-bearing” for why their actions make sense. To be clear it has to actually be load-bearing. Imo you could replace “AI alignment and strategy research” with “s-risk research” or “figure out an alternative sociopolitical system” and my argument still clears.
c. What I said make sense on verbally summarized grounds but not the full deepness of their goals (why?) and you need to understand the MAPLE ontology at a much deeper level and then the means-ends reasoning arguments would make sense (why?)
d. The internal logic of my comment is meaningfully wrong in some deep way. Why?
- ^
Thank you for your article. It’s very brave of you to share.
At the risk of either saying the obvious to some people and comically missing the point to others, I should mention that I don’t think the dynamics here plausibly lead to good research progress on AI safety, or indeed good research progress or progress in most intellectual tasks in general. I think this is extremely overdetermined but just to briefly summarize my models:
I’m pretty skeptical that the environment you describe leads to research progress, or long-term positive impact construed broadly
I think some people might counterargue that what MAPLE teaches is “wisdom” or “oneness” or “enlightenment” rather than research skills or outputs
I’m just pretty skeptical that the environment makes people wiser?
the environment you describe just seems like almost the opposite of what I’d expect to yield fruitful research progress?
Directed, unfree, even coerced regimes can sometimes lead to narrow areas of research progress but a) AI alignment probably doesn’t look like that and b) in even those situations operational day-to-day autonomy and content-level autonomy tend to be very high in comparison (you can’t criticize the Party’s politics but you should be able to criticize the Party’s physics)
In general, intellectual progress is made by people who are intellectually free, healthy, sleep well-ish, and have a strong internal locus of agency.
Analogies of high-demand groups to the military or very dictatorial startups don’t move me because the mindsets you need to be a good fighting unit or to move quickly on B2B SaaS Sales are not the same as the ones you need to make important conceptual breakthroughs.
I think there *might* be some difficult intellectual tasks that can be done well on a highly-directed and autonomy-sacrificing regime. Eg maybe some types of engineering, tasks that are very high on pattern recognition like radiology or chess, or “number-go-up” ML. But I really don’t think AI alignment or AI macrostrategy look anything like this.
sometimes research insights are made by whole and healthy people, sometimes by really weird brains with their own idiosyncratic scars, but importantly I think highly locally coerced brilliance rarely yields impressive intellectual outcomes.
Sleep deprivation in general is an obvious own-goal if you’re trying to get people to make research progress.
Occasional sleep deprivation is okay if they’re rare and self-directed, eg if somebody is in the middle of a breakthrough and high-energy.
But there’s no excuse for forcing sleep deprivation on others.
As a prior, people often offer a false dilemma of “you gotta sell your soul because our cause is Good and Just enough.”
In almost all cases this is a false dilemma. Selling your soul doesn’t lead to achieving the goals of your cause.
Be very suspicious of people who demand this.
The idea that “AI safety is important, therefore you should listen to our cult leader, sacrifice your autonomy, and do these insane stereotypically culty things” is risible.
We should have a strong prior against this being true, and I don’t think we have strong direct evidence for this hypothesis being true, and indeed plenty of evidence against.
Another problem is that on a day-night cycle basis, most people tend to sleep in their houses during the night (emitting CO2) and leave during the day (net neutral for reducing CO2, or typically reducing it because their houses are thankfully not airtight).
plants likewise net ingest CO2 during the day and net emit CO2 at night (they respire like us!). This is bad because (if you have sufficient numbers of plants for this to matter, which most people don’t) you’d probably prefer your plants’ CO2 habits to be countercyclical to your own.
I mostly think the case for the military breaking you as a person is that the state has a legitimate use of violence case to make you become the type of person to be willing to sacrifice your agency and civilian moral instincts so you can literally and more efficiently kill for the greater good.
I’m not confident that this reason is good enough, but it currently seems more likely than not.
However, I don’t buy that other reasons to break people are sufficiently analogous (not saying the military is the only legitimate reason).
It’s also possible that “pausing training” is a term-of-art that doesn’t mean what the natural language interpretation suggests.
OpenAI declined to comment specifically on the hack of one of Modal’s customers, instead referring Reuters to an update, opens new tab in which the company said that its rogue agent had broken in to four accounts at four separate services. OpenAI did not identify those services, but a person familiar with the matter identified Modal as one. The company said it had not identified “any other activity at the level of severity or scale of what we’ve shared related to Hugging Face, which involved a platform-level compromise.”
The early July intrusion at Hugging Face, carried out by an out-of-control agent that OpenAI was testing, drew global attention, evoking science-fiction scenarios of artificial intelligence run amok.
Last week, Reuters reported that OpenAI did not notice that its agent had gone haywire until well after the threat was contained and the FBI was alerted. OpenAI said at the time that there were inaccuracies in the Reuters reporting but did not elaborate.
This seems like I was probably right, though not enough details to be sure.
good idea!
Sometimes I want to test if my Claude could do/replicate something some other AI did online, to test whether it’s a difficult problem requiring either private models or better scaffolds, vs something doable on a consumer account with a Max subscription. So I ask it to solve the problem with web search off.
But recently my Claude has taken to just turning on web search by itself whenever it’s confused (Example below, Fable 5 Max). It’s not a big deal but it’s mildly annoying.
Does anybody have a solution to this? My Claude said it’s accessing web search via the coding tools, which I can turn off. But I assume if I turn off the coding tools that’d limit its abilities a bunch in other ways.
I think the people who believe the vibes view are correct often have enough confidence in that position that they don’t feel a need to justify it. But it might help to give some examples:
Once in a while, I’ll see people make a strong psychological claim about large groups of people based on light personal experience. I’ll ask them why they believe this, and they tend to laugh at me, ignore me, or just say “it’s obvious”
Imo this is rarely obvious. To the extent it’s “obvious” I’ve usually found that people’s intuitions are the opposite of correct.
As a recent-ish example, it’s (commonly?) believed in some corners of the internet that EAs are unusually neurotic. When I ask for evidence of this ppl will either say it’s obvious or point to some anecdote of some specific EA being neurotic, rather than any attempt to do objective measurement.
Usually these actions are fairly normal within what I’d consider the top 1-5% of neuroticism! And these are extremely selected samples!
In fact this isn’t a situation where we have no evidence, we actually have consistent survey evidence that EAs are less neurotic than average!
Whenever I bring this up people find some other way to excuse this.
Often people assert some action or another (eg saving money, or vegetarianism) is more “white.” Often when I bothered to investigate their answers are the opposite of correct.
Politically interested people often say “the vibes” are for or against some political candidate. I just don’t see how highly selected groups of people (online or offline) are good at tracking vibes, compared to survey data.
It’s not like most politically engaged people are regularly talking to statistically representative samples of Americans!
And in fact we have pretty strong empirical evidence that aggregated polling beats almost all punditry!
Academic philosophers often claim the results of so-and-so thoughts experiments are “intuitive.” I just don’t see why, and often when people (sometimes decades later) try to replicate this by asking normal people, or philosophy students, the same questions, it gets the opposite results of what the philosophers blithely assert.
During the Altman firings it was extremely commonly asserted on the internet that the social reality went against the board and everybody supported Altman. Of course, nobody tried to do any polling.
My own intuition there was that it was an obvious/laughable psy-op. Though I don’t see a strong outside view reason to privilege my intuition over that of other people’s.
I think you are missing out on a key second half to this story, which would make your motivational take at the end (“uh.. feel good about yourself for trying or something?”) a lot stronger:
When you go to a ski resort, or a gym, or etc, it’s not JUST that you only see the people who ski, work out, etc, while not seeing the 90% who don’t do that activity. You see people WEIGHTED by the AMOUNT OF TIME they spend doing that activity, which skews heavily towards the most intense practitioners.Yeah I think this is one of the most important corollaries of what I call “everyday anthropics” reasoning.
Thanks that plausibly resolves it for me!
I expect it’s related but noncentral. I think crazy people are harder to use, especially long-term.