Joe_Collman

Karma: 1,439

Joe_Collman 5 Apr 2024 23:33 UTC
2 points
0
on: What is the purpose and application of AI Debate?
Relevant here is Geoffrey Irving’s AXRP podcast appearance. (if anyone already linked this, I missed it)
I think Daniel Filan does a good job there both in clarifying debate and in questioning its utility (or at least the role of debate-as-solution-to-fundamental-alignment-subproblems). I don’t specifically remember satisfying answers to your (1)/(2)/(3), but figured it’s worth pointing at regardless.

Joe_Collman 28 Feb 2024 19:20 UTC
LW: 2 AF: 1
0
AF
on: Counting arguments provide no evidence for AI doom
Despite not answering all possible goal-related questions a priori, the reductionist perspective does provide a tractable research program for improving our understanding of AI goal development. It does this by reducing questions about goals to questions about behaviors observable in the training data.
[emphasis mine]

This might be described as “a reductionist perspective”. It is certainly not “the reductionist perspective”, since reductionist perspectives need not limit themselves to “behaviors observable in the training data”.
A more reasonable-to-my-mind behavioral reductionist perspective might look like this.
Ruling out goal realism as a good way to think does not leave us with [the particular type of reductionist perspective you’re highlighting].
In practice, I think the reductionist perspective you point at is:
- Useful, insofar as it answers some significant questions.
- Highly misleading if we ever forget that [this perspective doesn’t show us that x is a problem] doesn’t tell us [x is not a problem].

Joe_Collman 17 Feb 2024 7:05 UTC
LW: 2 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: Critiques of the AI control agenda
Sure, understood.
However, I’m still unclear what you meant by “This level of understanding isn’t sufficient for superhuman persuasion.”. If ‘this’ referred to [human coworker level], then you’re correct (I now guess you did mean this ??), but it seems a mildly strange point to make. It’s not clear to me why it’d be significant in the context without strong assumptions on correlation of capability in different kinds of understanding/persuasion.
I interpreted ‘this’ as referring to the [understanding level of current models]. In that case it’s not clear to me that this isn’t sufficient for superhuman persuasion capability. (by which I mean having the capability to carry out at least one strategy that fairly robustly results in superhuman persuasiveness in some contexts)

Joe_Collman 16 Feb 2024 19:56 UTC
LW: 2 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: Critiques of the AI control agenda
Do current models have better understanding of text authors than the human coworkers of these authors? I expect this isn’t true right now (though it might be true for more powerful models for people who have written a huge amount of stuff online). This level of understanding isn’t sufficient for superhuman persuasion.
Both “better understanding” and in a sense “superhuman persuasion” seem to be too coarse a way to think about this (I realize you’re responding to a claim-at-similar-coarseness).
Models don’t need to capable of a pareto improvement on human persuasion strategies, to have one superhuman strategy in one dangerous context. This seems likely to require understanding something-about-an-author better than humans, not everything-about-an-author better.
Overall, I’m with you in not (yet) seeing compelling reasons to expect a super-human persuasion strategy to emerge from pretraining before human-level R&D.
However, a specific [doesn’t understand an author better than coworkers] → [unlikely there’s a superhuman persuasion strategy] argument seems weak.
It’s unclear to me what kinds of understanding are upstream pre-requisites of at least one [get a human to do what you want] strategy. It seems pretty easy to miss possibilities here.
If we don’t understand what the model would need to infer from context in order to make a given strategy viable, it may be hard to provide the relevant context for an evaluation.
Obvious-to-me adjustments don’t necessarily help. E.g. giving huge amounts of context, since [inferences about author given input ( $x_{1}$ )] are not a subset of [inferences about author given input ( $x_{1}$ $\cup$ $x_{2}$ $\cup$ … $\cup$ $x_{1000}$ )].

Joe_Collman 8 Feb 2024 7:46 UTC
LW: 4 AF: 3
0
AF
in reply to: ryan_greenblatt’s comment on: Debating with More Persuasive LLMs Leads to More Truthful Answers
Thanks for the thoughtful response.
A few thoughts:
If length is the issue, then replacing “leads” with “led” would reflect the reality.
I don’t have an issue with titles like ”...Improving safety...” since it has a [this is what this line of research is aiming at] vibe, rather than a [this is what we have shown] vibe. Compare “curing cancer using x” to “x cures cancer”.
Also in that particular case your title doesn’t suggest [we have achieved AI control]. I don’t think it’s controversial that control would improve safety, if achieved.
I agree that this isn’t a huge deal in general—however, I do think it’s usually easy to fix: either a [name a process, not a result] or a [say what happened, not what you guess it implies] approach is pretty general.
Also agreed that improving summaries is more important. Quite hard to achieve given the selection effects: [x writes a summary on y] tends to select for [x is enthusiastic about y] and [x has time to write a summary]. [x is enthusiastic about y] in turn selects for [x misunderstands y to be more significant than it is].
Improving this situation deserves thought and effort, but seems hard. Great communication from the primary source is clearly a big plus (not without significant time cost, I’m sure). I think your/Buck’s posts on the control stuff are commendably clear and thorough.
I expect the paper itself is useful (I’ve still not read it). In general I’d like the focus to be on understanding where/how/why debate fails—both in the near-term cases, and the more exotic cases (though I expect the latter not to look like debate-specific research). It’s unsurprising that it’ll work most of the time in some contexts. Completely fine for [show a setup that works] to be the first step, of course—it’s just not the interesting bit.

Joe_Collman 8 Feb 2024 3:14 UTC
LW: 2 AF: 1
1
AF
in reply to: Joe_Collman’s comment on: Debating with More Persuasive LLMs Leads to More Truthful Answers
I’d be curious what the take is of someone who disagrees with my comment.
(I’m mildly surprised, since I’d have predicted more of a [this is not a useful comment] reaction, than a [this is incorrect] reaction)
I’m not clear whether the idea is that:
1. The title isn’t an overstatement.
2. The title is not misleading. (e.g. because “everybody knows” that it’s not making a claim of generality/robustness)
3. The title will not mislead significant amounts of people in important ways. It’s marginally negative, but not worth time/attention.
4. There are upsides to the current name, and it seems net positive. (e.g. if it’d get more attention, and [paper gets attention] is considered positive)
5. This is the usual standard, so [it’s fine] or [it’s silly to complain about] or …?
6. Something else.
I’m not claiming that this is unusual, or a huge issue on its own.
I am claiming that the norms here seem systematically unhelpful.
I’m more interested in the general practice than this paper specifically (though I think it’s negative here).
I’d be particularly interested in a claim of (4) - and whether the idea here is something like [everyone is doing this, it’s an unhelpful equilibrium, but if we unilaterally depart from it it’ll hurt what we care about and not fix the problem]. (this seems incorrect to me, but understandable)

Joe_Collman 7 Feb 2024 22:04 UTC
LW: 9 AF: 4
8
AF
on: Debating with More Persuasive LLMs Leads to More Truthful Answers
Interesting—I look forward to reading the paper.
However, given that most people won’t read the paper (or even the abstract), could I appeal for paper titles that don’t overstate the generality of the results. I know it’s standard practice in most fields not to bother with caveats in the title, but here it may actually matter if e.g. those working in governance think that you’ve actually shown “Debating with More Persuasive LLMs Leads to More Truthful Answers”, rather than “In our experiments, Debating with More Persuasive LLMs Led to More Truthful Answers”.
The title matters to those who won’t read the paper, and can’t easily guess at the generality of what you’ll have shown (e.g. that your paper doesn’t include theoretical results suggesting that we should expect this pattern to apply robustly or in general). Again, I know this is a general issue—this just happens to be a context where I can point this out with high confidence without having read the paper :).

Joe_Collman 28 Jan 2024 7:13 UTC
LW: 2 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: The case for ensuring that powerful AIs are controlled
Thanks for the link.
I find all of this plausible. However, I start to worry when we need to rely on “for all” assumptions based on intuition. (also, I worry in large part because domains are a natural way to think here—it’s when things feel natural that we forget we’re making assumptions)
I can buy that [most skills in a domain correlate quite closely] and that [most problematic skills/strategies exist in a small number of domains]. The ‘all’ versions are much less clear.

Joe_Collman 26 Jan 2024 7:27 UTC
LW: 6 AF: 4
1
AF
on: The case for ensuring that powerful AIs are controlled
Great post (I’ve not yet read it thoroughly, or thought for long).
The first concern that springs to mind:
- I expect a strong correlation between [humans are bad at [subversive strategy x]], [humans don’t tend to think of [subversive strategy x]], and [humans don’t tend to notice [subversive strategy x]].
  - My worry is more that we miss strategies that we’re bad at, than strategies the AI is extremely good at. It seems plausible for us to be entirely unaware of the existence of some subversive strategies—since they’ve never been human skills, and so don’t appear on our map.
  - It seems to me that the near-term danger is better described as [subversive strategies that don’t occur to humans], rather than [superhuman subversive strategies]. The latter will put people in mind of [strategies humans use, only better] - genius hacking/persuasion etc. I also want people considering [strategies that don’t occur to humans at all].
- On a minor-but-related note, I’m not too keen on writing/thinking in terms of “problematic domains” rather than “problematic skills” / “problematic strategies”. There’s no necessity for a subversive strategy to map nicely into something I have a label for—e.g. “hacking”, “persuasion”.
  - If we need to rule out all problematic strategies, I want to avoid language/thinking that may stop us considering problems outside our usual categories.
  - (I don’t think you’re saying anything substantively imprecise—here I’m only concerned about language and blind-spots)
One last thing that caught my eye:
...but humans are probably relatively more selected (by evolution) for resisting persuasion than for being good at modern science...
This seems a dangerous assumption, since humans have been selected to resist persuasion when resistance helped pass on their genes, which is very much not always. E.g. being persuaded of x when it’s likely that the rest of your tribe will be persuaded of x may well be helpful-to-your-genes, regardless of the truth of x or of the validity of the arguments. Humans were selected to believe useful fictions.
I note also that there’s a big difference between [human x has the skills and understanding necessary to resist being persuaded of y] and [human x will in fact resist being persuaded of y].

Joe_Collman 20 Jan 2024 22:36 UTC
2 points
0
on: What rationality failure modes are there?
Some examples here: mistakes with conservation of expected evidence

Joe_Collman 15 Jan 2024 1:15 UTC
7 points
2
in reply to: Matthew Barnett’s comment on: Against most AI risk analogies
For crying out loud, LLMs are already considered “AIs” by most people! How could they be a worse analogy for AI, across the board, than extraterrestrial beings that we have never come in contract with?
By tending to lead to overconfidence.
An aliens analogy is explicitly relying on [we have no idea what this will do]. It’s easy to imagine friendly aliens, just as it’s easy to imagine unfriendly ones, or entirely disinterested ones. The analogy is unlikely to lead to a highly specific, incorrect model.
This is not true for LLMs. It’s easy to assume that particular patterns will continue to hold—e.g. that it’ll be reasonably safe to train systems with something like our current degree of understanding.
To be clear, I’m not saying they’re worse in terms of information content: I’m saying they can be worse in the terms you’re using to object to analogies: “routinely conveying the false impression of a specific, credible model of AI”.
I think it’s correct that we should be very wary of the use of analogies (though they’re likely unavoidable).
However, the cases where we need to be the most wary are those that seem most naturally applicable—these are the cases that are most likely to lead to overconfidence. LLMs, [current NNs], or [current AI systems generally] are central examples here.
On asymmetric pushback, I think you’re correct, but that you’ll tend to get an asymmetry everywhere between [bad argument for conclusion most people agree with] and [bad argument for conclusion most people disagree with].
People have limited time. They’ll tend to put a higher value on critiquing invalid-in-their-opinion arguments when those lead to incorrect-in-their-opinion conclusions (at least unless they’re deeply involved in the discussion).
There’s also an asymmetry in terms of consequences-of-mistakes here: if we think that AI will be catastrophic, and are wrong, this causes a delay, a large loss of value, and a small-but-significant increase in x-risk; if we think that AI will be non-catastrophic, and are wrong, we’re dead.
Lack of pushback shouldn’t be taken as a strong indication that people agree with the argumentation used.
Clearly this isn’t ideal.
I do think it’s worth thinking about mechanisms to increase the quality of argument.
E.g. I think the ability to emoji react to particular comment sections is helpful here—though I don’t think there’s one that’s great for [analogy seems misleading] as things stand. Perhaps there should be a [seems misleading] react?? (I don’t think “locally invalid” covers this)

Joe_Collman 14 Jan 2024 6:21 UTC
2 points
0
on: OpenAI’s Preparedness Framework: Praise & Recommendations
Concrete suggestion: OpenAI should allow the Safety Advisory Group Chair and the head of the Preparedness Team to have “veto power” on model development and deployment decisions.
Quite possibly a good idea, but I think it’s less obvious than it seems at first glance:
Remember that a position’s having veto power will tend to have a large impact on selection for that position.
The comparison isn’t [x with veto power] vs [x without veto power].
It’s [x with veto power] vs [y without veto power].
If y would tend to have deeper understanding, more independence or more caution than x, it’s not obvious that giving the position veto power helps. Better to have someone who’ll spot problems and need to use persuasion, than someone who can veto but spots no problems.

Joe_Collman 12 Jan 2024 4:53 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Simulators
I’m using ‘simulation’ as it’s used in the post [the imitation of the operation of a real-world process or system over time]. The real-world process is the production of the string of tokens.
I still think that referring to what the LLM does in one step as “a simulation” is at best misleading. “a prediction” seems accurate and not to mislead in the same way.

Joe_Collman 12 Jan 2024 4:42 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Simulators
The point I’m making here is that in the terms of this post the LLM defines the transition function of a simulation.
I.e. the LLM acts on [string of tokens], to produce [extended string of tokens].
The simulation is the entire thing: the string of tokens changing over time according to the action of the LLM.
Saying “the LLM is a simulation” strongly suggests that a simulation process (i.e. “the imitation of the operation of a real-world process or system over time”) is occurring within the LLM internals.
Saying “GPT is a simulator” isn’t too bad—it’s like saying “The laws of physics are a simulator”. Loosely correct.
Saying “GPT is a simulation” is like saying “The laws of physics are a simulation”, which is at least misleading—I’d say wrong.
In another context it might not be too bad. In this post simulation has been specifically described as “the imitation of the operation of a real-world process or system over time”. There’s no basis to think that the LLM is doing this internally.
Unless we’re claiming that it’s doing something like that internally, we can reasonably say “The LLM produces a simulation”, but not “The LLM is a simulation”.
(oh and FYI, Janus is “they”—in the sense of actually being two people: Kyle and Laria)

Joe_Collman 12 Jan 2024 4:21 UTC
LW: 2 AF: 1
0
AF
in reply to: habryka’s comment on: Simulators
Yeah—I just noticed this ”...is the mechanics underlying our world.” on the tag page.
Agreed that it’s inaccurate and misleading.
I hadn’t realized it was being read this way.

Joe_Collman 12 Jan 2024 3:37 UTC
5 points
2
on: Saving the world sucks
Some relevant thoughts are in Nate’s Replacing Guilt sequence.
I can understand the sentiments expressed here—particularly in terms of dealing with these things at the age of 12.
However, I’d draw a distinction between:
1. Making the world better according to values that aren’t your own, from a sense of obligation.
2. Doing something to prevent the literal end of the world.
And I’d note that (2) does not rest on (1), on altruism (effective or otherwise), or on any particularly narrow moral view. Wanting some kind of non-paperclip-like world to continue existing isn’t a niche value.
Nietzsche or Ayn Rand would be among the last people to be guilted into saving the world, but may well do it anyway. This is not because they cared deeply about shrimp! (not to my knowledge, that is)
But of course there’s a lot to be said for understanding your values, and following a path you endorse on your own terms.

Joe_Collman 12 Jan 2024 2:37 UTC
2 points
0
in reply to: RogerDearnaley’s comment on: Simulators
I don’t disagree, but I don’t think that describing the process an LLM uses to generate a single token as a simulation is clarifying in this context.
I’m fairly sure the post is making no such claim, and I think it becomes a lot more likely that readers will have habryka’s interpretation if the word “simulation” is applied to LLM internals (and correctly conclude that this interpretation entails implausible claims).
I think “predictor” or the like is much better here.
Unless I’m badly misunderstanding, the post is taking a time-evolution-of-a-system view of the string of tokens—not of LLM internals.
I don’t think it’s claiming anything about what the internal LLM mechanism looks like.

Joe_Collman 12 Jan 2024 2:17 UTC
5 points
0
on: An even deeper atheism
The main case for optimism on human-human alignment under extreme optimization seems to be indirection: not that [what I want] and [what you want] happen to be sufficiently similar, but that there’s a [what you want] pointer within [what I want].
Value fragility doesn’t argue strongly against the pointer-based version. The tails don’t come apart when they’re tied together.
It’s not obvious that the values-on-reflection of an individual human would robustly maintain the necessary pointers (to other humans, to past selves, to alternative selves/others...), but it is at least plausible—if you pick the right human.
More generally, an argument along the lines of [the default outcome with AI doesn’t look too different from the default outcome without AI, for most people] suggests that we need to do better than the default, with or without AI. (I’m not particularly optimistic about human-human alignment without serious and principled efforts)

Joe_Collman 11 Jan 2024 19:10 UTC
LW: 4 AF: 2
0
AF
in reply to: habryka’s comment on: Simulators
Sure, but I don’t think anyone is claiming that there’s a similarity between a brain stepping forward in physical time and transformer internals. (perhaps my wording was clumsy earlier)
IIUC, the single timestep in the ‘physics’ of the post is the generation and addition of one new token.
I.e. GPT uses [some internal process] to generate a token.
Adding the new token is a single atomic update to the “world state” of the simulation.
The [some internal process] defines GPT’s “laws of physics”.
The post isn’t claiming that GPT is doing some generalized physics internally.
It’s saying that [GPT(input_states) --> (output_states)] can be seen as defining the physical laws by which a simulation evolves.
As I understand it, it’s making almost no claim about internal mechanism.
Though I think “GPT is a simulator” is only intended to apply if its simulator-like behaviour robustly generalizes—i.e. if it’s always producing output according to the “laws of physics” of the training distribution (this is imprecise, at least in my head—I’m unclear whether Janus have any more precise criterion).
I don’t think the post is making substantive claims that disagree with [your model as I understand it]. It’s only saying: here’s a useful way to think about the behaviour of GPT.

Joe_Collman 11 Jan 2024 7:15 UTC
LW: 4 AF: 2
0
AF
in reply to: habryka’s comment on: Simulators
Oh, hang on—are you thinking that Janus is claiming that GPT works by learning some approximation to physics, rather than ‘physics’?
IIUC, the physics being referred to is either through analogy (when it refers to real-world physics), or as a generalized ‘physics’ of [stepwise addition of tokens]. There’s no presumption of a simulation of physics (at any granularity).
E.g.:
Models trained with the strict simulation objective are directly incentivized to reverse-engineer the (semantic) physics of the training distribution, and consequently, to propagate simulations whose dynamical evolution is indistinguishable from that of training samples.
Apologies if I’m the one who’s confused :).
This just seemed like a natural explanation for your seeming to think the post is claiming a lot more mechanistically. (I think it’s claiming almost nothing)