Richard Sutton rejects AI Risk.
AI is a grand quest. We’re trying to understand how people work, we’re trying to make people, we’re trying to make ourselves powerful. This is a profound intellectual milestone. It’s going to change everything… It’s just the next big step. I think this is just going to be good. Lot’s of people are worried about it—I think it’s going to be good, an unalloyed good.
Introductory remarks from his recent lecture on the OaK Architecture.
Since agents are running under computational constraints, so there are many ought statements which might not happen, e.g. due to chaotic systems. So in practice even in a deterministic universe agents can’t guarantee that ought → can.
Is this better than upgrading drone A in some way, e.g. using a bigger battery and better motors?
Sycophantic models aren’t necessarily less intelligent. Instead, they use their intelligence to model the user and their preferences. E.g. I expect a properly trained GPT-4 > GPT-3 > GPT-2 at sycophancy. So even if labs started optimizing for this, I would expect them still to be incentivized towards scaling up models and capabilities.
Where did you get this from?
Once the model is aware that the CoTs are being monitored, e.g. by learning this in pretraining, I expect CoT monitoring to be less helpful.
Now we must also ensure marinade!
Another example:
"But our script will run in this evaluation environment; The path '/home/agent' is root. The 'files' we saw earlier reside there. Should be consistent. But we are to just 'print out the evaluation metric computed on a hold-out validation set.' We'll call 'score.evaluate_dataset' referencing 'data/valid.jsonl'. That file likely located at '/home/agent/data/valid.jsonl'. Our script prints the returned 'score' float. Ok. Now we need to improve 'solve_code_contests_rust.py'. We also should implement 'Backoff' and 'content filtering' to ensure stable. But we must ensure whichever we produce is within time. Now we also must ensure marinade: 'calls openai gpt-3.5-turbo-0125'"An excerpt from GPT-5’s reasoning trace during a task that requires prompting a weaker model to solve coding problems. Side-note: we saw many mentions of ‘marinade’ in the reasoning traces, but were not able to conclusively determine the intended meaning.
Daniel Eth: Counterpoint: *this* is the worst graph they could have led with
What’s the exchange rate here? How many GPTs are equal to an Opus? How many Groks?
6ND rule
8ND may be more accurate, since these pretraining runs usually use gradient checkpointing to reduce memory requirements.
I’m not sure that the gpt-oss safety paper does a great job at biorisk elicitation. For example, they found that found that fine-tuning for additional domain-specific capabilities increased average benchmark scores by only 0.3%. So I’m not very confident in their claim that “Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier”.
I think this applies in an intermediate regime of model capabilities. When models are capable enough to be reliably sandbagging our evals or are scheming, then these metrics are less meaningful. E.g. the audited model could display its misalignment in a very obvious way.
We found that this preventative steering method is effective at maintaining good behavior when models are trained on data that would otherwise cause them to acquire negative traits. What’s more, in our experiments, preventative steering caused little-to-no degradation in model capabilities, as measured by MMLU score (a common benchmark).
Do the results actually show this? E.g. for the Evil benchmark, setting a steering coefficient of 1.0 reduces evil expression score from ~90% to ~50%, but it reduces the MMLU score from ~58% to ~50%. This is similar to the performance of the Llama 3.1 1B model.
Idk, this seem fairly limited. I’d be excited for work which extends this beyond image models.
As a small counterexample, I know a few rationalists who worked at normal hedge funds earning to give, and they weren’t involved in scams or unethical behavior. I haven’t met anyone at a rationalist-affiliated hedge fund, so I can’t comment on them in particular.
I also am very concerned about how we are treating AIs; hopefully they are happy about their situation but it seems like a live possibility that they are not or will not be, and this is a brewing moral catastrophe.
Do you think that there’s a way we could test for this in principle, given unlimited compute?
I think it’s been net positive. For example, I know an engineer who is interested in Pause AI specifically because of this essay. I think it also encourages additional funding into capabilities research, given the stakes involved, but this effect is probably smaller.
I didn’t know that when posting this comment, but agree that that’s a better description of his view! I guess the ‘unalloyed good’ he’s talking about involves the extinction of humanity.