RSS

Jacob Pfau

Karma: 1,577

UK AISI Alignment Team

Se­quent: scale and au­toma­tion for higher con­fi­dence in alignment

Geoffrey Irving, Alex HT, Jesse Hoogland, Daniel Murfet, Jacob Pfau, Marco Cozzi and Stan van Wingerden
10 Jun 2026 15:37 UTC
276 points
2 comments11 min readLW link
(sequent.org)

Au­to­mated Align­ment is Harder Than You Think

Aleksandr Bowkis, Marie_DB, Jacob Pfau and Geoffrey Irving
14 May 2026 22:01 UTC
143 points
6 comments3 min readLW link
(arxiv.org)

From per­sonas to in­ten­tions: to­wards a sci­ence of mo­ti­va­tions for AI models

David Africa and Jacob Pfau
14 Apr 2026 12:26 UTC
77 points
5 comments7 min readLW link

Re­search Areas in Meth­ods for Post-train­ing and Elic­i­ta­tion (The Align­ment Pro­ject by UK AISI)

Jacob Pfau and Benjamin Hilton
1 Aug 2025 10:27 UTC
12 points
0 comments6 min readLW link
(alignmentproject.aisi.gov.uk)

Re­search Areas in Bench­mark De­sign and Eval­u­a­tion (The Align­ment Pro­ject by UK AISI)

Jacob Pfau and Benjamin Hilton
1 Aug 2025 10:26 UTC
10 points
0 comments9 min readLW link
(alignmentproject.aisi.gov.uk)

Re­search Areas in Prob­a­bil­is­tic Meth­ods (The Align­ment Pro­ject by UK AISI)

Jacob Pfau and Benjamin Hilton
1 Aug 2025 10:26 UTC
4 points
0 comments4 min readLW link
(alignmentproject.aisi.gov.uk)

Re­search Areas in Eval­u­a­tion and Guaran­tees in Re­in­force­ment Learn­ing (The Align­ment Pro­ject by UK AISI)

Jacob Pfau and Benjamin Hilton
1 Aug 2025 9:53 UTC
14 points
0 comments11 min readLW link
(alignmentproject.aisi.gov.uk)

The Align­ment Pro­ject by UK AISI

Mojmir, Benjamin Hilton, Jacob Pfau, Geoffrey Irving, Joseph Bloom, Tomek Korbak, David Africa and Edmund Lau
1 Aug 2025 9:52 UTC
29 points
0 comments2 min readLW link
(alignmentproject.aisi.gov.uk)

Un­ex­ploitable search: block­ing mal­i­cious use of free parameters

Jacob Pfau and Geoffrey Irving
21 May 2025 17:23 UTC
40 points
16 comments6 min readLW link

An al­ign­ment safety case sketch based on debate

Marie_DB, Jacob Pfau, Benjamin Hilton and Geoffrey Irving
8 May 2025 15:02 UTC
62 points
21 comments25 min readLW link
(arxiv.org)

UK AISI’s Align­ment Team: Re­search Agenda

Benjamin Hilton, Jacob Pfau, Marie_DB and Geoffrey Irving
7 May 2025 16:33 UTC
115 points
3 comments11 min readLW link

Prospects for Align­ment Au­toma­tion: In­ter­pretabil­ity Case Study

Jacob Pfau and Geoffrey Irving
21 Mar 2025 14:05 UTC
33 points
5 comments8 min readLW link

Au­dit­ing LMs with coun­ter­fac­tual search: a tool for con­trol and ELK

Jacob Pfau20 Feb 2024 0:02 UTC
28 points
6 comments10 min readLW link

LM Si­tu­a­tional Aware­ness, Eval­u­a­tion Pro­posal: Vio­lat­ing Imitation

Jacob Pfau26 Apr 2023 22:53 UTC
16 points
2 comments2 min readLW link

Early situ­a­tional aware­ness and its im­pli­ca­tions, a story

Jacob Pfau6 Feb 2023 20:45 UTC
29 points
6 comments3 min readLW link

Ja­cob Pfau’s Shortform

Jacob Pfau17 Jun 2022 16:40 UTC
3 points
42 comments1 min readLW link