Adversarial Training

TagLast edit: Jun 3, 2022, 1:30 AM by Ruby

Adversarial training, importance sampling, and anti-adversarial training for AI whistleblowing

BuckJun 2, 2022, 11:48 PM

42 points

20 votes

Overall karma indicates overall quality.

0 comments3 min readLW link

Behavioral red-teaming is unlikely to produce clear, strong evidence that models aren’t scheming

BuckOct 10, 2024, 1:36 PM

101 points

35 votes

Overall karma indicates overall quality.

4 comments13 min readLW link

Solving adversarial attacks in computer vision as a baby version of general AI alignment

Stanislav FortAug 29, 2024, 5:17 PM

98 points

41 votes

Overall karma indicates overall quality.

9 comments7 min readLW link

High-stakes alignment via adversarial training [Redwood Research report]

dmz, LawrenceC and Nate Thomas

May 5, 2022, 12:59 AM

142 points

64 votes

Overall karma indicates overall quality.

29 comments9 min readLW link

Can Generalized Adversarial Testing Enable More Rigorous LLM Safety Evals?

scasperJul 30, 2024, 2:57 PM

25 points

12 votes

Overall karma indicates overall quality.

0 comments4 min readLW link

AI Safety 101 - Chapter 5.2 - Unrestricted Adversarial Training

Charbel-RaphaëlOct 31, 2023, 2:34 PM

17 points

7 votes

Overall karma indicates overall quality.

0 comments19 min readLW link

Deep Forgetting & Unlearning for Safely-Scoped LLMs

scasperDec 5, 2023, 4:48 PM

127 points

54 votes

Overall karma indicates overall quality.

30 comments13 min readLW link

AXRP Episode 17 - Training for Very High Reliability with Daniel Ziegler

DanielFilanAug 21, 2022, 11:50 PM

16 points

6 votes

Overall karma indicates overall quality.

0 comments35 min readLW link

Takeaways from our robust injury classifier project [Redwood Research]

dmzSep 17, 2022, 3:55 AM

143 points

64 votes

Overall karma indicates overall quality.

12 comments6 min readLW link 1 review

Ironing Out the Squiggles

Zack_M_DavisApr 29, 2024, 4:13 PM

159 points

70 votes

Overall karma indicates overall quality.

36 comments11 min readLW link

Adversarial Robustness Could Help Prevent Catastrophic Misuse

aogDec 11, 2023, 7:12 PM

30 points

9 votes

Overall karma indicates overall quality.

18 comments9 min readLW link

Some thoughts on why adversarial training might be useful

Beth BarnesDec 8, 2021, 1:28 AM

9 points

4 votes

Overall karma indicates overall quality.

6 comments3 min readLW link

Oversight Leagues: The Training Game as a Feature

Paul BricmanSep 9, 2022, 10:08 AM

20 points

9 votes

Overall karma indicates overall quality.

6 comments10 min readLW link

EIS IX: Interpretability and Adversaries

scasperFeb 20, 2023, 6:25 PM

30 points

11 votes

Overall karma indicates overall quality.

8 comments8 min readLW link

Latent Adversarial Training (LAT) Improves the Representation of Refusal

alexandraabbas, nlpet and hal2k

Jan 6, 2025, 10:24 AM

21 points

13 votes

Overall karma indicates overall quality.

6 comments10 min readLW link

Continuous Adversarial Quality Assurance: Extending RLHF and Constitutional AI

Benaya KorenJul 8, 2023, 5:32 PM

6 points

7 votes

Overall karma indicates overall quality.

0 comments9 min readLW link

Can We Trust the Judge? A novel method of Modelling Human Bias and Systematic Error in Debate-Based Scalable Oversight

Andreea ZamanJul 19, 2025, 9:44 PM

1 point

1 vote

Overall karma indicates overall quality.

0 comments7 min readLW link

Enhancing Genomic Foundation Model Robustness through Iterative Black-Box Adversarial Training

Jeyashree Krishnan and Ajay Mandyam Rangarajan

Oct 14, 2025, 8:54 PM

8 points

5 votes

Overall karma indicates overall quality.

0 comments7 min readLW link

Does robustness improve with scale?

ChengCheng, niki.h, Ian McKenzie, Oskar Hollinsworth, Tom Tseng and AdamGleave

Jul 25, 2024, 8:55 PM

14 points

6 votes

Overall karma indicates overall quality.

0 comments1 min readLW link

(far.ai)

EIS XII: Summary

scasperFeb 23, 2023, 5:45 PM

19 points

11 votes

Overall karma indicates overall quality.

0 comments6 min readLW link

Loss Curves

James CamachoMay 6, 2025, 10:22 PM

16 points

5 votes

Overall karma indicates overall quality.

3 comments4 min readLW link

(github.com)

Beyond the Board: Exploring AI Robustness Through Go

AdamGleaveJun 19, 2024, 4:40 PM

41 points

12 votes

Overall karma indicates overall quality.

2 comments1 min readLW link

(far.ai)

A Solution to Sandbagging and other Self-Provable Misalignment: Constitutional AI Detectives

Knight LeeApr 14, 2025, 10:27 AM

−3 points

2 votes

Overall karma indicates overall quality.

2 comments4 min readLW link

LLM Sycophancy: grooming, proto-sentience, or both?

gturner4Oct 13, 2025, 12:58 AM

1 point

1 vote

Overall karma indicates overall quality.

0 comments2 min readLW link

EIS XI: Moving Forward

scasperFeb 22, 2023, 7:05 PM

19 points

11 votes

Overall karma indicates overall quality.

2 comments9 min readLW link

Intriguing Properties of gpt-oss Jailbreaks

zroe1 and Jack Sanderson

Aug 13, 2025, 7:42 PM

16 points

11 votes

Overall karma indicates overall quality.

0 comments10 min readLW link

(xlabaisecurity.com)

Latent Adversarial Training

Adam JermynJun 29, 2022, 8:04 PM

57 points

27 votes

Overall karma indicates overall quality.

13 comments5 min readLW link

No comments.

Keyboard shortcuts

Keys shown in yellow (e.g., ]) are accesskeys, and require a browser-specific modifier key (or keys).

Keys shown in grey (e.g., ?) do not require any modifier keys.

General
? Show keyboard shortcuts
Esc Hide keyboard shortcuts

Site navigation
h Go to Home (a.k.a. “Frontpage”) view
f Go to Featured (a.k.a. “Curated”) view
a Go to All (a.k.a. “Community”) view
m Go to Meta view
v Go to Tags view
c Go to Recent Comments view
r Go to Archive view
q Go to Sequences view
t Go to About page
u Go to User or Login page
o Go to Inbox page

Page navigation
, Jump up to top of page
. Jump down to bottom of page
/ Jump to top of comments section
s Search

Page actions
n New post or comment
e Edit current post

Post/comment list views
. Focus next entry in list
, Focus previous entry in list
; Cycle between links in focused entry
Enter Go to currently focused entry
Esc Unfocus currently focused entry
] Go to next page
[ Go to previous page
\ Go to first page
e Edit currently focused post

Editor
k Bold text
i Italic text
l Insert hyperlink
q Blockquote text

Appearance
= Increase text size
- Decrease text size
0 Reset to default text size
′ Cycle through content width settings
1 Switch to default theme [A]
2 Switch to dark theme [B]
3 Switch to grey theme [C]
4 Switch to ultramodern theme [D]
5 Switch to simple theme [E]
6 Switch to brutalist theme [F]
7 Switch to ReadTheSequences theme [G]
8 Switch to classic Less Wrong theme [H]
9 Switch to modern Less Wrong theme [I]
; Open theme tweaker
Enter Save changes and close theme tweaker
Esc Close theme tweaker (without saving)

Slide shows
l Start/resume slideshow
Esc Exit slideshow
→↓ Next slide
←↑ Previous slide
Space Reset slide zoom

Miscellaneous
x Switch to next view on user page
z Switch to previous view on user page
` Toggle compact comment list view
g Toggle anti-kibitzer