There was a paper I read a few years ago where they said if they played a really unusual strategy – not a good one; I think it was one a human would have recognized as a blunder – it was out of distribution enough that the AI gets confused and loses. Whatever came of that?
I guess this is karma for me ever having replied to a question with a link to lmgtfy[1].
I thought it would be clear from context that what I was asking for was a first-hand account of how (and whether) such adversarial strategies, which I read are simple enough to be possible to learn and implement unaided over-the-board, had impacted play in these no-stakes Go schools.
I don’t think that was clear at all. Personally, I thought the question was a sensible one on its own, and something I had wondered myself, and that’s why I took the time to look it up for you rather than downvote what looked like laziness - ‘whatever happened to that KataGo adversarial attack research, anyway? I haven’t heard about it in a while. Surely it hasn’t been fixed? I would’ve heard about it, I think, given how DRL agents are so fragile in general, that a robust fix to adversarial attacks in any DRL setting ought to be big news. But what’s the current state of play?’
But I have never seen anyone mention seeing someone go to the length of memorizing anti-KataGo strategies or deploying them ‘the real world’, aside from the documented example in this KG line of research of someone doing so just to prove that the circling hack can be deployed by a real human player against a live bot and is not intractable in practice (as many adversarial examples are very fragile or require near-superhuman capabilities to deploy correctly).
I would be shocked if anyone was doing so given that it’s a lot of work to win games against a few specific obsolete versions of one specific Go agent (the transfer to other agents is real but the success rate goes from ~100% to <5%, IIRC) where the human operator could just take over at some point when they recognize the weird thing going on, or where you could just quit and go find an easier game to cheat in yourself (such as against a sucker human player) rather than hacking their Go agent, given that the whole point is that they are lazy and cheating and trying to get a quick easy win.
There was a paper I read a few years ago where they said if they played a really unusual strategy – not a good one; I think it was one a human would have recognized as a blunder – it was out of distribution enough that the AI gets confused and loses. Whatever came of that?
Well, the obvious thing to do is to check the reverse citations. Or just ask a LLM: https://chatgpt.com/share/69f58633-01b4-83e8-b3b1-de42d3d196c9
FWIW, my understanding was that individual attacks could be fixed by further training or architectural tweaks, but you could still find new attacks and so the basic problem of adversarial robustness in DRL agents was nowhere close to being solved. The GPT-5.5 Pro Deep Research report says something similar. It looks like the best ref would be https://www.reddit.com/r/baduk/comments/14prv4f/katago_should_be_partially_resistant_to_cyclic/ + https://gomagic.org/david-wu-on-building-katago/#h-the-circular-group-problem-where-bots-still-misjudge-go
I guess this is karma for me ever having replied to a question with a link to lmgtfy[1].
I thought it would be clear from context that what I was asking for was a first-hand account of how (and whether) such adversarial strategies, which I read are simple enough to be possible to learn and implement unaided over-the-board, had impacted play in these no-stakes Go schools.
In my defense, that was like fifteen years ago, back when Google still reliably answered questions.
I don’t think that was clear at all. Personally, I thought the question was a sensible one on its own, and something I had wondered myself, and that’s why I took the time to look it up for you rather than downvote what looked like laziness - ‘whatever happened to that KataGo adversarial attack research, anyway? I haven’t heard about it in a while. Surely it hasn’t been fixed? I would’ve heard about it, I think, given how DRL agents are so fragile in general, that a robust fix to adversarial attacks in any DRL setting ought to be big news. But what’s the current state of play?’
But I have never seen anyone mention seeing someone go to the length of memorizing anti-KataGo strategies or deploying them ‘the real world’, aside from the documented example in this KG line of research of someone doing so just to prove that the circling hack can be deployed by a real human player against a live bot and is not intractable in practice (as many adversarial examples are very fragile or require near-superhuman capabilities to deploy correctly).
I would be shocked if anyone was doing so given that it’s a lot of work to win games against a few specific obsolete versions of one specific Go agent (the transfer to other agents is real but the success rate goes from ~100% to <5%, IIRC) where the human operator could just take over at some point when they recognize the weird thing going on, or where you could just quit and go find an easier game to cheat in yourself (such as against a sucker human player) rather than hacking their Go agent, given that the whole point is that they are lazy and cheating and trying to get a quick easy win.
I don’t think there is anyone using this to counter cheaters, but people do it for fun, and there is definitely more than one example.
Here is a video where Nick Sibicky beats katago with 10k playouts.
Here is a game on ogs where a player captures the entire board against katago micro.