An open mind is like a fortress with its gates unbarred and unguarded.
MathiasKB
Karma: 404
if it makes it easier, I can add the questions to manifold if you provide a list of questions and resolution criteria.
thanks for pointing that out, I’ve added a note in the description
There’s countries where cooperative firms are doing fine. Most of Denmark’s supermarket chains are owned by the cooperative coop. Denmark’s largest dairy producer Arla is a cooperative too. Both operate in a free market and are out-competing privately owned competitors.
Both also resort to many of the same dirty tricks traditionally structured firms are pulling. Arla, for example, has done tremendous harm to the plant-based industry through aggressive lobbying. Structuring firms as cooperatives doesn’t magically make them aligned.
Cicero, as it is redirecting its entire fleet: ‘What did you call me?’
Yeah, my original claim is wrong. It’s clear that KataGo is just playing sub-optimally outside of distribution, rather than punished for playing optimally under a different ruleset than its being evaluated.
Actually this modification shouldn’t matter. After looking into the definition of pass-alive, the dead stones in the adversarial attacks are clearly not pass-alive.
Under both unmodified and pass-alive modified tromp-taylor rules, KataGo would lose here and its surprising that self-play left such a weakness.
The authors are definitely onto something, and my original claim that the attack only works due to kataGo being trained under a different rule-set is incorrect.
No, the KataGo paper explicitly states at the start of page 4:
”Self play games used Tromp-Taylor rules [21] modified to not require capturing stones within pass-aliveterritory”Had KataGo been trained on unmodified Tromp-Taylor rules, the attack would not have worked. The attack only works because the authors are having KataGo play under a different ruleset than it was trained on.
If I have the details right, I am honestly very confused about what the authors are trying to prove with this paper. Given their Twitter announcement claimed that the rulesets were the same my best guess is simply that it was an oversight on their part.
(EDIT: this modification doesn’t matter, the authors are right, I am wrong. See my comment below)
As someone who plays a lot of go, this result looks very suspicious to me. To me it looks like the primary reason this attack works is due to an artifact of the automatic scoring system used in the attack. I don’t think this attack would be replicable in other games, or even KataGo trained on a correct implementation.
In the example included on the website, KataGo (White) is passing because it correctly identifies the adversary’s (Black) stones as dead meaning the entire outside would be its territory. Playing any move in KataGo’s position would gain no points (and lose a point under Japanese scoring rules), so KataGo passes.
The game then ends and the automatic scoring system designates the outside as undecided, granting white 0 points and giving black the win.
If the match were to be played between two human players, they would have to agree whether the outside territory belongs to white or not. If black were to claim their outside stones are alive the game would continue until both players pass and agree about the status of all territory (see ‘disputes’ in the AGA ruleset).
But in the adversarial attack, the game ends after the pass and black gets the win due to the automatic scoring system deciding the outcome. But the only reason that KataGo passed is that it correctly inferred that it was in a winning position with no way to increase its winning probability! Claiming that to be a successful adversarial attack rings a bit hollow to me.
I wouldn’t conclude anything from this attack, other than that Go is a game with a lot of edge-cases that need to be correctly handled.
EDIT: I just noticed the authors address this on the website, but I still think this significantly diminishes the ‘impressiveness’ of the adversarial attack. I don’t know the exact ruleset KataGo is trained under, but unless it’s the exact same as the ruleset used to evaluate the adversarial attack, the attack only works due to KataGo playing to win a different game than the adversary.
Evaluating the RCT is a chance to train the evaluation-muscle in a well-defined domain with feedback. I’ve generally found that the people who are best at evaluations in RCT’able domains, are better at evaluating the hard-to-evaluate claims as well.
Often the difficult to evaluate domains have ways of getting feedback, but if you’re not in the habit of looking for it, you’re less likely to find the creative ways to get data.
I think a much more common failure mode within this community, is that we get way overconfident beliefs about hard-to-evaluate domains, because there aren’t many feedback loops and we aren’t in the habit of looking for them.
Does anyone know of any zero-trust investigations on nuclear risk done in the EA/Rationalist community? Open phil has funded nuclear work, so they probably have an analysis somewhere that concluded it is a serious risk to civilization, but I haven’t ever looked into these analyses.
For each tweet the post found arguing their point, I can find two arguing the opposite. Yes, in theory tweets are data points, but in practice the author just uses them to confirm his already held beliefs.
I don’t think the real world is good enough either.
The fact that humans feel a strong sense of the tetris effect, suggest to me that the brain is constantly generating and training on synthetic data.
Another issue with greenwashing and safetywashing is that it gives people who earnestly care a false impression that they are meaningfully contributing.
Despite thousands of green initiatives, we’re likely to blow way past the 1.5c mark because the far majority of those initiatives failed to address the core causes of climate change. Each plastic-straw ban and reusable diaper gives people an incorrect impression that they are doing something meaningful to improve the climate.
Similarly I worry that many people will convince themselves that they are doing something meaningful to improve AI Safety, but because they failed to address the core issues they end up contributing nothing. I am not saying this as a pure hypothetical, I think this is already happening to a large extent.
I quit a well paying job to become a policy trainee working with AI in the European Parliament because I was optimizing for “do something which looks like contributing to AI safety”, with a strenuous at best model of how my work would actually lead to a world which creates safe AI. What horrified me during this was that a majority of people I spoke to in the field of AI policy seemed to be making similar errors as I was.Many of us justify our work this by pointing out the second-order benefits such as “policy work is field-building”, “This policy will help create better norms” or “I’m skilling up / getting myself to a place of influence”, and while these second order effects are real and important, we should be very sceptical of interventions whose first-order effects aren’t promising.
I apologize that this became a bit of a rant about AI Policy, but I have been annoyed with myself for making such basic errors and this post helped me put a word on what I was doing.
The primary question on my mind is something like this:
How much retraining is needed for Gato to learn a new task? Given a task, such as “Stack objects and compose a relevant poem” which combines skills it has already learned, yet is a fundamentally different task, does it quickly learn how to perform well at it?
If not, then it seems Deepmind ‘merely’ managed to get a single agent to do a bunch of tasks we were previously only able to do with multiple agents. If it is also quicker at learning new tasks in similar domains, than an agent trained solely to do it, then it seems like a big step towards general intelligence.
Hi Niplav, happy to hear you think that.
I just uploaded the pkl files that include the pandas dataframes for the metaculus questions and GPT’s completions for the best performing prompt to github. Let me know if you need anything else :)
https://github.com/MperorM/gpt3-metaculus
I think wife rolls of the tongue uniquely well here due to ‘wife’ rhyming with ‘life’, creating the pun. Outside of that I don’t buy it. In Denmark, wife-jokes are common despite wife being a two syllable word (kone) and husband-jokes are rare despite husband being a one syllable word (mand).
My model of why we see this has much more to do with gender norms and normalised misogyny than with catchiness of the words.
Good point, though I would prefer we name it Quality Adjusted Spouse Years :)
Fantastic to see this wonderful game be passed onto a new generation!
I can’t speak specifically about American politics, but I work for the equivalent of the democrats in the European Parliament and participate actively in municipal politics, which probably carries some resemblance.
The biggest barrier you have to overcome is getting your political party to agree you would be the best person to represent them in office. Join the political party you feel most closely aligned with and start showing up for its local events. You’ll be surprised at the influence you can have on local politics just by showing up.Your most difficult task is to convince your fellow party members to put their trust and weight behind you. Why should I support you? I need to instinctively feel that you’ll represent my interests and keep me safe, before I’m spending a minute volunteering for your campaign.
If you can do this, you’ll be able to figure out the rest as you go along.
This wouldn’t be the first time Deepmind pulled these shenanigans.
My impression of Deepmind is they like playing up the impressiveness of their achievements to give an impression of having ‘solved’ some issue, never saying anything technically false, while suspiciously leaving out relevant information and failing to do obvious tests of their models which would reveal a less impressive achievement.
For Alphastar they claimed ‘grandmaster’ level, but didn’t show any easily available stats which would make it possible to verify. As someone who was in Grandmaster league at the time of it playing (might even have run into it on ladder, some of my teammates did), its play at best felt like low grandmaster to me.
At their event showing an earlier prototype off, they had one player (TLO) play their off-race with which he certainly was not at a grandmaster level. The pro player (Mana) playing their main race beat it at the event, when they had it play with the same limited camera access humans have. I don’t remember all the details anymore, but I remember being continuously annoyed by suspicious omission after suspicious omission.
What annoys me most is that this still was a wildly impressive achievement! Just state in the paper: “we managed to reach grandmaster with one out of three factions”—Nobody has ever managed to create AI that played remotely as well as this!
Similarly Deepminds no-search chess engine is surely the furthest anyone has gotten without search. Even if it didn’t quite make grandmaster, just say so!