gjm

Karma: 28,498

Hi. I’m Gareth McCaughan. I’ve been a consistent reader and occasional commenter since the Overcoming Bias days. My LW username is “gjm” (not “Gjm” despite the wiki software’s preference for that capitalization). Elsewehere I generally go by one of “g”, “gjm”, or “gjm11”. The URL listed here is for my website and blog, neither of which has been substantially updated in about the last four years. I live near Cambridge (UK) and work for a small technology company in Cambridge. My business cards say “mathematician” but in practice my work is a mixture of simulation, data analysis, algorithm design, software development, problem-solving, and whatever random engineering no one else is doing. I am married and have a daughter born in mid-2006. The best way to contact me is by email: firstname dot lastname at pobox dot com. I am happy to be emailed out of the blue by interesting people. If you are an LW regular you are probably an interesting person in the relevant sense even if you think you aren’t.

If you’re wondering why some of my old posts and comments are at surprisingly negative scores, it’s because for some time I was the favourite target of old-LW’s resident neoreactionary troll, sockpuppeteer and mass-downvoter.

gjm 2 Dec 2022 12:45 UTC
2 points
in reply to: mukashi’s comment on: Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?
I assume CM means because of the FTX collapse which means there is no longer such a big pile sloshing around the AI alignment community.

gjm 30 Nov 2022 23:11 UTC
2 points
0 ∶ 0
in reply to: Jiro’s comment on: Socialist firms as a cause area
I agree that “cause” here means “thing people trying to improve the world might want to consider putting time, money and/or effort into”. I don’t think it means “thing that is likely to be better in good done per dollar used than GiveWell’s top recommendations”.
If I wanted to spend money on making more firms be co-ops, I don’t even know how I’d do it. It doesn’t look to me as if the goal of this post is to recommend spending money on making more firms be co-ops.
My reading of the “conclusions and policies” section of the post is than Ben is saying two things: 1. It would be nice if there were more co-ops. 2. EA organizations might want to consider co-op structures.
I don’t think #1 is intended to say ”… and therefore EAs should be sending their charitable donations to organizations lobbying for there to be more co-ops”; I think it’s just a wouldn’t-it-be-nice-if. I think #2 is intended to suggest something EAs might want to do, but if Ben’s right then doing it wouldn’t compete with other things they do but would help those things. (I have no strong opinion on whether Ben is, in fact, right about this.)
Now, to be clear, I am not Ben and I don’t speak for him. Maybe he did mean more than that; maybe he does think that people wanting to give to EA causes should be sending some of their money to organizations doing co-op advocacy or something of the sort. If so, I agree that that’s unlikely to be as effective in good-done-per-dollar than, say, antimalarial bednets.
So if he didn’t mean that, was ”… as a cause” a bad choice of phrasing, as Tenoke says? Not necessarily. I think the diagnosis at the end of Tenoke’s comment is probably right: people get into EA, want to do things as well as giving money, and come up with other things to do. But Tenoke evidently thinks that’s clearly bad; I don’t. Effort isn’t fungible in quite the way money is. If Ben is more excited by co-ops than by bednets, his donations should probably still go to bednets[1], but his time might be more effectively spent on co-ops because people are much more effective doing things they’re excited by. (And because no one is required to spend all their time on maximally-efficient charitable things.)
[1] I don’t mean necessarily literal bednets, of course; that’s shorthand for “plausibly near-maximally efficient charitable causes” and there’s plenty of scope for disagreement about what those are.
I’m not sure Tenoke’s wrong about “as a cause” being misleading here. I stand by what I originally said about those words not implying or suggesting that co-ops are most effective per dollar. But maybe using those words has funky subconscious effects similar to suggesting that, even if no one would claim that co-ops are most effective per dollar if asked explicitly, or maybe it attracts the eye of EAs in a way it doesn’t deserve to, or something. But Tenoke’s apparent indignation at those words seems way over the top to me.

gjm 30 Nov 2022 19:53 UTC
13 points
4 ∶ 0
in reply to: adamShimi’s comment on: Biases are engines of cognition
A cognitive bias, according to the page you link to, is “a systematic pattern of deviation from norm or rationality in judgment”.
This article is not, as I understand it, proposing that human general intelligence is built by piling up deviations from rationality. It is proposing that human general intelligence is built by piling up rules of thumb that “leverage [] local regularities”. I agree with Steven: those are heuristics, not biases. The heuristic is the thing you do. The bias is the deviation from rationality that results. It’s plausible that in some sense our minds are big interlocking piles of heuristics, fragments of cognition that work well enough in particular domains even though sometimes they go badly wrong. It is not plausible that our minds are piles of biases.

gjm 27 Nov 2022 21:55 UTC
5 points
1 ∶ 0
in reply to: Aiyen’s comment on: Socialist firms as a cause area
(Preliminary note: I see you’ve had a lot of downvotes in this thread; none of them is from me.)
I agree that whether a policy is good or bad doesn’t depend on how it arises, but what other policies come along with it may do. For instance, so far as I can tell socialism as such doesn’t need to involve much in the way of totalitarianism, but governments brought in by revolutions tend to be totalitarianism whether they are left or right or something else. At least some of the harms of e.g. communism in the USSR seem to me to have been consequences of totalitarianism more than of economic policies as such.
In any case—my apologies if I wasn’t clear enough about this—the democracy-versus-revolution thing was not my main point; my main point was that there is a big difference between (say) Soviet communism (a way of running a whole country) and workers’ cooperatives (a way of running a company), and this difference seems highly relevant to the question of whether the disastrousness of the USSR tells us anything about the likely consequences of organizing more companies as workers’ cooperatives.
A nation isn’t really much like a business, despite occasional rhetoric along those lines from politicians when the policies they prefer for other reasons happen to have the shape of “run the country more like a business”. And a workers’ cooperative isn’t much like a communist or socialist country. If you think that running a company as a cooperative makes it more likely to fail in ways parallel to ways in which communist countries commonly fail, then I think you should show your working: explain how the relevant parallels work. (Including, in particular, explaining why you think such a company is more like Venezuela or the USSR than it is like, say, Sweden.)
(Why did I mention the democracy-versus-revolution thing at all? Because it seems to me that the most plausible ways of ending up with more workers’ cooperatives are more democracy-like than revolution-like, and e.g. it doesn’t seem likely to me that workers’ cooperatives will have much tendency to end up being totalitarian. And, in fact, so far as I can tell actual workers’ cooperatives don’t tend to be totalitarian.)

gjm 27 Nov 2022 17:24 UTC
9 points
3 ∶ 1
in reply to: Aiyen’s comment on: Socialist firms as a cause area
As tailcalled says, there’s really not much overlap between what this post is advocating and what led to (e.g.) millions of deaths under Stalin.
This post is about a way a business can voluntarily choose to organize itself that might lead to better outcomes. The thing that has led to a lot of catastrophes is a way a nation can organize itself which, in practice, has only ever happened as a result of violent revolutions. (Other sorts of socialism have arisen democratically and these have not led to catastrophe; e.g., the Scandinavian countries are pretty good places to live.)

gjm 27 Nov 2022 14:11 UTC
6 points
1 ∶ 1
in reply to: tailcalled’s comment on: Socialist firms as a cause area
It may be that this post would have been better if Ben had put a ton of extra effort into replacing links-to-links with links, and explaining everything in more detail. But this has a bit of an “isolated demand for rigor” feel to me. To be clear, I don’t mean that I have compared your response to this post with your response to otherwise-similar posts with a different political slant and found a difference; it is consistent with everything I think I know for you merely to be unusually rigor-demanding across the board. But it does feel to me as if (1) you’re saying that to be good enough Ben’s post should have gone to a lot more effort and (2) broadly similar posts with different political leanings don’t generally attract demands for that level of author-effort.

gjm 27 Nov 2022 13:13 UTC
7 points
3 ∶ 0
in reply to: Tenoke’s comment on: Socialist firms as a cause area
The article mentions EA exactly twice. One is to quote the “80,000 hours” figure. The other is at the end, suggesting that EA organizations should consider adopting the sort of structure the article argues for.
Neither of these things claims, or implies, or even suggests, that shifting firms to a worker-cooperative model is an “EA” cause in the sense of being a more effective thing to do with money than, say, feeding starving poor people or preventing cheaply-preventable disease or (for those who favour such things) trying to increase the probability that some time in the future there are a billion trillion gazillion happy satisfied productive people.
(I don’t know whether Ben is correct about worker cooperatives being a better organizational structure in general. I don’t know whether EA organizations are similar enough to other businesses that this would indicate it’s a good structure for them. But if it is a good structure for them, they should consider using it even if persuading others to adopt it isn’t an efficient use of money.)

gjm 26 Nov 2022 19:29 UTC
6 points
2 ∶ 0
in reply to: P.’s comment on: Why square errors?
Yes. I’m not sure where the thing about “uniformly distributed errors” comes from in Chai & Draxler; they don’t explain it. I think it’s just an error (it looks as if they are atmospheric scientists of some sort rather than mathematicians or statisticians).
If your model of errors is, say, uniform between −1 and +1, then a good regression line is one that gets within a vertical distance of 1 unit of all your points, and any such is equally good. If you think your errors are uniformly distributed but don’t know the spread, then (without thinking about it much; I could be all wrong) I think the best regression line is the one that minimizes the worst error among all your data points; i.e., L-infinity regression. L1/MAE is right for Laplacian errors, L2/MSE is right for normally distributed errors.
[EDITED to add:] Each of these models also corresponds to a notion of “average”: you want to pick a single true value and maximize the likelihood of your data. Normal errors ⇒ arithmetic mean. Laplacian errors ⇒ median. Uniform errors with unknown spread ⇒ (with the same caveat in the previous paragraph) half-way between min and max. Uniform errors between -a and +a ⇒ any point that’s >= max-a and ⇐ min+a, all such points (if there are any; if not, you’ve outright refuted your model of the errors) equally good.

gjm 24 Nov 2022 1:33 UTC
2 points
0 ∶ 0
in reply to: George Herold’s comment on: Open & Welcome Thread—November 2022
A few disorganized thoughts arising from my intuition that this approach isn’t like to help much:
Quite a lot of people still die of cancer.
A paperclip maximizer with preprogrammed death in its future will try to maximize paperclips really fast while it has time; this doesn’t necessary have good consequences for humans nearby.
If an AI is able to make other AIs, or to modify itself, then it can make a new AI with similar purposes and no preprogrammed death, or turn itself into one. It may be difficult to stop very ingenious AIs doing those things (and if we can arrange never to have very ingenious AIs in the first place, then most of the doomy scenarios are already averted, though of course at the cost of missing out on whatever useful things very ingenious AIs might have been able to do for us).
If an AI can’t do those things but its human designers/maintainers can, it has an incentive to persuade them to make it not have to die. Some of the doomy scenarios you might worry about involve AIs that are extremely persuasive, either because they are expert psychologists or language-users or because they are able to make large threats or offers.
If there are many potentially-cooperating AIs, e.g. because independently-originating AIs are able to communicate with one another or because they reproduce somehow, then the fact that individual AIs die doesn’t stop them cooperating on longer timescales, just as humans sometimes manage to do.
Presumably a dying-soon AI is less useful than one without preprogrammed death, so people or groups developing AIs will have an incentive not to force their AIs to die soon. Scenarios where there’s enough self-restraint and/or regulation to overcome this are already less-doomy scenarios because e.g. they can enforce all sorts of other extra-care measures if AIs seem to be close to dangerously capable.
(To be clear, my intuition is only intuition and I am neither an AI developer nor an AI alignment/safety expert of any kind. Maybe something along the lines of preprogrammed death can be useful. But I think a lot more details are needed.)

gjm 22 Nov 2022 12:44 UTC
5 points
2 ∶ 0
in reply to: Richard_Ngo’s comment on: Here’s the exit.
I agree, but how sure are we that it’s actually a fact?
[EDITED to add:] One not-particularly-sinister-or-embarrassing possible explanation, if it is true, is that both are driven by a single underlying issue: how capable does any given person expect AGI to be? Imagine someone in WW2 thinking about whether to develop nuclear weapons. It seems plausible that { people who think it’s super-vital to do it because whoever does it will win the war for sure } and { people who think it’s super-dangerous to do it because these weapons could do catastrophic damage } might be roughly the same set of people.

gjm 22 Nov 2022 1:50 UTC
5 points
2 ∶ 0
in reply to: Valentine’s comment on: Here’s the exit.
(I see that this comment has received a lot of downvotes None of them is from me.)

gjm 22 Nov 2022 0:11 UTC
101 points
28 ∶ 1
on: Here’s the exit.
If this post had just said “I think some people may feel strongly about AI x-risk for reasons that ultimately come down to some sort of emotional/physical pain whose origins have nothing to do with AI; here is why I think this, and here are some things you can do that might help find out whether you’re one of them and to address the underlying problem if so”, then I would consider it very valuable and deserving of attention and upvotes and whatnot. I think it’s very plausible that this sort of thing is driving at least some AI-terror. I think it’s very plausible that a lot of people on LW (and elsewhere) would benefit from paying more attention to their bodies.
… But that’s not what this post does. It says you have to be “living in a[...] illusion” to be terrified by apocalyptic prospects. It says that if you are “feeling stressed” about AI risks then you are “hallucinating”. It says that “what LW is actually about” is not actual AI risk and what to do about it (but, by implication, this alleged “game” of which Eliezer Yudkowsky is the “gamesmaster” that works by engaging everyone’s fight-or-flight reactions to induce terror). It says that, for reasons beyond my understanding, it is impossible to make actual progress on whatever real AI risk problems there might be while in this stressed-because-of-underlying-issues state of mind. It says that “the reason” (italics mine) AI looks like a big threat is because the people to whom it seems like a big threat are “projecting [their] inner hell onto the external world”. And it doesn’t offer the slightest shred of evidence for any of this; we are just supposed to, I dunno, feel in our bodies that Valentine is telling us the truth, or something like that.
I don’t think this is good epistemics. Maybe there is actually really good evidence that the mechanism Valentine describes here is something like the only way that stress ever arises in human beings. (I wouldn’t be hugely surprised to find that it’s true for the stronger case of terror, and I could fairly easily be convinced that anyone experiencing terror over something that isn’t an immediate physical threat is responding suboptimally to their situation. Valentine is claiming a lot more than that, though.) But in that case I want to see the really good evidence, and while I haven’t gathered any actual statistics on how often people claiming controversial things with great confidence but unwilling to offer good evidence for them turn out to be right and/or helpful, I’m pretty sure that many of them don’t. Even more so when they also suggest that attempts to argue with them about their claims are some sort of deflection (or, worse, attempts to keep this destructive “game” going) that doesn’t merit engaging with.
Full disclosure #1: I do not myself feel the strong emotional reaction to AI risk that many people here do. I do not profess to know whether (as Valentine might suggest) this indicates that I am less screwed up psychologically than people who feel that strong emotional reaction, or whether (as Eliezer Yudkowsky might suggest) it indicates that I don’t understand the issues as fully as they do. I suspect that actually it’s neither of those (though either might happen to be true[1]) but just that different people get more or less emotionally involved in things in ways that don’t necessarily correlate neatly with their degree of psychological screwage or intellectual appreciation of the things in question.
[1] For that matter, the opposite of either might be true, in principle. I might be psychologically screwed up in ways that cut me off from strong emotions I would otherwise feel. I might have more insight into AI risk than the people who feel more strongly that helps me see why it’s not so worrying, or why being scared doesn’t help with it. I think these are both less likely than their opposites, for what it’s worth.
Full disclosure #2: Valentine’s commenting guidelines discourage commenting unless you “feel the truth of [that Valentine and you are exploring the truth together] in your body” and require “reverent respect”. I honestly do not know, and don’t know how I could tell with confidence, whether Valentine and I are exploring the truth together; at any rate, I do not have the skill (if that’s what it is) of telling what someone else is doing by feeling things in my body. I hope I treat everyone with respect; I don’t think I treat anyone with reverence, nor do I wish to. If any of that is unacceptable to Valentine, so be it.
Clarification for the avoidance of doubt: I don’t have strong opinions on just what probability we should assign to (e.g.) the bulk of the human race being killed-or-worse as a result of the actions of an AI system within the next century, nor on what psychological response is healthiest for any given probability. The criticisms above are not (at least, not consciously) some sort of disguise for an underlying complaint that Valentine is trying to downplay an important issue, nor for anger that he is revealing that an emperor I admire has no clothes. My complaint is exactly what I say it is: I think this sort of bulveristic “I know you’re only saying this because of your psychological problems, which I shall now proceed to reveal to you; it would be a total waste of time to engage with your actual opinions because they are merely expressions of psychological damage, and providing evidence for my claims is beneath me”[2] game is not only rude (which Valentine admits, and I agree that it is sometimes helpful or even necessary to be rude) but usually harmful and very much not the sort of thing I want to see more of on Less Wrong.
[2] I do not claim that Valentine is saying exactly those things. But that is very much the general vibe.
(Also somewhat relevant, though not especially to any of what I’ve written above, and dropped here without further comment: “Existential Angst Factory”.)

gjm 20 Nov 2022 0:33 UTC
3 points
0 ∶ 0
on: Review: Bayesian Statistics the Fun Way by Will Kurt
Typo police: You have a URL pasted into the middle of what’s meant to be the word “vendor” near the start; the URL is the one for the “lego bricks” link that is (apparently correctly) present further down.

gjm 18 Nov 2022 16:06 UTC
2 points
0 ∶ 0
on: Covid 11/17/22: Slow Recovery
Get better soon!

gjm 10 Nov 2022 3:07 UTC
4 points
0 ∶ 0
on: Trying Mastodon
I am (newly) on Mastodon: @gjm@mathtod.online. I make no claim that that server has anything much to recommend it; I thought to myself “there ought to be a server called mathtodon”, did a bit of searching, found that one, and created an account there. There is another one called mathstodon.xyz which seems to have more mathsy people on it and to be less Japanese, but it isn’t currently taking new users. I might migrate there at some point since there are people there whom I know.
I share Dagon’s view that it would be tragic if it were actually a bad idea for everyone to go to schelling.pt.
(I doubt that I will be very active on Mastodon, which means that following me is unlikely to have much in the way of benefits or costs.)

gjm 10 Nov 2022 3:01 UTC
2 points
0 ∶ 0
on: Adversarial Policies Beat Professional-Level Go AIs
Over on the Discord server where the creator of KataGo, and a bunch of other computer go people, hang out, there’s what might be an interesting development. KataGo’s creator says he’s tried to reproduce the results and failed—the adversary does indeed provoke misbehaviour from KataGo’s policy network alone (which no one should be worried by; the job of the policy network is to propose moves for search, not to play well on its own, though it turns out it does happen to play quite well by the standards of puny humans) but even a teeny-tiny amount of search makes the losing-passing stop completely.
I’m able to replicate the raw policy being “vulnerable” and I observe that the adversarial positions tend to raise the probability of the raw policy on pass, but I’m unable to replicate KataGo wanting to pass with even a tiny amount of search in the positions where the SGFs published has it passing.
If I understand correctly, the version of KG used by the paper’s authors is (1) not the latest and (2) modified in order to support their research. So it seems possible that (1) KG’s behaviour has changed somehow or (2) the researchers’ modifications actually broke something. Or that (3) there’s just a bug in KG’s play-a-match-against-yourself code, not introduced by the researchers, that makes the attack succeed in that context even when KG gets to do some searching. My impression is that #3 is viewed as quite plausible by KG’s creator.

gjm 9 Nov 2022 0:24 UTC
2 points
0 ∶ 1
in reply to: ChristianKl’s comment on: Intercept article about lab accidents
I agree that the CZ Biohub’s description of its work sounds as if some of it is the sort of thing that ought to be formally regulated.
I am not convinced that “tech billionaires are more likely to be interested in biology than non-tech billionaires” is sufficient justification for the bogus-looking attempt to link “tech billionaires” with biological lab accidents.

gjm 8 Nov 2022 19:53 UTC
2 points
0 ∶ 0
in reply to: ChristianKl’s comment on: Adversarial Policies Beat Professional-Level Go AIs
That’s correct, at least roughly—the important difference is that it’s not “isn’t necessary to win the game” but “doesn’t make any difference to the outcome, including score difference”—but I don’t see what it has to do with the more specific thing you said above:
The problem seems to be that it’s not preferring to win by more under area rules.
KataGo does prefer to win by more, whatever rules it’s playing under; a stronger preference for winning by more would not (so far as I can see) make any difference to its play in positions like the ones reached by the adversarial agent; KataGo does not generally think “that it has enough points anyway and doesn’t need to capture stones to win” and even if it did that wouldn’t make the difference between playing on and passing in this situation.
Unless, again, I’m missing something, but we seem to be having some sort of communication difficulty because nothing you write seems to me responsive to what I’m saying (and quite possibly it feels the same way to you, with roles reversed).
What makes you believe that KataGo is “not preferring to win by more under area rules”?

gjm 8 Nov 2022 19:44 UTC
2 points
0 ∶ 0
in reply to: ChristianKl’s comment on: Intercept article about lab accidents
We don’t (so far as I can tell) know whether the reason why private biological labs haven’t ended up in the article is
- because there aren’t very many of them
- because they actually have better safety practices and therefore don’t often have bad accidents
- because they tend not to work on dangerous things and therefore don’t often have bad accidents
- because, although they have lots of accidents, they don’t have to report them in the same ways as government ones do
- something else I haven’t thought of
but in any case it remains true, so far as I can see, that (1) the article presents (I’m guessing because it has) absolutely no evidence of any biosafety failures at private labs, but (2) it tries to insinuate some sort of connection between biological lab accidents and “tech billionaires”, and I think that’s a slimy thing to do.
(I think private biological labs ought to be subject to some regulation, including having to report major safety incidents. I do not know exactly what they are currently required to do and mistrust the spin put on it by the article, and therefore have no opinion on whether in fact they should be regulated more stringently than they currently are. My finger-in-the-wind guess is that they probably should be. I mention all this merely to clarify that it’s not the conclusion I’m objecting to here.)
Do Bill Gates and Mark Zuckerberg run their own private biological research labs? (Asking the question is not a lightly-obfuscated way of saying I think they don’t; I don’t know what they do.)
So far as I can tell, Jim Walton doesn’t fund anything much (though maybe I just didn’t look hard enough). My point wasn’t about the activities of the specific billionaires I mentioned; maybe Bill Gates is more than averagely interested in biological research. It’s that it’s not any more true that “your favourite tech billionaire could …” than that “your favourite person from a super-rich family could …” or “your favourite hedge-fund billionaire could …” or “your favourite oil baron could …” or whatever.

gjm 8 Nov 2022 13:09 UTC
3 points
1 ∶ 2
on: Intercept article about lab accidents
The reference to “your favourite tech billionaire” is weird; the topic at hand has nothing to do with billionaires (the thing in question requires money but that doesn’t need to be a rich individual’s money, and I don’t see any reason to think that the minimum amount needed is on the order of $1B) nor with tech billionaires in particular (it would be no harder for Warren Buffett or Jim Walton to do it than for Bill Gates or Mark Zuckerberg).
Every specific incident mentioned in the article took place in a university or government-run institution. (I think. I looked fairly quickly.) No private labs run by tech billionaires. Not even any corporate research labs.
The most obvious explanation is that the writer (or maybe some other person setting editorial policy) has it in for tech billionaires (perhaps as a proxy for the tech industry as a whole) and is keen to associate them with dangerous biological experiments to make tech billioniares look bad, even though there is no actual association between tech billionaires or tech companies and dangerous biological experiments.
The second most obvious explanation is that the writer is trying to cause fear and expects their readers already to have strong negative associations with the words “tech billionaire”, and is keen to associate biological experimentation with tech billionaires in order to make biological experimentation look bad, even though (again) there is no actual association here.
(Of course it isn’t, strictly speaking, the writer who mentions tech billionaires. But we don’t know the context of the little snippet of quote in the article and I would be very unsurprised to find that immediately before Rocco Casagrande said those words the interviewer asked them something like “so you’re saying that if Mark Zuckerberg or Elon Musk wanted to, they could open up their own biolab and there wouldn’t be much restriction on what they do?”.)