Garrett Baker

Karma: 4,792

I have signed no contracts or agreements whose existence I cannot mention.

They thought they found in numbers, more than in fire, earth, or water, many resemblances to things which are and become; thus such and such an attribute of numbers is justice, another is soul and mind, another is opportunity, and so on; and again they saw in numbers the attributes and ratios of the musical scales. Since, then, all other things seemed in their whole nature to be assimilated to numbers, while numbers seemed to be the first things in the whole of nature, they supposed the elements of numbers to be the elements of all things, and the whole heaven to be a musical scale and a number.

Metaph. A. 5, 985 b 27–986 a 2.

Garrett Baker May 21, 2025, 1:51 AM
2 points
0
in reply to: Guive’s comment on: AI Doomerism in 1879
Can you rephrase his argument in your own words? In particular, define what “tension” means.

Garrett Baker May 21, 2025, 1:50 AM
2 points
0
in reply to: Matthew Barnett’s comment on: AI Doomerism in 1879
I get the same result with Claude, but when I push at all it caves & says I understand you fine.

It seems a crux is what you mean by “tension”.

Garrett Baker May 21, 2025, 12:32 AM
6 points
5
in reply to: Matthew Barnett’s comment on: AI Doomerism in 1879

Please try to engage with what I’m actually saying, rather than continuing to misrepresent my position.

It seems everyone has this problem with your writing, have you considered speaking more clearly or perhaps considering people understand you fully and it is you who are wrong?

In this case, I believe it’s the latter, since

If someone says their high p(doom) is driven by short timelines, what they likely mean is that AGI is now expected to arrive via a certain method—namely, deep learning—that is perceived as riskier than what might have emerged under slower or more deliberate development. If that’s the case, it directly supports my core point.

Really? I thought your core point was

It’s perhaps worth highlighting the significant tension between two contrasting claims: on the one hand, the idea that modern AI doomerism was “anticipated” as early as the 19th century, and on the other, the idea that modern AI doom arguments are rationally grounded in a technical understanding of today’s deep learning systems.

In which case I did explain why there is no tension, as can be seen from my saying

So we can get the observed shift with most of the “highly technical DL-specific considerations” mainly updating the p(AGI soon) factor via the incredibly complicated and arcane practice of… extrapolating benchmark scores.

That is, it is a very strange thing to say there is a “significant tension” between having high p(doom | AGI soon) on first principles reasoning, and to have p(AGI soon) get updated by benchmark scores.

Garrett Baker May 20, 2025, 11:42 PM
8 points
2
in reply to: Matthew Barnett’s comment on: AI Doomerism in 1879
Yes, I agree your whole comment sucks. I know you know there is a difference between p(doom) and p(doom|AGI soon), and your reasons for having a high p(doom | AGI soon) and low p(doom) can be very different. Indeed a whole factor of p(AGI soon) different!

So we can get the observed shift with most of the “highly technical DL-specific considerations” mainly updating the p(AGI soon) factor via the incredibly complicated and arcane practice of… extrapolating benchmark scores.

Indeed, the fact AGI seems to be arriving so quickly is the main reason most people are worried!

This is not to say they like deep learning. There can be additional reasons deep learning is bad in their book, but is deep learning a core part of their arguments? Hell no! Do you know how I know? I’ve actually read them! Indeed, if you type site:arbital.greaterwrong.com “deep learning” into google, you get back two results. Compare with site:arbital.greaterwrong.com “utility function”, which gives you 5 pages. Now which do you think is more central to their high p(doom | AGI in 5 years)?

Garrett Baker May 20, 2025, 11:05 PM
6 points
9
in reply to: Matthew Barnett’s comment on: AI Doomerism in 1879

their primary concern seems to be driven by the nature of existing deep learning technologies. This is suggested by their use of the phrase “anything remotely like current techniques”, which suggests that their core worries stem largely from deep learning rather than all potential AI development pathways

You know better! Eliezer at least has been arguing these points far before DL!

Garrett Baker May 20, 2025, 10:12 PM
4 points
0
in reply to: Viliam’s comment on: School & Jobs are good SOLELY because people are lazy

I admit, this is a point that I am confused about: are employees generally expected to learn new skills, and if they do, are they expected to do that on company time or during their free time? Before I had kids, I automatically went with “free time”, and only started to reflect on that after the free time disappeared.

Why not just ask your company?

Lighthaven Sequences Reading Group #35 (Tuesday 5/20)

Garrett Baker, Aella, Ronny Fernandez, Ben Pace and Jozdien

May 19, 2025, 8:58 PM

8 points

0 comments1 min readLW link

Garrett Baker May 19, 2025, 5:02 PM
4 points
0
in reply to: Samuel Hapák’s comment on: Google Logo Ligature Bug
Reading out “google logo ligature” in a font reader seems more informative than skipping the logo entirely, which is what an unprintable unicode character would do.

Garrett Baker May 16, 2025, 5:13 PM
4 points
2
on: Consider Others’ Cost Tolerances

Rough, non-weird probabilities can be given as extra info in the form of words of estimative probability.

That is not such a good solution, people have very different intuitive interpretations of those words, from Superforecasting

Kent said he was pessimistic. He felt the odds were about 65 to 35 in favor of an attack. The official was startled. He and his colleagues had taken “serious possibility” to mean much lower odds.10

Disturbed, Kent went back to his team. They had all agreed to use “serious possibility” in the [National Intelligence Estimate (NIE)] so Kent asked each person, in turn, what he thought it meant. One analyst said it meant odds of about 80 to 20, or four times more likely than not that there would be an invasion. Another thought it meant odds of 20 to 80—exactly the opposite. Other answers were scattered between those extremes. Kent was floored. A phrase that looked informative was so vague as to be almost useless. Or perhaps it was worse than useless, as it had created dangerous misunderstandings. And what about all the other work they had done? Had they “been seeming to agree on five months’ worth of estimative judgments with no real agreement at all?” Kent wrote in a 1964 essay. “Were the NIEs dotted with ‘serious possibilities’ and other expressions that meant very different things to both producers and readers? What were we really trying to say when we wrote a sentence such as this?”11

And if you are concerned about not looking weird to a good friend, it seems like also not a very good option to send them a table of probability words to memorize before talking with you.

Garrett Baker May 15, 2025, 5:54 PM
5 points
0
in reply to: Saul Munn’s comment on: Saul Munn’s Shortform

~$5 for a cup of coffee — that’s about an order of magnitude cheaper.

Are you buying your coffee from a cafe every day or something? You can buy a pack of nice grounds for like $13, and that lasts more than a month (126 Tbsp/pack / (3 Tbsp/day) = 42 days/pack), totaling 30¢/day. Half the cost of a caffeine pill. And that’s if you don’t buy bulk.

Garrett Baker May 12, 2025, 5:03 PM
2 points
0
in reply to: Hopenope’s comment on: Orienting Toward Wizard Power
Yeah, I figured something like that would be going on if I was wrong, thanks!

Garrett Baker May 11, 2025, 8:42 PM
2 points
0
in reply to: Hopenope’s comment on: Orienting Toward Wizard Power
The wikipedia side-effect page says that the rate of seizures is between 0.01-0.1%, for comparison about 0.68% of the population has epilepsy, so I’m skeptical this ought to be such a concern. Am I reading these numbers incorrectly?

I can definitely believe the anxiety bit. It is a stimulant, and anxiety & depression are very very correlated.

Garrett Baker May 11, 2025, 4:00 AM
17 points
−2
on: Book Review: “Encounters with Einstein” by Heisenberg

How did Einstein and Heisenberg go so wrong?

I don’t know about Heisenberg, but a common answer for Einstein is the Institute for Advanced Study at Princeton, which took the best minds of the day, and freed them from practical concerns about justifying their research interests or talking to outside researchers, apparently causing them to be increasingly less productive & out of touch from the rest of their respective fields.

From Surely You’re Joking Mr. Feynman

When I was at Princeton in the 1940s I could see what happened to those great minds at the Institute for Advanced Study, who had been specially selected for their tremendous brains and were now given this opportunity to sit in this lovely house by the woods there, with no classes to teach, with no obligations whatsoever. These poor bastards could now sit and think clearly all by themselves, OK? So they don’t get an idea for a while: They have every opportunity to do something, and they’re not getting any ideas. I believe that in a situation like this a kind of guilt or depression worms inside of you, and you begin to worry about not getting any ideas. And nothing happens. Still no ideas come.

Nothing happens because there’s not enough real activity and challenge: You’re not in contact with the experimental guys. You don’t have to think how to answer questions from the students. Nothing!

Hamming also writes about the IAS

When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore?

The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn’t the way things go. So that is another reason why you find that when you get early recognition it seems to sterilize you. In fact I will give you my favorite quotation of many years. The Institute for Advanced Study in Princeton, in my opinion, has ruined more good scientists than any institution has created, judged by what they did before they came and judged by what they did after. Not that they weren’t good afterwards, but they were superb before they got there and were only good afterwards.

And notes a similar effect of nobel prizes on academics’ work

But let me say why age seems to have the effect it does. In the first place if you do some good work you will find yourself on all kinds of committees and unable to do any more work. You may find yourself as I saw Brattain when he got a Nobel Prize. The day the prize was announced we all assembled in Arnold Auditorium; all 3 winners [Shockley, Bardeen, & Brattain] got up and made speeches. The third one, Brattain, practically with tears in his eyes, said, “I know about this Nobel-Prize effect and I am not going to let it affect me; I am going to remain good old Walter Brattain.” Well, I said to myself, “That is nice.” But in a few weeks I saw it was affecting him. Now he could only work on great problems.

and he attributes the decline in Shannon’s career to a similar effect from getting tenure & a blank check from MIT

Shannon, I believe, ruined himself. In fact when he left Bell Labs [for MIT, which offered him tenure + a blank check], I said, “That’s the end of Shannon’s scientific career.” I received a lot of flak from my friends who said that Shannon was just as smart as ever. I said, “Yes, he’ll be just as smart, but that’s the end of his scientific career”, and I truly believe it was.

Of course, as we see above, he also attributes Shannon’s decline to not being willing to work on smaller problems. These two problems could exacerbate each other, so they’re not entirely incompatible.

Garrett Baker May 11, 2025, 1:38 AM
9 points
5
in reply to: Viliam’s comment on: Viliam’s Shortform
I had a class in college about the history of the atomic bomb. Our text book was Rhodes, and most of the stuff we learned about was the different competing theories of atoms, the experimental methods & math used to distinguish between them, math behind atomic chain reactions, and the scientists who did those things. It was great! Our tests were basically physics tests.

I’d like more history like that, and I’d like marginal movements in that direction in math class. For example, something like Radical Real Analysis but for everything.

Obviously schools won’t do this well, but they won’t do anything well. Any change they make nowadays will be oriented toward making the least common denominator happy, if I understand the current fad among teachers nowadays.

The much more interesting question is how could this be done well, and the more useful is why the current fad & how can it be changed, so saying something like “teach more history in math!” can be expected to go well.

Lighthaven Sequences Reading Group #34 (Tuesday 5/13)

Garrett Baker, Aella, Ronny Fernandez, Ben Pace and Jozdien

May 10, 2025, 7:42 AM

8 points

0 comments1 min readLW link

Garrett Baker May 10, 2025, 1:13 AM
2 points
0
in reply to: Daniel Kokotajlo’s comment on: 1a3orn’s Shortform
I agree, and we do see some american companies doing the same thing.

Garrett Baker May 10, 2025, 12:40 AM
22 points
3
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform
Re: open sourcing. My guess why they open source more is for verification purposes. Chinese labs have an earned reputation for scams. So a lab that announces a closed source chat site, to investors, could very well be a claude or openai or llama or gemini wrapper. However, a lab that releases the weights of their model, and “shows their work” by giving a detailed writeup of how they managed to train the model while staying under their reported costs is significantly more likely to be legitimate.

Garrett Baker May 9, 2025, 7:56 PM
4 points
0
in reply to: David Lorell’s comment on: Orienting Toward Wizard Power
This is true, but I read amitlevy49′s comment as having an implicit “and therefore anyone who wants that kind of natural drive should take bupropion”. I probably should’ve given more information in my response.

Garrett Baker May 9, 2025, 5:28 PM
17 points
0
in reply to: johnswentworth’s comment on: Orienting Toward Wizard Power
FYI (cc @Gram_Stone) the 2023 course website has (~~poor quality~~ edit:nevermind I was accessing them wrong) video lectures.
Edit 2: For future (or present) folks, I’ve also downloaded local mp4s of the slideshow versions of the videos here, and can share privately with those who dm, in case you want them too or the site goes down.

Garrett Baker May 9, 2025, 4:24 PM
2 points
0
in reply to: amitlevy49’s comment on: Orienting Toward Wizard Power
Not really, the hypothesis is that John has depression, and of all the antidepressants, bupropion is the best (if it works for you).

Garrett Baker

Lighthaven Se­quences Read­ing Group #35 (Tues­day 5/​20)

Lighthaven Se­quences Read­ing Group #34 (Tues­day 5/​13)

Lighthaven Sequences Reading Group #35 (Tuesday 5/20)

Lighthaven Sequences Reading Group #34 (Tuesday 5/13)