ryan_b

Karma: 5,259

ryan_b 29 Sep 2025 22:31 UTC
10 points
2
on: On Dwarkesh Patel’s Podcast With Richard Sutton
It feels to me like Sutton is too deep inside the experiential learning theory. When he says there is no evidence for imitation, this only makes sense if you imagine it strictly according to the RL theory he has in mind. He isn’t applying the theory to anything; he is inside the theory and interpreting everything according his understanding of it.

It did feel like there was a lot of talking past one another when Dwarkesh was clearly talking about the superintelligent behaviors everyone is interested in (doing science, math, and engineering) as his model for intelligence, and Sutton is blowing all of this off only to articulate quite late in the game that his perspective is that human infants are his model for intelligence. If this was cleared up early, it would probably have been more productive.
I have always found the concept of a p-zombie kind of silly, but now I feel like we might really have to investigate the question of an approximate i-zombie: if we have a computer than can output anything an intelligent human can, but we stipulate that the computer is not intelligent....and so on and so forth.

On the flip side, it feels kind of like a waste of time. Who would be persuaded by such a thing?

ryan_b 29 Sep 2025 20:43 UTC
6 points
0
on: I have decided to stop lying to Americans about 9/11
Good job saying a brave thing; on a US-based site with a plurality US membership, this was a risk. Well done.
Out of curiosity, how often do conversations about 9/11 come up? For the most part, we don’t discuss it that much among ourselves except during the anniversary, although I do make note that it was just a couple of weeks ago and indeed the traditional observance is literally just to talk about where we were and what we were doing at the time, which precisely when the observation about cheering would come up.
It may or may not surprise you that while there were basically no rooms cheering in the US, there was a substantial minority population that celebrated after the fact. Mostly these were people who hated finance and globalization (which the twin towers symbolized) or something about foreign policy (imperialism, colonialism, etc) or as some form of divine punishment (tolerating gay people or interracial marriage or what-have-you).
So, thank you for saying your piece. I appreciate the honesty.

ryan_b 24 Sep 2025 20:58 UTC
6 points
1
on: IABIED Review—An Unfortunate Miss
Strong upvote, I appreciate the inside-view context that you have from publishing a similar book. I bought it as a result of this review.

I cannot, alas, promise a side-by-side review. However, there are a couple of questions I am primed to look for, foremost among them right now: how much detail is invested in identifying the target audience? The impression I am getting so far is that it has been approximately defined as not us, but a lot of complaints seem to turn on this question. I see a lot of discussion about laymen but that’s an information level, not a target audience. I don’t know if I have seen much discussion of the target audiences at all outside of the AI policy area, come to think of it.

ryan_b 19 Sep 2025 21:12 UTC
3 points
1
on: ryan_b’s Shortform
The important information I should take from a strong trend is an axis, or a dimension, rather than the default takeaway of a direction.
I have the colloquial habit of talking about a trend as a direction, leaning on the implicit metaphor of physical space for whatever domain in which the trend appears. I’ve only just today started to realize that, while I am pretty sure the physical space (or, probably, geography and maps) metaphor is why I speak that way, there’s no reason not to lean into it as an explicit description within the abstract space for the domain. By this I mean that whatever it is we are talking about (the domain) has several important parts to it (dimensions), and taken together these form the space of the domain.
Returning to the direction v. dimension takeaway, this basically means that the important thing is the dimension (or dimensions) along which the trend moves, so it is worth looking at the opposite direction of the trend as well.

This is basically the same as the idea as taking good advice and reversing it, just looking at changes in the world instead.

ryan_b 26 Aug 2025 22:08 UTC
2 points
0
on: Underdog bias rules everything around me
Boiling this down for myself a bit, I want to frame this as a legibility problem: we can see our own limitations, but outsiders successes are much more visible than their limitations.

ryan_b 25 Aug 2025 20:16 UTC
5 points
0
in reply to: johnswentworth’s comment on: Yudkowsky on “Don’t use p(doom)”
I’m inclined to look at the blunt limitations of bandwidth on this one. The first hurdle is that p(doom) can pass through tweets and shouted conversations in bay area house parties.

ryan_b 25 Aug 2025 19:59 UTC
4 points
2
in reply to: Lukas Finnveden’s comment on: Yudkowsky on “Don’t use p(doom)”
I also think he objects to putting numbers on things, and I also avoid doing it. A concrete example: I explicitly avoid putting numbers on things in LessWrong posts. The reason is straightforward—if a number appears anywhere in the post, about half of the conversation in the comments will be on that number to the exclusion of the point of the post (or the lack of one, etc). So unless numbers are indeed the thing you want to be talking about, in the sense of detailed results of specific computations, they are positively distracting from the rest of the post for the audience.

I focused on the communication aspect in my response, but I should probably also say that I don’t really track what the number is when I actually go to the trouble of computing a prior, personally. The point of generating the number is clarifying the qualitative information, and then the point remains the qualitative information after I got the number; I only really start paying attention to what the number is if it stays consistent enough after doing the generate-a-number move that I recognize it as being basically the same as the last few times. Even then, I am spending most of my effort on the qualitative level directly.
I make an analogy to computer programs: the sheer fact of successfully producing an output without errors weighs much more than whatever the value of the output is. The program remains our central concern, and continuing to improve it using known patterns and good practices for writing code is usually the most effective method. Taking the programming analogy one layer further, there’s a significant chunk of time where you can be extremely confident the output is meaningless; suppose you haven’t even completed what you already know to be minimum requirements, and compile the program anyway, just to test for errors so far. There’s no point in running the program all the way to an output, because you know it would be meaningless. In the programming analogy, a focus on the value of the output is a kind of “premature optimization is the root of all evil” problem.
I do think this probably reflects the fact that Eliezer’s time is mostly spent on poorly understood problems like AI, rather than on stable well-understood domains where working with numbers is a much more reasonable prospect. But it still feels like even in the case where I am trying to learn something that is well-understood, just not by me, trying for a number feels opposite the idea of hugging the query, somehow. Or in virtue language: how does the number cut the enemy?

ryan_b 25 Aug 2025 18:09 UTC
5 points
3
in reply to: Lukas Finnveden’s comment on: Yudkowsky on “Don’t use p(doom)”
I can’t speak for Eliezer, but I can make some short comments about how I am suspicious of thinking in terms of numbers too quickly. I warn you beforehand my thoughts on the subject aren’t very crisp (else, of course, I could put a number on them!)

Mostly I feel like emphasizing the numbers too much fails to respect the process by which we generate them in the first place. When I go as far as putting a number on it, the point is to clarify my beliefs on the subject; it is a summary statistic about my thoughts, not the output of a computation (I mean it technically is, but not a legible computation process we can inspect and/or maybe reverse). The goal of putting a number on it, whatever it may be, is not to manipulate the number with numerical calculations any more than the goal of writing an essay to is grammatically manipulate the concluding sentence, in my view.
Through the summary statistic analogy, I think that I basically disagree with the idea of numbers providing a strong upside in clarity. While I agree that numbers as a format are generally clear, they are only clear as far as that number goes—they communicate very little about the process by which they were reached, which I claim is the key information we want to share.
Consider the arithmetic mean. This number is perfectly clear, insofar as it means there are some numbers which got added together and then divided by how many numbers were summed. Yet this tells us nothing about how many numbers there were, or what the values of the numbers themselves were, or how wide the range of numbers was, or what the possible values were; there are infinitely many variations behind just the mean. It is also true going from no number at all to a mean screens out infinitely many possibilities, and I expect that infinity is substantially larger than the number of possibilities behind any given average. I feel like the crux of my disagreement with the idea of emphasizing numbers is people who endorse them strongly look at the number of possibilities eliminated in the step of going from nothing to an average and think “Look at how much clarity we have gained!” whereas I look at the number of possibilities remaining and think “This is not clear enough to be useful.”
The problem gets worse when numbers are used to communicate. Supposing two people meet in a Bay Area House Party and tell each other their averages. If they both say “seven,” they’ll probably assume they agree, even though it is perfectly possible for the average of what to have literally zero overlap. This is the point at which numbers turn actively misleading, in the literal sense that before they exchanged averages they at least knew they knew nothing, and after exchanging averages they wrongly conclude they agree.
Contrast this with a more practical and realistic case where we might get two different answers on something like probabilities from a data science question. Because it’s a data science question we are already primed to ask questions about the underlying models and the data to see why the numbers are different. We can of course do the same with the example about averages, but in the context of the average even giving the number in the first place is a wasted step because we gain basically nothing until we have the data information (where the sum-of-all-n-divided-by-n is the model). By contrast, in the data science question we can reasonably infer that the models will be broadly similar, and that if they aren’t that information by itself likely points to the cruxes between them. As a consequence, getting the direct numbers is still useful; if two data science sources give very similar answers, they likely do agree very closely.
In sum, we collectively have gigantic uncertainty about the qualitative questions of models and data for whether AI can/will cause human extinction. I claim the true value of quantifying our beliefs, the put-a-number-on-it mental maneuver, is clarifying the qualitative questions. This is also what we really want to be talking about with other people. The trouble is the number we have put on all of this internally is what we communicate but does not contain the process for generating the number, and then the conversation invariably becomes about the numbers, and in my experience this actively obscures the key information we want to exchange.

ryan_b 22 Aug 2025 21:28 UTC
3 points
0
on: DeepSeek v3.1 Is Not Having a Moment
I suspect DeepSeek is unusually vulnerable to the problem of switching hardware because my expectation for their cost advantage fundamentally boils down to having invested a lot of effort in low-level performance optimization to reduce training/inference costs.

Switching the underlying hardware breaks all this work. Further, I don’t expect the Huawei chips to be as easy to optimize as the Nvidia H-series, because the H-series are built mostly the same way as Nvidia has always built them (CUDA), and Huawei’s Ascend is supposed to be a new architecture entirely. Lots of people know CUDA; only Huawei’s people know how the memory subsystem for Ascend works.

If I am right, it looks like they got hurt by bad timing this round same way as they benefited from good timing last round.

Edit: Finally found a reasonable description of what happened. They were programming Nvidia hardware in assembly. My hardware switch guess is confirmed—this has wiped out their primary advantage. If they continue to fade, I think we could fairly assess them as a casualty of politics.

ryan_b 19 Aug 2025 15:48 UTC
4 points
0
in reply to: Elizabeth’s comment on: Elizabeth’s Shortform
From that experience, what do you think of the learning value of being in a job you are not qualified for? More specifically, do you think you learned more from being in the job you weren’t qualified for than you did in in other jobs that matched your level better?

ryan_b 17 Aug 2025 13:24 UTC
45 points
23
on: My Interview With Cade Metz on His Reporting About Lighthaven
My days of not taking that person seriously sure are coming to a middle.

ryan_b 17 Aug 2025 13:21 UTC
2 points
0
on: “Buckle up bucko, this ain’t over till it’s over.”
I commonly make a similar transition that I describe as task orientation versus time orientation.

The transition happens when there is some project where there seems to some number of tasks to do and I expect it to be done (or get that step done, or whatever). This expectation then turns out to be wrong, usually because the steps fail directly or I didn’t have enough information about what needed to be done. Then I will explicitly switch to time orientation, which really just means that I will focus on making whatever progress is possible within the time window, or until complete.

One difference is that my experience isn’t sorted by problem difficulty per se. I mean it correlates with problem difficulty, but the real dividing line is how much attention I expected it to require versus how much it wound up requiring. Therefore it is gated by my behavior beforehand rather than by being a Hard Problem.

Counterintuitively I find time-orientation to be a very effective method of getting out of analysis paralysis. This seems like a difference to me because the “time to do some rationality” trigger associates heavily with the deconfusion class of analytical strategies (in my mind). I suspect the underlying mechanism here is that analysis in the context of normal problems is mainly about efficiency, but the more basic loop of action-update-action-update is consistently more effective when information is lacking.

I think there’s something to time orientation having the notion of bash-my-face-against-the-problem-to-gather-information-about-it which makes it more effective than analysis for me a lot of the time because it has the information gathering step built in explicitly, whereas my concepts of analysis are still mostly just received from the subjects where I picked them up, leaving them in separate buckets. This makes me vulnerable to the problem of using the wrong analytical methods because almost all presentations of analysis assume the information being analyzed, and I have a limited sense of the proverbial type signature as a result.

ryan_b 4 Aug 2025 23:25 UTC
2 points
0
in reply to: ACCount’s comment on: I am worried about near-term non-LLM AI developments
Ah, but is it a point-in-time sidegrade with a faster capability curve in the future? At the scale we are working now, even a marginal efficiency improvement threatens to considerably accelerate at least the conventional concerns (power concentration, job loss, etc).

ryan_b 30 Jul 2025 14:56 UTC
5 points
0
on: My Empathy Is Rarely Kind
So what happens when you move towards empathy with people you are more aligned with in the first place? Around here, for example?

ryan_b 22 Apr 2025 21:15 UTC
20 points
4
on: You Better Mechanize
No matter how much I try, I just cannot force myself to buy the premise of replacement of human labor as a reasonable goal. Consider the apocryphal quote:
If I had asked people what they wanted, they would have said faster horses. –Henry Ford
I’m clearly in the wrong here, because every CEO who talks about the subject talks about faster horses^[1], and here we have Mechanize whose goal is to build faster horses, and here is the AI community concerned about the severe societal impacts of digital horse shit.

Why, exactly, are all these people who are into accelerating technical development and automation of the economy working so hard at cramming the AI into the shape of a horse?
1. ^
  For clarity, faster horses here is a metaphor for the AI just replacing human workers at their existing jobs.

ryan_b 22 Apr 2025 14:21 UTC
2 points
0
on: Why Have Sentence Lengths Decreased?
- How do we count specialized language? By this I mean stuff like technical or scientific specialties, which are chock-full of jargon. The more specialized they are, the less they share with related topics. I would expect we do a lot more jargon generating now than before, and jargon words are mostly stand-ins for entire paragraphs (or longer) of explanation.
- Related to jargon: academic publishing styles. Among other things, academic writing style is notorious for being difficult for outsiders to penetrate, and making no accommodation for the reader at all (even the intended audience). I have the sense that papers in research journals have almost evolved in the opposite direction, all though I note my perception is based on examples of older papers with an excellent reputation, which is a strong survivorship bias. Yet those papers were usually the papers that launched new fields of inquiry; it seems to me they require stylistic differences like explaining intuitions because the information is not there otherwise.
- Unrelated to the first two, it feels like we should circle back to the relationship between speaking and writing. How have sentences and wordcount fared when spoken? We have much less data for this because it requires recording devices, but I seem to recall this being important to settling the question of whether the Iliad could be a written-down version of oral tradition. The trick there was they recorded some bards in Macedonia in the early 20th century performing their stories, transcribed the recordings, and then found them to be of comparable length to Homer. Therefore, oral tradition was ruled in.

ryan_b 22 Mar 2025 14:41 UTC
3 points
2
on: They Took MY Job?
Good fiction might be hard, but that doesn’t much matter to selling books. This thing is clearly capable of writing endless variations on vampire romances, Forgotten Realms or Magic the Gathering books, Official Novelization of the Major Motion Picure X, etc.
Writing as an art will live. Writing as a career is over.

ryan_b 6 Feb 2025 14:55 UTC
9 points
0
on: The future of humanity is in management
And all of this will happen far faster than it did in the past, so people won’t get a chance to adapt. If your job gets eliminated by AI, you won’t even have time to reskill for a new job before AI takes that one too.
I propose an alternative to speed as explanation: all previous forms of automation were local. Each factory had to be automated in bespoke fashion one at a time; a person could move from a factory that was automated to any other factory that had not been yet. The automation equipment had to be made some somewhere and then moved to where the automation was happening.
By contrast, AI is global. Every office on earth can be automated at the same time (relative to historical timescales). There’s no bottleneck chain where the automation has to be deployed to one locality, after being assembled in a different locality, from parts made in many different localities. The limitations are network bandwidth and available compute, both of which are shared resource pools and complements besides.

ryan_b 16 Jan 2025 15:46 UTC
4 points
3
on: How quickly could robots scale up?
I like this effort, and I have a few suggestions:
- Humanoid robots are much more difficult than non-humanoid ones. There are a lot more joints than in other designs; the balance question demands both more capable components and more advanced controls; as a consequence of the balance and shape questions, a lot of thought needs to go into wrangling weight ratios, which means preferring more expensive materials for lightness, etc.
- In terms of modifying your analysis, I think this cashes out as greater material intensity—the calculations here are done by weight of materials, we just need a way to account for the humanoid robot requiring more processing on all of those materials. We could say something like 1500kg of humanoid robot materials take twice as much processing/refinement as 1500kg of car materials (occasionally this will be about the same; for small fractions of the weight it will be 10x the processing, etc).
- The humanoid robots are more vulnerable to bottlenecks than cars. Specifically they need more compute and rare earth elements like neodymium, which will be tough because that supply chain is already strained by new datacenters and AI demands.
What links here?
- Noosphere89's comment on How AI Takeover Might Happen in 2 Years by joshc (11 Feb 2025 21:00 UTC; 2 points)

ryan_b 15 Jan 2025 21:00 UTC
2 points
0
on: A Novel Idea for Harnessing Magnetic Reconnection as an Energy Source
This is a fun idea! I was recently poking at field line reconnection myself, in conversation with Claude.

I don’t think the energy balance turns out in the idea’s favor. Here are the heuristics I considered:
- The first thing I note is what happens during reconnection: a bunch of the magnetic energy turns into kinetic and thermal energy. The part you plan to harvest is just the electric field part. Even in otherwise ideal circumstances, that’s a substantial loss.
- The second thing I note is that in a fusion reactor, the magnetic field is already being generated by the device, via electromagnets. This makes the process look like putting current into a magnetic field, then to break the magnetic field in order to get less current back out (because of the first note).
- The third thing I note is that reconnection is about the reconfiguration of the magnetic field lines. I’m highly confident that electric fields when the lines break define how the lines reconnect, so if you induct all the energy out the reconnection will look different than would have. Mostly this would cash out as a weaker magnetic field than it would be otherwise, driving more recharging of the magnetic field, making the balance worse.
All of that being said, Claude and ChatGPT both respond well to sanity checking. You can say directly something like: “Sanity check: is this consistent with thermodynamics?”

I also think that ChatGPT misleadingly treated the magnetic fields and electric fields as being separate because it was using an ideal MHD model, where this is common due to the simplifications the model makes. In my experience at least Claude catches a lot of confusion and oversights by asking specifically about the differences between the physics and the model.