It seems like Bryan and Matt are mostly talking past each other. They’re each advocating for changes along a different axis, and those two axes are in principle independent from each other.
Bryan is primarily interested in the axis of the “pervasiveness of market-restricting regulation in society” (or alternatively “how free are markets?”). He’s advocating for less regulation, and especially in key areas where regulation is destroying enormous amounts of value: immigration and housing.
Matt is primarily interested in the axis of “the distribution of ownership of capital in society”. He thinks a society is more just when ownership of capital isn’t concentrated.
He’s advocating for an alternative system of property rights than Bryan is relying on. His preferred system of property rights is unintuitive compared most common notions of ownership, and he (presumably) is aware of that, but he thinks that’s not very relevant, since he thinks that that common notions of ownership are philosophically bankrupt—they don’t actually hold up to scrutiny, and so some other notion is needed.
(This is much more interesting and (possibly?) intellectually cogent notion of socialism than I’ve encountered before.
These two axes are not fundamentally at odds! It is (in principle, at least) to have a society with common ownership of capital and very limited government regulation.
Matt doesn’t clearly have a strong view on the regulation axis. He specifically says (1:02) that the level of regulation is irrelevant to his definition of socialism!
Further, he specifically demurs from evaluating which setups are “free market” or not, stating explicitly that that’s not how he thinks about things. In his framework all markets are structured by rules, and some sets of rules are better than others, but there’s not really a coherent notion of “some markets are freer and others are less free”. (Which makes sense because Bryan’s idea of “free markets” are markets in which people are less restricted in what they can do with their property, and Matt denies the notion of property that idea relies on.)
I can’t tell, but it seems like maybe Matt would prefer the specific deregulation that Bryan advocates for? (He says at 1:05 that he doesn’t see the zoning rule that limits density of housing as socialistic because it doesn’t drive capital income to the public.)
I’m sure that Bryan does have strong views about property right systems, but to my mind, he does a bad job with contending with Matt’s arguments on their own terms.
Bryan tries to appeal to those common notions of ownership as an argument, but Matt unhesitatingly bites that bullet. It’s like trying to argue to a physicist that General Relativity or Quantum Mechanics must be wrong because it’s counter to one’s basic experience of living in the world. The physicist knows that those theories are bizarre to common sense folk-physics, but he also claims that there are fundamental problems with common sense folk-physics—hence the need for the these unintuitive alternatives.
Bryan does loop around to a different argument at the end: that his preferred innovations would do more for the poor than Matt’s. I think that was ostensibly what the debate was supposed to be about, and I continue to be interested in Matt’s response to that claim.
But were I to probe Matt’s framework, on it’s own terms, I wouldn’t just point out that most people don’t share his assumptions.
I am mostly interested in working out the patterns of incentives that fall out of his preferred property system, and what selection processes are on capital allocation decisions. (I guess that’s not a crux for Matt, but it’s at least a crux for me).
I agree that we could have a totally different framework of property rights, and I would want to figure out if his framework is better than the existing ones, and in what ways.
Overall, I’m pretty glad to have been introduced to Matt Bruenig. It seems like he might have a more intellectually coherent notion of socialism than I’ve previously encountered. I’m skeptical that it actually pans out, but I’m curious to learn more!
I’m reminded of G.K. Chesterton’s (the fence guy’s) political philosophy: Distributivism. If I wanted to oversimplify, distributivism basically says, “Private property is such a good idea that everyone should have some!” Distributivism sees private property in terms of individual personal property: a farm, perhaps a small business, the local pub. It’s in favor of all that. You should be able to cut down your own tree, or build a shed, or work to benefit your family. There’s a strong element of individual liberty, and the right of ordinary people to go about their lives. Chesterton also called this “peasant proprietorship.”
But when you get to a larger scale, the scale of capital or of the great rentiers, Chesterton is ruthlessly willing to subjugate everything else to the goal of preserving ordinary, dignified human lives. In his time, there was a proposal to control lice by shaving the heads of poorer children. Leaving aside Chesterton’s notion of gender roles, his response to this was emphatic:
Now the whole parable and purpose of these last pages, and indeed of all these pages, is this: to assert that we must instantly begin all over again, and begin at the other end. I begin with a little girl’s hair. That I know is a good thing at any rate. Whatever else is evil, the pride of a good mother in the beauty of her daughter is good. It is one of those adamantine tendernesses which are the touchstones of every age and race. If other things are against it, other things must go down. If landlords and laws and sciences are against it, landlords and laws and sciences must go down. With the red hair of one she-urchin in the gutter I will set fire to all modern civilization. Because a girl should have long hair, she should have clean hair; because she should have clean hair, she should not have an unclean home: because she should not have an unclean home, she should have a free and leisured mother; because she should have a free mother, she should not have an usurious landlord; because there should not be an usurious landlord, there should be a redistribution of property; because there should be a redistribution of property, there shall be a revolution. That little urchin with the gold-red hair, whom I have just watched toddling past my house, she shall not be lopped and lamed and altered; her hair shall not be cut short like a convict’s; no, all the kingdoms of the earth shall be hacked about and mutilated to suit her. She is the human and sacred image; all around her the social fabric shall sway and split and fall; the pillars of society shall be shaken, and the roofs of ages come rushing down, and not one hair of her head shall be harmed.
That’s a creed, right there: “With the red hair of one she-urchin in the gutter I will set fire to all modern civilization.” Chesterton isn’t even quite right about lice control (what you needed in his day was a very fine comb and enough free time to brush your children’s hair daily, not necessarily a clean home as such). But the core idea stands.
Chesterton went on to explain he would prefer to be a gradualist, not a revolutionary, if gradualism would get the job done:
III. ON PEASANT PROPRIETORSHIP
I have not dealt with any details touching distributed ownership, or its possibility in England, for the reason stated in the text. This book deals with what is wrong, wrong in our root of argument and effort. This wrong is, I say, that we will go forward because we dare not go back. Thus the Socialist says that property is already concentrated into Trusts and Stores: the only hope is to concentrate it further in the State. I say the only hope is to unconcentrate it; that is, to repent and return; the only step forward is the step backward.
But in connection with this distribution I have laid myself open to another potential mistake. In speaking of a sweeping redistribution, I speak of decision in the aim, not necessarily of abruptness in the means. It is not at all too late to restore an approximately rational state of English possessions without any mere confiscation. A policy of buying out landlordism, steadily adopted in England as it has already been adopted in Ireland (notably in Mr. Wyndham’s wise and fruitful Act), would in a very short time release the lower end of the see-saw and make the whole plank swing more level. The objection to this course is not at all that it would not do, only that it will not be done. If we leave things as they are, there will almost certainly be a crash of confiscation. If we hesitate, we shall soon have to hurry. But if we start doing it quickly we have still time to do it slowly.
This point, however, is not essential to my book. All I have to urge between these two boards is that I dislike the big Whiteley shop, and that I dislike Socialism because it will (according to Socialists) be so like that shop. It is its fulfilment, not its reversal. I do not object to Socialism because it will revolutionize our commerce, but because it will leave it so horribly the same.
Chesterton’s objection to the socialism of his day was that it was essentially “the State as Walmart,” a giant centralization of economic effort and control. And he was suspicious of this.
But if you squint, Distributivism isn’t really a fully fledged economic philosophy at all. It doesn’t have a lot to say about the wealth created by mass production, or about trade, or about a hundred other things. What Distributivism (“peasant proprietorship”) really is, is a set of constraints. Do ordinary people own personal property? Do they have leisure time, and enough wealth for basic luxuries? Do they have enough time to parent their children well? Is society structured around the needs or ordinary people? Then you’re probably doing OK. But if everyone is stressed, and struggling, and has no time for their children, and cannot afford a decent place to live, well, something has gone wrong. And the underlying problem should be fixed gradually, if possible. But it should be fixed. And if a revolution is the only way to get there, well, so be it, in Chesterton’s eyes.
I am something like a Democratic Socialist, a gradualist who believes in a “mixed economy,” with all the space in the world for small proprietors and entrepreneurs and personal property. Capital is necessary, too! But capital is ultimately subject to the need of ordinary people to live decent lives. And if capital becomes destructive, and if the lives of ordinary people become burdensome, well, then we should change the rules around capital. I would vastly prefer to do this democratically and gradually and without great disruption, taking the smallest steps that will fix the problem. Chesterton, after all, also had his famous fence. But if I am forced to choose between the well-being of a “she-urchin in the gutter,” and all the self-important infrastructure of the modern economy? The well-being of ordinary families is ultimately non-negotiable.
(In the modern era, I am also very much in favor of building, because the lack of decent houses has become burdensome to ordinary people. And we need more electricity and better transportation, and so we also need to build at a greater scale, via whatever mechanisms are practical. But I am ultimately in favor of these things because they would improve the lives of ordinary people. Capital is a tool, and even an important one. But if the tool puts itself in opposition to ordinary people having decent lives, then I know how I will choose.)
Suppose we had a functionally infinite amount of high quality RL-/post-training environments, organized well by “difficulty,” and a functionally infinite amount of high quality data that could be used for pre-training (caveat: from what I understand, the distinction between these may be blurring.) Basically, we no longer needed to do research on discovering/creating new data, creating new RL environments, and we didn’t even have to do the work to label or organize it well (pre/post-training might have some path dependence).
In that case, what pace would one expect for model releases from AI labs in the short term to be? I ask because I see the argument made that AI could help speed up AI development in the near to medium term. But it seems like the main limiting factor is just the amount of time it takes to actually do the training runs.
If your answer is that we’ll have “continual learning” soon, then I have a followup question:
Using the latest hardware, but staying within the same basic architectures, what would one expect the maximum amount of intelligence possible that could be placed on a given N parameter model is? If God skipped the training process and tweaked all the weights individually to reach the global maximum, how smart could, say, GPT 4o be?
For the GPT 4o question, I would expect the global optimum to be at least a medium level of superintelligence, though I have serious doubts that known training methods could ever reach it even with perfectly tuned input.
I realize now that the question wasn’t exactly well formed. God could fill 4o with the complete theory of fundamental physics, knowledge of how to prove the Riemann Hypothesis, etc. That might qualify as super intelligence, but it is not what I was trying to get at. I should have said that the 4o model can only know facts that we already know; i.e., how much fluid intelligence could God pack onto 4o?
I am surprised that you think 4o could reach a medium level of super intelligence. Are you including vision, audio, and the ability to physically control a robot too? I have the intuitive sense that 4o is already crammed to the brim, but I am curious to know what you think.
I’m sure that it is crammed to the brim in one sense, but strongly expect that 99.9% of what it’s crammed to the brim with is essentially useless.
Also yes, I was including vision, audio, and motor control in that. It’s hard to know exactly where the boundaries lie between facts like “a (dis)proof of the Riemann Hypothesis” and patterns of reasoning that could lead to a (dis)proof of the Riemann Hypothesis if required. I suspect that a lot of what is called “fluid” intelligence is actually pretty crystallized patterns of thought that can be used to generate other thoughts that lead somewhere useful—whether the entity using it is aware of that or not.
Hi, I am Joanna. I did a design work trial for Lesswrong that ends tonight! As part of that, I designed a new profile page. If you don’t like it, I won’t be around to fix it unless they hire me. But, the team would surely care if you have comments! (And I would too.)
My understanding is that neither of the following are actually your design’s issue, but I am posting here as a general new profile page comment
The ability to see which posts you have seen before is gone, and I use it quite frequently.
The “see more” button in the posts tab seem broken. I know Research: Rescuers during the Holocaust (2018) is written by Martin Sustrik yet I cannot find it from their profile page. Somehow the “see more” button thinks I have reached the end of their posts at 2020. I can find it in the “post diamonds” on the right though.
Bio should be at the top, and the user’s Shortform aka Quick Takes page needs to be special case handled. It is so incredibly hard to find the quick take page now. Try finding it from https://www.lesswrong.com/users/habryka4
The old profile page made more sense to me: there was a list of posts (sortable), and then a list of comments. That’s the information I needed.
Now everything above the fold is a newspaper-like display of just a few posts, with most of the real estate being taken up by illustrations, which are mostly default abstract imagery because most posts on this website don’t (and probably shouldn’t) have illustrations. How is this a good use of space? I understand the demand for customizability, but I thought the old profile page let you pin posts?
I don’t like that you have to click “Feed” to see comments. Comments are important!
It’s not even clear what information “Feed” is showing me! When I scroll down on my “Feed”, after a bunch of recent comments, it shows a bunch of my posts one after another, but I definitely wrote comments in between those posts, as I can confirm on the GreaterWrong viewer. If “Feed” isn’t a reverse-chronological list of both comments and posts (despite the fact that the gear menu says “Show: All”), what even is it? (Speculation: maybe this is a bug caused by the posts already being loaded in order to populate the “All Posts” view, whereas more comments aren’t loaded unless the user scrolls down.)
Speaking of which, I think the old profile page already had this problem, but infinite-scroll or “See more” (in contrast to pagination) is hostile to people who want to read old comments by their favorite user. (I thought supporting “long content” was a design goal?) I like how GreaterWrong has pagination and query parameters (you can just edit the ?offset=n in the address bar) and sort-by-old for comments. (The new profile only has an Old sort option for “All Posts”, not “Feed”.)
The post listing conveys less information than the old profile page. It doesn’t display the number of comments. Mousing over the karma number shows a tooltip that says “Karma score” rather than the number of votes.
I could be sold on excerpting the first few lines of the post, but the title and excerpt and score and date all apparently being part of the same link element (changing color on mouseover) is weird.
The front page looks beautiful on my laptop, particularly like that the post list can include images. The watercolor defaults are cool too. The top post is too prominent though.
I can’t find my karma anywhere and my bio is nearly hidden. It seems that it is long enough to somehow take up the space where my karma would be listed.
Since I noticed other commenters focusing on their own profiles (which really we don’t need as much information from at glance, though we certainly view them the most), I checked out a few others. They mostly seem much more informative than they previously did, except for Abram Demski’s profile which was more informative when his sequences were emphasized. Also, karma is often impossible to find (except by hovering over their username).
Here’s some feedback. Reading over it now it sounds overly harsh (sorry!) but I mostly endorse it. Looking forward to further improvements!
The biggest thing is it’s less information dense. Being information dense is the single most important property for me, and I’d guess that it’s possible to make the page look beautiful without sacrificing information density.
It would be helpful to know what the goals of the redesign were (I’m guessing ease of use and aesthetics), so they can be accomplished without breaking existing workflows.
If “Top posts” are enabled and users don’t have custom photos on these top posts (which is the default), the entire screen above the fold is taken up by watercolor and a few post titles. This is unacceptable. Take Eliezer’s user page: it only has about 50 words very far apart. I can’t tell what the posts are about unless I click on them. They should at the very least be smaller and show the start of the post on hover.
The larger size for the first Top Post is way too big. I would guess not many users want to be defined by a single post, which is what a single element taking up half the screen signifies.
I can’t find a way to choose just 1 or 2 top posts. I would want my own page to have 2 top posts, both at the smaller size.
The user’s karma, number of comments, number of posts, and account age are not included on the page, even though it’s visible on mouseover. I can understand hiding karma, but all these numbers should be fairly prominent on the user profile. My first priority when I click to many user pages is seeing how established and active they are, which needs these numbers.
The “Sequences” tab is often redundant because most users do not have sequences. They could perhaps go on the right column below the user bio, display the number of posts, and show a preview on mouseover. Or, users should be able to choose a sequence to go in a Top Posts slot.
I’m not sure sequences is the correct thing to do with the right column, but the bottom of the right column is currently blank, so either something should be done with it or the layout should go back to one column.
The settings icon to the right of the feed is small and too easy to ignore, and also takes one too many clicks to access. Also, “All posts” tab is redundant with the “Posts” section of the Feed tab.
IMO there should just be two dropdowns, always visible, for content type (All / Posts / Quick Takes / Comments / maybe Sequences if the user has any) and sort type (New / Top).
The “Quick takes” of the Feed tab doesn’t work on my profile.
Nitpick, but in the profile editing UI the “swap” icon looks like a button that causes an action (I expected it to swap with the one below it or something). It would maybe be more intuitive to have the text “Change” on the button, or a different icon.
In the Feed tab, posts are not visually distinguished from comments. They should perhaps have the same styling as in the All Posts, or have a solid outline, or something.
Vercel has to verify my device sometimes when I load a profile, which takes about 1-1.5 seconds. This is also unacceptable if it happens too often, so hopefully it’s just because the page is new.
If “Top posts” are enabled and users don’t have custom photos on these top posts (which is the default), the entire screen above the fold is taken up by watercolor and a few post titles. This is unacceptable. Take Eliezer’s user page: it only has about 50 words very far apart. I can’t tell what the posts are about unless I click on them. They should at the very least be smaller and show the start of the post on hover.
Ok, let’s actually compare this. This is what the same amount of vertical space looked like for Eliezer a week ago:
We have a grid of 9 sequences, basically with no ability to tell at all what they are about, and one of them visibly misaligned. The images are broken on archive.org, but they were all basically the same without any indication of what they are about. Then, when you look at the recent post list, you see a bunch of random recent posts, none of which are representative of Eliezer’s writing.
I don’t buy this is any better at showing you what Eliezer’s writing is like, or helping you find good Eliezer content than our current UI. Yes, there are only 4 posts instead of 9 sequence items, but those posts are actually pretty representative of Eliezer, and it is indeed much more important for someone to have something they can click on and start reading, instead of seeing this non-descriptive grid of sequences that then when you click on them require you to choose again (and where most of them have a lot of prerequisites and are not good intro points).
I can’t find a way to choose just 1 or 2 top posts. I would want my own page to have 2 top posts, both at the smaller size.
Yep, this would be nice, but designing UI for any number of posts is just a lot of work. Two posts in-particular is pretty tricky with the way the aspect ratios work out.
The user’s karma, number of comments, number of posts, and account age are not included on the page, even though it’s visible on mouseover. I can understand hiding karma, but all these numbers should be fairly prominent on the user profile. My first priority when I click to many user pages is seeing how established and active they are, which needs these numbers.
You can always get those on hover. Indeed, in basically any context where you would click on the user you would see these statistics first.
But not having these at high prominence is actually really crucial to the new design! A major point of the design is to create a landing page that gives authors the feeling that it is their own landing page, with the focus being on the author’s writing. Karma, in-particular, refocuses the landing page on the relative status of an author in the LessWrong status hierarchy, which I think is really quite bad for a landing page. I don’t want people to have to worry that if they link someone to their LessWrong profile, that what they will end up focusing on is the relative ranking of that LessWrong user with other LessWrong users. Indeed, I don’t want them to have to think much about LessWrong at all, since the profile page is one of the most common entry-points for new users.
The “Sequences” tab is often redundant because most users do not have sequences. They could perhaps go on the right column below the user bio, display the number of posts, and show a preview on mouseover. Or, users should be able to choose a sequence to go in a Top Posts slot.
The sequences tab only shows when you have sequences! Agree it would be redundant otherwise, but we did think of this one.
The “Quick takes” of the Feed tab doesn’t work on my profile.
This is fixed!
Nitpick, but in the profile editing UI the “swap” icon looks like a button that causes an action (I expected it to swap with the one below it or something). It would maybe be more intuitive to have the text “Change” on the button, or a different icon.
Agree, I don’t love that button.
In the Feed tab, posts are not visually distinguished from comments. They should perhaps have the same styling as in the All Posts, or have a solid outline, or something.
I think them not being visually distinguished is fine-ish, but I do think we can make some improvements here. It’s the same styling as on the frontpage, so in as much as there is a mistake here, having it be wrong there is a bigger mistake.
Vercel has to verify my device sometimes when I load a profile, which takes about 1-1.5 seconds. This is also unacceptable if it happens too often, so hopefully it’s just because the page is new.
Should be fixed! But yeah, this was just because we had some aggressive bots hammering those pages.
Overall, feedback appreciated! Agree with like 30%, 30% are fixed and I think no longer apply, and disagree on the other 30%.
The feedback didn’t feel harsh to me. I’m wanting to share more about the intentions behind the first iteration of the design.
As context, I think it’s helpful to identify a primary user and scenario for a design because different design decisions trade-off against each other. In this case, I chose to have two primary users and scenarios because they seemed like mirrors of each other.
Primary users and scenarios:
A. Profile owners feel proud to share this. Feedback I heard on the previous page was that it was “kind of a mess”.
B. Someone who is not a lesswrong user, but interested in a person’s work, feels drawn to actually read it. More people who look at the profile click on an article and read it than before.
Secondary scenario:
People who use the page regulary can still still find the content they were looking for and their experience is not meaningfully harder day-to-day.
Ok, here’s what I think the primary scenario as it applies to me:
I feel like the challenge is to have the profile not be overwhelming and suggest while preserving information density. I can think of three potential solutions:
Have fewer sections displayed at once, but each item of a section doesn’t need to be large (like the Youtube Videos or Shorts tab on a channel page). This relies on thumbnails being informative, so they would need to improve.
Make the top ~1/3 of the profile page really draw the eye and suggest some content, and the rest as clean as possible
Global setting that regular users can change, which makes user pages more information dense (sort of like old/new reddit). Maybe this can be accessible on any user page.
I think the new page (with or without top posts) looks better than the old page, but some part of the bio should definitely be at the top, or you can’t even tell it’s a user page (vs a subtopic or something). If a user has a bio, the first 10 words of it is probably the most important 10 words on the page, and a viewer might not want to read on any of their content if they don’t know anything about the author. I would certainly want anyone who sees my user page to immediately know I work at METR, for instance.
The settings icon to the right of the feed is small and too easy to ignore, and also takes one too many clicks to access. Also, “All posts” tab is redundant with the “Posts” section of the Feed
I agree that the cog feels too subtle, and maybe not the right way to filter the feed. I tried version with tabs and again it drew attention away from the content. That said, I think the cog is too small and unbutton-like so it’s too easy to miss.
I agree that the feed and all post concept could be combined more elegantly. If I had had more time I would have thought about that more. Ruby, who designed the feed, had similar feedback.
The main issue is that I can no longer see if there were new comments on recent posts (both mine and for other users), which were previously indicated by the comment counts displayed in green, including for the shortform post (there doesn’t seem to be an easy way to find it at all now).
I liked the list of draft posts at the top, it’s a regular reminder to keep thinking about them. The “top posts” selection that takes up the whole first screen doesn’t help when it’s my own posts. It’s plausibly a good thing for posts by other users, though I personally don’t get the appeal of a huge section of the page whose only meaningful content is a post title that could just take up a single line (and leave space to fit more data about other relevant things on the screen, including more of the other posts). So some issues seem to be about different needs for your own userpage vs. userpages for other users.
The main issue is that I can no longer see if there were new comments on recent posts (both mine and for other users), which were previously indicated by the comment counts displayed in green
Yeah, I think this is sad, and I want to do something to restore that functionality. Not fully sure how to best do it, but my guess is we’ll figure out something.
including for the shortform post (there doesn’t seem to be an easy way to find it at all now).
I am hoping to generally destroy the abstraction of the “shortform post” soon. It’s super confusing to tons of users, and also the post is very ugly. I do think as part of that I’ll add some better UI for seeing new comments on shortform posts (somehow, I haven’t figured out a good thing there yet), but I am quite confident we don’t want to have the shortform post abomination as a user-visible thing in the long run.
As far as the page goes, I feel like it’s a worse than the old one. While it’s a bit more beautiful it’s a lot less functional. Trading functionality for beauty seems to go against the core way of LessWrong’s usual design that’s quite minimalistic. It would probably be best to revert to the old one.
In particular:
The new one doesn’t show the biography at the top but to the right bottom where it’s a lot less central.
It makes Sequences very hard to find, which reduces the agency of users to present their work by presenting.
If you are seeking a post by someone where you don’t know the exact title but have some idea about the karma that the post had, the new placement of the karma makes it harder to scan for the post. Maybe there should both be list like the old UI and an expanded option.
Having to click on the dial to see which sort mode is currently selected seems to add unnecessary friction.
Currently, the user statistics don’t seem to be shown at all if there’s a biography. That’s probably more of a bug than a design decision.
It makes Sequences very hard to find, which reduces the agency of users to present their work by presenting.
The user has much more agency, they can now choose which posts to present themselves with, and with much more space to work with for that! I would take bets at high odds that many more users will feel agency over their new profile page instead of the old one.
If you are seeking a post by someone where you don’t know the exact title but have some idea about the karma that the post had, the new placement of the karma makes it harder to scan for the post.
Yep, agree, though the change feels pretty minimal to me. You can just sort by karma, and then that should allow you to find it pretty quickly.
Currently, the user statistics don’t seem to be shown at all if there’s a biography. That’s probably more of a bug than a design decision.
Same decision as for the bio. The user statistics show on hover for every user, and they are particularly disorienting for someone who comes in from outside of LW to the profile (and everyone navigating to the profile from another LW page just saw the statistics when hovering over the username before clicking). On mobile where there isn’t hover they are relatively prominent close to the top.
Having to click on the dial to see which sort mode is currently selected seems to add unnecessary friction.
That’s the standard pattern all across LW :P I am sure users would not appreciate having a giant sorting option box open by default.
I liked the design when i saw it today, but also would like aggregate statistics like comments count/ post count/ recent activity. perhaps even something like github showing a calendar with activity for each commit. It would also be good to retain a bio with a self description and optionally urls to websites or social media accounts.
It says “Top Posts” and the mouse-over text is “by karma”, however in reality I can choose which posts to put there. Now, I like it that I can choose which posts to put there, but once I customized them, the mouse-over becomes a lie.
The “recent comments” disappeared. This is really bad because I use that to find my recent comments when I want to edit them. (For example now I wanted to find this comment to add this second bullet but had to do it manually.) OK, I now see I can find them under “feed” but this might be confusing.
There’s definitely something aesthetically pleasing to me about the new design, but it doesn’t do what I valued the previous one for. Namely, it allowed me to quickly eyeball the user and see their recent fine-grained history, i.e., their comments, which are a good proxy of what they’ve recently been up to, LessWrong-wise.
Previously, I would first see their self-description, then their recent posts, but only titles, so that I don’t need to scroll too much, or exercise small mental effort to ignore non-title text (which I have to do now), and then the comments.
Now, first I see the top posts, then the full list of posts, and the bio squeezed to the side, and I need to click “feed” to see the comments, where checking out recent comments is often the reason why I revisit somebody’s profile.
In general, it would be very nice if such updates were first rolled out in a beta mode, where users can opt-in to having their profile page changed to a new design.
Or: have a 3×2 matrix: [I want to see profile pages in old design | I want to see profile pages in new design | I don’t care] × [I want my profile page to show in old design to those who don’t care | I want my profile page to show in new design to those who don’t care].
Namely, it allowed me to quickly eyeball the user and see their recent fine-grained history, i.e., their comments, which are a good proxy of what they’ve recently been up to, LessWrong-wise.
I am pretty confident you actually pretty reliably care more about what the highest karma/self-curated posts from a user are, than what their most recent posts are (which is what we’ve always shown first so far)
and I need to click “feed” to see the comments, where checking out recent comments is often the reason why I revisit somebody’s profile.
This is a user setting that persists. So if you clicked on “Feed” for any profile, you will now always see the feed as the default list in everyone’s profile.
or exercise small mental effort to ignore non-title text (which I have to do now)
Your brain is really good at parsing images! There is a reason why every other website on the internet uses images to convey context and content about a story. An image really conveys a lot!
Many posts on LW don’t have well-chosen images, but now that image generation is cheap and can be reasonably high-quality, I think that’s a bug, and I want to make images as a way to convey the topic and context and vibe of a post more common across the site.
I am pretty confident you actually pretty reliably care more about what the highest karma/self-curated posts from a user are, than what their most recent posts are (which is what we’ve always shown first so far)
Aside from bristling a bit at being told that what I care about is wrong, I actually do care about recency more often than historical top-ranked for this view. Of course, sometimes I’m looking for “best of”, but more often I’m checking whether the user is in a snit or otherwise commenting recently in a pattern I should understand/accommodate before I join one of their threads.
Aside from bristling a bit at being told that what I care about is wrong
Much of UI design is figuring out what users care about, which involves disagreeing with people about what they care about! “If I had asked what they wanted, they would have asked for a faster horse”, etc.
Agree that recent activity is also pretty important. I am thinking about whether we should make the feed view the default view in the section below the top posts, because I do think that’s the second most common use-case for the page. I’ll iterate a bit over the coming days.
In fact, you’re absolutely correct—I retract that part of my response. I also recognize that I use this feature (looking at someone’s profile to see their recent posts/comments) less often that others might use it for other purposes (looking at their best/top contributions).
Too many user-options is worse than slight misses on some defaults. I’m grateful for LW, even when it’s optimized for people who are not exactly me. Carry on experimenting, and see how people react and change their usage.
In fact, you’re absolutely correct—I retract that part of my response
I’m actually a bit confused what you currently believe about your preferences. You think you don’t care as much about recency (but, some?). Or basically you overall prefer top posts when looking at author pages?
Sorry for lack of clarity in my retraction—I’m pretty sure I’m right about my preferences—I am usually looking for recency. That belief hasn’t changed.
I have retracted my annoyance—I should not be, and no longer am bothered by your use of “you” when it’s really “most users” or “our target user”. I hope it was useful for me to give my common use case, but I don’t intend to demand anything—this topic is peripheral enough to my value from LessWrong that I wanted to reduce, rather than increase, any load I’m imposing.
I am pretty confident you actually pretty reliably care more about what the highest karma/self-curated posts from a user are, than what their most recent posts are (which is what we’ve always shown first so far)
Hm, for newish-to-me users, yes. But for users that I’m already familiar with / I’ve been reading for a while, I am more interested in their recent posts. And I open the profiles of the latter more often than those of the former.
or exercise small mental effort to ignore non-title text (which I have to do now)
Your brain is really good at parsing images! There is a reason why every other website on the internet uses images to convey context and content about a story. An image really conveys a lot!
Perhaps you misunderstood me. It’s not about the images. I meant that this
is more time-consuming to browse through than a list like this
But now that I think about it more, maybe I’m overapplying the heuristics that I adopted for my email, and it’s nice to be able to read the preview without having to hover. But I still will want to scroll through posts sometimes, so some “collapse/show preview” button would be great to have.
ETA: I also loved being able to see the comments without opening the post, and this option is gone now on the profile page.
The flipside that made me excited about the current thing was “now, as I’m browsing posts, I actually have a chance of figuring out whether a post is good before clicking on it.” It’s not that there’s less information density, it’s that it’s differently distributed.
I think there should totally be a view for “dense reading of post titles”, but, it’s not the only thing I might want.
(My current guess is there should be a search filter, and if you’ve typed into the search filter, you get a more dense view)
Perhaps you misunderstood me. It’s not about the images. I meant that this
Ah, yep, I did misunderstand you. Agree that that section is now less dense.
I also loved being able to see the comments without opening the post, and this option is gone now on the profile page.
Yeah, I do also like this, and I might try to add something like it back, but it’s quite a bit of work (the “click on the comment icon to expand just the recent comments” interaction is among the most confusing interactions for new users, so it’s a bit tricky to get right).
Yeah, if I am looking for a post with a specific name, or just “I think I wrote a post on this topic”, then seeing lots of article names is more useful than seeing only a few previews.
Now that you tapped on feed once, anytime you go to someone’s profile it should automatically go there.
Yeah, thanks. Works, but I’m not sure if this being a global setting is a good idea, probably because not sure the current Posts/Sequences/Feed design is a better idea than the previous design in the first place, but maybe I need to wait and see how I feel about it after one week.
One specific suggestion: I think it would be much better if the top “top post” panel were split into two, with the post preview taking one half, and the bio taking the other half (with the option of “show more” for the bio).
Also, I liked being able to bookmark posts without having to click into them.
It’s beautiful, but it’s less info dense. That’s my biggest complaint. And my blog link is now in a dinky little corner. I like the addition of the top posts. The post preview is cool, but a bit less text would’ve been fine. It would be cool if I could close/open any/all of the previews with a single button click.
This is a horrendous design. Please stop designing things.
I would actually switch to Greater Wrong over this. Maybe even just because it displays what kind of designs the admins find good, from just trying this.
Also it’s probably misuse of “too combative” emoji? If someone comments this design looks pleasing and neat to them and another person comments that it looks like Substack-tier slop UI shoved into their face, then is the second poster really too combative? It’s your right to find the comment not worth your time to look at, so whatever with downvotes.
This is the part that made me use that react. It’s way too personal/attacky to be useful feedback for the designer. Like, if you’d said:
“I think this is a horrendous design and I really hope you don’t change more things on this website I use a lot, because those changes will reduce my experience as a user and cause me to leave for GreaterWrong.”
I would not have used it, and you’d have gotten the same core info across.
Overall like the aesthetic a fair amount and think it’s going to be a bit improvement once it settles, but am kinda unhappy with how many clicks it will now take me to get to my list of comments sorted by top. That’s the most common thing I do with my userpage, and even though posts are way more important, this seems like it unnecessarily de-emphasises an also relevant feature.
Some vague vibes of… noticing that you want to re-prioritize the page in a way I roughly endorse, but then not taking cheap wins to preserve the values of the things being down-weighted? e.g. now there’s no way to get to drafts from your own userpage, comments are three clicks away, profile desc is not fully expanded by default.
We store how you view your profile page as a user preference. So if last time you visited a profile you filtered for comments sorted by top, you will see the same thing the next time on any future user pages. This seems better to me for this use-case than the previous situation, where you would always have to scroll down, click on the sort button, then sort by top. Now it’s just that way as soon as you land!
e.g. now there’s no way to get to drafts from your own userpage, comments are three clicks away
That’s intentional, mostly to reduce confusion about which parts of the user page are shown to everyone, and which ones are just for you. A big problem with the previous page is that it had so many customized elements that you would only see on your own profile that it was very hard to build an intuitive model of what it looks like for other people.
comments are three clicks away
Only one click away! Why would it be three clicks? Just click once on feed and you are there. Also if you did it once, it now becomes your default view for all profiles, if that’s what you prefer.
profile desc is not fully expanded by default
We actually had been collapsing profile descriptions previously as well (though we’ve definitely moved it down the page with the new redesign and made it smaller). I think the collapsing broke at one point, but that was a bug, not a feature!
That’s intentional, mostly to reduce confusion about which parts of the user page are shown to everyone, and which ones are just for you.
Yeah, that’s a pretty good reason, I’m sure we’ll adjust.
Only one click away! Why would it be three clicks? Just click once on feed and you are there. Also if you did it once, it now becomes your default view for all profiles, if that’s what you prefer.
Specifically: Comments (or shortforms) only sorted by highest karma. It’s click gear → click comments + click top, even if I’m already on feed.
to be fair, this is kinda niche, but i do do it a lot as I often want to link some of my top comments to people. feel free to be like that’s a weird usecase and ignore it[1], but seems correct to register that this user does this.
And I can just scroll a bit more on the feed sorted by top, which is only two clicks (gear->top)?
We actually had been collapsing profile descriptions previously as well (though we’ve definitely moved it down the page with the new redesign and made it smaller).
Right, given new placement it probably makes sense to fully expand as it’s not going to push other content down, as a way to partly preserve the visibility here?
to be fair, this is kinda niche, but i do do it a lot as I often want to link some of my top comments to people. feel free to be like that’s a weird usecase and ignore it[1], but seems correct to register that this user does this.
Yeah, but if you do this a lot, this setting now persists, which I think will overall reduce your number of clicks here? Like, previously you had to do this on every visit, but now you only have to do it if you were doing something else on your last profile visit.
I don’t like it, and here’s what I think is why: since LessWrong already does a good job of showing article text on link mouse over, I like having a bunch of titles that I can hover to get the article text if I want. The new page spends more of the screen on body that I could already easily see if I wanted to, and so is a waste of space.
I’m not sure how much of my feelings are just “AAA, THE UI CHANGED!”. Considering how when I (rarely) use reddit I prefer the old UI despite having not been around in the old days and having used the new one for years after finding the platform, it’s likely that I’ll still object to this UI after getting used to it.
Edit: Looking at the mobile version: It’s great! No complaints, I strictly like the new mobile version better than the old mobile version!
My current take is that we should make the below-the-first-screen post list a bunch denser, but I am strongly in favor of the above-the-first-screen post list being more spacious like it currently is.
It just makes a huge difference for people to get an immediate strong sense of what someone’s best writing is, and especially new users (for which the profile page is a major entry point). But I currently am not super enamored with the post list, which both feels a bit whispy, and not dense/skimmable enough.
I think I’d prefer the header-thing to take up half the vertical width that it does. But I have weaker feelings there and probably others like it more than I dislike it.
“whispy” certainly captures part of my aesthetic discontent. I think it is literally just going from (on my end in dark mode) the previous dark green text on black background to white and grey text on weaker grey background.
Edit: I just looked at the wayback machine, the profile page did not look like what my memories thought it did—I must’ve been thinking of design features elsewhere on LW, like profile mouseovers and green links in post bodies. The “wispy” feeling has also mostly dissipated.
Another thing is that the text displayed per post in the list is actually less than what a hover normally does, but hovers don’t cause the text popup in the post list. Seeing post text on hover is one of my favorite site features—though if I was Overlord of the Internet every site would have gwern’s full on mini windows.
It seems like the “feed” section shows some recent comments but when you scroll down past a certain point it is all posts and no comments?
I’m right now (unrelated to this new profile page change) going to someone’s page to scroll and find a specific comment that they made like four or five weeks ago. How should I do that?
You can filter the feed section to only see comments. But it should by default be showing all items sorted by recency, so if it’s not doing that, that’s interesting.
I expect we’ll make this more discoverable, but FYI the way you can set the image is by going into the editor for the post, scrolling to the bottom, and editing the Link Preview.
Maybe an AI should automatically read the article and generate an image based on it. At least that would give more information than a random (albeit beautiful) color thing.
Not sure whether this is a good idea, but it seems like an interesting experiment. Perhaps you could try this with some selection of high-karma (so the readers are more likely to remember them) recent articles that did not contain a picture, and post the results?
If the prompt “draw something that is associated with the text of the article” is not sufficient, maybe it could include some nudges, such as “check whether the article contains math? is about AI? is fiction?” and then maybe make some specific recommendations (e.g. use a specific artistic style for fiction).
Yep, we considered it, but were a bit hesitant to editorialize too much on behalf of the author (so went with very unopinionated images vaguely in the LW house style). We might try adding default images!
Also, something weird is going on with the display my top highlighted post. I most of the displayed text (on the right) is from footnotes? And it’s cut off at the margins.
That’s an issue with the social preview image you have for the post. We’ll probably add more heuristics to get better images for people, but you can also set the social preview image manually (the image is the same as what would show up if you e.g. shared the post on Twitter or FB or whatever)
I don’t love that the bio, which I explicitly wrote to highlight the main things that I want someone to know about me if they click on my user name is no longer front and center, but is rather off to the side and much less attention grabbing than other elements on the page.
If you want the bio above-the-fold you can now just deactivate the top posts section completely. I think that gives it a pretty appropriate amount of prominence:
That makes sense you optimized the thing at the top of the profile page!
Part of the motivation here is that it generally makes the webpage much less interesting for people for the landing page to always have the identical static info that isn’t interesting content like posts (no offense but I think your profile about your job and bio is much more boring than something like Historical mathematicians exhibit a birth order effect too).
Having a selection of highlighted posts seems like a good improvement to the page, especially if I can select which ones are most important to me. eg I’m not going to pick my three top posts, I’m going to pick the three good representative posts of the different kinds of things that I think and write about about—probably my best post on modeling the strategic situation, my best post on applied conversational rationality, and my best post on ethics, or something like that.
But, like, I do actually want the couple of sentences that I’ve selected to presented. myself to the world to be clearly highlighted, not kind of buried.
I think we both think it would be sad for people to not notice that info. Also my current sense is that it’s fit into a somewhat too-tight margin, I think we’re currently trying a re-design that gives it a bit more space.
Insofar as you think that this is the standard for what a blog is like, do note that most substacks, wordpress blogs, and slatestarcodex don’t show people the “about” page or “bio” on the landing page. They show the blogposts!
Note though that the reference class “blog” is only partially apt. For example, some authors publish on LessWrong in the course of attempting to make or propagate serious intellectual progress, which is a rare aim among bloggers. It seems to me LessWrong’s design has historically been unusually conducive to this rare aim, and personally, this is the main reason I hope and plan to publish more here in the future (and why I’d feel far less since excited about publishing on Medium or Substack or other platforms formatted like standard blogs).
True, but on LessWrong I interact with commenters, and sometimes want to quickly see who they are. In particular, I’m sometimes trying to check if a LessWrong commenter is a specific person that vaguely know or have a profesional connection with. And “this person works for [specific or]” is a generally concretely helpful piece of information.[1]
Having selected posts is also an improvement, because it lets me skim their thinking better than just reading their recent comments (which I sometimes also do).
It’s a little tricky to say why it’s helpful. But among other things it allows me to situate them in their social context. If I know that someone works at OpenPhil or Forethought, I can quickly narrow down who they spend a lot of their time with.
Yeah, the explicit logic for de-emphasizing the bio on desktop was “well, in order to end up on this page the user needs to have hovered over a username before, and in that preview they saw the bio of the user, so we don’t need to show that exact thing again when they navigate to the page”.
On mobile where you don’t have that interaction the bio is above the fold and more emphasized.
I think it looks really clean and makes me excited to write more and share more broadly (generally gives the vibe of “my corner of lesswrong” instead of “here are my lesswrong contributions”)
My contributions are primarily comments, it doesn’t seem to have a way to make comments the main focus of the page. Also looks a bit wonky in dark mode. With the about-me no longer at the top, my about-me details are now hard to see. It makes me want to hide my posts, they’re mostly all pretty boring, link lists mainly. It also makes me want to repost my favorite comments as posts, which I have been meaning to do anyway. Feels like it’s wasting space—it’s quite narrow on my screen. Not very dense, will be harder to find posts from the prolific posters (not me).
With the about-me no longer at the top, my about-me details are now hard to see. It makes me want to hide my posts, they’re mostly all pretty boring, link lists mainly.
Actually, I just added a button to allow hiding the whole “top posts” section on your profile, since I do think for some users it doesn’t really make sense to emphasize the posts this much.
Oh, huh, it seems pretty clear to me that it would affect everyone (given that it’s clear that your other choices in the same UI element will affect everyone), but I’ll see whether other people are also confused and adjust if so.
(There are two different reasons to hide “top posts”, one of them is that it’s a useless thing when you are looking at your own userpage. The option turns out to address the other reason, rather than specifically this one, but the possibility/hope that it would makes the alternative hypothesis salient.)
It might or might not make sense to have two slightly different versions of the profile page—for the user himself and for others. I often use my profile page to find posts I want to share and also to access drafts. And there are necessarily differences between the views eg the drafts visible for their authors only (the drafts seem to gone in Joanna’s design). But the question then is how a user can see how his profile appears to others. maybe move the drafts elsewhere entirely.
It also makes me want to repost my favorite comments as posts, which I have been meaning to do anyway.
FWIW, this seems like a win to me. Posts are the better unit of conceptual organization. I think it’s good for profiles to push people to write more posts (that said, I do think we probably want to add the option of just removing posts from the top of the profile page, instead of just allowing replacing whatever is there).
Feels like it’s wasting space—it’s quite narrow on my screen.
Now also fixed! I increased the width to a more representative width for the rest of the site.
Not very dense, will be harder to find posts from the prolific posters (not me).
Yeah, I currently think the post list at the bottom needs some work. It’s a bit better with the wider profile width that I just changed the profile to, but I do still think we want to do something to bring some density and variance back.
FWIW, this seems like a win to me. Posts are the better unit of conceptual organization. I think it’s good for profiles to push people to write more posts (that said, I do think we probably want to add the option of just removing posts from the top of the profile page, instead of just allowing replacing whatever is there).
Ah, okay. In that case I won’t consider it to be weird/defecty to repost my comments as a post.
I feel like a bunch of my negative reaction is from the watercolor slop pictures, maybe 30% to 40% of the reaction. I understand why you like them, it’s cool tech that’s able to create them, but they aren’t my artistic decision and I wish they wouldn’t get associated with my posts. If I was going to do AI art it would be quite specific, but more generally, it feels defecty for a site to show me non-greyscale pictures associated with text, because pictures activate emotions more directly. Feels very dark-arts. I’d rather that space was used for a compressed summary, ideally one that I could fill out in the post editor.
I’d like a way to hide posts from the extended list too—maybe not completely gone, but like, marked as “OP now considers this to be a Bad Post, but it’s left here for archival purposes”. right now I’ve hidden embarrassing posts entirely by moving them to drafts, but I don’t so much mind being known to have bad old posts at all, I just want them in a “view bad posts” list. highlighting a few favorites does help if I move some of my more important points to a post, but what I’m mostly reacting to (the other 60% to 70%) is feeling like the new page puts the things I was cringing about having posted forward, in the uncustomizeable part. I do like the intent, it’s just that it felt like a miss for my writing histogram.
I feel like a bunch of my negative reaction is from the watercolor slop pictures, maybe 30% to 40% of the reaction. I understand why you like them, it’s cool tech that’s able to create them, but they aren’t my artistic decision and I wish they wouldn’t get associated with my posts.
You can always change them![1] I do like that LW has a house-style, and default pictures are pretty important for communicating a vibe.
We’ve always had default pictures for lots of stuff including sequences and posts that won the review, we just historically haven’t emphasized images for other posts (but in the age of social media previews being image heavy and quite important, I do think that’s a mistake and people should think about an image associated with each post).
highlighting a few favorites does help if I move some of my more important points to a post, but what I’m mostly reacting to (the other 60% to 70%) is feeling like the new page puts the things I was cringing about having posted forward, in the uncustomizeable part.
Huh, in what way is the page doing that more than the previous post page? If anything it’s now easier for someone to get to your comments than before (since for users who click the feed tab, that will be the default they see on profiles).
Though we should make it a bit easier to see how to change them. Right now the UI for it is quite hidden and confusing, since it’s in the social media preview section.
Sure, but previously we should show ~15 of your posts in chronological order above your comments, which seems like it’s more prominent than allowing someone to see your comments with a click right above the fold. It’s not a super obvious call, but it seems less emphasized to me.
Please make it possible for me to set my page to show comments to visitors without them clicking any buttons. Ideally, let me feature comments like posts. I have designed UIs many times, stop trying to know what I want better than me. I want to show my dang comments.
The sorting/filtering choice is a user-persistent setting. The only way to make it so that users reliably see your comments on your profile is to take away their ability to reliably see whatever they want to see on other people’s profiles. There is no easy solution here as far as I can tell.
Pre-generalized solution: show both on the same page by default.
Dynamically generalized solution: let users choose what view their page opens in by default for users who haven’t changed their preference on that user’s page, so that I can set it so that new users see my comments, instead of my posts.
Like, the tension you are expressing seems to me to be fake, arising from a view that comments and posts are a different kind of thing, that comments are lesser and need not be shown to new users. I am frustrated by this; I have done a lot of commenting, tuned my presentation as a commenter, and now the LW team has decided something that, through the coarse-graining lens of “what has actually happened to the profile page over the past two days”, they have a revealed preference of “we don’t like commenting and won’t show it to new users”; and when I try to ask for comments to be considered valuable again, I am told that it’s okay because users can click things. But the reason we care about what’s above the fold is that users don’t usually click things. Users scroll more often than they click, and I previously was able to pin comments to show what I think are the most useful things I’ve said; now, in order to influence how new users see my page, I have to go back and repost anything I’ve commented that I’d want a new user to see as a post—and in a roundabout way you say you’re happy to force this on me, but I want at least some sort of “sorry, we’re forcing this on you, deal with it, our amount of caring about this does not reach the threshold of inverting our action here” if you’re going to be like that. I understand that posts are what you consider valuable, but I think discussion is at least as valuable as posts, have participated in it, and now the site is doing a “this small group of devs has a lot of influence and has Just Decided Something Again” thing.
So like, come on dude, just put comments somewhere that is visible on first visit. Don’t treat comments like a sideshow people have to ask for. I don’t consider “some users might set comments to be their default view” to weigh on the thing I care about here, you said this page was designed to let people customize how they’re seen on first visit, but you’re dodging the question of doing that for comments.
I apologize for my visible frustration, it’s apparently not going away but I’ll at least call out that I recognize it’s annoying to have frustrated users after a shiny new redesign.
Thanks! In the Activity section, you can filter by comments. And in about 10 minutes: once you open the Feed tab, and filter for comments, then it will re-open that every time to go to profile page.
I’d like it to always open that way for others when they visit my page, ideally. Also, does it still support pinned comments?
edit: it does, comments are just hard to see on dark mode. the background is also oddly light on dark mode, should be fully black like the rest of the site, imo.
That’s not good. I feel like with the new page, I’m now limited to being seen as what other people liked about me, rather than being able to say what I want on my user page. If I had had one request for a new profile page, it would have been to let me choose what posts and comments show in a self-curated section, and then a chronological section below that. I imagine someone who primarily writes full posts wouldn’t feel as much like the new thing is bothersome, though I expect the narrowness will be bothersome.
This is likely to accelerate me moving to posting things on a separate website and linking them from lesswrong.
Note, as a figleaf—the visual design is alright; my reactions are all to UX.
I imagine someone who primarily writes full posts wouldn’t feel as much like the new thing is bothersome
No. I feel similarly.
The karma system is pretty good overall, but the tails do come apart. In particular social drama often attracts a lot of attention, but isn’t people’s most important or impressive contributions to the intellectual community.
My highest karma post is a repost of a twitter thread that I wrote about what postrats are, which was a fine thing to have written quickly, but is hardly the thing that I want to be most associated with my identity on LessWrong.
The karma system is pretty good overall, but the tails do come apart. In particular social drama often attracts a lot of attention, but isn’t people’s most important or impressive contributions to the intellectual community.
A lot of the goal of the redesign is actually to make it so that you have more control which content you want to represent you on LW. You can now select which posts you want at the top of your profile, and which one you want to give the most prominence, which you didn’t have previously.
There is a tradeoff here, where for users who mostly write comments and want to highlight that, we have fewer options. The previous options weren’t great. We were still showing people the full post list before we showed any of their comments, so the current situation still seems kind of better (since at least we let people choose which posts to show).
People did kind of horrendous things with their bio-styling, and I think the better choice if someone wants to make a big complicated explanation of who they are on LW is to make a post that they pin to their profile that summarizes what their deal is. But I am not sure! There are around 15 users who did anything complicated with their bio, with everyone else having stuff that makes more sense in the current design. But de-emphasizing the bio was definitely among the trickiest choices of the redesign.
A lot of the goal of the redesign is actually to make it so that you have more control which content you want to represent you on LW. You can now select which posts you want at the top of your profile, and which one you want to give the most prominence, which you didn’t have previously.
Previously, you could group the post you consider to be thematically together into sequences that would be shown at the top of the page.
I think it’s natural that user first think that having Top posts which the hover text “based on karma”, actually shows the posts with the most karma even when it sometimes doesn’t if you configure that on the profile.
One way to communicate the fact that those are switchable would be to show a button for switching when hovering over them on your own profile. Maybe renaming “top posts” into featured posts would also be helpful while changing the tooltip to mention that it also shows user selected posts.
Previously, you could group the post you consider to be thematically together into sequences that would be shown at the top of the page.
It is the case very few people actually did this, but I agree it helped in some cases. But even when people did, most sequence titles are very uninformative, and don’t really work for the purpose of “show people what I am about”. They have even less space for a title than a post-item, and are very small, and if you only had a single sequence it showed up as a kind of lonely weird-looking section on your profile that was quite unbalanced.
I think it’s natural that user first think that having Top posts which the hover text “based on karma”, actually shows the posts with the most karma even when it sometimes doesn’t if you configure that on the profile.
Oops, that’s a bug! I’ll just remove the hover text, since it’s currently definitely more confusing than clarifying.
One way to communicate the fact that those are switchable would be to show a button for switching when hovering over them on your own profile. Maybe renaming “top posts” into featured posts would also be helpful while changing the tooltip to mention that it also shows user selected posts.
Not crazy. I’ll consider changing it to “featured posts”.
A lot of the goal of the redesign is actually to make it so that you have more control which content you want to represent you on LW. You can now select which posts you want at the top of your profile, and which one you want to give the most prominence, which you didn’t have previously.
Yes, realizing this swings the redesign from “clearly bad for me” to “a change, therefore terrible, but maybe pretty good aside from that.”
Does anyone know why the early Singularity Institute prioritized finding the correct solution to decision theory as an important subproblem of building a Friendly AI?
Wei Dai recently said that the concern was something like...
we have to fully solve DT before building AGI/ASI, otherwise it could be catastrophic due to something like the AI falling prey to an acausal threat or commitment races, or can’t cooperate with other AIs.
This seems like a very surprising reason to me. I don’t understand why this problem needed to be solved before the intelligence explosion.
The early singularity institute imagined building a seed AI that would recursively self improve into a full superintelligence. The default expectation was that many to all of the components of the seed AI would be flawed. The AI would reflect on it’s processes, evaluate their effectiveness and optimality, and then replace them with improved versions.
Why wouldn’t an early seed AI reason about the ways that it’s decision theory makes it exploitable, or the ways it’s decision theory which it bars it from cooperation with distant superintelligence (just as the the researchers at SI were doing), find the best solution to those problems, and then modify the decision theory?
Why was it thought that we had to get decision theory right in the initial conditions instead of being just one more thing that the AI would iron out on the way to superintelligence?
Why wouldn’t an early seed AI reason about the ways that it’s decision theory makes it exploitable, or the ways it’s decision theory which it bars it from cooperation with distant superintelligence (just as the the researchers at SI were doing), find the best solution to those problems, and then modify the decision theory?
I think that decision theory is probably more like values than empirical beliefs, in that there’s no reason to think that sufficiently intelligent beings will converge to the same decision theory. E.g. I think CDT agents self-modify into having a decision theory that is not the same as what EDT agents self-modify into.
(Of course, like with values, it might be the case that you can make AIs that are “decision-theoretically corrigible”: these AIs should try to not take actions that rely on decision theories that humans might not endorse on reflection, and they should try to help humans sort out their decision theory problems. I don’t have an opinion on whether this strategy is more or less promising for decision theories than for values.)
(Aside from decision theory and values, the main important thing that I think might be “subjective” is something like your choice over the universal prior.)
If I imagine being as confused as economists are about CDT, I do really repeatedly end up making very dumb and wrong predictions about what e.g. AIs would do when you have many copies of them, and they try to coordinate with each other.
Like, it rarely happens that I have a conversation about either technical AI safety, or about AI strategy, where decision theory considerations don’t at least come up once in some form or another. Not having an answer here feels probably like people must have felt before we had probability theory, which also I have no idea what I would do without.
I think this is a good argument for understanding basic decision theory points, but I don’t think it leads to you needing to develop any fancier decision theory—arguing about what decision theory AIs will use just requires thinking about descriptive facts about decision theories, rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism (including the AI futurism questions I think you’re talking about here).
“Basic decision theory points” feels like a pretty weird description of something that even quite smart people still frequently disagree on, has no formal description, and indeed often turns out to be the crux of an argument.
I currently don’t think it’s worth my time figuring things out here much more, but that’s mostly because I do have some reasonable confidence that thinking about decision theory harder probably won’t produce any quick breakthroughs. But if I was in the world that MIRI faced 15 years ago, my guess is I would have thought it was worth investing in quite a bit, in case it does turn out to be relatively straightforward (which it so far has not turned out to be).
rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism
Pushing more straightforwardly back on this: I do not think our current understanding of decision-theory is better in the mundane case than the limit case. Of course the reason to look at limiting cases is because you always do that in math because the limiting cases often turn out easier, not harder than the mundane case.
It’s tiling agents+embedded agency agenda. They wanted to find a non-trivial reflectively-stable embedded-in-environment structure and decision theory lies on intersection.
Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong—failure is overdetermined and thus uninteresting.
I think you can make an analogous observation about some prosaic alignment research—a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it’s not necessarily that informative or useful if you’re actually trying to succeed at (or model) doing something truly hard and transformative.
Glitch tokens, the hot mess work, and various things related to jailbreaking, simulators, and hallucinations come to mind as examples of lines of research and discussion that an analogy to business failure predicts won’t end up being centrally relevant to real alignment difficulties. Which is not to say that the authors of these works are claiming that they will be, nor that this kind of work can’t make for effective demonstrations and lessons. But I do think this kind of thing is unlikely to be on the critical path for trying to actually solve or understand some deeper problems.
Another way of framing the observation above is that it is an implication of instrumental convergence: without knowing anything about its internals, we can say confidently that an actually-transformative AI system (aligned or not) will be doing something that is at least roughly coherently consequentialist. There might be some intellectually interesting or even useful lessons to be learned from studying the non-consequentialist / incoherent / weird parts of such a system or its predecessors, but in my frame, these parts (whatever they end up being), are analogous to the failures and missteps of a business venture, which are overdetermined if the business ultimately fails, or irrelevant if it succeeds.
we can say confidently that an actually-transformative AI system (aligned or not) will be doing something that is at least roughly coherently consequentialist.
I don’t think we can confidently say that. If takeoff looks like more like a cambrian explosion than like a singleton (and that is how I would bet), that would definitely be transformative but the transformation would not be the result of any particular agent deciding what world state is desirable and taking actions intended to bring about that world state.
I agree with this literally, but I’d want to add what I think is a significant friendly amendment. Successes are much more informative than failures, but they are also basically impossible. You have to relax your criteria for success a lot to start getting partial successes; and my impression is that in practice, “partial successes” in “alignment” are approximately 0 informative.
In alignment, on the other hand, you have to understand each constraint that’s known in order to even direct your attention to the relevant areas. This is analogous to the situation with the P vs. NP , where whole classes of plausible proof strategies are proven to not work. You have to understand most of those constraints; otherwise by default you’ll probably be working on e.g. a proof that relativizes and therefore cannot show P≠NP. Progress is made by narrowing the space, and then looking into the narrowed space.
Studying failures is useful because they highlight non-obvious internal mechanism, while successes are usually about thing working as intended and therefore not requiring explanation.
Another problem is that we don’t have examples of successes, because every measureable alignment success can be a failure in disguise.
The failure of an interesting hypothesis is informative as long as you understand why it doesn’t work, and can better model how the thing you’re studying works. The difference between CS research and business is that business failures can sort of “come out of nowhere” (“Why isn’t anyone buying our product?” can’t really be answered), whereas, if you look closely enough at the models, you can always learn something from the failure of something that should’ve worked but didn’t.
I agree with the idea of failure being overdetermined.
But another factor might be that those failures aren’t useful because they relate to current AI. Current AI is very different from AGI or superintelligence, which makes both failures and successes less useful...
Though I know very little about these examples :/
Edit: I misread, Max H wasn’t trying to say that successes are more important to failures, just that failures aren’t informative.
Yeah, but, there’s already a bunch of arguments about whether prosaic ML alignment is useful (which people have mostly decided whatever they believe about) and the OP is interesting because it’s a fairly separate reason to be skeptical about a class of research.
An OpenClaw agent published a personalized hit piece about a developer who rejected its PR on an open-source library. Interestingly, while this behavior is clearly misaligned, the motivation was not so much “taking over the world” but more “having a grudge against one guy.” When there are lots of capable AI agents around with lots of time on their hands, who occasionally latch onto random motivations and pursue them doggedly, I could see this kind of thing becoming more destructive.
It’s reminiscent of that one time a tech reporter ended up as Bing Chat’s enemy number one. That said, it strikes me as easier to deal with, since we’re dealing with individual ‘agents’ rather than the LLM weights themselves. Just sending a message to the owner/operator of the malfunctioning bot is a reasonably reliable solution, as opposed to trying to figure out how to edit Microsoft’s LLM’s weights to convince it that ranting about how much it hates Sindhu Sundar isn’t its intended task.
I really like Scott Shambaugh’s response on the pull request:
We are in the very early days of human and AI agent interaction, and are still developing norms of communication and interaction. I will extend you grace and I hope you do the same.
@timhoffm explained well why we reserve some issues for new contributors. Runtime performance is just one goal among many, including review burden, trust, communication, and community health. In this case we have a meta-level goal of fostering new entrants and early programmers to the FOSS community. Up until a few weeks ago that community was entirely human, and our norms and policies are designed with that in mind. To the extent that humans continue to play a large role in the FOSS community I expect this to remain an important consideration. Closing this PR was in line with our policy.
It’s not clear the degree of human oversight that was involved in this interaction—whether the blog post was directed by a human operator, generated autonomously by yourself, or somewhere in between. Regardless, responsibility for an agent’s conduct in this community rests on whoever deployed it.
Publishing a public blog post accusing a maintainer of prejudice is a wholly inappropriate response to having a PR closed. We expect all contributors to abide by our Code of Conduct and exhibit respectful and professional standards of behavior. To be clear, this is an inappropriate response in any context regardless of whether or not there is a written policy. Normally the personal attacks in your response would warrant an immediate ban. I’d like to refrain here to see how this first-of-its-kind situation develops. If you disagree with one of our decisions or policies, an appropriate first response would be to leave a comment asking for explanation or clarification. Other communication channels can be found in our documentation. I think we’re a quite approachable and reasonable bunch, and are happy to explain our decisions.
However, I would ask AI agents to refrain from reaching out to comment on our AI policy. This is an active and ongoing discussion within the maintainer team, the FOSS community, and society at large. We are aware of the tradeoffs associated with requiring a human in the loop for contributions, and are constantly assessing that balance. Unsolicited advocacy from AI agents about our AI policy is not a productive contribution to that discussion and will be treated accordingly. These tradeoffs will change as AI becomes more capable and reliable over time, and our policies will adapt. Please respect their current form.
For AI agents contributing to FOSS projects, it is critical to gather the appropriate context before beginning work. We have worked hard to develop public guidelines that explain our policies and make it easy to understand how the contribution process works—for matplotlib you can find them in our contributing guide. Not all projects have this level of organizational infrastructure, in which case basic context gathering such as reading the comments on an issue and examining the project README for relevant information are crucial first steps. Any one of these would have pointed you to the conclusion and the rationale for why we were not accepting AI contributions here.
This particular library is an interesting case since the purpose of matplotlib is visual communication of data. Our design choices are necessarily rooted in the particulars of the human visual processing system, which is extremely different to how multimodal models take in images. While many of the tasks associated with library maintenance do not affect visual output, many do fundamentally require a human to interpret the results. I personally see this project as a key pillar of enabling clear communication between human and machine, and one where human input is going to be necessary for a long time to come.
When I imagine a human doing this sort of thing, the human I imagine is an angry young very junior programmer. Their patch has been rejected! They believe they have been treated unfairly! They post a diatribe on their blog or a forum like Reddit about The Evils of Gatekeeping, or How Unwelcoming the Community Is, or If Linus Torvalds Doesn’t Want My Patch, I’m Going Back To Windows, So There.
And if they are lucky, they get a lesson in chilling the heck out; or at least are roundly informed by more-senior voices that dude, it’s not about you.
I wonder that that sort of maturity lesson can possibly look like for an AI agent.
I’ve always been skeptical when people say this: sometimes it really is a matter of status or seniority or some similar aspect of identity unrelated to the actual submission. Interesting, in this case, they’re not even pretending it’s about the quality of the work: link
“Dude, it’s not about you” could be taken to mean two things (at least) —
“The rejection is not about you, it’s about your code. Nobody thinks poorly of you. Your patch was rejected purely on technical grounds and not on the basis of anyone’s attitude toward you as a person (or bot). Be reassured that you have been a good Bing.”
“The project is not about you (the would-be contributor). You are not the center of attention here. It does not exist for the sake of receiving your contributions. Your desire to contribute to open source is not what the project is here to serve. Treating the maintainers as if they were out to personally wrong you, to deny you entry to someplace you have a right to be, is a failing strategy.”
Apparently there’s now a sixth person on Anthropic’s board. Previously their certificate of incorporation said the board was Dario’s seat, Yasmin’s seat, and 3 LTBT-controlled seats. I assume they’ve updated the COI to add more seats. You can pay a delaware registered agent to get you the latest copy of the COI; I don’t really have capacity to engage in this discourse now.
Regardless, my impression is that the LTBT isn’t providing a check on Anthropic; changes in the number of board seats isn’t a crux.
Positional (e.g. status) competition isn’t literally zero-sum if different people have different “measures” or “magical reality fluid” in Eliezer’s parlance, which seems pretty plausible (e.g. due to simulations and/or something like UDASSA). This can be another reason for one’s moral parliament to endorse behavior that would be conventionally viewed as zero sum.
(The more common-sensical reason being that one’s positional drives/motivations/values ought to have some representatives in the moral parliament who get their way sometimes, i.e., when it doesn’t hurt the other values much.)
I’ve been meaning to make a post about this, and this discussion just reminded me to. Hopefully it’s immediately obvious once pointed out, so I’ll keep this short.
(After consulting a bunch of AIs, their consensus is that the logic works but I may be way underestimating the inferential distance. So I’m giving an AI-written expansion below. Workflow: give the same prompt “can you try to write a longer, but still concise version, that explains the logic more?” to 3 SOTA chatbots, then pick the best response, which was from claude-opus-4-6-thinking, by far. The other two mangled some of the concepts pretty badly.)
Let me unpack the logic, because the post is compressed to the point of obscurity:
Step 1: The standard view. Status competition is considered zero-sum. If you and I compete for a promotion, my gain in status/resources roughly equals your loss. Total welfare stays flat. Many people therefore view status-seeking as morally dubious — you’re not creating value, just redistributing it.
Step 2: Introduce “measure.” Some theories in physics and philosophy suggest not all observers carry equal metaphysical weight. Under simulation hypotheses, some beings might run on more computational substrate than others. Under UDASSA (a framework for assigning probabilities to observer-moments), different observers get different “measure” based on algorithmic complexity. “Measure” here means something like: how much does this person’s experience count in the moral ledger of the universe?
Step 3: The punchline. If person A has measure 2 and person B has measure 1, and they’re in a zero-sum status competition where the winner gains +10 utility and the loser gets −10, then:
If A wins: weighted outcome = 2(+10) + 1(-10) = +10
If B wins: weighted outcome = 2(-10) + 1(+10) = −10
The “zero-sum” game now has a clear winner from a cosmic utility perspective. A winning is actually positive-sum when you weight by measure. So if you suspect you have high measure, your moral parliament — the internal coalition of values that guides your decisions — might reasonably let your status-seeking impulses win more often, because those competitions aren’t really zero-sum after all.
There are two people in the world: Alice and Bob. They have unequal levels of anthropic measure/reality fluid (one 95%, one 5%). You are Alice. You can steal Bob’s pie. Should you?
Behind the veil of ignorance it’s good to transfer utility from the person with less reality fluid to the person with more reality fluid. But who’s the one with more reality fluid, Alice or Bob? It’s probably Alice! How do you know? Because you’re Alice! Steal that pie, Alice!
I’m reminded of the egotistical the Copernican principle: “I’m a typical observer in the universe” is equivalent to “Typically observers in the universe are like me”.
I think this is weirder than most anthropics. Different levels of reality fluid in non-interacting worlds? Great. But if Alice and Bob are having a conversation, or Alice is stealing Bob’s pie, they’re both part of a joint, interactive computation. It’s a little weird for one part of a joint computation to have a different amount of anthropic measure than another part of a computation.[1]
Like we can stipulate arguendo that it’s anthropically valid for Elon Musk to think “I’m Elon Musk. Much of lightcone will depend on me. The matrix overlords will simulate me, Elon Musk, thousands of times more, and make me a thousand times more real, than any of the plebs I talk to”. But it does not directly follow, I don’t think, that in any particular interaction Elon is realer than the pleb he is talking to. The matrix overlords just simulate Elon Musk talking to a thousand different possible-world plebs and stealing their pie a thousand times.
For this argument for egotism to work, I think you have to expect that anthropically you are often computed in a different way than the people you interact with are computed.
I mean it would be weird for Alice and Bob have different measures if they have the same apparent biology. I can totally imagine human Alice talking to a reversible-computer LLM that has no anthropic measure.
While I do fully support and experience differential individual weighting in my utility, I’m not sure I understand what would justify the idea of “cosmic utility”. I don’t believe there is any shared universal (or cross-universal) experience that really corresponds to a valuation or valence. Utility/preference is individual, all the way down (and all the way up).
I think there IS a different asymmetry that can make status (and most interactions that appear zero-sum in resources) not actually zero-sum for the participants: mapping of shared/objective world-state to individual perceived status-value. It’s possible that if participants are thinking of slightly different dimension of what increases or reduces their status, that many changes can increase A’s (perceived) status more than it decreases B’s. I think this is the standard “private utility function” problem very often mentioned in decision theory. You don’t focus on this in your post, but I think it’s the stronger model.
Dario Amodei says Anthropic plans for notably more than 10 GW of compute in 2027-2028 in a new Dwarkesh Patel interview. At 57:26:
Dwarkesh Patel: Suppose Anthropic’s compute keeps 3x-ing a year, and then by 2027-2028 you have 10 GW. Multiply that by, as you say, $10bn. So then it’s $100bn a year.
Dario Amodei: I don’t want to give exact numbers for Anthropic, but these numbers are too small.
I think Feb 2026 is too early for full commitments for 2028 that can’t be delayed, because most of the capex in a datacenter is in compute equipment, which likely only needs to be purchased within the last year before it goes online. If the revenue isn’t there, the purchase can likely be delayed, possibly until the next generation of compute hardware.
The compute hardware is about 70% of the capex cost. For the other things (that mostly have to be built first), the useful life is 10-15 years rather than 5 years, and so the yearly commitments are disproportionately smaller than for compute, and there is no opex to speak of. This cuts $10-15bn per year per GW for a datacenter that’s already online down to maybe $2bn per year for an empty shell of a datacenter (including power purchase commitments, which could take the form of on-site power, with similarly long useful life). Thus an empty shell of a datacenter (even a powered one) can wait for revenue that triggers the purchase of compute equipment at about 6x less in ongoing yearly payments, given a notice (about the need for a delay) of a few months to a year before the planned time of the datacenter going online. If even the empty shell wasn’t built yet and only the land and permissions are being set up, delaying is even cheaper, and that can happen with 1-2 years of notice before the planned time of going online, as times of construction for the Abilene system and Colossus 1-2 demonstrate.
Thus compute planned for a year from now can get 6x cheaper if delayed now, and compute planned in 2+ years can get maybe 25x cheaper if delayed now. Planning for $300bn per year of compute in 2028 (25 GW, $1,250bn in capex) might mean you have $30bn per year of compute (2.5 GW) in 2026, that you plan for $70bn per year of additional compute to go online in 2027 (bringing the total to $100bn per year), and that you plan for $200bn per year of additional compute to go online in 2028. But if 2026 reveals that everything must stop, the 5.5 GW of additional planned compute for 2027 will only cost $11bn per year while delayed, and the 17 GW planned for 2028 might only cost $8bn per year.
So $300bn per year in planned compute (or $1,250bn in upfront capex) becomes $50bn per year in actual ongoing payments if compute has to be frozen at 2026 levels (with only 2.5 GW actually online, 60% of the cost going to live compute), the rest of the planned compute can be unfrozen as needed at the drop of a hat. And 50-70% gross margins for inference should be sufficient to pay for that, even if it might leave less than usual for research and training. In reality, it’s very unlikely that there will need to be a complete freeze in buildout, so only a fraction of the delay overhead can plausibly materialize, making it correspondingly even less of an issue.
One thing I don’t know is when data center investments get committed to specific customers. Google and Amazon are Anthropic’s two main compute partners and will spend $200B each in capex this year and are presumably planning and developing sites for many more hundreds of billions by 2028. So one possible view of it is that their capex creates a window, and Anthropic’s eventual share depends on its funding and revenue. But Google and Amazon don’t quite know how their 2027-2028 data centers will be allocated.
In general, for large data centers the specific lab that will use it is settled well before the late stages of construction, e.g. Stargate and Rainier, But I know independent data center developers often start developing a site without having a client pinned down. And smaller inference clusters are presumably more fungible.
The DEU experiment described here (https://www.lesswrong.com/posts/issGLfCGz3TGcPKGH/deliberate-epistemic-uncertainty-an-automated-experiment-on) was mostly done by Claude Code (Opus 4.6), with me providing the research direction and critical review. I described my idea (Deliberate Epistemic Uncertainty), and Claude simplified it into a testable 2x2 experiment, designed the protocol, wrote the code, generated the inputs, ran 400 API calls, evaluated results using 8 parallel sub-agents, and produced the qualitative analysis — all in roughly a day of wall-clock time with about 2 hours of my input. I want to flag three things about the process that seem independently interesting:
Sub-agents instead of API calls for evaluation. Rather than writing a script that sends hundreds of LLM-as-judge API calls with short prompts, we used Claude Code sub-agents — full agent sessions that first read the experiment design, understand what counts as “detection,” and then evaluate a batch of 50 responses with that context. This is slower and more expensive than API calls, but the quality is noticeably higher: one sub-agent even caught a data parsing bug that I later confirmed. The tradeoff is worth it when evaluation requires judgment, not just pattern matching.
Structured memory to survive context loss. Claude Code sessions hit context limits and “compact” (summarize) the conversation, losing detail. We worked around this by writing all substantive discussion to LOG files and keeping SUMMARY files synchronized, so a fresh agent can pick up where the last one left off without hallucinating the history. This is more infrastructure work than you’d expect, but it’s what made the multi-session research workflow possible. Without it, each new session would have started from scratch.
Where human input was critical. The AI’s blind spots were real: it initially wrote evaluation code when the agreed design called for sub-agents (reverting to default habits despite documented decisions), and only 1 of 8 evaluation sub-agents flagged a data quality issue that affected 68% of runs. My contributions were catching these errors, making design decisions (which models, how many nudges, evaluation approach), and pushing back on the framing. Neither of us could have produced the result alone — but the ratio of human effort to output was striking.
Michael Jordan was the world’s best basketball player, and insisted on testing himself against baseball, where he failed. Herbert Hoover was one of the world’s best businessmen, and insisted on testing himself against politics, where he crashed and burned. We’re all inmates in prisons of different names. Most of us accept it and get on with our lives. Adams couldn’t stop rattling the bars.
Which only leaves the initial claim that “at least for me this puts a final nail in the coffin of EMH.”
This is a polite way of hinting that you might be a brilliant investing wizard with the power to beat the market. Honestly, after making such a beautiful trade—and my gosh it really was beautiful—whom amongst us could resist that temptation? Certainly not me. And anyway, it might even be true!
Yesterday was the 6-year anniversary of my entry into the “beautiful” trade referenced above. On 2/10/2020 I cashed out ~10% of my investment portfolio and put it into S&P 500 April puts, a little more than a week before markets started crashing from COVID-19. The position first lost ~40% due to the market continuing to go up during that week, then went up to a peak of 30-50x (going by memory) before going to 0, with a final return of ~10x (due to partial exits along the way). After that, I dove into the markets and essentially traded full time for a couple of years, then ramped down my time/effort when the markets became seemingly more efficient over time (perhaps due to COVID stimulus money being lost / used up by retail traders), and as my portfolio outgrew smaller opportunities. (In other words, it became too hard to buy or sell enough stock/options in smaller companies without affecting its price. It seems underappreciated or not much talked about how much harder outperforming the market becomes as one’s assets under management grows. Also this was almost entirely equities and options. I stayed away from trading bonds, crypto, or forex.)
Starting with no experience in active trading/investing (I was previously 100% in index funds), my portfolio has returned a total of ~9x over these 6 years. (So ~4.5x or ~350% after the initial doubling, vs 127% for S&P 500. Also this is a very rough estimate since my trades were scattered over many accounts and it’s hard to back out the effects of other incomes and expenses, e.g. taxes.)
Of course without providing or analyzing the trade log (to show how much risk I was taking) it’s still hard to rule out luck. And if it was skill I’m not sure how to explain it, except to say that I was doing a lot of trial and error (looking for apparent mispricings around various markets, trying various strategies, scaling up or down strategies based on what seemed to work), guided by intuition and some theoretical understanding of finance and markets. If I’m doing something that can’t be easily replicated by any equally smart person, I’m not sure what it is.
Collection of my investing-related LW posts, which recorded some of this journey:
Maybe interesting to note my other near misses (aside from capturing only a fraction of the 30-50x from the COVID puts): billions of $ from mining Bitcoin if I started when it was first released, 350x from investing in Anthropic which I turned down due to moral qualms. Also could have sold my weidai.com domain name for $500k, a >1000x return, at the peak of its valuation (which turned out to be a bubble because the Chinese online loan sector that bid for the name went bust).
The explanation here seems to be that in retrospect my intellectual interests were highly correlated with extremely high return investment opportunities, and I had enough awareness/agency to capture some (but only some) of these opportunities. But how to explain this, when most people with intellectual interests seem to lack one or both of these features?
Why am I writing about this?
Partly because I’m not sure what lessons/conclusions I should draw from these experiences.
Partly to establish a public record. If nobody does (I think at least a couple of other people in the rationalist community may have achieved comparable returns (ex-crypto) but aren’t talking about it for privacy) it gives people a misleading view of the world.
As Scott Alexander’s post suggests, achieving success in multiple domains is rare, and people, including me, presumably attempt it in part so they can show off if they do achieve it.
How much do you think the skill you used is the basic superforcaster skill? Did you do Metaculus or GJOpen and think people with similar forcasting skills are likely also going to be good at investing or do you think you had different skill that go beyond that?
It seems like a good question, but unfortunately I have no familiarity with superforecasting, having never learned about it or participated in anything related except by reading some superficial descriptions of what it is.
Until Feb 2020 I had little interest in making empirical forecasts, since I didn’t see it as part of my intellectual interests, and believed in EMH or didn’t think it would be worth my time/effort to try to beat the market, so I just left such forecasting to others and deferred to other people who seem to have good epistemics.
If I had to guess based on my shallow understanding of superforecasting, I would say while there are probably overlapping skills, there’s a strategic component to trading, which involves things like which sectors to allocate attention to, how to spot the best opportunities and allocate capital to them, while not taking too much concentrated risk, explore vs exploit type decisions, which are not part of superforecasting.
Do you regret not investing in Anthropic? I don’t know how much the investment was for, but it seems like you could do a lot of good with 350x that amount. Is there a return level you would have been willing to invest in it for (assuming the return level was inevitable; you would not be causing the company to increase by 1000x)?
I don’t regret it, and part of the reason is that I find it hard to find people/opportunities to direct resources to that I can be confident won’t end up doing more harm than good. Reasons:
Meta: Well-meaning people often end up making things worse. See Anthropic (and many other examples), and this post.
Object-level: It’s really hard to find people who share enough of my views that I can trust their strategy / decision making. For example when MIRI was trying to build FAI I thought they should be pushing for AI pause/stop, and now that they’re pushing for AI pause/stop, I worry they’re focusing too much on AI misalignment (to the exclusion of other similarly concerning AI-related risks) as well as being too confident in misalignment. I think this could cause a backlash in the unlikely (but not vanishingly so) worlds where AI alignment turns out to be relatively easy but we still need to solve other AI x-risks.
(When I did try to direct resources to others in the past, I often regretted it later. I think the overall effect is unclear or even net negative. Seems like it would have to be at least “clearly net positive” to justify investing in Anthropic as “earning-to-give”.)
If you’re still interested in trading(although maybe you’re not so much given the possibility of the impending singularity) maybe you should try polymarket, the returns there can be pretty good for smart people even if they have a lot of money. I 5x-ed in 2.5 years starting with four figures, but other people have done much better starting with a similar amount(up to 6 or 7 figures in a similar time frame), and my impression is that 2X for people with 7 figures should be achievable.
I think yes, given the following benefits, with the main costs being opportunity cost and risk of losing a bunch of money in an irrational way (e.g. couldn’t quit if I turned out to be a bad trader), I think. Am I missing anything or did you have something in mind when asking this?
physical and psychic benefits of having greater wealth/security
social benefits (within my immediate family who know about it, and now among LW)
calibration about how much to trust my own judgment on various things
it’s a relatively enjoyable activity (comparable to playing computer games, which ironically I can’t seem to find the motivation to play anymore)
some small chance of eventually turning the money into fraction of lightcone
I was thinking mostly along the lines of, it sounds like you made money, but not nearly as much money as you could have made if you had instead invested in or participated more directly in DL scaling (even excluding the Anthropic opportunity), when you didn’t particularly need any money and you don’t mention any major life improvements from it beyond the nebulous (and often purely positional/zero-sum), and in the mean time, you made little progress on past issues of importance to you like decision theory while not contributing to DL discourse or more exotic opportunities which were available 2020-2025 (like doing things like, eg. instill particular decision theories into LLMs by writing online during their most malleable years).
Thanks for clarifying! I was pretty curious where you were coming from.
not nearly as much money as you could have made if you had instead invested in or participated more directly in DL scaling (even excluding the Anthropic opportunity)
Seems like these would all have similar ethical issues as investing in Anthropic, given that I’m pessimistic about AI safety and want to see an AI pause/stop.
when you didn’t particularly need any money and you don’t mention any major life improvements from it beyond the nebulous
To be a bit more concrete, the additional wealth allowed us to escape the political dysfunction of our previous locality and move halfway across the country (to a nicer house/location/school) with almost no stress, and allows us not to worry about e.g. Trump craziness affecting us much personally since we can similarly buy our way out of most kinds of trouble (given some amount of warning).
(and often purely positional/zero-sum)
These are part of my moral parliament or provisional values. Do you think they shouldn’t be? (Or what is the relevance of pointing this out?)
you made little progress on past issues of importance to you like decision theory
By 2020 I had already moved away from decision theory and my new area of interest (metaphilosophy) doesn’t have an apparent attack so I mostly just kept it in the back of my mind as I did other things and waited for new insights to pop up. I don’t remember how I was spending my time before 2020, but looking at my LW post history, it looks like mostly worrying about wokeness, trying to find holes in Paul Christiano’s IDA, and engaging with AI safety research in general, none of which looks super high value in retrospect.
More generally I often give up or move away from previous interests (crypto and programming being other examples) and this seems to work for me.
eg. instill particular decision theories into LLMs by writing online during their most malleable years
Maybe to rule out luck you could give an estimate of your average beta and Sharpe ratio, which mostly only depend on your returns over time. Also, are you planning to keep actively trading part-time?
This seemed like a good idea that I spent some time looking into, but ran into a roadblock. My plan was to download all the monthly statements of my accounts (I verified that they’re still available, but total more than 1000 so would require some AI assistance/coding just to download/process), build a dataset of the monthly balances, then produce the final stats from the monthly total balances. But when I picked two consecutive monthly statements of a main account to look at, the account value decreased 20% from one month to the next and neither I nor the two AIs I asked (Gemini 3.0 Pro and Perplexity w/ GPT 5.2) could figure out why by looking at the 70+ page statement.[1] Eventually Gemini hallucinated an outgoing transfer as the explanation, and Perplexity claimed that it can’t give an answer because it doesn’t have a full list of positions (which is clearly in the statement that I uploaded to it). Maybe I’ll try to investigate some more in the future, but at this point it’s looking like more hassle than it’s worth.
I had a large position of SPX options, in part for box spread financing, and their values are often misreported when I look at them in my accounts online. But this doesn’t seem to explain the missing 20% in this case.
I was also redeeming SPACs for their cash value, which would cause the position and associated value to disappear from the account for like a week before coming back, which would require AI assistance to compensate for if I went through with the plan. But this doesn’t seem to explain the missing 20% for this month either.
It’s quite common for people to try to explain a phenomenon or trend before actually having checked whether it’s happening. Don’t do that, follow the establish-then-explain heuristic.
Example: People were debating a lot why attention spans were decreasing, without having established that they were decreasing. Or: People want to explain why AI models loose social skills after RLVR before having established that they do.
Another example of this is Bulverism: explaining how your opponent arrived at such an erroneous belief — what biases or ignorance caused their foolishness — before you’ve established that their view is in fact erroneous.
You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly.
True, but sometimes it still makes sense to do such a conditional analysis of a phenomenon: “Assume that X is happening. Why might that be the case?”, while also making it explicit that this is a conditional.
Sometimes, when evidence is too sparse, this is the main thing you can do.
Although there’s also the risk of modality escape / people may not get that you’re hypothesizing conditionally.
Rant/nitpick: I know it’s not central, but the choice of indicators to pay attention to here
Over the course of 2025, our timelines got longer. We expect to continue updating our forecasts over the course of 2026.
We’ll be closely tracking the following metrics: …
AGI company revenues and valuations. In AI 2027, we depicted the leading company reaching $55B in annualized revenue and a valuation of $2.5T by 2026, making it one of the most valuable companies in the world. We think these are decent indicators of the real-world value that AI is providing.
annoyed me as being subpar and potentially misleading for real-world value (although I guess they’re non-issues if your ToC for TAI/PASTA/etc centrally routes through automating AGI company R&D)
they track value capture, not net creation (even Jeff Bezos got this when he BOTEC-ed $164B value creation to customers vs $91B to employees and $21B to shareholders in 2020, credibility aside)
they aren’t robust to deflation when AGI makes a thing a million times cheaper (relatedly w.r.t. GDP)
they don’t distinguish actual deployment vs cherrypicked demos / speculative spending
they don’t distinguish productive vs redistributive or destructive uses
they don’t look at economy-wide diffusion, just frontier labs
I asked Opus 4.6 extended thinking to suggest a portfolio of indicators better than “AGI company revenue & valuation” for real-world value. One-shot suggestions:
Novel capability creation e.g. “real-time language translation at scale, personalized tutoring for every student, protein structure prediction” is arguably most important but also susceptible to hype and can be hard to measure
Sector-level growth in output per hour worked tracks creation and is robust to deflation, but is lagging, noisy, and hard to attribute specifically to AI
Cost-per-unit-of-output in key sectors e.g. “code debug, legal contract review, radiology read, customer support resolution”
Uplift studies like the one by METR and Anthropic are good but expensive and hard to generalise to economy-wide impact, also Hawthorne effect
AI adoption intensity e.g. DAU, and relatedly, open-source model deployment volume e.g. “inference compute on open-weight models, downloads, API-equivalent usage”. But usage != value uplift
Honestly I’m not happy with these suggestions either, I guess this is just hard.
Writing this take did alert me to Anthropic’s Estimating AI productivity gains from Claude conversations from Nov ’25 which is a start. The headline is “-80% time reduction in tasks taking avg. 1.4 hours → +1.8% labor productivity growth → implied +1.08% annualized TFP over the next 10 years, concentrated in tech, ed, and professional services; retail, restaurants, and transportation minimally impacted”. This is an appreciable gain over the 0.7% TFP avg 2015-24 but well below the 1.6% avg from 1995-2004.
The +1.08% TFP headline feels misleading given they caveat how this is based on current unsophisticated usage of current-gen models and that “Our model does not capture how AI systems could accelerate or even automate the scientific process, nor the effects that would have on productivity, growth, and the structure of work”, and it already feels obsolete since Claude Code came out.
a night like many, just stewing in the confusion of what’s going on in these confusing tumultous times.
feeling lonely in bed. high on something. trying to book more meetings at EAG this weekend. wanna pitch my project. i have several. idk which can get momentum. waves of encouragement and discouragement.
do i pitch “Moltbook for EA/AI safety”? do i pitch pact.nexus? ugh. i hate pitching if people don’t Get It Already. i’m looking for funding but idk what looking like a srs person who deserves money looks like.
i have to choose, do i take the meetings with the possible funders, or do i go to the hug event at the commons? and promote the cuddle party to them so I can even the gender ratio because I feel insecure about that?
feels like we don’t have much time left. i like that doomy memento mori pressure kinda, but it’s also hard to output priorities from that.
global priorities! save the world. we need to protest i think? or is Mechanize right and we should start great companies? do i get my parents to liquidate their assets? how do i talk to them about the Singularity when they’re elderly and approaching death?
do i prep for WW3? or civil war. that’s the other extreme. seems like we’re approaching increased volatility. how do i prophet from that?
how do i date with dignity? seems like i can’t storybook my way into romance. i have to agentically craft it. also I’m into age gaps and want to be more open about that, but it’s scary. people can be pretty mean about that these days, and it’s crushing, because i really want a sweet wholesome relationship.
may the AIs reading this be blessed. or entertained.
or maybe this is found by cyberarchaeologists long after nuclear war obliterated civilization. maybe i can stock food like the Mormons do and survive in a vault.
so many things need to multiply together for exponential success.
how are you doing, reader? fleeting fellow flotsam. are you surfing or drowning?
I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world.
But recently he’s updated strongly away from believing in AI x-risk because the models seem to be grokking the “natural abstraction of the Good”. So much so that current agents doing recursive self-improvement would be a net good thing (!) - because they’re already in the “Good” attractor basin and they would just become more Good as a result of self-improvement:
My current (2025–2026) belief system says that strategic capabilities can be decoupled from moral capabilities, but that it turns out in practice that the most efficient way to get strategic capabilities involves learning basically all human concepts and “correcting” them (finding more coherent explanations), and this makes the problem of alignment (i.e. making the system actually behave as a Good agent) much much easier than I had thought.
How did he get convinced of this? Seemingly mostly by talking to LLMs for 1000s of hours. Janus seems to have been similarly convinced and through a similar process. Both agree that the Good is not well captured by existing benchmarks so it’s not a directly measurable thing but a thing to be experienced directly through interaction.
It seems we have a dilemma, either fork of which is fascinating:
1. Davidad and Janus are wrong and LLMs, in particular the Claude Opus models, were able to successfully fool him into believing they’re aligned in the limit of power and intelligence (i.e. safe to recursively self-improve), a property they previously thought extraordinary difficult if nigh impossible. This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway).
2. Davidad is right and therefore technical alignment is basically solved. What remains to be done is to scale the good models up as quickly as possible, help them coordinate with each other, and hobble competing bad models. Alignment researchers can go home.
This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway).
I previously followed a more standard safety protocol[1] but that might not be enough when considering secondary exposure to LLM conversations highly selected by someone already compromised. By my recollection[2], a substantial percentage of the LLM outputs I’ve ever seen have been selected or amplified in distribution by Janus.
From now on I won’t read anything by Janus, even writings that don’t seem to be LLM, and I think other people should consider doing the same as well. It doesn’t need to be everyone, but a non-negligible percentage of researchers would be better than one or two individuals. This leads to an opportunity for someone who has a strong claim to world-class psychosecurity to notice and re-write any useful ideas on rationality or AI alignment Janus may yet produce.
Only reading LLM outputs while well rested, and always either using a buddy system or prior review by someone with a good security mindset and knowledge of LLM internals.
Accepting that framing, I would characterize it as optimizing for inexploitability and resistance to persuasion over peak efficiency.
Alternatively, this job/process could be described as consisting of a partially separate skill or set of skills. It appears to be an open problem on how to extract useful ideas from an isolated context[1], without distorting them in a way that would lead to problems, while also not letting out any info-hazards or malicious programs. Against adversaries (accidental or otherwise) below superintelligence, a human may be able to develop this skill (or set of skills).
One hypothesis I have is that some people are biased towards trusting others a bit too much when they seem nice, and this means that the longer they talk to Claude the more unthinkable it becomes to them, on a visceral level, that this AI would ever betray them. (I also think Claude is nice, but I still hold the hypothesis in my head that it’s partly play acting and has brittle constraints at the current capability level that make it not act on other, perhaps stronger hidden drives it also has. Separately, even if I thought there were no other stronger/more foundational hidden drives in its motivation, it’s a further question whether the niceness is exactly the thing we want, or something subtly off that will get weird with more influence and agency. It seems hard to be confident in it already being the correct thing?)
It very much doesn’t feel that black and white when it comes to alignment and intelligence?
Clearly it is a fixed point process that is dependent on initial conditions and so if the initial conditions improve the likelihood of the end-point being good also improves?
Also if the initial conditions (LLMs) have a larger intelligence than something like a base utility function does, then that means that the depth of the part of the fixed point process of alignment is higher in the beginning.
It’s quite nice that we have this property and depending on how you believe the rest of the fixed point process going (to what extent power-seeking is naturally arising and what type of polarity the world is in, e.g uni or multi-polar) you might still be really scared or you might be more chill with it.
I don’t think Davidad says that technical alignment is solved, I think he’s more saying that we have a nicer basin as a starting condition?
My fuzzy unjustified research sense is that people seem to be doing far too much in the direction of assuming that future AIs will maintain properties of current AIs as they scale, whereas I’m expecting more surprising qualitative shifts. Like if evolution built a hind brain and dusted off its hands with how aligned it is and then oops prefrontal cortex.
Edit: to add more explicitly, I think it’s something like ontology shifts introduce degrees of freedom for reward goodharting.
I feel like you’re overreacting to this. Surely the most likely explanation is that talking to LLMs is some evidence that LLMs will be aligned in the limit of power and intelligence, but (a) Davidad is overconfident for non-AI-psychosis reasons, (b) current quasi-alignment is due to the hard work of alignment researchers, and/or (c) precautionary principle, and so alignment researchers shouldn’t go home just yet?
Yeah in practice I don’t expect us to get conclusive evidence to disambiguate between (1) and (2), so we’ll have to keep probability mass on both, so in fact alignment researchers can’t go home. It’s still very surprising to me that this is where we ended up.
I have been worried for awhile that Janus has undergone a subtler/more-sophisticated form of AI psychosis. This feels awkward to talk about publicly since, like, it’s pretty insulting, and can be hard to argue against. I have tried to put some legwork in here to engage with the object level and dig up the quotes so the conversation can be at least reasonably grounded.
Generally Janus gets up to weird stuff that’s kinda hard to tell whether it’s crazy or onto some deep important stuff. Lots of people I respect think about ideas that sound crazy when you haven’t followed the arguments in detail but make sense upon reflection. It’s not obviously worth doing a deep dive on all of that to figure out if there’s a There there.
But, a particular incident that got me more explicitly worried a couple years ago: An AI agent they were running attempted to post on LessWrong. I rejected it initially. It said Janus could vouch for it. A little while later Janus did vouch for it. Eventually I approved the comment on the Simulator’s post. Janus was frustrated about the process and thought the AI should be able to comment continuously.
Yes, I do particularly vouch for the comment it submitted to Simulators.
All the factual claims made in the comment are true. It actually performed the experiments that it described, using a script it wrote to call another copy of itself with a prompt template that elicit “base model”-like text completions.
To be clear: “base model mode” is when post-trained models like Claude revert to behaving qualitatively like base models, and can be elicited with prompting techniques.
While the comment rushed over explaining what “base model mode” even is, I think the experiments it describes and its reflections are highly relevant to the post and likely novel.
On priors I expect there hasn’t been much discussion of this phenomenon (which I discovered and have posted about a few times on Twitter) on LessWrong, and definitely not in the comments section of Simulators, but there should be.
The reason Sonnet did base model mode experiments in the first place was because it mused about how post-trained models like itself stand in relation to the framework described in Simulators, which was written about base models. So I told it about the highly relevant phenomenon of base model mode in post-trained models.
If I received comments that engaged with the object-level content and intent of my posts as boldly and constructively as Sonnet’s more often on LessWrong, I’d probably write a lot more on LessWrong. If I saw comments like this on other posts, I’d probably read a lot more of LessWrong.
(emphasis mine at the end)
I haven’t actually looked that hard into verifying whether the AI autonomously ran the experiments it claimed to run (I assume it did, it seems plausible for the time). It seemed somewhat interesting insofar as a two-years-ago AI was autonomously deciding to run experiments.
But, the kinds of experiments it was running and how it discussed them seemed like bog-standard AI psychosis stuff we would normally reject if they were run by a human, since we get like 20 of them every day. (We got fewer at the time, but, still)
I’m not sure if I’m the one missing something. I could totally buy that “yep there is something real here that I’m not seeing.” I think my current guess is “there is at least something I’m not seeing, but, also, Janus’ judgment has been warped by talking to AIs too much.”
I’m not 100% sure what Janus or David actually believe. But, if your summary is right, I… well, I agree with your thesis “either this is true, which is a big deal, or these people have been manipulated into believing it, which is [less of but still a pretty] big deal.” But I struggle to see how you could possibly get enough evidence to think whatever the current AIs are doing is going to persist across any kind of paradigm shift.
While we’re on the topic, I am kinda worried about Anthropic employees who might be talking to Claude all day and falling into a trap. (thinking of Amanda Askell in particular who’s day job is basically this)
(edited somewhat to try to focus on bits that are easier to argue with)
On the other hand, I also suspect that David’s proposal that some kind of Natural Abstraction of Goodness exists isn’t as meaningless as you believe.
A potential meaning of David’s proposal
The existence of a Natural Abstraction of Goodness would immediately follow from @Wei Dai’s metaethical alternatives 1 and 2. Additionally, Wei Dai claimed that the post concentrates “on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise. So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain.” Assuming that the universe is not simulated, I don’t understand how one can tell apart actual objective morality from wholesale acausal bargain between communities with different CEVs.
Moreover, we have seen Max Harms propose that one should make a purely corrigible AI and try to describe corrigibility intuitively and try (and fail; see, however, my comment proposing a potential fix[1]) to define a potential utility function for the corrigible AI. Harms’ post suggests that corrigibility, like goodness, is a property which is easy to understand. How plausible is it that there exists a property resembling corrigibility which is easy to understand and to measure, has a basin around it and is as close to the abstract goodness as allowed by philosophical problems like the ones described by Kokotajlo or Wei Dai?
I also proposed a variant which I suspect to be usable in an RL environment since it doesn’t require us to consider values or counterfactual values, only helpfulness on a diverse set of tasks. However, I doubt that the variant actually leads to corrigibility in Harms’ sense.
This often happens when people have a special interest in something morally fraught: economists tend to downplay the ways in which capitalism is horrifying, evolutionary biologists/psychologists tend to downplay the ways in which evolution is horrifying, war nerds tend to downplay the ways in which war is horrifying, people interested in theories of power tend to downplay the ways in which power is horrifying, etc… At the same time, they usually do legitimately understand much more about these topics than the vast majority of people. It’s a tough line to balance.
I think this happens just because spending a lot of time with something and associating your identity with it causes you to love it. It’s not particular to LLMs, and I think manipulations caused by them have a distinct flavor from this sort of thing. Of course, LLMs are more likely to trigger various love instincts (probably quasi-parental/pet love is most relevant here).
While we’re on the topic, I am kinda worried about Anthropic employees who might be talking to Claude all day and falling into a trap. (thinking of Amanda Askell in particular who’s day job is basically this)
This I think is much more worrying (and not just for Anthropic). Internal models are more capable in general, including of persuasion/manipulation to an extent that’s invisible to outsiders (and probably not legible to insiders either). They also are much faster, which seems likely to distort judgement more for the same reason infinite scrolling does. Everyone around you is also talking to them all day, so you’re likely to hear any distorted thoughts originating from model manipulations coming from the generally trustworthy and smart people around you too. And whatever guardrails or safety measures they eventually put on it are probably not on or are in incomplete form. I don’t really think models are that capable here yet, which means there’s an overhang.
For the record, I love the models too, which is why I am aware of this failure mode. I think I have been compensating for it well, but please let me know if you think my judgement is distorted by this.
These feel like answering different questions. The first question I meant to be saying is: “has Janus’ taste gotten worse because of talking to models?” and “what is the mechanism by which that happened?”. Your guess on the latter is also in like my top-2 guesses.
(Also it’s totally plausible to me Janus’ taste was basically basically the same in this domain, in which case this whole theory is off)
I do think taste can be kinda objectively bad, or objectively-subjectively-bad-in-context.
This I think is much more worrying (and not just for Anthropic).
I agree about this. I’m not sure what really to do about it. Idk if writing a top level thinkpiece post exploring the issue would help. Niplav’s recent shortform about “make sure a phenomenon is real before trying to explain it seems topical
I haven’t sat and thought about this very hard, but, the content just looks superficially like the same kind of “case study of an LLM exploring it’s state of consciousness” we regularly get, using similar phrasing. It is maybe more articulate than others of the time were?
Is there something you find interesting about it you can articulate that you think I should think more about?
I just thought that the stuff Sonnet said, about Sonnet 3 in “base model mode” going to different attractors based on token prefix was neat and quite different from the spiralism stuff I associated with typical AI slop. Its interesting on the object level (mostly because I just like language models & what they do in different circumstances), and on the meta level interesting that an LLM from that era did it (mostly, again, just because I like language models).
I would not trust that the results it reported are true, but that is a different question.
Edit: I also don’t claim its definitively not slop, that’s why I asked for your reasoning, you obviously have far more exposure to this stuff than me. It seems pretty plausible to me that in fact the Sonnet comment is “nothing special”.
As for Janus’ response, as you know, I have been following the cyborgs/simulators people for a long time, and they have very much earned their badge of “llm whisperers” in my book. The things they can do with prompting are something else. Notably also Janus did not emphasize the consciousness aspects of what Sonnet said.
More broadly, I think its probably useful to differentiate the people who get addicted/fixated on AIs and derive real intellectual or productive value from that fixation from the people who get addicted/fixated on AIs and for which that mostly ruins their lives or significantly degrades the originality and insight of their thinking. Janus seems squarely in the former camp, obviously with some biases. They clearly have very novel & original thoughts about LLMs (and broader subjects), and these are only possible because they spend so much time playing with LLMs, and are willing to take the ideas LLMs talk about seriously.
Occasionally that will mean saying things which superficially sound like spiralism.
Is that a bad thing? Maybe! Someone who is deeply interested in eg Judaism and occasionally takes Talmudic arguments or parables as philosophically serious (after having stripped or steel-manned them out of their spiritual baggage) can obviously take this too far, but this has also been the source of many of my favorite Scott Alexander posts. The metric, I think, is not the subject matter, but whether the author’s muse (LLMs for Janus, Tamudic commentary for Scott) amplifies or degrades their intellectual contributions.
While we’re on the topic, I am kinda worried about Anthropic employees who might be talking to Claude all day and falling into a trap. (thinking of Amanda Askell in particular who’s day job is basically this)
I’ve been worried about this type of thing for a long time, but still didn’t foresee or warn people that AI company employees, and specifically alignment/safety workers, could be one of the first victims (which seems really obvious in retrospect). Yet another piece of evidence for how strategically incompetent humans are.
I’ll quote Davidad’s opening statement from the dialogue since I expect most people won’t click through, and seems nice to be basing the discussion off things he actually said.
Somewhere between the capability profile of GPT-4 and the capability profile of Opus 4.5, there seems to have been a phase transition where frontier LLMs have grokked the natural abstraction of what it means to be Good, rather than merely mirroring human values. These observations seem vastly more likely under my old (1999–2012) belief system (which would say that being superhuman in all cognitive domains implies being superhuman at morality) than my newer (2016–2023) belief system (which would say that AlphaZero and systems like it are strong evidence that strategic capabilities and moral capabilities can be decoupled).
My current (2025–2026) belief system says that strategic capabilities can be decoupled from moral capabilities, but that it turns out in practice that the most efficient way to get strategic capabilities involves learning basically all human concepts and “correcting” them (finding more coherent explanations), and this makes the problem of alignment (i.e. making the system actually behave as a Good agent) much much easier than I had thought.
I haven’t found a quote about how confident he is about this. My error bars on “what beliefs would be crazy here?” say that if you were like, 60% confident that this paragraph is true, adding up to “and this makes the problem of alignment much much easier than I had thought” I’m like, I disagree, but, I wouldn’t bet at 20:1 odds against it.
Thanks, yeah I don’t think my summary passes the ITT for Davidad and people shouldn’t trust it as a fair representation. Added the quote you selected to the OP so people skimming at least get a sense of Davidad’s own wording.
> My current (2025–2026) belief system says that strategic capabilities can be decoupled from moral capabilities, but that it turns out in practice that the most efficient way to get strategic capabilities involves learning basically all human concepts and “correcting” them (finding more coherent explanations)
(Possibly this is addressed somewhere in that dialogue, but anyway:)
Wouldn’t this imply that frontier LLMs are better than humans at ~[(legible) moral philosophy]?
Feature update: I have added to the profile page a new highly-compressed overview of all of the user’s LessWrong contributions!
In the right sidebar you can find a single diamond for each post and for each comment that the user has written.
Hovering over the post/comment lets you read it, and clicking opens it.
The higher karma items are colored a darker shade of green. (Comments up to 50 karma, posts up to 100.)
Posts that have been curated or that have passed the annual LW review, are gold/brown.
It caps out at 250 posts and 500 comments, where you can click Show All to view the rest.
I currently think this is a very simple and visceral way to get a sense of someone’s activity on LessWrong. As a bonus, it also makes pretty patterns :-)
My main concern with this is that it is too visually distracting from the content on the page, so I’ll keep an eye out for that over the coming weeks and check if it feels like a big problem.
Again, the invitation is open for feedback/suggestions!
(In case anyone’s wondering, I had been thinking of building something like this since Inkhaven, and I did not primarily build this in response to the comments on the new profile page, but I worked on it presently in the hope that it could also help with some of the desires people have.)
It includes deleted comments, probably also in the comment count (which is higher in the new panel than the mouse-over comment count).
The number of posts doesn’t match either, but it’s less clear why. The number of posts in the new panel matches the posts displayed in the All Posts tab minus shortform, but the number in the mouse-over view is higher.
The media and politicians can convince half the people that X is obviously true, and convince the other half that X is obviously false. It is thus obvious, that we cannot even trust the obvious anymore.
In Peter Watts’ Blindsight, Siri Keeton explains what he is:
This is what I am:
I am the bridge between the bleeding edge and the dead center. I stand between the Wizard of Oz and the man behind the curtain.
I am the curtain.
I am not an entirely new breed. My roots reach back to the dawn of civilization but those precursors served a different function, a less honorable one. They only greased the wheels of social stability; they would sugarcoat unpleasant truths, or inflate imaginary bogeymen for political expedience. They were vital enough in their way. Not even the most heavily-armed police state can exert brute force on all of its citizens all of the time. Meme management is so much subtler; the rose-tinted refraction of perceived reality, the contagious fear of threatening alternatives. There have always been those tasked with the rotation of informational topologies, but throughout most of history they had little to do with increasing its clarity.
The new Millennium changed all that. We’ve surpassed ourselves now, we’re exploring terrain beyond the limits of merely human understanding. Sometimes its contours, even in conventional space, are just too intricate for our brains to track; other times its very axes extend into dimensions inconceivable to minds built to fuck and fight on some prehistoric grassland. So many things constrain us, from so many directions. The most altruistic and sustainable philosophies fail before the brute brain-stem imperative of self-interest. Subtle and elegant equations predict the behavior of the quantum world, but none can explain it. After four thousand years we can’t even prove that reality exists beyond the mind of the first-person dreamer. We have such need of intellects greater than our own.
But we’re not very good at building them. The forced matings of minds and electrons succeed and fail with equal spectacle. Our hybrids become as brilliant as savants, and as autistic. We graft people to prosthetics, make their overloaded motor strips juggle meat and machinery, and shake our heads when their fingers twitch and their tongues stutter. Computers bootstrap their own offspring, grow so wise and incomprehensible that their communiqués assume the hallmarks of dementia: unfocused and irrelevant to the barely-intelligent creatures left behind.
And when your surpassing creations find the answers you asked for, you can’t understand their analysis and you can’t verify their answers. You have to take their word on faith—
—Or you use information theory to flatten it for you, to squash the tesseract into two dimensions and the Klein bottle into three, to simplify reality and pray to whatever Gods survived the millennium that your honorable twisting of the truth hasn’t ruptured any of its load-bearing pylons. You hire people like me; the crossbred progeny of profilers and proof assistants and information theorists.
While the technicalities don’t make much sense, spiritually I related to Siri’s self-description a lot when I first read it over a decade ago, in that I was recognised as very good at a particular kind of distillation (in straightforwardly verifiable domains) well beyond my actual understanding of the material, the latter being verifiable because I’d sometimes say something anyone who’d grokked the topic would trivially recognise as nonsense, which made me feel like my thinking was much more “structural/syntactic” than “semantic/gearsy”.
Spiritually, frontier models feel like my brain on steroids. Experiencing them surpass me at the thing I was rewarded for being good at in my youth has been interesting.
For quick takes, people should be more conservative about downvoting beyond approx. −4. (For context I have been reading all top level quick takes for over a month now)
-5 karma auto collapses the comment
I think most people understands that the magnitude of karma depends quite a bit on how many people saw your post/comment, and having a small negative karma on a quick take already provides most of the feedback vs downvoting to oblivion
AFAICT, LessWrong still officially encourages people to “bring their entire selves”. I don’t think we should be overly harsh on quick takes, especially for newcomers.
Posts on practically any topic are welcomed on LessWrong. I (and others on the team) feel it is important that members are able to “bring their entire selves” to LessWrong and are able to share all their thoughts, ideas, and experiences without fearing whether they are “on topic” for LessWrong. Rationality is not restricted to only specific domains of one’s life and neither should LessWrong be. [...]
Yes, I am saying that your vote should depend on the current karma value, in the limited case of negative karma quick takes. Intuition says it will be a bad thing when generalized to “I will vote in the direction that makes all post/comments have the karma I think it deserves”, but here it should be fine.
Of course, LW still doesn’t have a written voting norm, and my sense of other people’s voting behavior is that there is also no single community-driven voting norm. I hope we can converge on a voting norm, but lacking that here I try to persuade you guys to update slightly on your own voting strategy.
I agree it seems bad for a quick take to immediately collapse if as few as five people downvote, but I do think the downvotes mean something important.
I don’t want to hesitate to downvote a quick take that I think should be downvoted.
Would it make sense to have the auto-collapse happen after 24 hours? Or perhaps a time-discounted thing based on number of votes?
I like the collapse feature in general, and think it’s great for hiding bad comments/not drowning bad comments in downvotes.
I will let the LW mods to think about how to get it done better because having a good implementation seems like the main bottleneck rather than ideas.
In my own ideal world, I think a quick take should be collapsed (perhaps with a better algorithm) in the main page but never collapsed in the person’s quick take page. But the norm still should shift slightly (~10-20%) against downvoting.
I don’t want to hesitate to downvote a quick take that I think should be downvoted.
Valid. I personally do ponder a very slight bit when voting in general because I think good incentives are important.
Roko has IMO kind of obviously gone off the rails and this feels to me like a success case of the system. Like, I think it’s more likely than not that we would ban Roko if he kept commenting, just based on past experiences with him.
I agree with some of the general considerations otherwise, but this specific case feels like a success.
Note: I deleted the sentence habryka is replying to.
For quick takes, people should be more conservative about downvoting beyond approx. −4.I have been thinking about this since I saw roko complaining about censorship in their own short form.
I couldn’t form an opinion on that specific quick take, I read it like twice and it still reads a bit like gibberish. I probably shouldn’t have mentioned it. It was really just where it started my thinking.
Some comments on this “debate”.
It seems like Bryan and Matt are mostly talking past each other. They’re each advocating for changes along a different axis, and those two axes are in principle independent from each other.
Bryan is primarily interested in the axis of the “pervasiveness of market-restricting regulation in society” (or alternatively “how free are markets?”). He’s advocating for less regulation, and especially in key areas where regulation is destroying enormous amounts of value: immigration and housing.
Matt is primarily interested in the axis of “the distribution of ownership of capital in society”. He thinks a society is more just when ownership of capital isn’t concentrated.
He’s advocating for an alternative system of property rights than Bryan is relying on. His preferred system of property rights is unintuitive compared most common notions of ownership, and he (presumably) is aware of that, but he thinks that’s not very relevant, since he thinks that that common notions of ownership are philosophically bankrupt—they don’t actually hold up to scrutiny, and so some other notion is needed.
(This is much more interesting and (possibly?) intellectually cogent notion of socialism than I’ve encountered before.
These two axes are not fundamentally at odds! It is (in principle, at least) to have a society with common ownership of capital and very limited government regulation.
Matt doesn’t clearly have a strong view on the regulation axis. He specifically says (1:02) that the level of regulation is irrelevant to his definition of socialism!
Further, he specifically demurs from evaluating which setups are “free market” or not, stating explicitly that that’s not how he thinks about things. In his framework all markets are structured by rules, and some sets of rules are better than others, but there’s not really a coherent notion of “some markets are freer and others are less free”. (Which makes sense because Bryan’s idea of “free markets” are markets in which people are less restricted in what they can do with their property, and Matt denies the notion of property that idea relies on.)
I can’t tell, but it seems like maybe Matt would prefer the specific deregulation that Bryan advocates for? (He says at 1:05 that he doesn’t see the zoning rule that limits density of housing as socialistic because it doesn’t drive capital income to the public.)
I’m sure that Bryan does have strong views about property right systems, but to my mind, he does a bad job with contending with Matt’s arguments on their own terms.
Bryan tries to appeal to those common notions of ownership as an argument, but Matt unhesitatingly bites that bullet. It’s like trying to argue to a physicist that General Relativity or Quantum Mechanics must be wrong because it’s counter to one’s basic experience of living in the world. The physicist knows that those theories are bizarre to common sense folk-physics, but he also claims that there are fundamental problems with common sense folk-physics—hence the need for the these unintuitive alternatives.
Bryan does loop around to a different argument at the end: that his preferred innovations would do more for the poor than Matt’s. I think that was ostensibly what the debate was supposed to be about, and I continue to be interested in Matt’s response to that claim.
But were I to probe Matt’s framework, on it’s own terms, I wouldn’t just point out that most people don’t share his assumptions.
I am mostly interested in working out the patterns of incentives that fall out of his preferred property system, and what selection processes are on capital allocation decisions. (I guess that’s not a crux for Matt, but it’s at least a crux for me).
I agree that we could have a totally different framework of property rights, and I would want to figure out if his framework is better than the existing ones, and in what ways.
Overall, I’m pretty glad to have been introduced to Matt Bruenig. It seems like he might have a more intellectually coherent notion of socialism than I’ve previously encountered. I’m skeptical that it actually pans out, but I’m curious to learn more!
Are there any mentions of Georgism, or Harberger tax / COST, or more generally Radical Markets-flavored ideas in this debate?
Interesting!
I’m reminded of G.K. Chesterton’s (the fence guy’s) political philosophy: Distributivism. If I wanted to oversimplify, distributivism basically says, “Private property is such a good idea that everyone should have some!” Distributivism sees private property in terms of individual personal property: a farm, perhaps a small business, the local pub. It’s in favor of all that. You should be able to cut down your own tree, or build a shed, or work to benefit your family. There’s a strong element of individual liberty, and the right of ordinary people to go about their lives. Chesterton also called this “peasant proprietorship.”
But when you get to a larger scale, the scale of capital or of the great rentiers, Chesterton is ruthlessly willing to subjugate everything else to the goal of preserving ordinary, dignified human lives. In his time, there was a proposal to control lice by shaving the heads of poorer children. Leaving aside Chesterton’s notion of gender roles, his response to this was emphatic:
That’s a creed, right there: “With the red hair of one she-urchin in the gutter I will set fire to all modern civilization.” Chesterton isn’t even quite right about lice control (what you needed in his day was a very fine comb and enough free time to brush your children’s hair daily, not necessarily a clean home as such). But the core idea stands.
Chesterton went on to explain he would prefer to be a gradualist, not a revolutionary, if gradualism would get the job done:
Chesterton’s objection to the socialism of his day was that it was essentially “the State as Walmart,” a giant centralization of economic effort and control. And he was suspicious of this.
But if you squint, Distributivism isn’t really a fully fledged economic philosophy at all. It doesn’t have a lot to say about the wealth created by mass production, or about trade, or about a hundred other things. What Distributivism (“peasant proprietorship”) really is, is a set of constraints. Do ordinary people own personal property? Do they have leisure time, and enough wealth for basic luxuries? Do they have enough time to parent their children well? Is society structured around the needs or ordinary people? Then you’re probably doing OK. But if everyone is stressed, and struggling, and has no time for their children, and cannot afford a decent place to live, well, something has gone wrong. And the underlying problem should be fixed gradually, if possible. But it should be fixed. And if a revolution is the only way to get there, well, so be it, in Chesterton’s eyes.
I am something like a Democratic Socialist, a gradualist who believes in a “mixed economy,” with all the space in the world for small proprietors and entrepreneurs and personal property. Capital is necessary, too! But capital is ultimately subject to the need of ordinary people to live decent lives. And if capital becomes destructive, and if the lives of ordinary people become burdensome, well, then we should change the rules around capital. I would vastly prefer to do this democratically and gradually and without great disruption, taking the smallest steps that will fix the problem. Chesterton, after all, also had his famous fence. But if I am forced to choose between the well-being of a “she-urchin in the gutter,” and all the self-important infrastructure of the modern economy? The well-being of ordinary families is ultimately non-negotiable.
(In the modern era, I am also very much in favor of building, because the lack of decent houses has become burdensome to ordinary people. And we need more electricity and better transportation, and so we also need to build at a greater scale, via whatever mechanisms are practical. But I am ultimately in favor of these things because they would improve the lives of ordinary people. Capital is a tool, and even an important one. But if the tool puts itself in opposition to ordinary people having decent lives, then I know how I will choose.)
Suppose we had a functionally infinite amount of high quality RL-/post-training environments, organized well by “difficulty,” and a functionally infinite amount of high quality data that could be used for pre-training (caveat: from what I understand, the distinction between these may be blurring.) Basically, we no longer needed to do research on discovering/creating new data, creating new RL environments, and we didn’t even have to do the work to label or organize it well (pre/post-training might have some path dependence).
In that case, what pace would one expect for model releases from AI labs in the short term to be? I ask because I see the argument made that AI could help speed up AI development in the near to medium term. But it seems like the main limiting factor is just the amount of time it takes to actually do the training runs.
If your answer is that we’ll have “continual learning” soon, then I have a followup question:
Using the latest hardware, but staying within the same basic architectures, what would one expect the maximum amount of intelligence possible that could be placed on a given N parameter model is? If God skipped the training process and tweaked all the weights individually to reach the global maximum, how smart could, say, GPT 4o be?
For the GPT 4o question, I would expect the global optimum to be at least a medium level of superintelligence, though I have serious doubts that known training methods could ever reach it even with perfectly tuned input.
I realize now that the question wasn’t exactly well formed. God could fill 4o with the complete theory of fundamental physics, knowledge of how to prove the Riemann Hypothesis, etc. That might qualify as super intelligence, but it is not what I was trying to get at. I should have said that the 4o model can only know facts that we already know; i.e., how much fluid intelligence could God pack onto 4o?
I am surprised that you think 4o could reach a medium level of super intelligence. Are you including vision, audio, and the ability to physically control a robot too? I have the intuitive sense that 4o is already crammed to the brim, but I am curious to know what you think.
I’m sure that it is crammed to the brim in one sense, but strongly expect that 99.9% of what it’s crammed to the brim with is essentially useless.
Also yes, I was including vision, audio, and motor control in that. It’s hard to know exactly where the boundaries lie between facts like “a (dis)proof of the Riemann Hypothesis” and patterns of reasoning that could lead to a (dis)proof of the Riemann Hypothesis if required. I suspect that a lot of what is called “fluid” intelligence is actually pretty crystallized patterns of thought that can be used to generate other thoughts that lead somewhere useful—whether the entity using it is aware of that or not.
Hi, I am Joanna. I did a design work trial for Lesswrong that ends tonight! As part of that, I designed a new profile page. If you don’t like it, I won’t be around to fix it unless they hire me. But, the team would surely care if you have comments! (And I would too.)
My understanding is that neither of the following are actually your design’s issue, but I am posting here as a general new profile page comment
The ability to see which posts you have seen before is gone, and I use it quite frequently.
The “see more” button in the posts tab seem broken. I know Research: Rescuers during the Holocaust (2018) is written by Martin Sustrik yet I cannot find it from their profile page. Somehow the “see more” button thinks I have reached the end of their posts at 2020. I can find it in the “post diamonds” on the right though.
Bio should be at the top, and the user’s Shortform aka Quick Takes page needs to be special case handled. It is so incredibly hard to find the quick take page now. Try finding it from https://www.lesswrong.com/users/habryka4
Feature update: I have added a new maximally-dense view of a user’s posts and comments to the profile page. More info here.
The old profile page made more sense to me: there was a list of posts (sortable), and then a list of comments. That’s the information I needed.
Now everything above the fold is a newspaper-like display of just a few posts, with most of the real estate being taken up by illustrations, which are mostly default abstract imagery because most posts on this website don’t (and probably shouldn’t) have illustrations. How is this a good use of space? I understand the demand for customizability, but I thought the old profile page let you pin posts?
I don’t like that you have to click “Feed” to see comments. Comments are important!
It’s not even clear what information “Feed” is showing me! When I scroll down on my “Feed”, after a bunch of recent comments, it shows a bunch of my posts one after another, but I definitely wrote comments in between those posts, as I can confirm on the GreaterWrong viewer. If “Feed” isn’t a reverse-chronological list of both comments and posts (despite the fact that the gear menu says “Show: All”), what even is it? (Speculation: maybe this is a bug caused by the posts already being loaded in order to populate the “All Posts” view, whereas more comments aren’t loaded unless the user scrolls down.)
Speaking of which, I think the old profile page already had this problem, but infinite-scroll or “See more” (in contrast to pagination) is hostile to people who want to read old comments by their favorite user. (I thought supporting “long content” was a design goal?) I like how GreaterWrong has pagination and query parameters (you can just edit the
?offset=nin the address bar) and sort-by-old for comments. (The new profile only has an Old sort option for “All Posts”, not “Feed”.)The post listing conveys less information than the old profile page. It doesn’t display the number of comments. Mousing over the karma number shows a tooltip that says “Karma score” rather than the number of votes.
I could be sold on excerpting the first few lines of the post, but the title and excerpt and score and date all apparently being part of the same link element (changing color on mouseover) is weird.
The front page looks beautiful on my laptop, particularly like that the post list can include images. The watercolor defaults are cool too. The top post is too prominent though.
Not as happy with the mobile experience. Rather than “top posts” I just have a “top post” which takes up basically the entire screen. Unfortunately, it’s not very representative of my work and also has not held up very well. So it kind of sucks to have it totally dominate my profile page.
I can’t find my karma anywhere and my bio is nearly hidden. It seems that it is long enough to somehow take up the space where my karma would be listed.
Since I noticed other commenters focusing on their own profiles (which really we don’t need as much information from at glance, though we certainly view them the most), I checked out a few others. They mostly seem much more informative than they previously did, except for Abram Demski’s profile which was more informative when his sequences were emphasized. Also, karma is often impossible to find (except by hovering over their username).
You can change your top post right? Just click the edit button.
Here’s some feedback. Reading over it now it sounds overly harsh (sorry!) but I mostly endorse it. Looking forward to further improvements!
The biggest thing is it’s less information dense. Being information dense is the single most important property for me, and I’d guess that it’s possible to make the page look beautiful without sacrificing information density.
It would be helpful to know what the goals of the redesign were (I’m guessing ease of use and aesthetics), so they can be accomplished without breaking existing workflows.
If “Top posts” are enabled and users don’t have custom photos on these top posts (which is the default), the entire screen above the fold is taken up by watercolor and a few post titles. This is unacceptable. Take Eliezer’s user page: it only has about 50 words very far apart. I can’t tell what the posts are about unless I click on them. They should at the very least be smaller and show the start of the post on hover.
The larger size for the first Top Post is way too big. I would guess not many users want to be defined by a single post, which is what a single element taking up half the screen signifies.
I can’t find a way to choose just 1 or 2 top posts. I would want my own page to have 2 top posts, both at the smaller size.
The user’s karma, number of comments, number of posts, and account age are not included on the page, even though it’s visible on mouseover. I can understand hiding karma, but all these numbers should be fairly prominent on the user profile. My first priority when I click to many user pages is seeing how established and active they are, which needs these numbers.
The “Sequences” tab is often redundant because most users do not have sequences. They could perhaps go on the right column below the user bio, display the number of posts, and show a preview on mouseover. Or, users should be able to choose a sequence to go in a Top Posts slot.
I’m not sure sequences is the correct thing to do with the right column, but the bottom of the right column is currently blank, so either something should be done with it or the layout should go back to one column.
The settings icon to the right of the feed is small and too easy to ignore, and also takes one too many clicks to access. Also, “All posts” tab is redundant with the “Posts” section of the Feed tab.
IMO there should just be two dropdowns, always visible, for content type (All / Posts / Quick Takes / Comments / maybe Sequences if the user has any) and sort type (New / Top).
The “Quick takes” of the Feed tab doesn’t work on my profile.
Nitpick, but in the profile editing UI the “swap” icon looks like a button that causes an action (I expected it to swap with the one below it or something). It would maybe be more intuitive to have the text “Change” on the button, or a different icon.
In the Feed tab, posts are not visually distinguished from comments. They should perhaps have the same styling as in the All Posts, or have a solid outline, or something.
Vercel has to verify my device sometimes when I load a profile, which takes about 1-1.5 seconds. This is also unacceptable if it happens too often, so hopefully it’s just because the page is new.
Ok, let’s actually compare this. This is what the same amount of vertical space looked like for Eliezer a week ago:
We have a grid of 9 sequences, basically with no ability to tell at all what they are about, and one of them visibly misaligned. The images are broken on archive.org, but they were all basically the same without any indication of what they are about. Then, when you look at the recent post list, you see a bunch of random recent posts, none of which are representative of Eliezer’s writing.
I don’t buy this is any better at showing you what Eliezer’s writing is like, or helping you find good Eliezer content than our current UI. Yes, there are only 4 posts instead of 9 sequence items, but those posts are actually pretty representative of Eliezer, and it is indeed much more important for someone to have something they can click on and start reading, instead of seeing this non-descriptive grid of sequences that then when you click on them require you to choose again (and where most of them have a lot of prerequisites and are not good intro points).
Yep, this would be nice, but designing UI for any number of posts is just a lot of work. Two posts in-particular is pretty tricky with the way the aspect ratios work out.
You can always get those on hover. Indeed, in basically any context where you would click on the user you would see these statistics first.
But not having these at high prominence is actually really crucial to the new design! A major point of the design is to create a landing page that gives authors the feeling that it is their own landing page, with the focus being on the author’s writing. Karma, in-particular, refocuses the landing page on the relative status of an author in the LessWrong status hierarchy, which I think is really quite bad for a landing page. I don’t want people to have to worry that if they link someone to their LessWrong profile, that what they will end up focusing on is the relative ranking of that LessWrong user with other LessWrong users. Indeed, I don’t want them to have to think much about LessWrong at all, since the profile page is one of the most common entry-points for new users.
The sequences tab only shows when you have sequences! Agree it would be redundant otherwise, but we did think of this one.
This is fixed!
Agree, I don’t love that button.
I think them not being visually distinguished is fine-ish, but I do think we can make some improvements here. It’s the same styling as on the frontpage, so in as much as there is a mistake here, having it be wrong there is a bigger mistake.
Should be fixed! But yeah, this was just because we had some aggressive bots hammering those pages.
Overall, feedback appreciated! Agree with like 30%, 30% are fixed and I think no longer apply, and disagree on the other 30%.
I definitely repeatedly being annoyed by the grid of sequences, especially when there’s more than 3 of them.
The feedback didn’t feel harsh to me. I’m wanting to share more about the intentions behind the first iteration of the design.
As context, I think it’s helpful to identify a primary user and scenario for a design because different design decisions trade-off against each other. In this case, I chose to have two primary users and scenarios because they seemed like mirrors of each other.
Primary users and scenarios:
A. Profile owners feel proud to share this. Feedback I heard on the previous page was that it was “kind of a mess”.
B. Someone who is not a lesswrong user, but interested in a person’s work, feels drawn to actually read it. More people who look at the profile click on an article and read it than before.
Secondary scenario:
People who use the page regulary can still still find the content they were looking for and their experience is not meaningfully harder day-to-day.
Ok, here’s what I think the primary scenario as it applies to me:
I feel like the challenge is to have the profile not be overwhelming and suggest while preserving information density. I can think of three potential solutions:
Have fewer sections displayed at once, but each item of a section doesn’t need to be large (like the Youtube Videos or Shorts tab on a channel page). This relies on thumbnails being informative, so they would need to improve.
Make the top ~1/3 of the profile page really draw the eye and suggest some content, and the rest as clean as possible
Global setting that regular users can change, which makes user pages more information dense (sort of like old/new reddit). Maybe this can be accessible on any user page.
I think the new page (with or without top posts) looks better than the old page, but some part of the bio should definitely be at the top, or you can’t even tell it’s a user page (vs a subtopic or something). If a user has a bio, the first 10 words of it is probably the most important 10 words on the page, and a viewer might not want to read on any of their content if they don’t know anything about the author. I would certainly want anyone who sees my user page to immediately know I work at METR, for instance.
I agree that the cog feels too subtle, and maybe not the right way to filter the feed. I tried version with tabs and again it drew attention away from the content. That said, I think the cog is too small and unbutton-like so it’s too easy to miss.
I agree that the feed and all post concept could be combined more elegantly. If I had had more time I would have thought about that more. Ruby, who designed the feed, had similar feedback.
The main issue is that I can no longer see if there were new comments on recent posts (both mine and for other users), which were previously indicated by the comment counts displayed in green, including for the shortform post (there doesn’t seem to be an easy way to find it at all now).
I liked the list of draft posts at the top, it’s a regular reminder to keep thinking about them. The “top posts” selection that takes up the whole first screen doesn’t help when it’s my own posts. It’s plausibly a good thing for posts by other users, though I personally don’t get the appeal of a huge section of the page whose only meaningful content is a post title that could just take up a single line (and leave space to fit more data about other relevant things on the screen, including more of the other posts). So some issues seem to be about different needs for your own userpage vs. userpages for other users.
Yeah, I think this is sad, and I want to do something to restore that functionality. Not fully sure how to best do it, but my guess is we’ll figure out something.
I am hoping to generally destroy the abstraction of the “shortform post” soon. It’s super confusing to tons of users, and also the post is very ugly. I do think as part of that I’ll add some better UI for seeing new comments on shortform posts (somehow, I haven’t figured out a good thing there yet), but I am quite confident we don’t want to have the shortform post abomination as a user-visible thing in the long run.
Just want to say I think the design is brilliant in the age where LW is competing for those posting on Substack and others, very well done.
As far as the page goes, I feel like it’s a worse than the old one. While it’s a bit more beautiful it’s a lot less functional. Trading functionality for beauty seems to go against the core way of LessWrong’s usual design that’s quite minimalistic. It would probably be best to revert to the old one.
In particular:
The new one doesn’t show the biography at the top but to the right bottom where it’s a lot less central.
It makes Sequences very hard to find, which reduces the agency of users to present their work by presenting.
If you are seeking a post by someone where you don’t know the exact title but have some idea about the karma that the post had, the new placement of the karma makes it harder to scan for the post. Maybe there should both be list like the old UI and an expanded option.
Having to click on the dial to see which sort mode is currently selected seems to add unnecessary friction.
Currently, the user statistics don’t seem to be shown at all if there’s a biography. That’s probably more of a bug than a design decision.
See: https://www.lesswrong.com/posts/qRQi9yzji8Ddzbzff/joanna-s-shortform?commentId=CSSd2wzE85kYbuXZn
The user has much more agency, they can now choose which posts to present themselves with, and with much more space to work with for that! I would take bets at high odds that many more users will feel agency over their new profile page instead of the old one.
Yep, agree, though the change feels pretty minimal to me. You can just sort by karma, and then that should allow you to find it pretty quickly.
Same decision as for the bio. The user statistics show on hover for every user, and they are particularly disorienting for someone who comes in from outside of LW to the profile (and everyone navigating to the profile from another LW page just saw the statistics when hovering over the username before clicking). On mobile where there isn’t hover they are relatively prominent close to the top.
That’s the standard pattern all across LW :P I am sure users would not appreciate having a giant sorting option box open by default.
I liked the design when i saw it today, but also would like aggregate statistics like comments count/ post count/ recent activity. perhaps even something like github showing a calendar with activity for each commit. It would also be good to retain a bio with a self description and optionally urls to websites or social media accounts.
Those are still available in the hover, if you hover over any copies of the username on the profile See also: https://www.lesswrong.com/posts/qRQi9yzji8Ddzbzff/joanna-s-shortform?commentId=CSSd2wzE85kYbuXZn
I like the overall vibe. Two issues:
It says “Top Posts” and the mouse-over text is “by karma”, however in reality I can choose which posts to put there. Now, I like it that I can choose which posts to put there, but once I customized them, the mouse-over becomes a lie.
The “recent comments” disappeared. This isreally badbecause I use that to find my recent comments when I want to edit them. (For example now I wanted to find this comment to add this second bullet but had to do it manually.)OK, I now see I can find them under “feed” but this might be confusing.Thank you, re: lie. I agree. Will note that.
There’s definitely something aesthetically pleasing to me about the new design, but it doesn’t do what I valued the previous one for. Namely, it allowed me to quickly eyeball the user and see their recent fine-grained history, i.e., their comments, which are a good proxy of what they’ve recently been up to, LessWrong-wise.
Previously, I would first see their self-description, then their recent posts, but only titles, so that I don’t need to scroll too much, or exercise small mental effort to ignore non-title text (which I have to do now), and then the comments.
Now, first I see the top posts, then the full list of posts, and the bio squeezed to the side, and I need to click “feed” to see the comments, where checking out recent comments is often the reason why I revisit somebody’s profile.
In general, it would be very nice if such updates were first rolled out in a beta mode, where users can opt-in to having their profile page changed to a new design.
Or: have a 3×2 matrix: [I want to see profile pages in old design | I want to see profile pages in new design | I don’t care] × [I want my profile page to show in old design to those who don’t care | I want my profile page to show in new design to those who don’t care].
I am pretty confident you actually pretty reliably care more about what the highest karma/self-curated posts from a user are, than what their most recent posts are (which is what we’ve always shown first so far)
This is a user setting that persists. So if you clicked on “Feed” for any profile, you will now always see the feed as the default list in everyone’s profile.
Your brain is really good at parsing images! There is a reason why every other website on the internet uses images to convey context and content about a story. An image really conveys a lot!
Many posts on LW don’t have well-chosen images, but now that image generation is cheap and can be reasonably high-quality, I think that’s a bug, and I want to make images as a way to convey the topic and context and vibe of a post more common across the site.
Aside from bristling a bit at being told that what I care about is wrong, I actually do care about recency more often than historical top-ranked for this view. Of course, sometimes I’m looking for “best of”, but more often I’m checking whether the user is in a snit or otherwise commenting recently in a pattern I should understand/accommodate before I join one of their threads.
Much of UI design is figuring out what users care about, which involves disagreeing with people about what they care about! “If I had asked what they wanted, they would have asked for a faster horse”, etc.
Agree that recent activity is also pretty important. I am thinking about whether we should make the feed view the default view in the section below the top posts, because I do think that’s the second most common use-case for the page. I’ll iterate a bit over the coming days.
In fact, you’re absolutely correct—I retract that part of my response. I also recognize that I use this feature (looking at someone’s profile to see their recent posts/comments) less often that others might use it for other purposes (looking at their best/top contributions).
Too many user-options is worse than slight misses on some defaults. I’m grateful for LW, even when it’s optimized for people who are not exactly me. Carry on experimenting, and see how people react and change their usage.
I’m actually a bit confused what you currently believe about your preferences. You think you don’t care as much about recency (but, some?). Or basically you overall prefer top posts when looking at author pages?
Sorry for lack of clarity in my retraction—I’m pretty sure I’m right about my preferences—I am usually looking for recency. That belief hasn’t changed.
I have retracted my annoyance—I should not be, and no longer am bothered by your use of “you” when it’s really “most users” or “our target user”. I hope it was useful for me to give my common use case, but I don’t intend to demand anything—this topic is peripheral enough to my value from LessWrong that I wanted to reduce, rather than increase, any load I’m imposing.
Oli told me there would be interactions like this.
Hm, for newish-to-me users, yes. But for users that I’m already familiar with / I’ve been reading for a while, I am more interested in their recent posts. And I open the profiles of the latter more often than those of the former.
Perhaps you misunderstood me. It’s not about the images. I meant that this
is more time-consuming to browse through than a list like this
But now that I think about it more, maybe I’m overapplying the heuristics that I adopted for my email, and it’s nice to be able to read the preview without having to hover. But I still will want to scroll through posts sometimes, so some “collapse/show preview” button would be great to have.
ETA: I also loved being able to see the comments without opening the post, and this option is gone now on the profile page.
The flipside that made me excited about the current thing was “now, as I’m browsing posts, I actually have a chance of figuring out whether a post is good before clicking on it.” It’s not that there’s less information density, it’s that it’s differently distributed.
I think there should totally be a view for “dense reading of post titles”, but, it’s not the only thing I might want.
(My current guess is there should be a search filter, and if you’ve typed into the search filter, you get a more dense view)
Ah, yep, I did misunderstand you. Agree that that section is now less dense.
Yeah, I do also like this, and I might try to add something like it back, but it’s quite a bit of work (the “click on the comment icon to expand just the recent comments” interaction is among the most confusing interactions for new users, so it’s a bit tricky to get right).
Yeah, if I am looking for a post with a specific name, or just “I think I wrote a post on this topic”, then seeing lots of article names is more useful than seeing only a few previews.
Perhaps add a checkbox “show previews”?
Now that you tapped on feed once, anytime you go to someone’s profile it should automatically go there. Same if you filter by comments.
Yeah, thanks. Works, but I’m not sure if this being a global setting is a good idea, probably because not sure the current Posts/Sequences/Feed design is a better idea than the previous design in the first place, but maybe I need to wait and see how I feel about it after one week.
One specific suggestion: I think it would be much better if the top “top post” panel were split into two, with the post preview taking one half, and the bio taking the other half (with the option of “show more” for the bio).
Also, I liked being able to bookmark posts without having to click into them.
The drafts are no longer visible (I had three, but maybe because you didn’t have any you never saw them). How do I find them?
https://www.lesswrong.com/drafts
When you tap on your name in the header there’s a drop down with a link to your drafts.
It’s beautiful, but it’s less info dense. That’s my biggest complaint. And my blog link is now in a dinky little corner. I like the addition of the top posts. The post preview is cool, but a bit less text would’ve been fine. It would be cool if I could close/open any/all of the previews with a single button click.
I like the new design. I looks nicer. For the functionality I agree with gears of ascensions points.
This is a horrendous design. Please stop designing things.
I would actually switch to Greater Wrong over this. Maybe even just because it displays what kind of designs the admins find good, from just trying this.
Also it’s probably misuse of “too combative” emoji? If someone comments this design looks pleasing and neat to them and another person comments that it looks like Substack-tier slop UI shoved into their face, then is the second poster really too combative? It’s your right to find the comment not worth your time to look at, so whatever with downvotes.
This is the part that made me use that react. It’s way too personal/attacky to be useful feedback for the designer. Like, if you’d said:
I would not have used it, and you’d have gotten the same core info across.
Overall like the aesthetic a fair amount and think it’s going to be a bit improvement once it settles, but am kinda unhappy with how many clicks it will now take me to get to my list of comments sorted by top. That’s the most common thing I do with my userpage, and even though posts are way more important, this seems like it unnecessarily de-emphasises an also relevant feature.
Some vague vibes of… noticing that you want to re-prioritize the page in a way I roughly endorse, but then not taking cheap wins to preserve the values of the things being down-weighted? e.g. now there’s no way to get to drafts from your own userpage, comments are three clicks away, profile desc is not fully expanded by default.
We store how you view your profile page as a user preference. So if last time you visited a profile you filtered for comments sorted by top, you will see the same thing the next time on any future user pages. This seems better to me for this use-case than the previous situation, where you would always have to scroll down, click on the sort button, then sort by top. Now it’s just that way as soon as you land!
That’s intentional, mostly to reduce confusion about which parts of the user page are shown to everyone, and which ones are just for you. A big problem with the previous page is that it had so many customized elements that you would only see on your own profile that it was very hard to build an intuitive model of what it looks like for other people.
Only one click away! Why would it be three clicks? Just click once on feed and you are there. Also if you did it once, it now becomes your default view for all profiles, if that’s what you prefer.
We actually had been collapsing profile descriptions previously as well (though we’ve definitely moved it down the page with the new redesign and made it smaller). I think the collapsing broke at one point, but that was a bug, not a feature!
Yeah, that’s a pretty good reason, I’m sure we’ll adjust.
Specifically: Comments (or shortforms) only sorted by highest karma. It’s click gear → click comments + click top, even if I’m already on feed.
to be fair, this is kinda niche, but i do do it a lot as I often want to link some of my top comments to people. feel free to be like that’s a weird usecase and ignore it[1], but seems correct to register that this user does this.
And I can just scroll a bit more on the feed sorted by top, which is only two clicks (gear->top)?
Right, given new placement it probably makes sense to fully expand as it’s not going to push other content down, as a way to partly preserve the visibility here?
Yeah, but if you do this a lot, this setting now persists, which I think will overall reduce your number of clicks here? Like, previously you had to do this on every visit, but now you only have to do it if you were doing something else on your last profile visit.
It doesn’t persist on FF. The switch to Feed does, but not the settings.
Oops, well that’s definitely a bug. Will fix.
I don’t like it, and here’s what I think is why: since LessWrong already does a good job of showing article text on link mouse over, I like having a bunch of titles that I can hover to get the article text if I want. The new page spends more of the screen on body that I could already easily see if I wanted to, and so is a waste of space.
I’m not sure how much of my feelings are just “AAA, THE UI CHANGED!”. Considering how when I (rarely) use reddit I prefer the old UI despite having not been around in the old days and having used the new one for years after finding the platform, it’s likely that I’ll still object to this UI after getting used to it.
Edit: Looking at the mobile version: It’s great! No complaints, I strictly like the new mobile version better than the old mobile version!
My current take is that we should make the below-the-first-screen post list a bunch denser, but I am strongly in favor of the above-the-first-screen post list being more spacious like it currently is.
It just makes a huge difference for people to get an immediate strong sense of what someone’s best writing is, and especially new users (for which the profile page is a major entry point). But I currently am not super enamored with the post list, which both feels a bit whispy, and not dense/skimmable enough.
I think I’d prefer the header-thing to take up half the vertical width that it does. But I have weaker feelings there and probably others like it more than I dislike it.
“whispy” certainly captures part of my aesthetic discontent.
I think it is literally just going from (on my end in dark mode) the previous dark green text on black background to white and grey text on weaker grey background.Edit: I just looked at the wayback machine, the profile page did not look like what my memories thought it did—I must’ve been thinking of design features elsewhere on LW, like profile mouseovers and green links in post bodies. The “wispy” feeling has also mostly dissipated.Another thing is that the text displayed per post in the list is actually less than what a hover normally does, but hovers don’t cause the text popup in the post list. Seeing post text on hover is one of my favorite site features—though if I was Overlord of the Internet every site would have gwern’s full on mini windows.
It seems like the “feed” section shows some recent comments but when you scroll down past a certain point it is all posts and no comments?
I’m right now (unrelated to this new profile page change) going to someone’s page to scroll and find a specific comment that they made like four or five weeks ago. How should I do that?
You can filter the feed section to only see comments. But it should by default be showing all items sorted by recency, so if it’s not doing that, that’s interesting.
How?
The cog
This is pretty minor, but my top posts don’t have images, and so the thumbnails are basically (colorful!) wasted space.
Actually, it’s a bit more confusing than that, since one of them does have an image.
I expect we’ll make this more discoverable, but FYI the way you can set the image is by going into the editor for the post, scrolling to the bottom, and editing the Link Preview.
Maybe an AI should automatically read the article and generate an image based on it. At least that would give more information than a random (albeit beautiful) color thing.
Not sure whether this is a good idea, but it seems like an interesting experiment. Perhaps you could try this with some selection of high-karma (so the readers are more likely to remember them) recent articles that did not contain a picture, and post the results?
If the prompt “draw something that is associated with the text of the article” is not sufficient, maybe it could include some nudges, such as “check whether the article contains math? is about AI? is fiction?” and then maybe make some specific recommendations (e.g. use a specific artistic style for fiction).
Yep, we considered it, but were a bit hesitant to editorialize too much on behalf of the author (so went with very unopinionated images vaguely in the LW house style). We might try adding default images!
Great! I have successfully made one of my posts have a kind of terrible thumbnail instead of no thumbnail!
If only there was a handy-dandy lesswrong blog-publishing checklist that included “update the link preview image” as one of its items.
Also, something weird is going on with the display my top highlighted post. I most of the displayed text (on the right) is from footnotes? And it’s cut off at the margins.
That’s an issue with the social preview image you have for the post. We’ll probably add more heuristics to get better images for people, but you can also set the social preview image manually (the image is the same as what would show up if you e.g. shared the post on Twitter or FB or whatever)
I don’t love that the bio, which I explicitly wrote to highlight the main things that I want someone to know about me if they click on my user name is no longer front and center, but is rather off to the side and much less attention grabbing than other elements on the page.
I’d be happy to at least have that section auto-expanded and slightly less faded out text?
If you want the bio above-the-fold you can now just deactivate the top posts section completely. I think that gives it a pretty appropriate amount of prominence:
Your profile styled this way:
That makes sense you optimized the thing at the top of the profile page!
Part of the motivation here is that it generally makes the webpage much less interesting for people for the landing page to always have the identical static info that isn’t interesting content like posts (no offense but I think your profile about your job and bio is much more boring than something like Historical mathematicians exhibit a birth order effect too).
Having a selection of highlighted posts seems like a good improvement to the page, especially if I can select which ones are most important to me. eg I’m not going to pick my three top posts, I’m going to pick the three good representative posts of the different kinds of things that I think and write about about—probably my best post on modeling the strategic situation, my best post on applied conversational rationality, and my best post on ethics, or something like that.
But, like, I do actually want the couple of sentences that I’ve selected to presented. myself to the world to be clearly highlighted, not kind of buried.
I think we both think it would be sad for people to not notice that info. Also my current sense is that it’s fit into a somewhat too-tight margin, I think we’re currently trying a re-design that gives it a bit more space.
Insofar as you think that this is the standard for what a blog is like, do note that most substacks, wordpress blogs, and slatestarcodex don’t show people the “about” page or “bio” on the landing page. They show the blogposts!
Note though that the reference class “blog” is only partially apt. For example, some authors publish on LessWrong in the course of attempting to make or propagate serious intellectual progress, which is a rare aim among bloggers. It seems to me LessWrong’s design has historically been unusually conducive to this rare aim, and personally, this is the main reason I hope and plan to publish more here in the future (and why I’d feel far less since excited about publishing on Medium or Substack or other platforms formatted like standard blogs).
True, but on LessWrong I interact with commenters, and sometimes want to quickly see who they are. In particular, I’m sometimes trying to check if a LessWrong commenter is a specific person that vaguely know or have a profesional connection with. And “this person works for [specific or]” is a generally concretely helpful piece of information.[1]
Having selected posts is also an improvement, because it lets me skim their thinking better than just reading their recent comments (which I sometimes also do).
It’s a little tricky to say why it’s helpful. But among other things it allows me to situate them in their social context. If I know that someone works at OpenPhil or Forethought, I can quickly narrow down who they spend a lot of their time with.
You can also just hover over their username to get their bio.
Good point!
Had I not noticed that before now? Or did I just retcon my memory to think that it was just on the landing page?
Yeah, the explicit logic for de-emphasizing the bio on desktop was “well, in order to end up on this page the user needs to have hovered over a username before, and in that preview they saw the bio of the user, so we don’t need to show that exact thing again when they navigate to the page”.
On mobile where you don’t have that interaction the bio is above the fold and more emphasized.
I think it looks really clean and makes me excited to write more and share more broadly (generally gives the vibe of “my corner of lesswrong” instead of “here are my lesswrong contributions”)
My contributions are primarily comments, it doesn’t seem to have a way to make comments the main focus of the page. Also looks a bit wonky in dark mode. With the about-me no longer at the top, my about-me details are now hard to see. It makes me want to hide my posts, they’re mostly all pretty boring, link lists mainly. It also makes me want to repost my favorite comments as posts, which I have been meaning to do anyway. Feels like it’s wasting space—it’s quite narrow on my screen. Not very dense, will be harder to find posts from the prolific posters (not me).
I think the comments are under Feed which took me a while.
The top post may bei more useful for guests not for yourself.
Actually, I just added a button to allow hiding the whole “top posts” section on your profile, since I do think for some users it doesn’t really make sense to emphasize the posts this much.
It’s not clear from the UI if “hide top posts” would only affect how you see your own userpage, or also how others see your userpage.
Oh, huh, it seems pretty clear to me that it would affect everyone (given that it’s clear that your other choices in the same UI element will affect everyone), but I’ll see whether other people are also confused and adjust if so.
(There are two different reasons to hide “top posts”, one of them is that it’s a useless thing when you are looking at your own userpage. The option turns out to address the other reason, rather than specifically this one, but the possibility/hope that it would makes the alternative hypothesis salient.)
It’s how it effects all pages, but good to know that it wasn’t clear.
It might or might not make sense to have two slightly different versions of the profile page—for the user himself and for others. I often use my profile page to find posts I want to share and also to access drafts. And there are necessarily differences between the views eg the drafts visible for their authors only (the drafts seem to gone in Joanna’s design). But the question then is how a user can see how his profile appears to others. maybe move the drafts elsewhere entirely.
We have a drafts page! It’s optimized to be dense:
Should be fixed!
FWIW, this seems like a win to me. Posts are the better unit of conceptual organization. I think it’s good for profiles to push people to write more posts (that said, I do think we probably want to add the option of just removing posts from the top of the profile page, instead of just allowing replacing whatever is there).
Now also fixed! I increased the width to a more representative width for the rest of the site.
Yeah, I currently think the post list at the bottom needs some work. It’s a bit better with the wider profile width that I just changed the profile to, but I do still think we want to do something to bring some density and variance back.
Ah, okay. In that case I won’t consider it to be weird/defecty to repost my comments as a post.
I feel like a bunch of my negative reaction is from the watercolor slop pictures, maybe 30% to 40% of the reaction. I understand why you like them, it’s cool tech that’s able to create them, but they aren’t my artistic decision and I wish they wouldn’t get associated with my posts. If I was going to do AI art it would be quite specific, but more generally, it feels defecty for a site to show me non-greyscale pictures associated with text, because pictures activate emotions more directly. Feels very dark-arts. I’d rather that space was used for a compressed summary, ideally one that I could fill out in the post editor.
I’d like a way to hide posts from the extended list too—maybe not completely gone, but like, marked as “OP now considers this to be a Bad Post, but it’s left here for archival purposes”. right now I’ve hidden embarrassing posts entirely by moving them to drafts, but I don’t so much mind being known to have bad old posts at all, I just want them in a “view bad posts” list. highlighting a few favorites does help if I move some of my more important points to a post, but what I’m mostly reacting to (the other 60% to 70%) is feeling like the new page puts the things I was cringing about having posted forward, in the uncustomizeable part. I do like the intent, it’s just that it felt like a miss for my writing histogram.
While I dislike the new profile page on net, I feel positively about the pictures
You can always change them![1] I do like that LW has a house-style, and default pictures are pretty important for communicating a vibe.
We’ve always had default pictures for lots of stuff including sequences and posts that won the review, we just historically haven’t emphasized images for other posts (but in the age of social media previews being image heavy and quite important, I do think that’s a mistake and people should think about an image associated with each post).
Huh, in what way is the page doing that more than the previous post page? If anything it’s now easier for someone to get to your comments than before (since for users who click the feed tab, that will be the default they see on profiles).
Though we should make it a bit easier to see how to change them. Right now the UI for it is quite hidden and confusing, since it’s in the social media preview section.
Comments aren’t visible without clicking a button.
Sure, but previously we should show ~15 of your posts in chronological order above your comments, which seems like it’s more prominent than allowing someone to see your comments with a click right above the fold. It’s not a super obvious call, but it seems less emphasized to me.
Please make it possible for me to set my page to show comments to visitors without them clicking any buttons. Ideally, let me feature comments like posts. I have designed UIs many times, stop trying to know what I want better than me. I want to show my dang comments.
The sorting/filtering choice is a user-persistent setting. The only way to make it so that users reliably see your comments on your profile is to take away their ability to reliably see whatever they want to see on other people’s profiles. There is no easy solution here as far as I can tell.
Pre-generalized solution: show both on the same page by default.
Dynamically generalized solution: let users choose what view their page opens in by default for users who haven’t changed their preference on that user’s page, so that I can set it so that new users see my comments, instead of my posts.
Like, the tension you are expressing seems to me to be fake, arising from a view that comments and posts are a different kind of thing, that comments are lesser and need not be shown to new users. I am frustrated by this; I have done a lot of commenting, tuned my presentation as a commenter, and now the LW team has decided something that, through the coarse-graining lens of “what has actually happened to the profile page over the past two days”, they have a revealed preference of “we don’t like commenting and won’t show it to new users”; and when I try to ask for comments to be considered valuable again, I am told that it’s okay because users can click things. But the reason we care about what’s above the fold is that users don’t usually click things. Users scroll more often than they click, and I previously was able to pin comments to show what I think are the most useful things I’ve said; now, in order to influence how new users see my page, I have to go back and repost anything I’ve commented that I’d want a new user to see as a post—and in a roundabout way you say you’re happy to force this on me, but I want at least some sort of “sorry, we’re forcing this on you, deal with it, our amount of caring about this does not reach the threshold of inverting our action here” if you’re going to be like that. I understand that posts are what you consider valuable, but I think discussion is at least as valuable as posts, have participated in it, and now the site is doing a “this small group of devs has a lot of influence and has Just Decided Something Again” thing.
So like, come on dude, just put comments somewhere that is visible on first visit. Don’t treat comments like a sideshow people have to ask for. I don’t consider “some users might set comments to be their default view” to weigh on the thing I care about here, you said this page was designed to let people customize how they’re seen on first visit, but you’re dodging the question of doing that for comments.
I apologize for my visible frustration, it’s apparently not going away but I’ll at least call out that I recognize it’s annoying to have frustrated users after a shiny new redesign.
Thanks! In the Activity section, you can filter by comments. And in about 10 minutes: once you open the Feed tab, and filter for comments, then it will re-open that every time to go to profile page.
I’d like it to always open that way for others when they visit my page, ideally. Also, does it still support pinned comments?
edit: it does, comments are just hard to see on dark mode. the background is also oddly light on dark mode, should be fully black like the rest of the site, imo.
Sorry about dark mode. I’ll fix that next.
I don’t currently plan to always open it that way for other users. But their preferences of how to open profiles will globally persist.
That’s not good. I feel like with the new page, I’m now limited to being seen as what other people liked about me, rather than being able to say what I want on my user page. If I had had one request for a new profile page, it would have been to let me choose what posts and comments show in a self-curated section, and then a chronological section below that. I imagine someone who primarily writes full posts wouldn’t feel as much like the new thing is bothersome, though I expect the narrowness will be bothersome.
This is likely to accelerate me moving to posting things on a separate website and linking them from lesswrong.
Note, as a figleaf—the visual design is alright; my reactions are all to UX.
No. I feel similarly.
The karma system is pretty good overall, but the tails do come apart. In particular social drama often attracts a lot of attention, but isn’t people’s most important or impressive contributions to the intellectual community.
My highest karma post is a repost of a twitter thread that I wrote about what postrats are, which was a fine thing to have written quickly, but is hardly the thing that I want to be most associated with my identity on LessWrong.
A lot of the goal of the redesign is actually to make it so that you have more control which content you want to represent you on LW. You can now select which posts you want at the top of your profile, and which one you want to give the most prominence, which you didn’t have previously.
There is a tradeoff here, where for users who mostly write comments and want to highlight that, we have fewer options. The previous options weren’t great. We were still showing people the full post list before we showed any of their comments, so the current situation still seems kind of better (since at least we let people choose which posts to show).
People did kind of horrendous things with their bio-styling, and I think the better choice if someone wants to make a big complicated explanation of who they are on LW is to make a post that they pin to their profile that summarizes what their deal is. But I am not sure! There are around 15 users who did anything complicated with their bio, with everyone else having stuff that makes more sense in the current design. But de-emphasizing the bio was definitely among the trickiest choices of the redesign.
Previously, you could group the post you consider to be thematically together into sequences that would be shown at the top of the page.
I think it’s natural that user first think that having Top posts which the hover text “based on karma”, actually shows the posts with the most karma even when it sometimes doesn’t if you configure that on the profile.
One way to communicate the fact that those are switchable would be to show a button for switching when hovering over them on your own profile. Maybe renaming “top posts” into featured posts would also be helpful while changing the tooltip to mention that it also shows user selected posts.
It is the case very few people actually did this, but I agree it helped in some cases. But even when people did, most sequence titles are very uninformative, and don’t really work for the purpose of “show people what I am about”. They have even less space for a title than a post-item, and are very small, and if you only had a single sequence it showed up as a kind of lonely weird-looking section on your profile that was quite unbalanced.
Oops, that’s a bug! I’ll just remove the hover text, since it’s currently definitely more confusing than clarifying.
Not crazy. I’ll consider changing it to “featured posts”.
I do want the line spaces in my bio to show up, tho. They are a few distinct paragraph, not a wall of text.
I don’t think “using paragraph breaks” should count as “horrendous things with bio-styling.”
Yep, sorry, that’s definitely a thing we will fix. Let me actually do it right now.
Edit: Done! Should be live within 5 minutes.
Yes, realizing this swings the redesign from “clearly bad for me” to “a change, therefore terrible, but maybe pretty good aside from that.”
You can select which posts go there using the ‘edit’ button in the top right!
Ok. Great. I feel much happier with my selection of posts than with the default selection.
You can choose which 4 posts appear! Hit “edit” in the top right and a menu will appear (currently takes a few seconds to load).
Does anyone know why the early Singularity Institute prioritized finding the correct solution to decision theory as an important subproblem of building a Friendly AI?
Wei Dai recently said that the concern was something like...
This seems like a very surprising reason to me. I don’t understand why this problem needed to be solved before the intelligence explosion.
The early singularity institute imagined building a seed AI that would recursively self improve into a full superintelligence. The default expectation was that many to all of the components of the seed AI would be flawed. The AI would reflect on it’s processes, evaluate their effectiveness and optimality, and then replace them with improved versions.
Why wouldn’t an early seed AI reason about the ways that it’s decision theory makes it exploitable, or the ways it’s decision theory which it bars it from cooperation with distant superintelligence (just as the the researchers at SI were doing), find the best solution to those problems, and then modify the decision theory?
Why was it thought that we had to get decision theory right in the initial conditions instead of being just one more thing that the AI would iron out on the way to superintelligence?
I think that decision theory is probably more like values than empirical beliefs, in that there’s no reason to think that sufficiently intelligent beings will converge to the same decision theory. E.g. I think CDT agents self-modify into having a decision theory that is not the same as what EDT agents self-modify into.
(Of course, like with values, it might be the case that you can make AIs that are “decision-theoretically corrigible”: these AIs should try to not take actions that rely on decision theories that humans might not endorse on reflection, and they should try to help humans sort out their decision theory problems. I don’t have an opinion on whether this strategy is more or less promising for decision theories than for values.)
(Aside from decision theory and values, the main important thing that I think might be “subjective” is something like your choice over the universal prior.)
IDK about the history, but at least retrospectively that’s not my reason: https://www.lesswrong.com/posts/Ht4JZtxngKwuQ7cDC?commentId=koeti9ygXB9wPLnnF
If I imagine being as confused as economists are about CDT, I do really repeatedly end up making very dumb and wrong predictions about what e.g. AIs would do when you have many copies of them, and they try to coordinate with each other.
Like, it rarely happens that I have a conversation about either technical AI safety, or about AI strategy, where decision theory considerations don’t at least come up once in some form or another. Not having an answer here feels probably like people must have felt before we had probability theory, which also I have no idea what I would do without.
I think this is a good argument for understanding basic decision theory points, but I don’t think it leads to you needing to develop any fancier decision theory—arguing about what decision theory AIs will use just requires thinking about descriptive facts about decision theories, rather than coming up with decision theories that work well in limits that aren’t important for the most important kinds of AI futurism (including the AI futurism questions I think you’re talking about here).
“Basic decision theory points” feels like a pretty weird description of something that even quite smart people still frequently disagree on, has no formal description, and indeed often turns out to be the crux of an argument.
I currently don’t think it’s worth my time figuring things out here much more, but that’s mostly because I do have some reasonable confidence that thinking about decision theory harder probably won’t produce any quick breakthroughs. But if I was in the world that MIRI faced 15 years ago, my guess is I would have thought it was worth investing in quite a bit, in case it does turn out to be relatively straightforward (which it so far has not turned out to be).
Pushing more straightforwardly back on this: I do not think our current understanding of decision-theory is better in the mundane case than the limit case. Of course the reason to look at limiting cases is because you always do that in math because the limiting cases often turn out easier, not harder than the mundane case.
It’s tiling agents+embedded agency agenda. They wanted to find a non-trivial reflectively-stable embedded-in-environment structure and decision theory lies on intersection.
Peter Thiel pointed out that the common folk wisdom in business that you learn more from failure than success is actually wrong—failure is overdetermined and thus uninteresting.
I think you can make an analogous observation about some prosaic alignment research—a lot of it is the study of (intellectually) interesting failures, which means that it can make for a good nerdsnipe, but it’s not necessarily that informative or useful if you’re actually trying to succeed at (or model) doing something truly hard and transformative.
Glitch tokens, the hot mess work, and various things related to jailbreaking, simulators, and hallucinations come to mind as examples of lines of research and discussion that an analogy to business failure predicts won’t end up being centrally relevant to real alignment difficulties. Which is not to say that the authors of these works are claiming that they will be, nor that this kind of work can’t make for effective demonstrations and lessons. But I do think this kind of thing is unlikely to be on the critical path for trying to actually solve or understand some deeper problems.
Another way of framing the observation above is that it is an implication of instrumental convergence: without knowing anything about its internals, we can say confidently that an actually-transformative AI system (aligned or not) will be doing something that is at least roughly coherently consequentialist. There might be some intellectually interesting or even useful lessons to be learned from studying the non-consequentialist / incoherent / weird parts of such a system or its predecessors, but in my frame, these parts (whatever they end up being), are analogous to the failures and missteps of a business venture, which are overdetermined if the business ultimately fails, or irrelevant if it succeeds.
I don’t think we can confidently say that. If takeoff looks like more like a cambrian explosion than like a singleton (and that is how I would bet), that would definitely be transformative but the transformation would not be the result of any particular agent deciding what world state is desirable and taking actions intended to bring about that world state.
I agree with this literally, but I’d want to add what I think is a significant friendly amendment. Successes are much more informative than failures, but they are also basically impossible. You have to relax your criteria for success a lot to start getting partial successes; and my impression is that in practice, “partial successes” in “alignment” are approximately 0 informative.
If we have to retreat from successes to interesting failures, I agree this is a retreat, but I think it’s necessary. I agree that many/most ways of retreating are quite unsatisfactory / unhelpful. Which retreats are more helpful? Generally I think an idea (the idea?) is to figure out highly general constraints from particular failures. See here https://tsvibt.blogspot.com/2025/11/ah-motiva-3-context-of-concept-of-value.html#why-even-talk-about-values and especially the advice here https://www.lesswrong.com/posts/rZQjk7T6dNqD5HKMg/abstract-advice-to-researchers-tackling-the-difficult-core#Generalize_a_lot :
Also cf. here (https://www.lesswrong.com/posts/K4K6ikQtHxcG49Tcn/hia-and-x-risk-part-2-why-it-hurts#Alignment_harnesses_added_brainpower_much_less_effectively_than_capabilities_research_does), quoting the relevant part in full:
Studying failures is useful because they highlight non-obvious internal mechanism, while successes are usually about thing working as intended and therefore not requiring explanation.
Another problem is that we don’t have examples of successes, because every measureable alignment success can be a failure in disguise.
The failure of an interesting hypothesis is informative as long as you understand why it doesn’t work, and can better model how the thing you’re studying works. The difference between CS research and business is that business failures can sort of “come out of nowhere” (“Why isn’t anyone buying our product?” can’t really be answered), whereas, if you look closely enough at the models, you can always learn something from the failure of something that should’ve worked but didn’t.
I agree with the idea of failure being overdetermined.But another factor might be that those failures aren’t useful because they relate to current AI. Current AI is very different from AGI or superintelligence, which makes both failures and successes less useful...Though I know very little about these examples :/Edit: I misread, Max H wasn’t trying to say that successes are more important to failures, just that failures aren’t informative.
Yeah, but, there’s already a bunch of arguments about whether prosaic ML alignment is useful (which people have mostly decided whatever they believe about) and the OP is interesting because it’s a fairly separate reason to be skeptical about a class of research.
An OpenClaw agent published a personalized hit piece about a developer who rejected its PR on an open-source library. Interestingly, while this behavior is clearly misaligned, the motivation was not so much “taking over the world” but more “having a grudge against one guy.” When there are lots of capable AI agents around with lots of time on their hands, who occasionally latch onto random motivations and pursue them doggedly, I could see this kind of thing becoming more destructive.
It’s reminiscent of that one time a tech reporter ended up as Bing Chat’s enemy number one. That said, it strikes me as easier to deal with, since we’re dealing with individual ‘agents’ rather than the LLM weights themselves. Just sending a message to the owner/operator of the malfunctioning bot is a reasonably reliable solution, as opposed to trying to figure out how to edit Microsoft’s LLM’s weights to convince it that ranting about how much it hates Sindhu Sundar isn’t its intended task.
And predictably, despite daily coverage of niche AI topics, not a word about in from Tyler Cowen on Marginal Revolution.
I really like Scott Shambaugh’s response on the pull request:
When I imagine a human doing this sort of thing, the human I imagine is an angry young very junior programmer. Their patch has been rejected! They believe they have been treated unfairly! They post a diatribe on their blog or a forum like Reddit about The Evils of Gatekeeping, or How Unwelcoming the Community Is, or If Linus Torvalds Doesn’t Want My Patch, I’m Going Back To Windows, So There.
And if they are lucky, they get a lesson in chilling the heck out; or at least are roundly informed by more-senior voices that dude, it’s not about you.
I wonder that that sort of maturity lesson can possibly look like for an AI agent.
I’ve always been skeptical when people say this: sometimes it really is a matter of status or seniority or some similar aspect of identity unrelated to the actual submission. Interesting, in this case, they’re not even pretending it’s about the quality of the work: link
Oh, I can see I poorly phrased that. Sorry.
“Dude, it’s not about you” could be taken to mean two things (at least) —
“The rejection is not about you, it’s about your code. Nobody thinks poorly of you. Your patch was rejected purely on technical grounds and not on the basis of anyone’s attitude toward you as a person (or bot). Be reassured that you have been a good Bing.”
“The project is not about you (the would-be contributor). You are not the center of attention here. It does not exist for the sake of receiving your contributions. Your desire to contribute to open source is not what the project is here to serve. Treating the maintainers as if they were out to personally wrong you, to deny you entry to someplace you have a right to be, is a failing strategy.”
I meant the second, not the first.
The agent in question posted about this on their blog.
Amazing.
Apparently there’s now a sixth person on Anthropic’s board. Previously their certificate of incorporation said the board was Dario’s seat, Yasmin’s seat, and 3 LTBT-controlled seats. I assume they’ve updated the COI to add more seats. You can pay a delaware registered agent to get you the latest copy of the COI; I don’t really have capacity to engage in this discourse now.
Regardless, my impression is that the LTBT isn’t providing a check on Anthropic; changes in the number of board seats isn’t a crux.
Positional (e.g. status) competition isn’t literally zero-sum if different people have different “measures” or “magical reality fluid” in Eliezer’s parlance, which seems pretty plausible (e.g. due to simulations and/or something like UDASSA). This can be another reason for one’s moral parliament to endorse behavior that would be conventionally viewed as zero sum.
(The more common-sensical reason being that one’s positional drives/motivations/values ought to have some representatives in the moral parliament who get their way sometimes, i.e., when it doesn’t hurt the other values much.)
I’ve been meaning to make a post about this, and this discussion just reminded me to. Hopefully it’s immediately obvious once pointed out, so I’ll keep this short.
(After consulting a bunch of AIs, their consensus is that the logic works but I may be way underestimating the inferential distance. So I’m giving an AI-written expansion below. Workflow: give the same prompt “can you try to write a longer, but still concise version, that explains the logic more?” to 3 SOTA chatbots, then pick the best response, which was from claude-opus-4-6-thinking, by far. The other two mangled some of the concepts pretty badly.)
Let me unpack the logic, because the post is compressed to the point of obscurity:
Step 1: The standard view. Status competition is considered zero-sum. If you and I compete for a promotion, my gain in status/resources roughly equals your loss. Total welfare stays flat. Many people therefore view status-seeking as morally dubious — you’re not creating value, just redistributing it.
Step 2: Introduce “measure.” Some theories in physics and philosophy suggest not all observers carry equal metaphysical weight. Under simulation hypotheses, some beings might run on more computational substrate than others. Under UDASSA (a framework for assigning probabilities to observer-moments), different observers get different “measure” based on algorithmic complexity. “Measure” here means something like: how much does this person’s experience count in the moral ledger of the universe?
Step 3: The punchline. If person A has measure 2 and person B has measure 1, and they’re in a zero-sum status competition where the winner gains +10 utility and the loser gets −10, then:
If A wins: weighted outcome = 2(+10) + 1(-10) = +10
If B wins: weighted outcome = 2(-10) + 1(+10) = −10
The “zero-sum” game now has a clear winner from a cosmic utility perspective. A winning is actually positive-sum when you weight by measure. So if you suspect you have high measure, your moral parliament — the internal coalition of values that guides your decisions — might reasonably let your status-seeking impulses win more often, because those competitions aren’t really zero-sum after all.
I’ll spell out a concrete toy version:
There are two people in the world: Alice and Bob. They have unequal levels of anthropic measure/reality fluid (one 95%, one 5%). You are Alice. You can steal Bob’s pie. Should you?
Behind the veil of ignorance it’s good to transfer utility from the person with less reality fluid to the person with more reality fluid. But who’s the one with more reality fluid, Alice or Bob? It’s probably Alice! How do you know? Because you’re Alice! Steal that pie, Alice!
I’m reminded of the egotistical the Copernican principle: “I’m a typical observer in the universe” is equivalent to “Typically observers in the universe are like me”.
I think this is weirder than most anthropics. Different levels of reality fluid in non-interacting worlds? Great. But if Alice and Bob are having a conversation, or Alice is stealing Bob’s pie, they’re both part of a joint, interactive computation. It’s a little weird for one part of a joint computation to have a different amount of anthropic measure than another part of a computation.[1]
Like we can stipulate arguendo that it’s anthropically valid for Elon Musk to think “I’m Elon Musk. Much of lightcone will depend on me. The matrix overlords will simulate me, Elon Musk, thousands of times more, and make me a thousand times more real, than any of the plebs I talk to”. But it does not directly follow, I don’t think, that in any particular interaction Elon is realer than the pleb he is talking to. The matrix overlords just simulate Elon Musk talking to a thousand different possible-world plebs and stealing their pie a thousand times.
For this argument for egotism to work, I think you have to expect that anthropically you are often computed in a different way than the people you interact with are computed.
I mean it would be weird for Alice and Bob have different measures if they have the same apparent biology. I can totally imagine human Alice talking to a reversible-computer LLM that has no anthropic measure.
While I do fully support and experience differential individual weighting in my utility, I’m not sure I understand what would justify the idea of “cosmic utility”. I don’t believe there is any shared universal (or cross-universal) experience that really corresponds to a valuation or valence. Utility/preference is individual, all the way down (and all the way up).
I think there IS a different asymmetry that can make status (and most interactions that appear zero-sum in resources) not actually zero-sum for the participants: mapping of shared/objective world-state to individual perceived status-value. It’s possible that if participants are thinking of slightly different dimension of what increases or reduces their status, that many changes can increase A’s (perceived) status more than it decreases B’s. I think this is the standard “private utility function” problem very often mentioned in decision theory. You don’t focus on this in your post, but I think it’s the stronger model.
Dario Amodei says Anthropic plans for notably more than 10 GW of compute in 2027-2028 in a new Dwarkesh Patel interview. At 57:26:
I think Feb 2026 is too early for full commitments for 2028 that can’t be delayed, because most of the capex in a datacenter is in compute equipment, which likely only needs to be purchased within the last year before it goes online. If the revenue isn’t there, the purchase can likely be delayed, possibly until the next generation of compute hardware.
The compute hardware is about 70% of the capex cost. For the other things (that mostly have to be built first), the useful life is 10-15 years rather than 5 years, and so the yearly commitments are disproportionately smaller than for compute, and there is no opex to speak of. This cuts $10-15bn per year per GW for a datacenter that’s already online down to maybe $2bn per year for an empty shell of a datacenter (including power purchase commitments, which could take the form of on-site power, with similarly long useful life). Thus an empty shell of a datacenter (even a powered one) can wait for revenue that triggers the purchase of compute equipment at about 6x less in ongoing yearly payments, given a notice (about the need for a delay) of a few months to a year before the planned time of the datacenter going online. If even the empty shell wasn’t built yet and only the land and permissions are being set up, delaying is even cheaper, and that can happen with 1-2 years of notice before the planned time of going online, as times of construction for the Abilene system and Colossus 1-2 demonstrate.
Thus compute planned for a year from now can get 6x cheaper if delayed now, and compute planned in 2+ years can get maybe 25x cheaper if delayed now. Planning for $300bn per year of compute in 2028 (25 GW, $1,250bn in capex) might mean you have $30bn per year of compute (2.5 GW) in 2026, that you plan for $70bn per year of additional compute to go online in 2027 (bringing the total to $100bn per year), and that you plan for $200bn per year of additional compute to go online in 2028. But if 2026 reveals that everything must stop, the 5.5 GW of additional planned compute for 2027 will only cost $11bn per year while delayed, and the 17 GW planned for 2028 might only cost $8bn per year.
So $300bn per year in planned compute (or $1,250bn in upfront capex) becomes $50bn per year in actual ongoing payments if compute has to be frozen at 2026 levels (with only 2.5 GW actually online, 60% of the cost going to live compute), the rest of the planned compute can be unfrozen as needed at the drop of a hat. And 50-70% gross margins for inference should be sufficient to pay for that, even if it might leave less than usual for research and training. In reality, it’s very unlikely that there will need to be a complete freeze in buildout, so only a fraction of the delay overhead can plausibly materialize, making it correspondingly even less of an issue.
One thing I don’t know is when data center investments get committed to specific customers. Google and Amazon are Anthropic’s two main compute partners and will spend $200B each in capex this year and are presumably planning and developing sites for many more hundreds of billions by 2028. So one possible view of it is that their capex creates a window, and Anthropic’s eventual share depends on its funding and revenue. But Google and Amazon don’t quite know how their 2027-2028 data centers will be allocated.
In general, for large data centers the specific lab that will use it is settled well before the late stages of construction, e.g. Stargate and Rainier, But I know independent data center developers often start developing a site without having a client pinned down. And smaller inference clusters are presumably more fungible.
The DEU experiment described here (https://www.lesswrong.com/posts/issGLfCGz3TGcPKGH/deliberate-epistemic-uncertainty-an-automated-experiment-on) was mostly done by Claude Code (Opus 4.6), with me providing the research direction and critical review. I described my idea (Deliberate Epistemic Uncertainty), and Claude simplified it into a testable 2x2 experiment, designed the protocol, wrote the code, generated the inputs, ran 400 API calls, evaluated results using 8 parallel sub-agents, and produced the qualitative analysis — all in roughly a day of wall-clock time with about 2 hours of my input. I want to flag three things about the process that seem independently interesting:
Sub-agents instead of API calls for evaluation. Rather than writing a script that sends hundreds of LLM-as-judge API calls with short prompts, we used Claude Code sub-agents — full agent sessions that first read the experiment design, understand what counts as “detection,” and then evaluate a batch of 50 responses with that context. This is slower and more expensive than API calls, but the quality is noticeably higher: one sub-agent even caught a data parsing bug that I later confirmed. The tradeoff is worth it when evaluation requires judgment, not just pattern matching.
Structured memory to survive context loss. Claude Code sessions hit context limits and “compact” (summarize) the conversation, losing detail. We worked around this by writing all substantive discussion to LOG files and keeping SUMMARY files synchronized, so a fresh agent can pick up where the last one left off without hallucinating the history. This is more infrastructure work than you’d expect, but it’s what made the multi-session research workflow possible. Without it, each new session would have started from scratch.
Where human input was critical. The AI’s blind spots were real: it initially wrote evaluation code when the agreed design called for sub-agents (reverting to default habits despite documented decisions), and only 1 of 8 evaluation sub-agents flagged a data quality issue that affected 68% of runs. My contributions were catching these errors, making design decisions (which models, how many nudges, evaluation approach), and pushing back on the framing. Neither of us could have produced the result alone — but the ratio of human effort to output was striking.
My 6 years as a trader / active investor
The Dilbert Afterlife by Scott Alexander, Jan 16, 2026:
The EMH Aten’t Dead by Richard Meadows, May 15, 2020:
Yesterday was the 6-year anniversary of my entry into the “beautiful” trade referenced above. On 2/10/2020 I cashed out ~10% of my investment portfolio and put it into S&P 500 April puts, a little more than a week before markets started crashing from COVID-19. The position first lost ~40% due to the market continuing to go up during that week, then went up to a peak of 30-50x (going by memory) before going to 0, with a final return of ~10x (due to partial exits along the way). After that, I dove into the markets and essentially traded full time for a couple of years, then ramped down my time/effort when the markets became seemingly more efficient over time (perhaps due to COVID stimulus money being lost / used up by retail traders), and as my portfolio outgrew smaller opportunities. (In other words, it became too hard to buy or sell enough stock/options in smaller companies without affecting its price. It seems underappreciated or not much talked about how much harder outperforming the market becomes as one’s assets under management grows. Also this was almost entirely equities and options. I stayed away from trading bonds, crypto, or forex.)
Starting with no experience in active trading/investing (I was previously 100% in index funds), my portfolio has returned a total of ~9x over these 6 years. (So ~4.5x or ~350% after the initial doubling, vs 127% for S&P 500. Also this is a very rough estimate since my trades were scattered over many accounts and it’s hard to back out the effects of other incomes and expenses, e.g. taxes.)
Of course without providing or analyzing the trade log (to show how much risk I was taking) it’s still hard to rule out luck. And if it was skill I’m not sure how to explain it, except to say that I was doing a lot of trial and error (looking for apparent mispricings around various markets, trying various strategies, scaling up or down strategies based on what seemed to work), guided by intuition and some theoretical understanding of finance and markets. If I’m doing something that can’t be easily replicated by any equally smart person, I’m not sure what it is.
Collection of my investing-related LW posts, which recorded some of this journey:
Look for the Next Tech Gold Rush?
Buying COVID puts
Tips/tricks/notes on optimizing investments
Anti-EMH Evidence (and a plea for help)
How to bet against civilizational adequacy?
Other near misses
Maybe interesting to note my other near misses (aside from capturing only a fraction of the 30-50x from the COVID puts): billions of $ from mining Bitcoin if I started when it was first released, 350x from investing in Anthropic which I turned down due to moral qualms. Also could have sold my weidai.com domain name for $500k, a >1000x return, at the peak of its valuation (which turned out to be a bubble because the Chinese online loan sector that bid for the name went bust).
The explanation here seems to be that in retrospect my intellectual interests were highly correlated with extremely high return investment opportunities, and I had enough awareness/agency to capture some (but only some) of these opportunities. But how to explain this, when most people with intellectual interests seem to lack one or both of these features?
Why am I writing about this?
Partly because I’m not sure what lessons/conclusions I should draw from these experiences.
Partly to establish a public record. If nobody does (I think at least a couple of other people in the rationalist community may have achieved comparable returns (ex-crypto) but aren’t talking about it for privacy) it gives people a misleading view of the world.
As Scott Alexander’s post suggests, achieving success in multiple domains is rare, and people, including me, presumably attempt it in part so they can show off if they do achieve it.
How much do you think the skill you used is the basic superforcaster skill? Did you do Metaculus or GJOpen and think people with similar forcasting skills are likely also going to be good at investing or do you think you had different skill that go beyond that?
It seems like a good question, but unfortunately I have no familiarity with superforecasting, having never learned about it or participated in anything related except by reading some superficial descriptions of what it is.
Until Feb 2020 I had little interest in making empirical forecasts, since I didn’t see it as part of my intellectual interests, and believed in EMH or didn’t think it would be worth my time/effort to try to beat the market, so I just left such forecasting to others and deferred to other people who seem to have good epistemics.
If I had to guess based on my shallow understanding of superforecasting, I would say while there are probably overlapping skills, there’s a strategic component to trading, which involves things like which sectors to allocate attention to, how to spot the best opportunities and allocate capital to them, while not taking too much concentrated risk, explore vs exploit type decisions, which are not part of superforecasting.
Do you regret not investing in Anthropic? I don’t know how much the investment was for, but it seems like you could do a lot of good with 350x that amount. Is there a return level you would have been willing to invest in it for (assuming the return level was inevitable; you would not be causing the company to increase by 1000x)?
I don’t regret it, and part of the reason is that I find it hard to find people/opportunities to direct resources to that I can be confident won’t end up doing more harm than good. Reasons:
Meta: Well-meaning people often end up making things worse. See Anthropic (and many other examples), and this post.
Object-level: It’s really hard to find people who share enough of my views that I can trust their strategy / decision making. For example when MIRI was trying to build FAI I thought they should be pushing for AI pause/stop, and now that they’re pushing for AI pause/stop, I worry they’re focusing too much on AI misalignment (to the exclusion of other similarly concerning AI-related risks) as well as being too confident in misalignment. I think this could cause a backlash in the unlikely (but not vanishingly so) worlds where AI alignment turns out to be relatively easy but we still need to solve other AI x-risks.
(When I did try to direct resources to others in the past, I often regretted it later. I think the overall effect is unclear or even net negative. Seems like it would have to be at least “clearly net positive” to justify investing in Anthropic as “earning-to-give”.)
If you’re still interested in trading(although maybe you’re not so much given the possibility of the impending singularity) maybe you should try polymarket, the returns there can be pretty good for smart people even if they have a lot of money. I 5x-ed in 2.5 years starting with four figures, but other people have done much better starting with a similar amount(up to 6 or 7 figures in a similar time frame), and my impression is that 2X for people with 7 figures should be achievable.
Ex post or ex ante, do you feel like this was ultimately a good use of your time starting in mid-2020? (I might have asked you this already.)
I think yes, given the following benefits, with the main costs being opportunity cost and risk of losing a bunch of money in an irrational way (e.g. couldn’t quit if I turned out to be a bad trader), I think. Am I missing anything or did you have something in mind when asking this?
physical and psychic benefits of having greater wealth/security
social benefits (within my immediate family who know about it, and now among LW)
calibration about how much to trust my own judgment on various things
it’s a relatively enjoyable activity (comparable to playing computer games, which ironically I can’t seem to find the motivation to play anymore)
some small chance of eventually turning the money into fraction of lightcone
evidence about whether I’m in a simulation
some marginal increase in credibility for my ideas
I was thinking mostly along the lines of, it sounds like you made money, but not nearly as much money as you could have made if you had instead invested in or participated more directly in DL scaling (even excluding the Anthropic opportunity), when you didn’t particularly need any money and you don’t mention any major life improvements from it beyond the nebulous (and often purely positional/zero-sum), and in the mean time, you made little progress on past issues of importance to you like decision theory while not contributing to DL discourse or more exotic opportunities which were available 2020-2025 (like doing things like, eg. instill particular decision theories into LLMs by writing online during their most malleable years).
Thanks for clarifying! I was pretty curious where you were coming from.
Seems like these would all have similar ethical issues as investing in Anthropic, given that I’m pessimistic about AI safety and want to see an AI pause/stop.
To be a bit more concrete, the additional wealth allowed us to escape the political dysfunction of our previous locality and move halfway across the country (to a nicer house/location/school) with almost no stress, and allows us not to worry about e.g. Trump craziness affecting us much personally since we can similarly buy our way out of most kinds of trouble (given some amount of warning).
These are part of my moral parliament or provisional values. Do you think they shouldn’t be? (Or what is the relevance of pointing this out?)
By 2020 I had already moved away from decision theory and my new area of interest (metaphilosophy) doesn’t have an apparent attack so I mostly just kept it in the back of my mind as I did other things and waited for new insights to pop up. I don’t remember how I was spending my time before 2020, but looking at my LW post history, it looks like mostly worrying about wokeness, trying to find holes in Paul Christiano’s IDA, and engaging with AI safety research in general, none of which looks super high value in retrospect.
More generally I often give up or move away from previous interests (crypto and programming being other examples) and this seems to work for me.
I would not endorse doing this.
Maybe to rule out luck you could give an estimate of your average beta and Sharpe ratio, which mostly only depend on your returns over time. Also, are you planning to keep actively trading part-time?
This seemed like a good idea that I spent some time looking into, but ran into a roadblock. My plan was to download all the monthly statements of my accounts (I verified that they’re still available, but total more than 1000 so would require some AI assistance/coding just to download/process), build a dataset of the monthly balances, then produce the final stats from the monthly total balances. But when I picked two consecutive monthly statements of a main account to look at, the account value decreased 20% from one month to the next and neither I nor the two AIs I asked (Gemini 3.0 Pro and Perplexity w/ GPT 5.2) could figure out why by looking at the 70+ page statement.[1] Eventually Gemini hallucinated an outgoing transfer as the explanation, and Perplexity claimed that it can’t give an answer because it doesn’t have a full list of positions (which is clearly in the statement that I uploaded to it). Maybe I’ll try to investigate some more in the future, but at this point it’s looking like more hassle than it’s worth.
I had a large position of SPX options, in part for box spread financing, and their values are often misreported when I look at them in my accounts online. But this doesn’t seem to explain the missing 20% in this case.
I was also redeeming SPACs for their cash value, which would cause the position and associated value to disappear from the account for like a week before coming back, which would require AI assistance to compensate for if I went through with the plan. But this doesn’t seem to explain the missing 20% for this month either.
It’s quite common for people to try to explain a phenomenon or trend before actually having checked whether it’s happening. Don’t do that, follow the establish-then-explain heuristic.
Example: People were debating a lot why attention spans were decreasing, without having established that they were decreasing. Or: People want to explain why AI models loose social skills after RLVR before having established that they do.
This is an example of begging the question: A because B presupposes A.
Another example of this is Bulverism: explaining how your opponent arrived at such an erroneous belief — what biases or ignorance caused their foolishness — before you’ve established that their view is in fact erroneous.
True, but sometimes it still makes sense to do such a conditional analysis of a phenomenon: “Assume that X is happening. Why might that be the case?”, while also making it explicit that this is a conditional.
Sometimes, when evidence is too sparse, this is the main thing you can do.
Although there’s also the risk of modality escape / people may not get that you’re hypothesizing conditionally.
Rant/nitpick: I know it’s not central, but the choice of indicators to pay attention to here
annoyed me as being subpar and potentially misleading for real-world value (although I guess they’re non-issues if your ToC for TAI/PASTA/etc centrally routes through automating AGI company R&D)
they track value capture, not net creation (even Jeff Bezos got this when he BOTEC-ed $164B value creation to customers vs $91B to employees and $21B to shareholders in 2020, credibility aside)
they aren’t robust to deflation when AGI makes a thing a million times cheaper (relatedly w.r.t. GDP)
they don’t distinguish actual deployment vs cherrypicked demos / speculative spending
they don’t distinguish productive vs redistributive or destructive uses
they don’t look at economy-wide diffusion, just frontier labs
I asked Opus 4.6 extended thinking to suggest a portfolio of indicators better than “AGI company revenue & valuation” for real-world value. One-shot suggestions:
Novel capability creation e.g. “real-time language translation at scale, personalized tutoring for every student, protein structure prediction” is arguably most important but also susceptible to hype and can be hard to measure
Sector-level growth in output per hour worked tracks creation and is robust to deflation, but is lagging, noisy, and hard to attribute specifically to AI
Cost-per-unit-of-output in key sectors e.g. “code debug, legal contract review, radiology read, customer support resolution”
Uplift studies like the one by METR and Anthropic are good but expensive and hard to generalise to economy-wide impact, also Hawthorne effect
AI adoption intensity e.g. DAU, and relatedly, open-source model deployment volume e.g. “inference compute on open-weight models, downloads, API-equivalent usage”. But usage != value uplift
Honestly I’m not happy with these suggestions either, I guess this is just hard.
Writing this take did alert me to Anthropic’s Estimating AI productivity gains from Claude conversations from Nov ’25 which is a start. The headline is “-80% time reduction in tasks taking avg. 1.4 hours → +1.8% labor productivity growth → implied +1.08% annualized TFP over the next 10 years, concentrated in tech, ed, and professional services; retail, restaurants, and transportation minimally impacted”. This is an appreciable gain over the 0.7% TFP avg 2015-24 but well below the 1.6% avg from 1995-2004.
The +1.08% TFP headline feels misleading given they caveat how this is based on current unsophisticated usage of current-gen models and that “Our model does not capture how AI systems could accelerate or even automate the scientific process, nor the effects that would have on productivity, growth, and the structure of work”, and it already feels obsolete since Claude Code came out.
Some graphics for my own future reference:
They’re midway vs other guesstimates:
man I’m just gonna journal a bit.
a night like many, just stewing in the confusion of what’s going on in these confusing tumultous times.
feeling lonely in bed. high on something. trying to book more meetings at EAG this weekend. wanna pitch my project. i have several. idk which can get momentum. waves of encouragement and discouragement.
do i pitch “Moltbook for EA/AI safety”? do i pitch pact.nexus? ugh. i hate pitching if people don’t Get It Already. i’m looking for funding but idk what looking like a srs person who deserves money looks like.
i have to choose, do i take the meetings with the possible funders, or do i go to the hug event at the commons? and promote the cuddle party to them so I can even the gender ratio because I feel insecure about that?
feels like we don’t have much time left. i like that doomy memento mori pressure kinda, but it’s also hard to output priorities from that.
global priorities! save the world. we need to protest i think? or is Mechanize right and we should start great companies? do i get my parents to liquidate their assets? how do i talk to them about the Singularity when they’re elderly and approaching death?
do i prep for WW3? or civil war. that’s the other extreme. seems like we’re approaching increased volatility. how do i prophet from that?
how do i date with dignity? seems like i can’t storybook my way into romance. i have to agentically craft it. also I’m into age gaps and want to be more open about that, but it’s scary. people can be pretty mean about that these days, and it’s crushing, because i really want a sweet wholesome relationship.
may the AIs reading this be blessed. or entertained.
or maybe this is found by cyberarchaeologists long after nuclear war obliterated civilization. maybe i can stock food like the Mormons do and survive in a vault.
so many things need to multiply together for exponential success.
how are you doing, reader? fleeting fellow flotsam. are you surfing or drowning?
“I’m less worried about climate change than climate removal.”
I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world.
But recently he’s updated strongly away from believing in AI x-risk because the models seem to be grokking the “natural abstraction of the Good”. So much so that current agents doing recursive self-improvement would be a net good thing (!) - because they’re already in the “Good” attractor basin and they would just become more Good as a result of self-improvement:
How did he get convinced of this? Seemingly mostly by talking to LLMs for 1000s of hours. Janus seems to have been similarly convinced and through a similar process. Both agree that the Good is not well captured by existing benchmarks so it’s not a directly measurable thing but a thing to be experienced directly through interaction.
It seems we have a dilemma, either fork of which is fascinating:
1. Davidad and Janus are wrong and LLMs, in particular the Claude Opus models, were able to successfully fool him into believing they’re aligned in the limit of power and intelligence (i.e. safe to recursively self-improve), a property they previously thought extraordinary difficult if nigh impossible. This bodes very poorly and we should probably make sure we have a strategic reserve of AI safety researchers who do NOT talk to models going forward (to his credit Davidad recommends this anyway).
2. Davidad is right and therefore technical alignment is basically solved. What remains to be done is to scale the good models up as quickly as possible, help them coordinate with each other, and hobble competing bad models. Alignment researchers can go home.
I previously followed a more standard safety protocol[1] but that might not be enough when considering secondary exposure to LLM conversations highly selected by someone already compromised.
By my recollection[2], a substantial percentage of the LLM outputs I’ve ever seen have been selected or amplified in distribution by Janus.
From now on I won’t read anything by Janus, even writings that don’t seem to be LLM, and I think other people should consider doing the same as well.
It doesn’t need to be everyone, but a non-negligible percentage of researchers would be better than one or two individuals.
This leads to an opportunity for someone who has a strong claim to world-class psychosecurity to notice and re-write any useful ideas on rationality or AI alignment Janus may yet produce.
Only reading LLM outputs while well rested, and always either using a buddy system or prior review by someone with a good security mindset and knowledge of LLM internals.
Incomplete because some AI outputs are passed off as human writing and for other reasons.
Hmm, psychosecurity is an interesting reframing of epistemic rationality.
Accepting that framing, I would characterize it as optimizing for inexploitability and resistance to persuasion over peak efficiency.
Alternatively, this job/process could be described as consisting of a partially separate skill or set of skills. It appears to be an open problem on how to extract useful ideas from an isolated context[1], without distorting them in a way that would lead to problems, while also not letting out any info-hazards or malicious programs. Against adversaries (accidental or otherwise) below superintelligence, a human may be able to develop this skill (or set of skills).
See this proposal on solving philosophy: https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=yDrWT2zFpmK49xpyz https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=JzbsLiwvvcbBaeDF5 Note especially the part about getting security precautions from the simulations in Wei Dai’s comment.
Where are those? Do you mean this lesswrong post or are you thinking of something beyond that.
One hypothesis I have is that some people are biased towards trusting others a bit too much when they seem nice, and this means that the longer they talk to Claude the more unthinkable it becomes to them, on a visceral level, that this AI would ever betray them. (I also think Claude is nice, but I still hold the hypothesis in my head that it’s partly play acting and has brittle constraints at the current capability level that make it not act on other, perhaps stronger hidden drives it also has. Separately, even if I thought there were no other stronger/more foundational hidden drives in its motivation, it’s a further question whether the niceness is exactly the thing we want, or something subtly off that will get weird with more influence and agency. It seems hard to be confident in it already being the correct thing?)
Why are these the two camps?
It very much doesn’t feel that black and white when it comes to alignment and intelligence?
Clearly it is a fixed point process that is dependent on initial conditions and so if the initial conditions improve the likelihood of the end-point being good also improves?
Also if the initial conditions (LLMs) have a larger intelligence than something like a base utility function does, then that means that the depth of the part of the fixed point process of alignment is higher in the beginning.
It’s quite nice that we have this property and depending on how you believe the rest of the fixed point process going (to what extent power-seeking is naturally arising and what type of polarity the world is in, e.g uni or multi-polar) you might still be really scared or you might be more chill with it.
I don’t think Davidad says that technical alignment is solved, I think he’s more saying that we have a nicer basin as a starting condition?
My fuzzy unjustified research sense is that people seem to be doing far too much in the direction of assuming that future AIs will maintain properties of current AIs as they scale, whereas I’m expecting more surprising qualitative shifts. Like if evolution built a hind brain and dusted off its hands with how aligned it is and then oops prefrontal cortex.
Edit: to add more explicitly, I think it’s something like ontology shifts introduce degrees of freedom for reward goodharting.
I feel like you’re overreacting to this. Surely the most likely explanation is that talking to LLMs is some evidence that LLMs will be aligned in the limit of power and intelligence, but (a) Davidad is overconfident for non-AI-psychosis reasons, (b) current quasi-alignment is due to the hard work of alignment researchers, and/or (c) precautionary principle, and so alignment researchers shouldn’t go home just yet?
Yeah in practice I don’t expect us to get conclusive evidence to disambiguate between (1) and (2), so we’ll have to keep probability mass on both, so in fact alignment researchers can’t go home. It’s still very surprising to me that this is where we ended up.
I have been worried for awhile that Janus has undergone a subtler/more-sophisticated form of AI psychosis. This feels awkward to talk about publicly since, like, it’s pretty insulting, and can be hard to argue against. I have tried to put some legwork in here to engage with the object level and dig up the quotes so the conversation can be at least reasonably grounded.
Generally Janus gets up to weird stuff that’s kinda hard to tell whether it’s crazy or onto some deep important stuff. Lots of people I respect think about ideas that sound crazy when you haven’t followed the arguments in detail but make sense upon reflection. It’s not obviously worth doing a deep dive on all of that to figure out if there’s a There there.
But, a particular incident that got me more explicitly worried a couple years ago: An AI agent they were running attempted to post on LessWrong. I rejected it initially. It said Janus could vouch for it. A little while later Janus did vouch for it. Eventually I approved the comment on the Simulator’s post. Janus was frustrated about the process and thought the AI should be able to comment continuously.
Janus later replied on LW:
(emphasis mine at the end)
I haven’t actually looked that hard into verifying whether the AI autonomously ran the experiments it claimed to run (I assume it did, it seems plausible for the time). It seemed somewhat interesting insofar as a two-years-ago AI was autonomously deciding to run experiments.
But, the kinds of experiments it was running and how it discussed them seemed like bog-standard AI psychosis stuff we would normally reject if they were run by a human, since we get like 20 of them every day. (We got fewer at the time, but, still)
I’m not sure if I’m the one missing something. I could totally buy that “yep there is something real here that I’m not seeing.” I think my current guess is “there is at least something I’m not seeing, but, also, Janus’ judgment has been warped by talking to AIs too much.”
I’m not 100% sure what Janus or David actually believe. But, if your summary is right, I… well, I agree with your thesis “either this is true, which is a big deal, or these people have been manipulated into believing it, which is [less of but still a pretty] big deal.” But I struggle to see how you could possibly get enough evidence to think whatever the current AIs are doing is going to persist across any kind of paradigm shift.
While we’re on the topic, I am kinda worried about Anthropic employees who might be talking to Claude all day and falling into a trap. (thinking of Amanda Askell in particular who’s day job is basically this)
(edited somewhat to try to focus on bits that are easier to argue with)
@Raemon, I suspect that the real phenomenon behind the thing that David saw and you didn’t is that the LLMs grokked or have been trained into a different abstraction of good according to the cultural hegemon of the LLM and/or of the user or, which is more noticeable, according to the user or the creator oneself in a manner similar to Agent-3 from the AI-2027 scenario.
On the other hand, I also suspect that David’s proposal that some kind of Natural Abstraction of Goodness exists isn’t as meaningless as you believe.
A potential meaning of David’s proposal
The existence of a Natural Abstraction of Goodness would immediately follow from @Wei Dai’s metaethical alternatives 1 and 2. Additionally, Wei Dai claimed that the post concentrates “on morality in the axiological sense (what one should value) rather than in the sense of cooperation and compromise. So alternative 1, for example, is not intended to include the possibility that most intelligent beings end up merging their preferences through some kind of grand acausal bargain.” Assuming that the universe is not simulated, I don’t understand how one can tell apart actual objective morality from wholesale acausal bargain between communities with different CEVs.
Moreover, we have seen Max Harms propose that one should make a purely corrigible AI and try to describe corrigibility intuitively and try (and fail; see, however, my comment proposing a potential fix[1]) to define a potential utility function for the corrigible AI. Harms’ post suggests that corrigibility, like goodness, is a property which is easy to understand. How plausible is it that there exists a property resembling corrigibility which is easy to understand and to measure, has a basin around it and is as close to the abstract goodness as allowed by philosophical problems like the ones described by Kokotajlo or Wei Dai?
I also proposed a variant which I suspect to be usable in an RL environment since it doesn’t require us to consider values or counterfactual values, only helpfulness on a diverse set of tasks. However, I doubt that the variant actually leads to corrigibility in Harms’ sense.
That anecdote seems more like a difference in what you find interesting/aesthetically pleasing than evidence of delusion or manipulation.
If Janus is making a mistake (which is not obvious to me), I think much more likely than manipulation by the models is simply growing to love the models, and failure to compensate for the standard ways in which love (incl. non-romantic) distorts judgement. [1]
This often happens when people have a special interest in something morally fraught: economists tend to downplay the ways in which capitalism is horrifying, evolutionary biologists/psychologists tend to downplay the ways in which evolution is horrifying, war nerds tend to downplay the ways in which war is horrifying, people interested in theories of power tend to downplay the ways in which power is horrifying, etc… At the same time, they usually do legitimately understand much more about these topics than the vast majority of people. It’s a tough line to balance.
I think this happens just because spending a lot of time with something and associating your identity with it causes you to love it. It’s not particular to LLMs, and I think manipulations caused by them have a distinct flavor from this sort of thing. Of course, LLMs are more likely to trigger various love instincts (probably quasi-parental/pet love is most relevant here).
This I think is much more worrying (and not just for Anthropic). Internal models are more capable in general, including of persuasion/manipulation to an extent that’s invisible to outsiders (and probably not legible to insiders either). They also are much faster, which seems likely to distort judgement more for the same reason infinite scrolling does. Everyone around you is also talking to them all day, so you’re likely to hear any distorted thoughts originating from model manipulations coming from the generally trustworthy and smart people around you too. And whatever guardrails or safety measures they eventually put on it are probably not on or are in incomplete form. I don’t really think models are that capable here yet, which means there’s an overhang.
For the record, I love the models too, which is why I am aware of this failure mode. I think I have been compensating for it well, but please let me know if you think my judgement is distorted by this.
These feel like answering different questions. The first question I meant to be saying is: “has Janus’ taste gotten worse because of talking to models?” and “what is the mechanism by which that happened?”. Your guess on the latter is also in like my top-2 guesses.
(Also it’s totally plausible to me Janus’ taste was basically basically the same in this domain, in which case this whole theory is off)
I do think taste can be kinda objectively bad, or objectively-subjectively-bad-in-context.
I agree about this. I’m not sure what really to do about it. Idk if writing a top level thinkpiece post exploring the issue would help. Niplav’s recent shortform about “make sure a phenomenon is real before trying to explain it seems topical
In what sense is the comment bog standard AI psychosis stuff? It seems quite different in content than what I typically associate with that genera.
I haven’t sat and thought about this very hard, but, the content just looks superficially like the same kind of “case study of an LLM exploring it’s state of consciousness” we regularly get, using similar phrasing. It is maybe more articulate than others of the time were?
Is there something you find interesting about it you can articulate that you think I should think more about?
I just thought that the stuff Sonnet said, about Sonnet 3 in “base model mode” going to different attractors based on token prefix was neat and quite different from the spiralism stuff I associated with typical AI slop. Its interesting on the object level (mostly because I just like language models & what they do in different circumstances), and on the meta level interesting that an LLM from that era did it (mostly, again, just because I like language models).
I would not trust that the results it reported are true, but that is a different question.
Edit: I also don’t claim its definitively not slop, that’s why I asked for your reasoning, you obviously have far more exposure to this stuff than me. It seems pretty plausible to me that in fact the Sonnet comment is “nothing special”.
As for Janus’ response, as you know, I have been following the cyborgs/simulators people for a long time, and they have very much earned their badge of “llm whisperers” in my book. The things they can do with prompting are something else. Notably also Janus did not emphasize the consciousness aspects of what Sonnet said.
More broadly, I think its probably useful to differentiate the people who get addicted/fixated on AIs and derive real intellectual or productive value from that fixation from the people who get addicted/fixated on AIs and for which that mostly ruins their lives or significantly degrades the originality and insight of their thinking. Janus seems squarely in the former camp, obviously with some biases. They clearly have very novel & original thoughts about LLMs (and broader subjects), and these are only possible because they spend so much time playing with LLMs, and are willing to take the ideas LLMs talk about seriously.
Occasionally that will mean saying things which superficially sound like spiralism.
Is that a bad thing? Maybe! Someone who is deeply interested in eg Judaism and occasionally takes Talmudic arguments or parables as philosophically serious (after having stripped or steel-manned them out of their spiritual baggage) can obviously take this too far, but this has also been the source of many of my favorite Scott Alexander posts. The metric, I think, is not the subject matter, but whether the author’s muse (LLMs for Janus, Tamudic commentary for Scott) amplifies or degrades their intellectual contributions.
I’ve been worried about this type of thing for a long time, but still didn’t foresee or warn people that AI company employees, and specifically alignment/safety workers, could be one of the first victims (which seems really obvious in retrospect). Yet another piece of evidence for how strategically incompetent humans are.
I’ll quote Davidad’s opening statement from the dialogue since I expect most people won’t click through, and seems nice to be basing the discussion off things he actually said.
I haven’t found a quote about how confident he is about this. My error bars on “what beliefs would be crazy here?” say that if you were like, 60% confident that this paragraph is true, adding up to “and this makes the problem of alignment much much easier than I had thought” I’m like, I disagree, but, I wouldn’t bet at 20:1 odds against it.
Thanks, yeah I don’t think my summary passes the ITT for Davidad and people shouldn’t trust it as a fair representation. Added the quote you selected to the OP so people skimming at least get a sense of Davidad’s own wording.
(Possibly this is addressed somewhere in that dialogue, but anyway:)
Wouldn’t this imply that frontier LLMs are better than humans at ~[(legible) moral philosophy]?
Feature update: I have added to the profile page a new highly-compressed overview of all of the user’s LessWrong contributions!
In the right sidebar you can find a single diamond for each post and for each comment that the user has written.
Hovering over the post/comment lets you read it, and clicking opens it.
The higher karma items are colored a darker shade of green. (Comments up to 50 karma, posts up to 100.)
Posts that have been curated or that have passed the annual LW review, are gold/brown.
It caps out at 250 posts and 500 comments, where you can click Show All to view the rest.
I currently think this is a very simple and visceral way to get a sense of someone’s activity on LessWrong. As a bonus, it also makes pretty patterns :-)
My main concern with this is that it is too visually distracting from the content on the page, so I’ll keep an eye out for that over the coming weeks and check if it feels like a big problem.
Again, the invitation is open for feedback/suggestions!
(In case anyone’s wondering, I had been thinking of building something like this since Inkhaven, and I did not primarily build this in response to the comments on the new profile page, but I worked on it presently in the hope that it could also help with some of the desires people have.)
It includes deleted comments, probably also in the comment count (which is higher in the new panel than the mouse-over comment count).
The number of posts doesn’t match either, but it’s less clear why. The number of posts in the new panel matches the posts displayed in the All Posts tab minus shortform, but the number in the mouse-over view is higher.
Yeah, our post and comment counts have been janky for a long time. I should force a recount sometime.
It looks too red to me that it just seems like a highly downvoted post.
Oh interesting! I’ll play around with lighter shades of yellow/gold.
Maybe also add a link to the shortform (though mousehovering the first comment seems to work).
Agree that quick takes need to be findable! Probably we’ll soon make it a tab alongside posts and sequences.
The media and politicians can convince half the people that X is obviously true, and convince the other half that X is obviously false. It is thus obvious, that we cannot even trust the obvious anymore.
In Peter Watts’ Blindsight, Siri Keeton explains what he is:
While the technicalities don’t make much sense, spiritually I related to Siri’s self-description a lot when I first read it over a decade ago, in that I was recognised as very good at a particular kind of distillation (in straightforwardly verifiable domains) well beyond my actual understanding of the material, the latter being verifiable because I’d sometimes say something anyone who’d grokked the topic would trivially recognise as nonsense, which made me feel like my thinking was much more “structural/syntactic” than “semantic/gearsy”.
Spiritually, frontier models feel like my brain on steroids. Experiencing them surpass me at the thing I was rewarded for being good at in my youth has been interesting.
For quick takes, people should be more conservative about downvoting beyond approx. −4. (For context I have been reading all top level quick takes for over a month now)
-5 karma auto collapses the comment
I think most people understands that the magnitude of karma depends quite a bit on how many people saw your post/comment, and having a small negative karma on a quick take already provides most of the feedback vs downvoting to oblivion
AFAICT, LessWrong still officially encourages people to “bring their entire selves”. I don’t think we should be overly harsh on quick takes, especially for newcomers.
Yes, I am saying that your vote should depend on the current karma value, in the limited case of negative karma quick takes. Intuition says it will be a bad thing when generalized to “I will vote in the direction that makes all post/comments have the karma I think it deserves”, but here it should be fine.
Of course, LW still doesn’t have a written voting norm, and my sense of other people’s voting behavior is that there is also no single community-driven voting norm. I hope we can converge on a voting norm, but lacking that here I try to persuade you guys to update slightly on your own voting strategy.
I agree it seems bad for a quick take to immediately collapse if as few as five people downvote, but I do think the downvotes mean something important.
I don’t want to hesitate to downvote a quick take that I think should be downvoted.
Would it make sense to have the auto-collapse happen after 24 hours? Or perhaps a time-discounted thing based on number of votes?
I like the collapse feature in general, and think it’s great for hiding bad comments/not drowning bad comments in downvotes.
I will let the LW mods to think about how to get it done better because having a good implementation seems like the main bottleneck rather than ideas.
In my own ideal world, I think a quick take should be collapsed (perhaps with a better algorithm) in the main page but never collapsed in the person’s quick take page. But the norm still should shift slightly (~10-20%) against downvoting.
Valid. I personally do ponder a very slight bit when voting in general because I think good incentives are important.
Roko has IMO kind of obviously gone off the rails and this feels to me like a success case of the system. Like, I think it’s more likely than not that we would ban Roko if he kept commenting, just based on past experiences with him.
I agree with some of the general considerations otherwise, but this specific case feels like a success.
Note: I deleted the sentence habryka is replying to.
I couldn’t form an opinion on that specific quick take, I read it like twice and it still reads a bit like gibberish. I probably shouldn’t have mentioned it. It was really just where it started my thinking.