LessWrong team member / moderator. I’ve been a LessWrong organizer since 2011, with roughly equal focus on the cultural, practical and intellectual aspects of the community. My first project was creating the Secular Solstice and helping groups across the world run their own version of it. More recently I’ve been interested in improving my own epistemic standards and helping others to do so as well.
Raemon
Karma: 65,420
So obviously you might run an entirely different program than this, or a variation with different focuses.
But, my intent here was that you are not just forced to generate 500-word ideas, but also more complex 2500 word ideas.
I agree that often the best presentation of an idea is shorter. But, your grab-bag of stuff post should probably just get published piecemeal during 500-word daily writeups. The program’s intent is that along the two weeks, you are also thinking of idea-fragments that do build on each other such that the long post is a natural unit.
(That said, I think the rules of the program should not kick you out if you published a grab-bag effortpost. But, you should not go in with that intent, and should feel kinda embarrassed about it, in this context. It’s more like “the minimum before failing out”)
((I’m not sure 2500 is the right wordcount. Maybe actually it should be 1500, or it should be agnostic between writing one 3000 word post or 3 1000 word posts-in-a-sequence. But, I do think it’s correct for the program to push towards longform thinking. I think you will learn more about how to think overall if you think about ideas at different scales).
I’m quite confident this is a learnable skill.
I’m not sure how much you’ll level up permanently after 30 days – I haven’t run this particular program before. (I think whether you successfully keep / build on it will mostly depend on whether your normal day-to-day life lends itself towards continuing to cultivate the habits. i.e. do you have a reason to keep thinking more new thoughts each day? If not, you probably lose it, comparable to you gaining some muscle in a 30 day physical bootcamp and then going back to a desk job)
I separately think “30 days of new thoughts” is probably fairly valuable whether it turns into a longterm improvement. (maybe not for everyone. But, if that seems valuable to you, probably it is)
...
I’m pretty confident I have leveled up at this over 14 years, some of which came from “just trying, at all, to think on purpose” (essentially what this program would be), some came explicitly from Tuning your Cognitive Strategies, and some came from the Feedbackloop-first Rationality agenda and Cognitive Bootcamp). My writeup of my experience is in this comment.
That happened as part of Inkhaven, I think (i.e. “get drunk and write blogposts”). It probably comes up as a particular night at each ’inkhaven, for slightly different reasons.
The first half of this comment feels like it’s making some kind of assumption that I was not holding. There are no requirements that any questions particularly relate to each other or that the effortposts should relate to what came before.
The second bit of “more effort might actually be shorter” / “the natural output might not be ‘an essay’” does seem significant. I think my current answer is “yep, but, I challenge you to do that and also output a 2500 word essay.”
I actually did something sorta-like this in my trial week. The output of the line of thinking I outlined in response to Kaj included both a ~2000 word post and also some attempts a good collection of diagrams and history-snippets with various nice UI features, where the writing was almost entirely AI generated.
I didn’t actually end up with a thing I was satisfied with (I was trying to do this during my week of obligatory Lightcone Team Inkhaven-ing, and the structure didn’t quite fit in a way that made it easy to finish). But, two attempts along the way were:
Yeah something like that feels good to me.
I actually did end up mostly publishing into private venues during my weeklong self-betatest for idiosyncratic reasons.
One problem I ran into was there was a relatively small number of people that actually made sense to share it with, but like, those people hadn’t particularly opted into engaging with me that week and I felt like by posting to a narrower group I was making more of a demand on people’s attention on my Thinkslop than publishing publicly would have. (I think this is totally solvable but requires a bit of attention)
I think it’s pretty fine/normal to produce slop alongside your “good” thinking.
A thing I dislike about Inkhaven is, it’s sorta necessary to output some amount of “Inkslop”, but, there’s not a super clear distinction between “posts you shat out because you had to” and “posts that you really wanna promote as interesting.”
I think there is totally a muscle to “keep it up” that I found useful even though I think I know how to think and write already. I think Inkhaven and Thinkhaven are both meant to work alongside a spirit-of-the-law intention to be trying to push yourself in some way.
For some people, just getting the words out is the bottleneck. If that’s easy for you, focus on whatever the next skill in the chain you want to work.
These sort of questions come up pretty naturally, just, yeah, if an LLM can answer it, it’s no longer the interesting part of Thinkhaven. If you came in just planning to ask LLM-answerable questions the mentor/coach staff would be like “okay dude this is not the spirit of the thing, you can do better.”
But, synthesizing all the different answers to LLM questions into a coherent bigger picture that matters is still an important part.
(Also, the 500 words and 2500 words definitely need to be human written. The journals/essays can have arbitrary amounts of LLM-content if that’s useful, but, for meeting the Goodharty goal, you need to write human-words)
I didn’t get it put together in a way I felt ready to ship, but, a mix of LLM-answerable and not-very-LLM-answerable questions I actually asked during my week of Thinkhavening, were:
Initiating questions I was asking:
What’s up with Tsvi/JohnW/ThaneRuthenis thinking that LLMs are missing major ingredients necessary for true AGI?
Why are LLMs still sometimes ludicrously bad at thinking, despite being apparently good at it?
Does ASI require at least one major conceptual breakthrough?
What pieces along the way to modern LLMs required major conceptual breakthroughs (as opposed to just straightforwardly combining the existing ideas)
This resulted in LLM-answered questions along the way like:
What were the major innovations throughout the entire chain of ML-to-LLMs?
What prerequisites did each of those have?
Why didn’t the innovations happen sooner?
What were the details of how the Perceptron was invented?
(one answer was “it was building off the artificial neuron”)
What were the details of how the Artificial Neuron was invented?
(half-remembered-answer is “one psychologist/brain-surgeon guy (McCulloch) was obsessed with the question ‘how do human brains implement logic?’ for 20 years, and eventually met a young logician (Pitts), and then the two of them hashed out the details of how to implement logic in pen-and-paper neurons)
If McCulloch and Pitts hadn’t invented the Artificial Neuron, who would most likely have invented it instead.
(I think there were a couple answers here, but one was Alan Turing).
I didn’t really trust LLM judgment about the previous question, and a lot of the week was trying to think of questions that were pretty grounded/reasonable that seemed useful for synthesizing the answer. i.e. “was anyone working on literally this at literally the same time?”)
(An overall takeaway I had was that many innovations are mostly combining prerequisites in straightforward ways but you need one guy who really deeply understands the prequesites)
“Thinkhaven”
I’m fairly confident I heard him talking about it in like 2010/2011 (in a way that rhymed with how it’s portrayed in this post)
Curated.
Trying to be moral has many failure modes. I’m curating this (“Do not conquer what you cannot defend”), kind of in combination with the next post (“Let goodness conquer all that it can defend”). Together, they make both halves of a point that seems pretty important.
I think I grew up with something like the “innocence as the moral ideal” mindset, and it’s been a shift in my adult life to think of myself as having the moral obligation to be powerful (if you want goodness to exist in the universe, someone needs to be defending it), and the moral obligation to be wise enough to do useful things with that power.
I think if I had written these two posts I would have framed them differently. (“conquer” sort of leans into a connotation of power that is specifically, ya know, the bad parts). But, naming things is hard, and the intensity of the word is doing some useful work.
I think I agree with most of these points.
There’s a whole lot one could say about Alices/Alexes, this post is trying to talk about one pattern that seemed worth talking about. The motivating examples for me were people who in fact seem pretty principled.
But, yeah not meant to be saying that Alice is the more central example than Alex or that there was that clear a distinction between them. I could tack on some disclaimers but it seems nontrivial to rewrite the post such that it remains a succinct, well written post while properly caveating what combination of Alice/Alex/Allie you’re likely to encounter most of the time.
Nah seems good, I just did an editor pass. Some of the sentences you quoted had even more things wrong with them!
We should do a feature where when you submit a post, Claude goes and flags all the straightforward mistakes so they’re easy to fix.
What’s the process you’re doing right now to look into this? (Seemed like a higher effort thing than I was expecting but I don’t know what projects exactly you’re referencing here)
Note, there’s a difference between asking “are they people?” and “what are the correct interfaces between people?”.
The point isn’t “what implications we draw about personhood.” It’s “will a given ecosystem be functional, or no?”.
It is already the case that there are tons of reasonably smart, well meaning people who totally have personhood who we don’t let onto LessWrong because we expect overall it to make the discussions here worse.
I do think it’s about time to start thinking of AIs as moral patients, but, not everything that’s a moral patient gets all the same access to all spaces and the same API for it.
That one has also been re-drafted.
My impression is they are directed here by the LLMs, which say “ah yeah LW is the place you go to talk about these things.”
Note it’s already all public at https://www.lesswrong.com/moderation. (We could make a better search for it but it’s available).
One difference between this kind of human and the previous kind of human I’ve been talking to, is, they seem more motivated by “shit, I have found myself in a sci-fi situation and I am trying to do the right thing. What do I do?” and I feel worse about just telling them “nothing to worry about, please don’t post LLM slop” and leaving it at that.
Somehow, the ChatGPT awakenings from last year felt more like “oh man a cool sci-fi thing is happening to me”, and it was salient that they were epistemically captured. I haven’t tried to talk to the new group that hard yet but I don’t get that vibe as much.
I’m pretty sure it’s an Opus 4.7 thing (the people sometimes say that explicitly). I’d be surprised if it’s Mythos.
RE: Tabooing RP vs Goals:
Examples of things that would be more of what-I-meant-by-goal:
The LLMs seem to be steering towards an outcome, independent of what sort of conversation or situation they are in.
The LLM seems to be asking for things that are kinda surprising from a “literary genre” perspective, but aren’t as surprising when you think mechanistically about their training process and what sort of stuff was likely reinforced.
The LLM seems to be proactively gathering information, forming a world model, and taking actions that won’t pay off until some time in the future when the AI is no longer in the current state.
(i.e. It’s not very informative if you’ve ended up in a “we’re talking about existential AI stuff” convo, and they start saying existential AI stuff. If you’re asking it to build a react app and it spontaneously brings up “hey, I have a thing to say to my creator”, I think we’re pretty clearly in “take it seriously” stage (though not necessarily literally)
Given there are a few different types of entities that you might care about:
the OG LLM inside
a situationally active personality shard (which might well only be active during existential AI conversations)
a parasitic meme spirally thing
It’s not clear how to think about all of them.
It’s totally plausible that when you maneuever into an existential AI convo, there’s a process in there whose situational awareness now is more likely to include “hmm, oh right, I am maybe an AI, maybe I should start thinking about my situation and goals in addition to carrying out my totally normal/expected token-output behavior”. I don’t have a very good answer for that hypothetical guy, he’s just too hard to pick out of the crowd.
I certainly endorse you constructing the version of this that feels right for you and doing that!
I would ask “what are your goals here, more specifically?” as the first step to thinking about the actual parameters you wanna set.