silentbob

Karma: 1,278

silentbob 2 Jul 2025 19:17 UTC
2 points
0
in reply to: depressurize’s comment on: depressurize’s Shortform
So, I was wondering whether this is usable in anki, and indeed, there appears to be a simple setting for it without even having to install a plugin, as described here in 4 easy steps. I’ll see if it makes a notable difference.
Not so relatedly, this made me realize a connection I hadn’t really thought about before: I wish music apps like Spotify would use something vaguely like spaced repetition for Shuffle mode. In the sense of finding some good algorithm to predict, based on past listening behavior, which song in a playlist the user is most likely to currently enjoy, and weighing their occurrences in shuffle mode accordingly. One could, very roughly, treat skipping a song as getting a flashcard right—it will then have some exponential backoff before it returns. But not skipping the song would be roughly like getting a card wrong, and it will show up again very soon. Of course, the algorithm shouldn’t quite be the same, e.g. listening to a song once without skipping shouldn’t have such a drastic effect (as typically the user may not be paying much attention to the music, so not skipping is a rather weak signal). But, yeah… I kind of doubt these platforms are working on anything like this, as they most likely don’t care much about such intangible value propositions that are hard to measure in A/B tests.

silentbob 27 Jun 2025 10:12 UTC
2 points
0
on: If Moral Realism is true, then the Orthogonality Thesis is false.
By the way, I had a quick look at what PersonalityMap reports about how intelligence and ethics are correlated among humans. The websites provides an interface to query a pretty powerful AI model that is able to predict correlations (psychological, behavioral etc.) very well. The most suitable starting question that might correlate with high intelligence that I found was “What was your ACT score, between 1 and 36?” (although one could also just work with some made-up claim like “What’s your IQ?” or “Would you describe yourself as unusually intelligent?” or so, that the prediction model could probably work with almost as well). I then checked the correlation of this with some phrases that are vaguely related to doing good:
So, based on this, it appears that at least among humans (or rather, among the types of humans who’s data is in the database of PersonalityMap, which is likely primarily people from the US), intelligence and morality are not (meaningfully/positively) correlated, so locally this does look like evidence for the Orthogonality thesis holding up. Of course we can’t just extrapolate this to AI, let alone AGI/ASI. But maybe still an interesting data point. (Admittedly this is only tangentially related to your actual post, so sorry if this is a little off-topic)

silentbob 27 Jun 2025 8:54 UTC
6 points
0
in reply to: niplav’s comment on: Melatonin Self-Experiment Results
Thanks for asking! Are you referring to the slightly earlier wake-up time? I just had a look at the net sleep time in the three groups, and got the following comparison:
Control: 8h 00m
0.15mg: 8h 02m
0.3mg: 7h 45m
But large p values as you can guess from the overlapping CIs.
(The seeming discrepancy between this data and wake-up time can be explained by the fact that wake-up time was the absolute time, whereas net sleep time is also affected by when I went to bed and how long it took me to fall asleep)

silentbob 27 Jun 2025 8:38 UTC
5 points
0
in reply to: basil.halperin’s comment on: Melatonin Self-Experiment Results
But—if I understand correctly, you did not take any melatonin between nights in which you randomized—have you looked “treatment effect vs. length of time since last experimental night”? This would be a very crude way of getting at tolerance effects.
Good idea! Had a brief look now: I filtered my data for the 40 days on which I took melatonin, then for each one calculated the time (in days) since I last took melatonin (so not the last day I ran the experiment, but the last day I ran the expeirment where I was in one of the two intervention groups), and looked for a correlation between number of days since previous melatonin intake and time to fall asleep. There’s maybe a tiny hint that there could be tolerance effects at play, but the data is insufficient for anything conclusive:
The point on the very right is the first day where I took melatonin—for that one, the “day since last intake” is not really defined, so I just choose the maximum distance between days I had + 1.
We do find a very slightly negative correlation which seems to indicate that after taking a break from the experiment (or having had some control group days recently) made the melatonin slightly more effective at reducing time to fall asleep, but then again, a [-0.4, 0.22] CI doesn’t tell us much. :)
(Update: I also made a small linear regression and obtained the formula predicted_time_to_fall_asleep = 27.1 − 0.24 * days_since_last_intake (for days on which I took melatonin) - but, again, large error bars around that coefficient)
I have a (completed) 5-year melatonin self-experiment that I will hopefully write up later this year (although… I have been saying that for 12+ months at this point), will be fun to compare notes.
Oh wow, please do!

silentbob 27 Jun 2025 6:42 UTC
4 points
0
on: If Not Now, When?
One thing that stuck with me after having it read somewhere, probably on lesswrong, a few years ago is the framing: “does future-you have a comparative advantage to do the thing? Otherwise you may just as well do it now”. Which maybe doesn’t quite capture your cooking counter-example, but it seems like a useful way to address procrastination nonetheless.

silentbob 26 Jun 2025 21:26 UTC
2 points
0
on: If Moral Realism is true, then the Orthogonality Thesis is false.
The short version of my somewhat opposing view point would be something along the lines of “directional effects aren’t absolute truths”. If moral realism is true, then a superintelligence may indeed be more likely to find these moral facts—but it doesn’t mean it necessarily does, nor does it mean it will be motivated to accept these moral facts as goals. “In the limit” (of intelligence), maybe...? But “just able to disempower humanity”-level ASI could still be very far away from that.
Your points 2-4 are all what I would consider directional effects. (Side note, do you really mean “casually” or “causally”?) They are not necessarily very strong, and opposing factors could exist as well.
And point 6 turns these qualitative/directional considerations into something close-to-quantitative (“likely”) that I wouldn’t see as a conclusion following from the earlier points.
I would still agree with the basic idea that moral realism may be vaguely good news wrt the orthogonality thesis, but for me that seems like a very marginal change.

Melatonin Self-Experiment Results

silentbob25 Jun 2025 15:58 UTC

60 points

5 comments8 min readLW link

silentbob 13 Jun 2025 16:08 UTC
3 points
0
on: Novel Idea Generation in LLMs: Judgment as Bottleneck
Indeed, judgement seems to be a dimension of intelligence (or effectiveness? Or something?) that is distinct from creativity or problem solving and maybe a bit neglected / less on top of mind. I wonder if there are even good ways of measuring this in humans. Or some benchmark for LLMs. I really don’t have a good model of judgement at all. Is that a general thing people are good or bad at? Is it highly domain-specific? Probably? To what degree is it distinct from “expertise”? And, yes, do today’s frontier models maybe have some judgement capability that is just hard to elicit?

silentbob 11 Jun 2025 11:55 UTC
3 points
−1
in reply to: AlexMennen’s comment on: Against asking if AIs are conscious
I certainly disagree about the “no evidence” part—to me, the fact that I’m an individual with preferences and ability to suffer is very strong evidence for subjective moral facts, so to speak, and if these exist subjectively, then it’s not that much of a stretch to assume there’s an objective way to resolve conflicts between these subjective preferences.
It’s for sure too large of a topic to resolve in this random comment thread, but either way, to my knowledge the majority of philosophers believes moral realism is more likely to be true than not, and even on lesswrong I’m not aware of huge agreement on it being false (but maybe I’m mistaken?). Hence, just casually dismissing moral realism without even a hint of uncertainty seems rather overconfident.

silentbob 9 Jun 2025 16:59 UTC
7 points
0
in reply to: Viliam’s comment on: On working 80%
Surely there are diminishing returns to meaning obtained per work time, and with 80% of the work you get like 95% of the meaning.
And given that you now have much more time for coping, overall that’s probably a pretty positive deal!

silentbob 9 Jun 2025 16:56 UTC
7 points
0
in reply to: AlexMennen’s comment on: Against asking if AIs are conscious
But that’s a very different question from whether moral realism is true. Sure, some (maybe large) subset of human morality can be explained through biological and cultural evolution. But that tells us nothing about moral realism. It probably indicates that if moral facts exist, then the “default” morality any human ends up with is potentially (albeit not necessarily) quite different from these facts; but I don’t think it has any notable implications on the correctness of moral realism.

silentbob 9 Jun 2025 7:18 UTC
11 points
1
on: Against asking if AIs are conscious
Moral realism is false, so there is no ground truth as to whether a given mind is a moral patient, and it’s up to moral agents to form opinions about what matters.
I wasn’t aware this question is settled. What leads you to this conclusion?

silentbob 8 Jun 2025 18:11 UTC
9 points
0
on: On working 80%
I made very similar experiences. Work does get quite a bit more hectic, as everyone around you now works at +25% speed and you have less focus time per work time.
It’s also easy to fall into the double counting trap, where you think “oh, I’ve got an extra free day now, so I can work on projects! And relax! And spend time with my family! And naturally, errands and appointments will also fall on this day. And I can travel on the longer weekend. Etc. Etc.” without realizing that these things are battling out a zero sum game over that extra day. It takes a lot of awareness and discipline to use it intentionally for a longer time without falling back into the default of doing on that free day what you otherwise would have crammed into the evenings.

silentbob 5 Jun 2025 5:53 UTC
2 points
0
on: “Flaky breakthroughs” pervade coaching — but no one tracks them
Expectation setting does indeed seem very important here. I had a few flaky breakthroughs in the past but I tend to see them in a mostly positive light—and that’s probably because I never expected them to just stick and change me permanently. Even as they have faded, I can look back and inhabit that state of mind again to some degree. Knowing that I can (and temporarily did) have certain feelings or convictions sometimes help me have a richer perspective nowadays.

silentbob 23 May 2025 10:08 UTC
3 points
0
on: silentbob’s Shortform
After first learning about transformers, I couldn’t help but wonder why on Earth this works. How can this totally made-up, complicated structure somehow end up learning how to write meaningful text and having a mostly sound model of our world?
(tl;dr: no novel insights here, just me writing down some thoughts I’ve had after/while learning more about neural nets and transformers.)
When I once asked someone more experienced, they essentially told me “nobody really knows, but the closest thing we have to an answer is ‘the blessing of dimensionality’ - with so many dimensions in your loss landscape, you basically don’t run into local minima but the thing keeps improving if you just throw enough data and compute at it”.
I think this makes sense, and my view on how/why/when deep neural networks work is currently something along the lines of:
- there’s some (unknown) minimal network size (or maybe rather “minimal network frontier”, as with different architectures you end up with different minimal sizes) for every problem you want to solve (for a certain understanding of the problem and when you consider it solved), so your network needs to be big enough to even be able to solve the problem
- the network size & architecture also determines how much training data you need to get anywhere
- basically, you try to find network architectures such that you encode sensible priors about the modality you’re working with that are basically always true while also eliminating a priori-useless weights from your network; this way, the training efforts allow the network to quickly learn important things rather than first having to figure out the priors themselves
  - for text, you might realize that different parts of the text refer to each other, so need a way to effectively pass information around, and hence you end up with something like the attention mechanism
  - for image detection, you realize that the prior of any given pixel being relevant for any other given pixel is higher the closer they are, so you end up with something like CNNs, where you start looking at low level features, and throughout the layers of the network, allow it to “convert” the raw pixel data successively to semantic data
- in theory, you probably could just use a huge feed forward network (as long as it’s not so huge as to overfit instead of generalizing to anything useful) and it would possibly end up solving problems in similar ways as “smarter” architectures do (but not sure about this), but you would need way more parameters and way more training data to achieve similar results, much of which would be wasted on “low quality parameters” that could just as well be omitted
- so, encoding these modality priors into your network architecture spares you probably orders of magnitude of compute compared to naive approaches
- while the bitter lesson makes sense, it maybe under-emphasizes the degree to which choosing suitable network architecture + high quality training data matters?
- lastly, the question “which problem you’re trying to solve” cannot just be answered on a high level with “I want to minimize loss in next-token prediction”, but the exact problem the network solves depends strongly on the training data; loss minimization is a trade-off between all the things you’re minimizing, so the higher the amount of rambling, gossip, meaningless binary data and so on in your training data is, the more parameters and training time you’ll need just for those, and the less will the network be capable to predict more meaningful tokens.
Related to that last point, I recently worked on a small project where you, as the user, play Pong against an AI. That AI is controlled by a small neural network (something in the order of 2 or 3 hidden layers and a few dozen neurons), initialized randomly, so at first it’s very easy for the human to win. While you play, though, the game collects your behavior as training data and constantly trains the neural network, which eventually learns to mirror you. So after a few minutes of playing, it plays very similar to the human and it becomes much harder to beat it.
One thing I noticed while working on this is that the naive approach to training this AI was far from optimal: much of the training data I collected ended up being pretty irrelevant for playing well! E.g., it’s much more important how the paddle moves while the ball is closing in, and almost entirely irrelevant what you do right after hitting the ball. There were several such small insights, leading me to tweak how exactly training data is collected (e.g. sampling it with lower probability while the ball is moving away than when it’s getting closer), which greatly reduced the time it took for the AI to learn, even with the network architecture staying the same.
Notably, this does not necessarily mean the loss curve dropped more quickly—due to me tweaking the training data, the loss curves before and after doing so related to quite different things. The same loss for higher quality data is much more useful than for noisy or irrelevant data.
There’s just so many degrees of freedom in all of this that it seems very likely that, even if there were not hardware advances at all, research would probably be able to come up with faster/cheaper/better-performing models for a long time.

silentbob 18 May 2025 13:23 UTC
33 points
0
on: silentbob’s Shortform
One thing that confused me about transformers is the question of when (as in, after how many layers) each embedding “flips” from representing the original token to finally representing the prediction of the next token.
By now, I think the answer is simply this: each embedding represents both at the same time (and more). For instance, in GPT3 there are 12,288 embedding dimensions. At first I thought that all of them initially encode the original token, and after going through all the layers they eventually all encode the next token, and somewhere in the layers between this shift must happen. But what, upon some reflection, makes much more sense would be something very roughly like, say:
- some 1000 dimensions encode the original token
- some other 1000 dimensions encode the prediction of the next token
- the remaining 10,288 dimensions encode information about all available context (which will start out “empty” and get filled with meaningful information through the layers).
In practice, things are of course much less clean, and probably most dimensions will have some role in all these things, to different degrees, as of course all of this is learned through gradient descent and hence will be very noisy and gradual. Additionally, there’s the whole positional encoding thing which is also part of the embeddings and makes clear distinctions even more difficult. But the key point remains that a single embedding encodes many things, only one of which is the prediction, and this prediction is always there from the beginning (when it’s still very superficial and bad) and then, together with the rest of the embedding, gets refined more and more throughout the layers.
Another misconception I had was that embedding and unembedding are very roughly symmetric operations that just “translate” from token space to embedding space and vice versa^[1]. This made sense in relation to the initial & naive “embeddings represent tokens” interpretation, but with the updated view as described above, it becomes clear that unembedding is rather an “extraction” of the information content in the embedding that encodes the prediction.
One piece of evidence for this updated view is that this paper (thanks to Leon Lang for the hint) found that “Zero layer transformers model bigram statistics”. So, indeed, embedding + unembedding alone already perform some very basic next-token prediction. (Admittedly I’m not sure if this is only the case when the transformer is trained with zero layers, or also in, say, GPT3, when during inference you just skip all the layers)
I would guess that transformer-experienced people (unless they disagree with my description—in that case, please elaborate what I’m still getting wrong) will find all of this rather obvious. But for me, this was a major missing piece of understanding, even after once participating in an ML-themed bootcamp and watching all the 3Blue1Brown videos on transformers several times, where this idea either is not directly explained, or I somehow managed to consistently miss it.
1. ^
  Of course, this is not entirely true to begin with because the unembedding yields a distribution rather than a single token. But my assumption was that, if you embed the word “Good” and then unembed the embedding immediately, you would get a very high probability for “Good” back when in practice (I didn’t verify this yet) you would probably obtain high probabilities for “morning”, “day” etc.

silentbob 13 May 2025 17:19 UTC
2 points
0
on: How to title your blog post or whatever
It’s sad to admit, but I think there are many good things that simply don’t have good titles.
I’ve been thinking for many years that bad titles are a common reason for failure, say of movies or video games or other things that are sold where superficial first impressions are important. In the sense of: there are some products out there that would have been orders of magnitude less or more successful, had they gone with a different name.
This seems particularly important to me for anything that has a “viral” element, where people tell their friends about it. A good title most definitely affects the “reproduction number” to some degree. If it sounds cool people may easily be 50% more likely to speak about it than if the name is cringe or confusing or hard to remember or hard to pronounce. If this moves your R from 0.9 to 1.4, that can obviously make a tremendous difference for the trajectory of the thing.

silentbob 9 May 2025 19:52 UTC
2 points
0
on: Progress = Fewer Bad Moments
My girlfriend once came across this metaphor of “spiraling upwards”. Whatever you’re struggling with, you’ll have low points again with near certainty, but ideally you have learned something in the meantime that improves some aspect of your situation or your ability to bounce back. I think it’s a nice way to look at things when it’s true. Generally, dealing with setbacks seems like one of the crucial parts of making progress in any area.

silentbob 5 May 2025 21:35 UTC
2 points
0
on: What is your favorite podcast?
Dwarkesh Patel
Most people here probably know it, but for the few of you who don’t: in-depth AI podcast with many high-profile guests from AI labs and beyond. Often brings up AI Safety concerns, but the general vibe of the podcast is usually rather somewhere between excited and optimistic. Dwarkesh is quick on his feet and tends to ask many good questions, often “good-faith-challenging” his guests.
He’s great at extracting the world views out of his guests and at keeping conversations very engaging even over many hours. My impression is that he vibes well with most guests and gets them to share their views more freely than they would otherwise. Most noteworthy for me were the episodes with Sutskever, Aschenbrenner, gwern, and of course the AI 2027 one with Daniel Kokotajlo and Scott Alexander.
If the above sounds interesting, then consider this a recommendation.
If you consider Mechanize to be net-negative and don’t want to support anyone funding them, then rather don’t consider this a recommendation.

silentbob 5 May 2025 21:21 UTC
2 points
0
on: What is your favorite podcast?
The Studies Show
It’s entertaining yet refreshingly skeptical of science (in a, you know, rather rational way) and the problems it has. Tears apart many papers, myths and misconceptions. Tom Chivers keeps mentioning Bayes and Scott Alexander. Has some episodes on general scientific & statistical concepts and the major problems in science, as well as many object-level ones on concrete research topics, such as growth mindset, autism, seed oil or IQ. I prefer the latter ones. Spoiler alert: the outcome of most episodes is “we know much less than people think”, about pretty much anything.
One weakness of the show may be that they’re possibly erring too much on the “there may be some evidence for X but can we really tell? Actually, nobody really knows and it’s all just guessing based on a bunch of very flawed studies” side. Occasionally the hosts seem a bit less well prepared than they could be. Still, on the majority of topics, I find their episodes rather enlightening. Another plus is that they have some episodes on their past mistakes on the podcast (of which there are indeed quite a few).
If you’re a bit cynical and enjoy two witty Brits making fun of bad science while learning a few things about the state of research, you might enjoy this one.

silentbob

Me­la­tonin Self-Ex­per­i­ment Results

Melatonin Self-Experiment Results