joseph_c

Karma: 181

joseph_c Sep 4, 2025, 3:18 AM
3 points
2 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: Remap your caps lock key
For Linux users on US Keyboards, you might want to try making Caps Lock the multi key (also called the compose key). On Cinnamon this can be done by going to Keyboard > Layouts > Options… > Position of Compose key, and other desktop environments probably have similar settings.

This lets me type umlauts (ä, ü, ö), foreign currencies (£, €, ¥,), copyright/trademark (©, ™), and a bunch of other stuff. For example, “ü” is made by typing Compose, u, and ” in sequence. I also added the line
```
<Multi_key> <backslash> : "λ"
```
to my ~/.XCompose file so that I can type λ efficiently; this is useful when writing Lisp code.

joseph_c Sep 1, 2025, 4:44 AM
4 points
5 votes
Overall karma indicates overall quality.
1
1 vote
Agreement karma indicates agreement, separate from overall quality.
on: My AI Predictions for 2027
Why the Architecture of LLMs Makes Them Bad at Deep Thinking: They’re Too Wide
GPT-3 is 96 layers deep (where each layer is only a few “operations”), but 49,152 “neurons” wide at the widest. This is an insanely wide, very shallow network. This is for good reasons: wide networks are easier to run efficiently on GPUs, and apparently deep networks are hard to train.
I don’t find this argument compelling, because the human brain is much wider and possibly shallower than GPT-3. Humans have a conscious reaction time of about 200 milliseconds, while neurons take about 1ms to influence their neighbors, meaning an upper bound on the depth of a conscious reaction is 200 neurons.

joseph_c Aug 29, 2025, 10:06 PM
3 points
2 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: The Gabian History of Mathematics

Thanks to Hilbert’s list, a lot of progress was made toward formalising proofs, logic, consistency and other similar concepts.

Hmmm… I don’t think this accurately describes the era of mathematics starting around the 1920s. In fact, I would argue that the correct era would be about 1910-1937, starting with Russell and Whitehead’s Principia Mathematica and ending with the proof that the lambda calculus is exactly as powerful as the Turing machine.

This era was focused on applying logic to computation. It saw the development of type theory and the foundations of computation. Some aspects, like the halting problem, were related to logical consistency, but I think the more important breakthroughs had to do with formalizing computation.

joseph_c Aug 24, 2025, 7:16 PM
21 points
9 votes
Overall karma indicates overall quality.
−2
2 votes
Agreement karma indicates agreement, separate from overall quality.
on: Shorter Tokens Are More Likely
So, I had a hypothesis last night that training on a different scoring rule might solve this problem (because it could encourage uncertain probabilities to be lower, and thus it would be easier to filter them out without making short tokens undeservedly more likely).

I ended up forking your code, and this morning trained an LLM on the Shakespeare dataset using the $α$ -ReLU loss (from Speeding Up Entmax). The $α$ -ReLU loss is a proper scoring rule based off of the Tsallis entropy.

My results were the following:

Letter Score Rule Top-K Control Test T=1.0 Test T=0.8 Test T=0.1

c Cross-entropy 200 3.09% 5.29% 6.64% 7.26%

c $α$ -ReLU 200 3.41% 4.66% 4.75% 4.33%

c $α$ -ReLU $\infty$ 3.41% 3.87% 4.00% 3.49%

All models were trained for 1500 iterations. The control was trained without the special C-tokenizer, while the tests were. In the paper, $α$ -ReLU is parameterized by $α$ which controls the Tsallis exponent, and $τ$ , which is a constant shift in the ReLU. I chose to set $α = 1.5$ and $τ = 0$ for all the experiments (which is the default for the $α$ -ReLU python library).

The $α$ -ReLU trained neural network does NOT seem to exhibit the same trend of lower temperatures leading to higher probability of words starting with “c”. And, when the top-k is set to $\infty$ , most of the difference between it and the control also disappears.

So, perhaps we should just be training with a different scoring rule!

You can find my fork at https://github.com/cooljoseph1/nanoGPT-tokenizer-experiment/.

EDIT: I realized I was calculating temperature unfairly for the $α$ -ReLU sampling. After fixing it, it is actually worse than the cross-entropy trained network:

Letter Score Rule Top-K Control Test T=1.0 Test T=0.8 Test T=0.1

c Cross-entropy 200 3.09% 5.29% 6.64% 7.26%

c $α$ -ReLU 200 3.41% 4.66% 7.54% 31.66%

c $α$ -ReLU $\infty$ 3.41% 3.87% 5.94% 31.66%

Letter	Score Rule	Top-K	Control	Test T=1.0	Test T=0.8	Test T=0.1
c	Cross-entropy	200	3.09%	5.29%	6.64%	7.26%
c	$α$ -ReLU	200	3.41%	4.66%	4.75%	4.33%
c	$α$ -ReLU	$\infty$	3.41%	3.87%	4.00%	3.49%

Letter	Score Rule	Top-K	Control	Test T=1.0	Test T=0.8	Test T=0.1
c	Cross-entropy	200	3.09%	5.29%	6.64%	7.26%
c	$α$ -ReLU	200	3.41%	4.66%	7.54%	31.66%
c	$α$ -ReLU	$\infty$	3.41%	3.87%	5.94%	31.66%

joseph_c Aug 24, 2025, 1:36 AM
4 points
3 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: Shorter Tokens Are More Likely
I remember reading a paper about how aiming for a certain entropy per token made LLMs sound more human. I think it might have been this paper? This marginalization of later tokens might be the reason why—aiming for a certain entropy would encourage lower probability tokens more often than a fixed temperature would while still avoiding “noisy” tokens.

joseph_c Aug 11, 2025, 12:53 AM
4 points
3 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: Run-time Steering Can Surpass Post-Training: Reasoning Task Performance
This seems especially easy to do with RWKV. Perhaps I will try that out sometime this week, but probably not. I did something like that (but not specifically for reasoning) two years ago for MIT’s Splash.

My code from Splash is on Github if anyone else wants to give it a try before I (maybe) get to it. The code is very bad, though. It’s just something I wrote for myself for the most part (so it doesn’t have any kind of documentation, and it has lots of dead code lying around). So, it might be easier to just write your own code.

joseph_c Jul 21, 2025, 5:12 AM
1 point
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Brendan Long’s comment on: LLMs Can’t See Pixels or Characters
It worked for me on the second attempt (also using ChatGPT).

Attempt 1:

Spell out “riedenheit”, i.e. peace, letter by letter with spaces separating them

Sure! Here’s “Zufriedenheit” (German for peace or contentment) spelled out with spaces:
Z U F R I E D E N H E I T

Attempt 2:

Spell out riedenheit, i.e. peace, letter by letter with spaces separating them

Sure! Here’s “riedenheit” spelled out letter by letter with spaces:
R I E D E N H E I T

joseph_c Jul 14, 2025, 3:47 PM
1 point
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: Arrow theorem is an artifact of ordinal preferences
Suppose every citizen truthfully provides a normalized cardinal valuation for each alternative: that is, valuations are positive, and for each citizen the sum of the valuations for all alternatives is one. Then, the sum of the valuations by alternative is a consistent measure of the social value for every alternative (Dhillon, Bouveret and Lemaître, 1999).
Why not have the sum of the squares of the valuations sum to one? By the Cauchy-Schwartz inequality, one’s utility is maximized by reporting a vector of valuations parallel to one’s true preferences, so you can get rid of the requirement for honesty.

joseph_c Mar 2, 2025, 7:54 AM
1 point
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: [PAPER] Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations
I’m a little confused about which computation you’re trying to sparsify. The paper seems to be written in the context of the technique where one uses sparse autoencoders to extract “features” from the embedding space of large language models which are hopefully interpretable. (Please correct me if I’m wrong about that!)
The goal would seem to be, then, to sparsify the computation of the language model. However, the method in your paper seems to sparsify the computation of the autoencoders themselves, not the language model. Shouldn’t the goal be to sparsify the language model’s computation? If so, why not use weight pruning? What is JSAE better at?

joseph_c Dec 28, 2024, 9:36 PM
2 points
2 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: James Camacho’s comment on: If all trade is voluntary, then what is “exploitation?”
Isn’t $\beta$ proportional to the inverse temperature, and so should be smaller now (with easier, more frequent trading)?

joseph_c Oct 28, 2024, 5:45 PM
8 points
3 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: On Shifgrethor
A math textbook leaving certain results as an exercise for the reader?
I think this is usually actually one of (1) the author not wanting to write out the proof (because it’s boring/tedious) or (2) a proof that would make a good exercise because it is easy enough if you understand the big ideas (and coming up with good exercises is not always easy).

joseph_c Oct 3, 2024, 10:10 PM
1 point
5 votes
Overall karma indicates overall quality.
1
3 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: Raemon’s comment on: “Slow” takeoff is a terrible term for “maybe even faster takeoff, actually”
Gradual/Sharp

joseph_c Aug 23, 2024, 12:34 AM
4 points
3 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: joseph_c’s Shortform
I recently came across Backpack Language Models and wanted to share it in case any AI interpretability people have not seen it. (I have yet to see this posted on LessWrong.)

The main difference between a backpack model and an LLM is that it enforces a much stricter rule to map inputs’ embeddings to output logits. Most LLMs allow the output logits to be an arbitrary function of the inputs’ embeddings; a backpack model requires the output logits to be a linear transformation of a linear combination of the input embeddings. The weights for this linear combination are parameterized by a transformer.

The nice thing about backpack models is that they are somewhat easier to interpret/edit/control: The output logits are a linear combination of the inputs’ embeddings, so you can directly observe how changing the embeddings changes the outputs.

joseph_c’s Shortform

joseph_cAug 23, 2024, 12:34 AM

3 points

1 vote

Overall karma indicates overall quality.

1 comment1 min readLW link

joseph_c Jun 27, 2024, 3:13 AM
2 points
3 votes
Overall karma indicates overall quality.
−1
1 vote
Agreement karma indicates agreement, separate from overall quality.
on: Childhood and Education Roundup #6: College Edition
Most students, 48 percent, claimed to be Native American on their application....
According to Intelligent.com Managing Editor Kristen Scatton, the prevalence of applicants who claim Native American ancestry is possibly due to the popular narrative that for many Americans, a small percentage of their DNA comes from a Native American tribe.
Maybe these students are purposely misinterpreting “Native American” to be someone who was born and raised in the United States, perhaps with ancestors born and raised in the US as well. This is actually the older sense of the term “Native American”, found, for example, in the name of the Native American Party back in the mid-1800s.

joseph_c May 31, 2024, 4:53 PM
4 points
4 votes
Overall karma indicates overall quality.
1
1 vote
Agreement karma indicates agreement, separate from overall quality.
on: Less Anti-Dakka
includeIt is written in More Dakka:
If something is a good idea, you need a reason to not try doing more of it.
Taken at face value, it implies the contrapositive:
If something is a bad idea, you need a reason to not try doing less of it.
This is not the contrapositive. It is not even the opposite.

joseph_c Apr 30, 2024, 4:17 AM
1 point
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: UDT1.01: The Story So Far (1/10)
Parfit’s Hitchhiker: You’re stranded in the desert and Omega comes up. It will give you a ride out of the desert iff it predicts you’d give it 10,000 dollars upon reaching civilization again. You get a ride. When in civilization again, do you go over to the bank and withdraw some money? Well, policies which pay up in this specific situation get (value of a life − 10,000 dollars) more than policies which don’t pay in this specific situation, which just die.
Why is this called Parfit’s Hitchhiker? Who is the Parfit it is referring to? Where was this scenario first written up? (I’m trying to dig up the original reference.)

joseph_c Mar 21, 2024, 4:00 AM
1 point
1 vote
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
in reply to: the gears to ascension’s comment on: I was raised by devout Mormons, AMA &| Offer Advice
Not for Mormons. They don’t believe in an omnipresent God.

joseph_c Mar 19, 2024, 9:46 PM
64 points
43 votes
Overall karma indicates overall quality.
52
23 votes
Agreement karma indicates agreement, separate from overall quality.
on: Increasing IQ by 10 Points is Possible
Well, what are your actual steps? Or is this just advertisement?

joseph_c Mar 13, 2024, 5:43 PM
2 points
2 votes
Overall karma indicates overall quality.
0
0 votes
Agreement karma indicates agreement, separate from overall quality.
on: I was raised by devout Mormons, AMA &| Offer Advice
Do you still live in Utah?
Did your family cut you off?
Do you know about [r/exmormon](https://old.reddit.com/r/exmormon/)?

joseph_c

Why the Architecture of LLMs Makes Them Bad at Deep Thinking: They’re Too Wide

joseph_c’s Shortform