A counterexample is a variant of the typical mind fallacy: if you perfectly understand something, it feels so obvious that you tend to forget what was not obvious before. So professors are regularly worse at explaining a difficult concept than teaching assistants: because the latter still know what makes it hard to understand.
Does glossing over large inferential distances correlate with poor understanding? I’d expect no. A teacher who is leaving big chunks of an argument as exercises for the reader is a poor communicator, but their failure is in modeling the listener and not necessarily understanding the subject. They may talk like that about everything.
Your point is more broad and you acknowledged it’s not always true, but this specific failure mode doesn’t seem correlated with understanding. You can understand something very well but will fail at walking someone to your conclusion if you make false assumptions about your audience’s knowledge or processing speed. See bad physics professors as an example of this.
Is this anecdotal, or research-based? I’ve always heard somewhat the opposite regarding your second point. Although, I could not find any significant sources on the subject.
One day I realized that when asked to explain things to someone, I reach for the precise words in my own private vocabulary to match the idea in my head, as opposed to finding the words that unlock the idea in theirs. Sometimes, when I try to visualize this shift in behavior within myself, I imagine it as the difference between a hammer and a river.
A hammer is a tool that strikes at one point until it pierces straight through, as if nothing were there in the first place. The wood can crack, and the absence becomes a pressure. The hammer’s journey starts and ends in one direction, the only evidence of an other being the force of its strike.
A river meets its banks, and finds the cleanest path around them, with a certain cosmic patience. If its course stands to be corrected, it travels onward, pivoting to the lesson of its substrate. And, the river doesn’t stop at the soil; it nourishes life near and far, endlessly cycling water, as an event, not an object.
Here’s a question:
Do we think in terms of language, or is there something more atavistic about thinking? Is there a language of the mind? Does this lie in the subconscious, the conscious, both, or neither?
Steerling-8B: The First Inherently Interpretable Language Model
This is probably worth a deeper discussion, but Guide Labs is claiming that their new model is “the first interpretable model that can trace any token it generates to its input context, concepts a human can understand, and its training data”.
Reading the blog, Steerling is basically just a discrete diffusion model where the final hidden states are passed through a sparse autoencoder before the LM head.
They also appear to apply a loss that aligns the SAE’s activations with labelled concepts (correct me if I’m wrong). However, this seems like an obvious example of The Most Forbidden Technique, and could make the model appear interpretable without the attributed concepts actually having causal effect on the model’s decisions.
Can we get some input from interpretability folks? I’m obviously bearish.
There is a lot of information, but the most clear take-away is that the majority of those surveyed support AI regulation.
Another result that surprises (and concerns) me is this side note:
A question that was interesting, but didn’t lead to a larger conclusion, was asking what actually happens when you ask a tool like ChatGPT a question. 45% think it looks up an exact answer in a database, and 21% think it follows a script of prewritten responses.
A recent paper probed LLMs and located both value features (representing the expected reward) and “dopamine” features (representing the reward prediction error). These features are embedded in sparse sets of neurons, and were found to be critical for reasoning performance.
Could these findings have any implications for model welfare?
If a model had mechanisms for “feeling good and bad”, I imagine they would look similar to this.
The paperclip maximizer will be an ad click maximizer.
Clarification: Since so much research goes into building models that maximize advertising results (even being the reason Google started building TPUs), an ad click maximizer is the most likely case of blatant outer alignment failure from a poorly chosen objective.
My point is that since so much research goes into building models that maximize advertising results, an ad click maximizer is the most likely case of blatant outer alignment failure. Ad clicks will be its end goal.
Being worse at explaining things will make those around you less able to understand your points, which may make them seem less intelligent.
Therefore, being worse at explaining may make you feel more intelligent relative to those around you.
My intuition was that if you are on the more insecure end of the spectrum, it can make you feel less intelligent.
Probably depends on the reaction of those who didn’t understand.
“This sounds really complicated. I am not smart enough to understand it.”
“This is some bullshit that doesn’t make sense.”
While not always the case, poor explanations usually point to poor understanding. Same with cluttered writing.
A counterexample is a variant of the typical mind fallacy: if you perfectly understand something, it feels so obvious that you tend to forget what was not obvious before. So professors are regularly worse at explaining a difficult concept than teaching assistants: because the latter still know what makes it hard to understand.
Does glossing over large inferential distances correlate with poor understanding? I’d expect no. A teacher who is leaving big chunks of an argument as exercises for the reader is a poor communicator, but their failure is in modeling the listener and not necessarily understanding the subject. They may talk like that about everything.
Your point is more broad and you acknowledged it’s not always true, but this specific failure mode doesn’t seem correlated with understanding. You can understand something very well but will fail at walking someone to your conclusion if you make false assumptions about your audience’s knowledge or processing speed. See bad physics professors as an example of this.
Is this anecdotal, or research-based? I’ve always heard somewhat the opposite regarding your second point. Although, I could not find any significant sources on the subject.
One day I realized that when asked to explain things to someone, I reach for the precise words in my own private vocabulary to match the idea in my head, as opposed to finding the words that unlock the idea in theirs. Sometimes, when I try to visualize this shift in behavior within myself, I imagine it as the difference between a hammer and a river.
A hammer is a tool that strikes at one point until it pierces straight through, as if nothing were there in the first place. The wood can crack, and the absence becomes a pressure. The hammer’s journey starts and ends in one direction, the only evidence of an other being the force of its strike.
A river meets its banks, and finds the cleanest path around them, with a certain cosmic patience. If its course stands to be corrected, it travels onward, pivoting to the lesson of its substrate. And, the river doesn’t stop at the soil; it nourishes life near and far, endlessly cycling water, as an event, not an object.
Here’s a question:
Do we think in terms of language, or is there something more atavistic about thinking? Is there a language of the mind? Does this lie in the subconscious, the conscious, both, or neither?
Steerling-8B: The First Inherently Interpretable Language Model
This is probably worth a deeper discussion, but Guide Labs is claiming that their new model is “the first interpretable model that can trace any token it generates to its input context, concepts a human can understand, and its training data”.
Reading the blog, Steerling is basically just a discrete diffusion model where the final hidden states are passed through a sparse autoencoder before the LM head.
They also appear to apply a loss that aligns the SAE’s activations with labelled concepts (correct me if I’m wrong). However, this seems like an obvious example of The Most Forbidden Technique, and could make the model appear interpretable without the attributed concepts actually having causal effect on the model’s decisions.
Can we get some input from interpretability folks? I’m obviously bearish.
Link to release post: https://www.guidelabs.ai/post/steerling-8b-base-model-release/
The Searchlight Institute recently released a survey of Americans’ views and usage of AI:
https://www.searchlightinstitute.org/research/americans-have-mixed-views-of-ai-and-an-appetite-for-regulation/
There is a lot of information, but the most clear take-away is that the majority of those surveyed support AI regulation.
Another result that surprises (and concerns) me is this side note:
A recent paper probed LLMs and located both value features (representing the expected reward) and “dopamine” features (representing the reward prediction error). These features are embedded in sparse sets of neurons, and were found to be critical for reasoning performance.
Could these findings have any implications for model welfare?
If a model had mechanisms for “feeling good and bad”, I imagine they would look similar to this.
The paper in question: https://arxiv.org/abs/2602.00986
The paperclip maximizer will be an ad click maximizer.
Clarification: Since so much research goes into building models that maximize advertising results (even being the reason Google started building TPUs), an ad click maximizer is the most likely case of blatant outer alignment failure from a poorly chosen objective.
Only during the early phases. Once it has sufficient capital (power, resources), it won’t need to. well-modeled by https://www.decisionproblem.com/paperclips/ .
My point is that since so much research goes into building models that maximize advertising results, an ad click maximizer is the most likely case of blatant outer alignment failure. Ad clicks will be its end goal.