Reconstruction loss is the CE loss of the patched model
If this is accurate then I agree that this is not the same as “the KL Divergence between the normal model and the model when you patch in the reconstructed activations”. But Fengyuan described reconstruction score as:
measures how replacing activations changes the total loss of the model
which I still claim is equivalent.
I think just showing would be better than reconstruction score metric because is very noisy.
there is a validation metric called reconstruction score that measures how replacing activations change the total loss of the model
That’s equivalent to the KL metric. Would be good to include as I think it’s the most important metric of performance.
Patch loss is different to L2. It’s the KL Divergence between the normal model and the model when you patch in the reconstructed activations at some layer.
It would be good to benchmark the normalized and baseline SAEs using the standard metrics of patch loss and L0.
What is Neuron Activity?
Does anything think this could actually be done in <20 years?
materialism where we haven’t discovered all the laws of physics yet — specifically, those that constitute the sought-for materialist explanation of consciousness
It seems unlikely that new laws of physics are required to understand consciousness? My claim is that understanding consciousness just requires us to understand the algorithms in the brain.
Without that real explanation, “atoms!” or “materialism!”, is just a label plastered over our ignorance.
Agreed. I don’t think this contradicts what I wrote (not sure if that was the implication).
The Type II error of behaving as if these and future systems are not conscious in a world where they are in fact conscious.
Consciousness does not have a commonly agreed upon definition. The question of whether an AI is conscious cannot be answered until you choose a precise definition of consciousness, at which point the question falls out of the realm of philosophy into standard science.
This might seem like mere pedantry or missing the point, because the whole challenge is to figure out the definition of consciousness, but I think it is actually the central issue. People are grasping for some solution to the “hard problem” of capturing the je ne sais quoi of what it is like to be a thing, but they will not succeed until they deconfuse themselves about the intangible nature of sentience.
You cannot know about something unless it is somehow connected the causal chain that led to the current state of your brain. If we know about a thing called “consciousness” then it is part of this causal chain. Therefore “consciousness”, whatever it is, is a part of physics. There is no evidence for, and there cannot ever be evidence for, any kind of dualism or epiphenomenal consciousness. This leaves us to conclude that either panpsychism or materialism is correct. And causally-connected panpsychism is just materialism where we haven’t discovered all the laws of physics yet. This is basically the argument for illusionism.
So “consciousness” is the algorithm that causes brains to say “I think therefore I am”. Is there some secret sauce that makes this algorithm special and different from all currently known algorithms, such that if we understood it we would suddenly feel enlightened? I doubt it. I expect we will just find a big pile of heuristics and optimization procedures that are fundamentally familiar to computer science. Maybe you disagree, that’s fine! But let’s just be clear that that is what we’re looking for, not some other magisterium.
If consciousness is indeed sufficient for moral patienthood, then the stakes seem remarkably high from a utilitarian perspective.
Agreed. If your utility function is that you like computations similar to the human experience of pleasure and you dislike computations similar to the human experience of pain (mine is!). But again, let’s not confuse ourselves by thinking there’s some deep secret about the nature of reality to uncover. Your choice of meta-ethical system is of the same type signature as your choice of favorite food.
The subfaction veto only applies to faction level policy. The faction veto is decided by pure democracy within the faction.
I would guess in most scenarios most subfactions would agree when to use the faction veto. Eg. all the Southern states didn’t want to end slavery.
Yes, Garrett is referring to this post: https://www.lesswrong.com/posts/yi7shfo6YfhDEYizA/more-people-getting-into-ai-safety-should-do-a-phd
Presumably the factions (eg. Southern states) also have sub factions, so maybe a better system would be described with the recursive acronym DVDF:
Democracy with Veto for each faction, plus DVDF within Factions.
What’s up with Tim Cook not using buzzwords like AI and ML? There is definitely something cool and aloof about refusing to get sucked into the latest hype train and I guess Apple are the masters of branding.
Anthropic now has a highly impressive model, impressive enough that it seems as if it breaks at least the spirit of their past commitments on how far they will push the frontier.
Why do you not discuss this further? I want to hear your thoughts.
Would be nice to create prediction markets for some of these. Especially interested in the ones about pausing development.
The students are correct to take this as evidence against the theory. However they can go back to the whiteboard, gain a full understanding of the theory, correct their experiment and subsequently collect overwhelming evidence to overcome their current distrust of the theory.
Yup I think I agree. However I could see this going wrong in some kind of slow takeoff world where the AI is already in charge of many things in the world.
That video is pretty awesome. Would be great if you could make it a 4 part band by singing at the same time.
If an AI somehow (implicitly, in practice) kept track of all the plausible H’s, i.e., those with high probability under P(H | D), then there would be a perfectly safe way to act: if any of the plausible hypotheses predicted that some action caused a major harm (like the death of humans), then the AI should not choose that action. Indeed, if the correct hypothesis H* predicts harm, it means that some plausible H predicts harm. Showing that no such H exists therefore rules out the possibility that this action yields harm, and the AI can safely execute it.
This idea seems to ignore the problem that the null action can also entail harm. In a trolley problem this AI would never be able to pull the lever.
Maybe you could get around this by saying that it compares the entire wellbeing of the world with and without its intervention. But still in that case if it had any uncertainty as to which way had the most harm, it would be systematically biased toward inaction, even when the expected harm was clearly less if it took action.
I’m confused—why are you so confident that we should avoid processed food. Isn’t the whole point of your post that we don’t know whether processed oil is bad for you? Where’s the overwhelming evidence that processed food in general is bad?