Fair enough. I’ll retract and revise to macro-organisms.
derelict5432
Karma: 126
The world optimized itself.
I think you have a good point buried in this post, but this doesn’t seem like it. The world didn’t optimize itself. Humans optimized for chicken volume, taste, and other parameters, and generally ignored the suffering and well-being of the chickens. It wasn’t ‘the world’ and it didn’t just optimize. It optimized certain variables at the expense of others.
Also, I’d push back on the idea (popularized probably most effectively by Pinker) that we’re doing ‘so much better’ now than we were 200 years ago. Again, for certain parameters, yes. For many others, no. The biggest factor in these discussions that you’re sort of pointing to is risk. We have increased per capita GDP, health outcomes, etc. But we have also massively increased X risk. We now have multiple non-negligible vectors for causing widespread catastrophe and possible extinction, not just among humans, but against all life on the planet. That in itself complicates the picture of how well we’re doing.
Which sort of dovetails with your point, which is that many times things are going great, until they aren’t. But you don’t specifically frame it as an increase in risk. You seem to be attributing it to a sort of semi-random unpredictable switch from doing well to doing poorly, as a sort of inevitable cost of optimization. I don’t think that’s right.
I also don’t have a strong sense of how pervasive underconfidence is, and my predictions would be relatively weak. But there are of course dynamics like Imposter Syndrome that are relatively common. One of my closest friends will rarely take a firm stand on any claim. He seems to do this in the name of fairness and epistemic humility, but it seems like a drastic overcorrection.
On the second point, sure. A lot of the people who need to correct for overconfidence are precisely the one that wield disproportionate influence and power (which their overconfidence almost certainly helped them attain). They’re also probably the ones least likely to calibrate, because incentives.
And there’s an interesting dynamic, especially in public discussions, where the tone is probably more properly adjusted for the viewers/readers than it is for the person you’re actually engaged with. I agree that changing an individual’s mind is very, very difficult. What I’m talking about is the danger of conflating being nice to the person and being nice to the ideas in a way that projects fairness and respect of the ideas that are completely unwarranted.
A perfectly civil and respectful public debate with a person who believes that non-white races are subhuman, or a climate change denialist, or full-on anti-vaxxer likely does more harm than good, if the ideas are given full respect and consideration. For a private discussion, I’d agree that the strategy should shift significantly, if the goal is to convince that person. In the first case, those beliefs need to be confronted with aggression, not politeness. They aren’t just epistemically terrible, they’re actively dangerous. Treating them with the veneer of legitimacy is likely to confuse bystanders who will come away with the wrong impression.
A few thoughts:
The post seems dangerously close to advocating a general lowering of confidence across the board. I don’t think that’s what you’re advocating, but it kind of sounds that way. Most people are wrongly overconfident, but the prescription for that is not for everyone to dial it down. Instead, we all need to be better calibrated. Some people need to actually up their belief confidence levels.
Overconfident belief is very often highly adaptive. That’s probably why it is so prevalent. Again, that makes a coarse-grain corrective potentially problematic if there’s a conflict between utility and truth-seeking.
I like most of your prescriptions for avoiding confirmation bias, but ‘be nice’ is a tough one. Bad beliefs are bad. Very bad beliefs are responsible for a huge amount of suffering, injustice, death, etc. They need to be choked out in the shallow end of the pool. There is a moral and rational obligation to not be nice to bad beliefs. Now, it’s difficult to tease apart being nice to beliefs and being nice to people. A lot of the time people take their beliefs very personally. So this is difficult territory to navigate. But you don’t want to be so careful and considerate that you give aid and comfort to beliefs that underpin the very worst outcomes. So ‘be nice’ as a sort of blanket prescription seems a little too generic, though I agree it should be the default when dealing with the person.
Not sure I understand your question. Are you basically asking what a placebo is?
Something like the placebo effect may certainly be a big part of why some woo things work.
This is almost certainly the case, and you give it a bit of short shrift. You wonder out loud why you would get a particular placebo effect with a particular practice. There are lots of potential reasons. Those need to be studied and the evidence needs to be documented.
That’s the basic problem with woo. What really makes it woo is that the causal mechanisms for its efficacy are highly suspect and cannot be identified through rigorous intervention/experimentation.
Prayer has a lot of well-demonstrated physical and mental positive outcomes. What do you think of prayer? Why do you think it works?
Sorry, I stopped reading because of the disingenuous shifting very early on.
They should be able to reliably diagnose diseases they are trained to diagnose.
Okay, agree. What’s their reliability for all diseases they are trained to diagnose?
Failure to diagnose uncommon diseases is rampant
Oh, okay, we’re going to focus on their reliability to diagnose uncommon diseases. So how do they do with that?
A survey of patients with rare diseases found that, in about half of cases, patients received at least one incorrect diagnosis, and two thirds required visits to at least three different doctors before being diagnosed.
Okay, we somehow went from general reliability, to uncommon, to rare without skipping a beat. You’re talking about different things. This lack of basic consistency undermined your credibility immediately.
Refuse to ‘understand’ things unless they are very clear. I don’t really know how to do this, because I don’t know what the alternative is like—being steadfastly confused about things seems to come naturally to me and I don’t know how else to be, but maybe you have both affordances available here and could lean one way or the other.
I’m interpreting this to mean: don’t be dazzled by bullshit. Don’t just nod along to authoritative-sounding, jargon-filled rhetoric simply because it has the veneer of deepness. Understand here is in quotes. I think it just means in this case to go along.
In grad school, there were many colloquia where a visiting professor would give a presentation. Everyone would nod along. A few polite questions would be asked, and then after some light applause we’d partake of butter cookies and punch. Meanwhile, I would often be sitting there wondering what the hell just happened. What were they really saying? If they were saying X, wasn’t that just complete nonsense? But it would take me a while to think through it and parse it to find out the content really was vacuous.
Over the years I’ve found a good technique is to try to translate complex-sounding claims into their simplest form. Sometimes this is difficult. Some complex things are necessarily complex. In academia, though, complexity is very often a smokescreen, and the simplicity translation often exposes this.
Agree. The term ‘post-scarcity’ focuses purely on instrumental goals: abundant energy, longevity, etc. Transhumanism tends to focus on the same kinds of things: immortality, invulnerability, power. These are instrumental. What are you going to do with all that power and longevity? They’re often putting the cart before the horse. Make great art? Pursue truth? Mitigate suffering? Enhance personal rights? The are the ends. Power is the means, but very often it’s treated as an end in itself.
Where is this quotation from? I don’t see a link.
Also: “isolated the relevant code”. That phrase could be doing a lot of heavy lifting here, right? It’s one thing to sift through a million lines of code and identify a bug. It’s another to be handed three lines that contain a bug and find it. Needle in a haystack vs. needle with two pieces of straw. If the methodology was identical, okay. But I’d like to see a side-by-side comparison of methodology here.
Is it your position that it is not obvious that a new species can causally drive another species extinct without being orders of magnitude more intelligent? Because the earth has had millions of existence proofs of that in the history of life.
I found the Dwarkesh—Terrence Tao conversation very frustrating, in particular the section on advancements in science, re heliocentrism and evolution. These guys talked about it epistemically as if everyone is a scientist. And in response to Dwarkesh’s question about why Darwin’s ideas took 200 years longer than Newton’s, even though they were simpler, Tao basically boiled it down to science communication. What?
The biggest causal impediment to the acceptance of both theories was religion and religious thought. It’s not even close. Darwin delayed publication for years because he was terrified of the societal and cultural blowback. Evolution dissolves the need for an interventionist creator and lumps us in with the rest of nature. It frames our ancestors as simple creatures akin to worms and bacteria. All the eloquence in the world isn’t going to overcome the entrenchment of the prevailing religious worldview. Ridiculous.
I agree all these reasons are weak. What I’ve seen a lot less of are good reasons in favor of potential AI consciousness. Maybe that would just be a recap of functionalist arguments, idk.
Incidentally, I have particular disdain for the ‘simulation is not instantiation’ argument. My favorite counterexample is: sure, a simulation of a hurricane does not get you wet, but does a simulation of a calculator add and multiply numbers? Or does a simulation/emulation of an Atari game system let you play Adventure? In other words, simulations of computation are instantiations of computation.
Sorry, I’m afraid I don’t understand what your analogy is supposed to map to. What is Grog in the context of our conversation? You seem to admit at the end that LLMs are not really at all like Grog, in that Grog has no underlying bedrock of understanding, while modern LLMs do.
Thus, there’s more to knowledge than lists of facts. It’s ways that the facts all connect to each other in an interconnected web, and it’s ways to think about things, etc.
I’ll agree with this definition. If you’ll agree that knowledge can exist in written form and textbooks often embody exactly what you describe. They are very rarely ‘lists of facts’. More often than not, they are logically curated, organized explanations of phenomenon and events, along with rich descriptions of their connections and interactions. You seem to be preferentially upselling knowledge that is stored in synaptic weights while drastically downplaying knowledge recorded in other mediums. Why?
So according to you, a system that could acquire new facts, record them, access them, and use them, continuously in this way would not constitute ‘real’ continuous learning. It could conceivably fill its database with the actionable knowledge of 1000 yet unwritten textbooks, but that wouldn’t be ‘real’ to you.
“wholly new ways of conceptualizing and navigating the world, not just keeping track of what’s going on” are learnable and storable in the way I describe.
How is this type of learning not open-ended? What is limiting it?
Your third criteria seems to be related to unsupervised learning, specifically self-play. Not sure why you’d limit continual learning in this way, either.
You seem to be putting somewhat arbitrary constraints on what constitutes continual learning. Generally, if the system’s knowledge base is fixed, it’s incapable of continuing to learn. If it has the capacity to acquire new knowledge and skills, by whatever means, it continues to learn. You’re narrowing that general idea without really justifying why.
I tell an LLM my favorite color. As long as that information is in its context window, it has access to it. As soon as that context rolls off or goes away, the LLM no longer has access to that information.
I build an agent with scaffolding that has a database. I tell it my favorite color. The agent records it in the database. The weights of the LLM are still fixed, but during its base training it learned how to access information. So if I ask it at any point in the future what my favorite color is, it knows. It access the information in the database.
Do you consider this continual learning? If not, why not?
This seems more like a within-distribution problem: the player is encountering a game that is composed of pieces that are very alike the pieces of the games they’ve previously encountered, and the rules follow a similar logic.
Well, that’s one of the big questions, isn’t it? Seems fairly clear there’s no hard boundary between in-distribution and out-of-distribution. Is the cure for cancer and the way to discover it going to be completely OOD? Or is it going to lean heavily on existing knowledge of cell biology, genetics, and all previous cancer research? The common phrasing is ‘standing on the shoulders of giants’. This is pretty well accepted as the way new inventions and discoveries happen. Not as radically alien knowledge that emerges from a vacuum, but an incremental step up using a mountain of existing knowledge bases (analogous to a game composed of pieces very alike ones they’ve previously encountered). Very large discoveries or paradigm shifts are likely more OOD, but the vast bulk of new science is fairly incremental and I would think the sort of problems you’d consider within-distribution. No?
I actually implemented my own private benchmark last year to try to test this with different models. The domain was a toy OOD task where the system had access to three possible tools that performed simple transformations on a configuration of binary values in a particular spatial arrangement. Stage 1 was exploration. The system was given a certain number of steps to probe with the tools (which were chosen randomly from a subset prior to each trial). After the experimentation stage, the system was required to use the tools to perform a transformation on a random arrangement to make it match a target one.
The exercise of building a benchmark was a great learning experience for me. My main takeaway was that differences in performance were nearly all driven by differences in scaffolding, and not so much the base model. This made me fairly disillusioned about benchmarks in general. Made me suspect that gains in benchmarks like ARC-AGI are mostly driven by scaffolding improvements. Maybe someone here has much more insight into that.
But it also made me think that the problem is probably not some far-out radically intractable problem. You mention continual learning and long time horizons. Just generally for OOD tasks, the system needs to be able to log results, generate and revise hypotheses, and carry out Bayesian updates in an iterative manner. Whether that can be cracked reliably for increasingly difficult problems with relatively straightforward scaffolding, or the base models need to be radically improved along with scaffolding, I don’t really know. Maybe for the much more difficult problems (like a Theory of Everything or a cure for the common cold) those advances are very far out. I would think though, that for simple and medium-difficulty problems, the frontier labs are already well on their way.
Not sure why the go-to examples for out-of-distribution problems tend to be the extreme of an entirely new theory or invention. To make progress on this problem, we’d want to identify minimally-OOD problems and benchmark those, wouldn’t we?
Melanie Mitchell and collaborators showed weaknesses in LLM on OOD tasks with simple perturbations in the alphabet for string-analogy tasks. This seems like the sort of example we should generally be thinking about and testing, because they’re likely much more tractable, toy domains or simple ad hoc tasks that deviate from strong biases in the training distribution.
I lol’d here, but maybe there’s a serious point?
You mention upright stance as part of the nursing or thermo hypothesis. Why wouldn’t the uniqueness to humans be largely a part of the upright stance? They don’t feature nearly as prominently in a tree-dense environment or if you’re hunched down on all fours most of the time, right? For a sexually-selected trait, they need to be displayed. Sexual selection also seems to correspond strongly with the onset of fertility.