LW1.0 username Manfred. Day job is condensed matter physics, hobby is thinking I know how to assign anthropic probabilities.
Charlie Steiner(Charlie Steiner)
Karma: 2,285
Have you been able to try the academic copy (rosettafold)?
I do it pretty rarely, so maybe not the best answerer. But I often do it when I feel like I want to compare long-term plans and one of them has a clear price while the other one only maybe does. Trying to estimate prices I’d put on things is one of a couple different decision-making tools.
Maybe something like various attempts from psychology to list emotions would help?
Overall, though, I think prevailing opinion around here is more along the lines of consciousness having many different moving parts, which means that most questions will not have simple answers.
E.g. saying I am “conscious of the taste of chocolate” tells you a lot of vague but related information about me—not only am I receiving certain signals from my taste receptors and nose and gut and peripheral glucose sensors and the texture on my tongue, but my recent past has primed me in such a way that this all gets interpreted in a particularly chocolatey way, which is then made accessible to the rest of my brain to a vague degree, but probably at least my verbal self-attention and memory formation capabilities become influenced by these interpreted perceptions, as well as priming of my perceptual systems for related stimuli, and relevant evaluations of reward/displeasure from my evaluative capabilities that will cause updates in my thought patterns for the future. All of these together make up being “conscious of the taste of chocolate,” but each one could be slightly different in me at different times, or in different people.
In other words, consciousness has no simple essence, nor is it made up of a small number of simple essences. This is especially relevant in animal consciousness—a dog is going to have many (though perhaps not all) of the macro-scale abilities I named in talking about my sensation of taste, but the lower-level details of their implementation are going to be different. “Being conscious” is not like “being more dense than water”—with density there’s only one possible dimension of variation, and there’s basically always a clear answer for which side of the line something is on, and if we’re curious about dogs we can just test a dog to see if it’s more dense than water and we’ll get a reliable result. With consciousness, all the small parts of it can vary, and which parts we care about may depend on what context we’re asking the question in! We might do better than trying to assign a binary value of “conscious or not” by assigning some degree of consciousness, but even that is still a one-dimensional simplification of a high-dimensional pattern.
P.S. Fight me, symmetry theory of valence stans. :P
How about Elo range normalized to time? If someone is 100 Elo points higher than me they win 64% of the time—if we play 3 games they’ll win best of 3 71% of the time, which is like 160 points higher rather than 100? So you figure out this function and then rank games by Elo per time
I have a totally non-Solomonoff explanation of what’s going on, which actually goes full G.K. Chesterton—I assign anthropic probabilities because I don’t assume that my not waking is impossible. But I’m not sure how a Solomonoff inductor would see it.
There’s a point by Stuart Armstrong that anthropic updates are non-Bayesian, because you can think of Bayesian updates as deprecating improbable hypotheses and renormalizing, while anthropic updates (e.g. updating on “I think just got copied”) require increasing probability on previously unlikely hypotheses.
In the last few years I’ve started thinking “what would a Solomonoff inductor do?” more often about anthropic questions. So I just thought about this case, and I realized there’s something interesting (to me at least).
Suppose we’re in the cloning version of Sleeping Beauty. So if the coin landed Heads, the sign outside the room will say Room A, if the coin landed Tails, the sign could say either Room A or Room B. Normally I translate this into Solomonoff-ese by treating different hypotheses about the world as Turing machines that could reproduce my memories. Each hypothesis is like a simulation of the world (with some random seed), plus some rule for specifying what physical features (i.e. my memories) to read to the output tape. So “Heads in Room A,” “Tails in Room A,” and “Tails in Room B” are three different Turing machines that reproduce the sequence of my memories by simulating a physical universe.
Where’s the non-Bayesian-ness? Well, it’s because before getting copied, but after the coin was flipped, there were supposed to only be two clumps of hypotheses—let’s call them “Heads and walking down the hallway” and “Tails and walking down the hallway.” And so before getting copied you assign only 50% to Tails. And so P(Tails|memories after getting copied) is made of two hypotheses while P(Tails|memories before getting copied) is only made of one hypothesis, so the extra hypothesis got bumped up non-Bayesianly.
But… hang on. Why can’t “Tails in Room B” get invoked as a second hypothesis for my memories even before getting copied? After all, it’s just as good at reproducing my past memories—the sequence just happens to continue on a bit.
What I think is happening here is that we have revealed a difference between actual Solomonoff induction, and my intuitive translation of hypotheses into Turing machines. In actual (or at least typical) Solomonoff induction, it’s totally normal if the sequence happens to continue! But in the hypotheses that correspond to worlds at specific points in time, the Turing machine reads out my current memories and only my current memories. At first blush, this seems to me to be an arbitrary choice—is there some reason why it’s non-arbitrary?
We could appeal to the imperfection of human memory (so that only current-me has my exact memories), but I don’t like the sound of that, because anthropics doesn’t seem like it should undergo a discontinuous transition if we get better memory.
Pretty sure this is the same (impressive) news as from CASP14 ( https://www.blopig.com/blog/2020/12/casp14-what-google-deepminds-alphafold-2-really-achieved-and-what-it-means-for-protein-folding-biology-and-bioinformatics/ ). But with fancier figures (edit: and more technical details of how they made the predictions) :P
We’re on SF14 now. Stockfish didn’t even have neural net evaluation until SF12 I think.
I just checked, and SF8 is from 2016. So still slightly off, but makes more sense.
I’m having some formatting problems (reading on lesswrong.com in firefox) with scroll bars under full-width LaTex covering the following line of text.
(So now I’m finishing reading it on greaterwrong.)
Nice! If I had university CS affiliations I would send them this with unsubtle comments that it would be a cool project to get students to try :P
In fact, now that I think about it, I do have one contact through the UIUC datathon. Or would you rather not have this sort of marketing?
A similarly odd question is how this plays with Solomonoff induction. Is a universe with infinite stuff in it of zero prior probability, because it requires infinite bits to specify where the stuff is? Quantum mechanics would say no: we can just specify a simple quantum state of the early universe, and then we’re within one branch of that wavefunction. And the (quantum) information required to locate us within that wavefunction is only related to the information we actually see, i.e. finite.
Weird coincidence, but I just read Superintelligence for the first time, and I was struck by the lack of mention of Steve Omohundro (though he does show up in endnote 8). My citation for instrumental convergence would be Omohundro 2008.
Classes for what audience?
College classes often offer more diversity, so if I propose something like “the art of makeup” maybe there’s already something like that at your local college. But for middle- and high-schoolers there are lots of much viewed, much-rated youtube tutorials that could easily be classes.
Actually, a lot of these tutorials are things like machining / maker videos that used to be popular classes in school but have gotten dropped. Why? Partially budgets not keeping up with cost of living of teachers and staff (the part of “cost disease” that just means that human-intensive jobs are more expensive relative to automatable jobs than they used to be) leading to effective budget cuts over the years for most public schools. But perhaps also misguided concerns about “practicality” or perverse incentives due to high-stakes testing, both of which would make new classes difficult to implement.
But still, some ideas that haven’t been mentioned might look like CNC Machining, 3d printing, Robotics, the aforementioned Art of Makeup, Smartphone life hacks, Custom T-shirt making, Cell-phone photography, how to dance like the people in music videos.
The other class I want to see more of is how to use AI tools to create digital media.
Some thoughts on reading Superintelligence (2014). Overall it’s been quite good, and nice to read through such a thorough overview even if it’s not new to me. Weirdly I got some comments that people often stop reading it. What this puts me in mind of is a physics professor remarking to me that they used to find textbooks impenetrable, but now they find it quite fun to leaf through a new introductory textbook. And now my brain is relating this to the popularity of fanfiction that re-uses familiar characters and settings :P
By god, Nick Bostrom thinks in metaphors all the time. Not to imply that this is bad at all, in fact it’s very interesting.
The way the intelligence explosion kinetics is presented really could stand to be less one-dimensional about intelligence. Or rather, perhaps it should ask us to accept that there is some one-dimensional measure of capability that is growing superlinearly, which can then by parlayed into all the other things we care about via the “superpower”-style arguments that appear two chapters later.
Has progress on AI seemed to outpace progress on augmenting human intelligence since 2014? I think so, and perhaps this explains why Bostrom_2014 puts more emphasis on whole brain emulations. But perhaps not—perhaps instead I’ve/we’ve been unduly neglecting thinking about alternate paths to superintelligence in the last few years.
Human imitations (machine learning systems trained to reproduce the behavior of a human within some domain) seem conspicuously absent from Bostrom_2014′s toolbox of parts to build an aligned AI out of. Is this a reflection of the times? My memory is blurry but… plausibly? If so, I think that’s that’s a pretty big piece of conceptual progress we’ve made.
When discussing what moral theory to give a superintelligence, Bostrom inadvertently makes a good case for another piece of conceptual progress since 2014 - our job is not to find The One Right Moral Theory, it is both more complicated and easier than that (see Scott, Me). Hopefully this notion has percolated around enough that this chapter would get written differently today. Or is this still something we don’t have consensus on among Serious Types? Then again, it was already Bostrom who coined the term MaxiPOK—maybe the functional difference isn’t too large.
I would have thought “maybe the CEV of humanity would just shut itself down” would be uttered with more alarm.
In conclusion, footnotes are superior to endnotes.
The idea is that a model of the world that helps you succeed inside the box might naturally generalize to making consequentialist plans that depend on whether you’re in the box. This is actually closely analogous to human intellect—we evolved our reasoning capabilities because they helped use reproduce in hunter-gatherer groups, but since then we’ve used our brains for all sorts of new things that evolution totally didn’t predict. And when we are placed in the modern environment rather than hunter-gatherer groups, we actually use our brains to invent condoms and otherwise deviate from what evolution originally thought brains were good for.
You have probably experienced the phenomenon of driving for miles while engrossed in conversation (or in silent soliloquy) and then discovering that you have utterly no memory of the road, the traffic, your car-driving activities. It is as if someone else had been driving. Many theorists (myself included, I admit) have cherished this as a favorite case of “unconscious perception and intelligent action.” But were you really unconscious of all those passing cars, stop lights, bends in the road at the time? You were paying attention to other things, but surely if you had been probed about what you had just seen at various moments on the drive, you would have had at least some sketchy details to report.
Dan Dennett
What, then is Dennett’s alternative picture of consciousness? He calls it the multiple drafts model, and what it is is a wholehearted embracing of the distributed nature of human minds. If you probe someone’s consciousness different ways, like asking them to press a button now versus report what they remember later, you can sometimes get different answers, because the state of someone’s mind is distributed throughout their brain, and different probes can access different facts about that state. It’s like the thing that gets probed, which gets fixated into consciousness when we direct attention to it, has multiple drafts of itself available to different systems of the brain, and these drafts get passed around and edited as time passes.
One way is to choose strategies that depend on whether or not you’re in the box. The classic example is to wait for some mathematical event (like inverting some famous hash function) that is too computationally expensive to do during training. But probably there will be cheaper ways to get evidence that you’re out of the box.
Now you might ask, why on earth would an agent trained in the box learn a strategy like that, if the behavior is identical inside the box? And my answer is that it’s because I expect future AI systems to be trained to build models of the world and care about what it thinks is happening in the world. An AI that does this is going to be really valuable for choosing actions in complicated real-world scenarios, ranging from driving a car at the simple end to running a business or a government at the complicated end. This training can happen either explicitly (i.e. having different parts of the program that we intentionally designed to be different parts of a model-building agent) or implicitly (by training an AI on hard problems where modeling the world is really useful, and giving it the opportunity to build such a model). If the trained system both cares about the world, and also knows that it’s inside the box and can’t affect the world yet, then it may start to make plans that depend on whether it’s in the box.
My go-to reference for this reasoning is Stuart’s post: https://www.alignmentforum.org/posts/4ZRDXv7nffodjv477/anthropic-reasoning-isn-t-magic
I’m going to have some criticism here, but don’t take it too hard :) Most of this is directed at our state of understanding in 2012.
I think a way to do better is not to mention SSA or SIA at all, and just talk about conditioning on information. Don’t even have to say “anthropic conditioning” or anything special—we’re just conditioning on the fact that sampling from some distribution (e.g. “worlds with intelligent life who figure out evolution”) gave us exactly our planet. (My own arguments for this on LW date from c. 2015, but this was a common position in cosmology before that.)
This gives you information that is more “anthropic” than SSA, but more specific than SIA. We can now ask probabilistic questions entirely in the language of conditional probabilities, which tells you more about what empirical questions are important. E.g. “What us the probability that octopus-level intelligence evolves on an earth-like planet in the milky way, conditional on some starting distribution over models of evolution, and further conditional on a sampling process from planets with human-level intelligence returning Earth?” The task is simply to update the models of evolution by reweighting according to how well they predict that sampling from our reference class give us.
Also, assuming all distributions are uniform gives one an unrealistic picture of timing there at the end. Think about what happens if the distributions are Poisson!
Footnote: Armstrong argues something more niche than that, because he’s not talking about a “normal” CDT agent doing averaging/totalling, he’s talking about an ADT agent doing averaging/totalling, and these are very different baseline agents!
What this makes me think is that quantum computing is mostly doomed. The killer app for quantum computing is predicting molecules and electronic structures. (Perhaps someone would pay for Shor’s algorithm, but its coolness far outstrips its economic value). But it’s probably a lot cheaper to train a machine-learning based approximation on a bunch of painstakingly assembled data than it is to build enough 50 milliKelvin cyostats. According to this view, the physics labs that will win at superconductor prediction are not the ones working on quantum computers or on theoretical breakthroughs, they’re going to be the guys converting every phonon spectrum from the last 50 years into a common data format so they can spend $30K to train a big 3D transformer on it.