I’m just reading out my mechanistic priors here (mostly not answering about specifically covid, from specific data).
In the same way that masks made sense in February of 2020 my reaction was “like duh? the viral particles are bigger than the particles that well fitted N95 masks already keep out? how could it NOT be helpful? are people really just this dumb?”
The immune system is much more complicated than masks, and has exponential growth INSIDE the mechanism, so it is way more understandable that people wouldn’t “get it”...
But applying the same kind of “there is a general system that generally works” reasoning here...
I. PHASES OF INFECTION OVER TIME
When you get virally infected WITHOUT a vaccine, the reason it takes “N days” to fight off a virus is because your immune system initially fights a LOSING war against the virus until an “antibody-reliant super weapon” can be constructed, kinda like a Manhattan Project, but more focused.
Once your immune system gets specifically able to fight the specific virus the fight gets much shorter and your immune system nearly always wins.
(Sometimes not: HIV is an exception, for example, because it evolves faster than your immune system can complete super-weapon projects to target the recent/past/obsolete version of your own infection in your own body. Herpes is a partial exception because SOMEHOW(?) it hides in a nook invisible to our entire immune system? Biology is rife with exceptional details.)
N days until disease goes away =
“time to design antibodies after exposure but before symptoms” +
“more time to design antibodies during symptoms while still losing the fight” +
“more time to USE the antibodies to win the fight at the end”
= “1-3 weeks for influenza (and mostly on the shorter end of that range)”.
MOST of the time is spent on research.
The last term in the equation above is generally pretty small. It would be even smaller if it was not performing a naively astonishing come-from-behind victory maneuver against a vast horde of gazillions of infected cells and viral particles.
The “time to form antibodies” varies across people, but the reason for the “3 week delay” between two vaccines is because the vaccine designers wanted to systematically wait until like 99% of the population would have finished this response, even sickly people. This way the first response and the second response would be biologically distinct exposures, but also the treatment with both would count be “the same for everyone and logically a unit in most people’s minds” which has nice political and PR properties.
II. IDEALIZED OPERATION
Some very healthy people could probably get away with a delay of possibly just a week?
If there existed many competent doctors with schedules that were wide open (or just an autodoc in every home) the autodoc could measure things directly and give you a second vaccine in the very hour that the immune response is good enough, and this might be only 5 days for some people? Reason: because some people get better from influenza pretty darn fast.
So suppose that someone got infected with the same strain of influenza right after they fought it off once already? Like they drink 4 ounces of “give you influenza fluid” for science...
Then they fast forward through most of the earlier immune machinery’s operation. They ALREADY have super weapons against exactly that influenza strain already designed, and design is the slowest step.
So the first few cycles of the SECOND infection can be thought of like something from a medieval anime war story that is an amusingly fast win for the good guys :-)
There are 4 (the inoculating dose) “ninjas with mind control magic scrolls” (viruses) who sneak into the small feudal nation (you), and while they are bespelling the first 4 gate keepers or farmers or whatever according to a basic magical mind control doubling schedule… There is a dog nearby that already has the scent of exactly those ninjas and exactly their specific paper for their specific mind control scrolls (from the previous infection), so the dog smells the ninjas and then howls, and howls spread quickly until they are heard several miles away at the superweapon depot.
The howls brings in killer robots with special algorithms to pierce every hiding trick the ninjas might employ (same as the first dog used, basically) and ends the game almost instantly. The killer robots take out the “at most 64-256 ninjas” and the invasion is over. EVEN if the robots had had to face 1,073,741,824 (==2^30) ninjas the robots would have STILL won (because robots are awesome). That would have been a long fight.
However, because the dog knew the scent, and the robot got there, and the robot knew the scent as well, the response is super early, and super fast, and the fight doesn’t even take that long, and all together it prevents most of the possible damage to the little kingdom.
So a lot potentially hinges on your own personal vitality, and exactly which strain infects you. If the “smell” of the invaders isn’t the same as before, the event will be BOTH more damaging AND take longer to deal with.
III. APPLICATION OF THE MODEL
If you are healthy and the strains are close to what the vaccine targeted, then your average infection period will be like… half a day? Maybe 2 days?
Also, you will almost never notice it, because the metaphorical robots will kill the ninjas before there are enough ninjas to cause much damage to the countryside and form a ninja army to attack the lungs, or the brain, or whatever.
(Part of why covid is so freaking horrible is that it is infectious BEFORE the symptoms show up because the lab that created it probably used serial passage on humanized lab animals to make sure that the fit between the spike protein and human ACE receptors is very efficient. From the very beginning, in December, in Wuhan, it transmitted human-to-human BEFORE the fever starts, which means that the temp checking system that defeated Version 1 of SARS wasn’t going to be good enough for this new version of SARS. If not for this detail, it wouldn’t even be on my radar as a thing to worry about that you could pass covid during these quickly suppressed microinfections that vaccinated immune systems handle automatically all the time.)
HOWEVER ALSO, clinical studies of “people who are vaccinated but still somehow got sick” will therefore, over time, come to be based on either people who are sickly or who have new strains, or both.
(For example, like 20% of the US might be classified as some-kind-of-alcoholic, because even just 1 drink of alcohol wipes out a large percent of your immune cells for hours or days, and then your hematopoietic cells rebuild them at the cost of telomere shorting. Such people often have relatively wimpy immune systems, especially late in life.)
So if you’re healthy and not exposed to brand new strains, the time during which you might be kinda infectious MIGHT be roughly 1.5 days in (while awesome robots kill the kind of ninjas they were programmed for)? Maybe less than 1.5 days? Maybe hours?
(I have half-heartedly looked for evidence of vaccinated healthy people being “the only logically possible link in a transmission chain” and have not found much evidence of such things. My search was not thorough, however.)
BUT if you are exposed to newly antibody-escaping strains, or are just “not healthy in the first place”, or both, then you could look at naive clinical studies (that might not be careful about admitting or explaining the straightforward theoretical implications of their sampling methods), to get a sense of how your own bad luck situation might play out.
If you are unlucky, then studies performed on other unlucky people could show a rough infection, amenable to therapeutic interventions, that occurs partly in a hospital where “data collection to power formal studies” is easier and cheaper to collect.
Long story short: assuming a healthy immune system, and no immune escape from a new virus variant, your infectious period is probably much shorter (or even essentially non-existent) after you are successfully vaccinated.
This might be wrong. These are just priors, and a general model. (HIV is an exception to this model that I already know about, for example.)
However, I think there are not many ways that (1) the model has an essentially broken structure with respect to the immune system in general or (2) covid has some additional specific wrinkle to its biology such that it routes around the normal immune responses.
If someone existed in a conversation with me, in a similar state of “working almost entirely from priors”, and they put $10 at risk while I put $20 at risk, and they won my $20 if they found something on the internet that defeated my model or basic conclusions, but if they couldn’t find anything definite within a couple hours I would win their $10… then I would accept the bet… and be happy to learn how I was wrong <3
On the open internet, “against all comers”, bets work differently, and mean more. I’m not offering this bet here.
All I’m reporting here is my subjective “conversational bet tolerance” for “my priors, and that the priors are basically adequate to model this situation without extensively consulting with the internet first”.
V. POLITICS AND NOVELTY
The US essentially doesn’t have a public health system anymore.
The giving of free vaccines is what such a system would do if it existed, but for covid vaccines, this was done as an emergency one-off, because...
...so basically none of what I said about vaccinated immune systems applies to a hypothetical “very new strain” where a single one could arise in a place with a lot of infected people and then evade hypothetical border controls AND also evade current antibodies produced by the current vaccine.
To account for this risk you would have to have a positive predictive evolutionary model?
And I can’t do that, and don’t know of anyone who can. Maybe nothing will evolve? That’s what I hope.
If hope isn’t good enough, and you don’t count political solutions you basically have to fall back to really generic solutions like “retreat from the world (maybe as a project)” or else maybe just “cryonics and crossed fingers”?
Or abandon the constraint of not “being allowed to modify the current political disaster directly” and then like… uh… gain power… and fix border controls relevant to public health (and public health in general) directly for real?
Or just hope that Obama cleans house somehow eventually via more backroom deals or something?
Or take extra advantage of the brief cessation in the covid horror, while vaxxed against the current strain… because there’s no coherently structurally certain reason that the next wave won’t have already started up again by next Christmas?
If I couldn’t get a vaccine, and was (justifiably) relying on the herd to protect me from plagues, I would be SUPER angry at the herd for not doing this properly.
Your housemate has my sympathy.
I noticed (while reading your great modeling exercise about an important topic) a sort of gestalt presumption of “one big compartment (which is society itself)” in the write-up and this wasn’t challenged by the end.
Maybe this is totally valid? The Internet is a series of tubes, but most of the tubes connect to each other eventually, so it is kinda like all one big place maybe? Perhaps we shall all be assimilated one day.
But most of my thoughts about modeling how to cope with differences in preference and behavior focus a lot on the importance of spacial or topological or social separations to minimize conflicts and handle variations in context.
My general attitude is roughly: things in general are not “well mixed” and (considering how broken various things can be in some compartments) thank goodness for that!
This is a figure from this research where every cell basically represents a spacially embedded agent, and agents do iterated game playing with their neighbors and then react somehow.
In many similar bits of research (which vary in subtle ways, and reveals partly what the simulation maker wanted to see) a thing that often falls out is places where most agents are being (cooperatively?) gullible, or (defensively?) cheating, or doing tit-for-tat… basically you get regions of tragic conflict, and regions of simple goodness, and tit-for-tat is often at the boundaries (sometimes converting cheaters through incentives… sometimes with agents becoming complacent because T4T neighbors cooperate so much that it is easy to relax into gullibility… and so on).
A lot depends on the details, but the practical upshot for me is that it is helpful to remember that the right thing in one placetime is not always the right thing everywhere or forever.
Arguably, “reminding people about context” is just a useful bravery debate position local to my context? ;-)
With a very simple little prisoner’s dilemma setup, utility is utility, and it is clear what “the actual right thing” is: lots of bilateral cooperate/cooperate interactions are Simply Good.
However in real life there is substantial variation in cultures and preferences and logistical challenges and coordinating details and so on.
It is pretty common, in my experience, for people to have coping strategies for local problems that they project out on others who are far from them, which they imagine to be morally universal rules. However, when particular local coping strategies are transported to new contexts, they often fail to translate to actual practical local benefits, because the world is big and details matter.
Putting on a sort of “engineering hat”, my general preference then is to focus on small specific situations, and just reason about “what ought to be done here and now” directly based on local details the the direct perception of objective goodness.
The REASON I would care about “copying others” is generally either (1) they figured out objectively good behavior that I can cheaply add to my repertoire, or (2) they are dangerous monsters who will try to hurt me of they see me acting differently. (There are of course many other possibilities, and subtleties, and figuring out why people are copying each other can be tricky sometimes.)
Your models here seem to be mostly about social contagion, and information cascades, and these mechanisms read to me as central causes of “why ‘we’ often can’t have nice things in practice” …because cascading contagion is usually anti-epistemic and often outright anti-social.
You’re having dinner with a party of 10 at a Chinese restaurant. Everyone else is using chop sticks. You know how to use chop sticks but prefer a fork. Do you ask for a fork? What if two other people are using a fork?
I struggled with this one because I will tend to use a chopstick at Chinese restaurants for fun, and sometimes I’m the only one using them, and several times I’ve had the opportunity to teach someone how to use them. The alternative preference in this story would be COUNTERFACTUAL to my normal life in numerous ways.
Trying to not fight the hypothetical too much, I could perhaps “prefer a fork” (as per the example) in two different ways:
(1) Maybe I “prefer a fork” as a brute fact of what makes me happy for no reason. In this case, you’re asking me about “a story person’s meta-social preferences whose object-level preferences are like mine but modified for the story situation” and I’m a bit confused by how to imagine that person answering the rest of the question. After making an imaginary person be like me but “prefer a fork as a brute emotional fact”… maybe the new mind would also be different in other ways as well? I couldn’t even figure out an answer to the question, basically. If this was my only way to play along, I would simply have directly “fought the hypothetical” forthrightly.
(2) However, another way to “prefer a fork” would be if the food wasn’t made properly for eating with a chopstick. Maybe there’s only rice, and the rice is all non-sticky separated grains, and with a chopstick I can only eat one grain at a time. This is a way that I could hypothetically “still have my actual dietary theories intact” and naturally “prefer a fork”… and in this external situation I would probably ask for a fork no matter how unfun or “not in the spirit of the experience” it seems? Plausibly, I would be miffed, and explain things to people close to me who had the same kind of rice, and I would predict that they would realize I was right, nod at my good sense, and probably ask the waiter to give them a fork as well.
But in that second try to generate an asnwer, it might LOOK like the people I predicted might copy me would be changing because “I was +1 to fork users and this mapped through a well defined social behavior curve feeling in them” but in my mental model the beginning of the cascade was actually caused by “I verbalized a real fact and explained an actually good method of coping with the objective problem” and the idea was objectively convincing.
I’m not saying that peer pressure should always be resisted. It would probably be inefficient for everyone to think from first principles all the time about everything. Also there are various “package deal” reasons to play along with group insanity, especially when you are relatively weak or ignorant or trying to make a customer happy or whatever. But… maybe don’t fall asleep while doing so, if you can help it? Elsewise you might get an objectively bad result before you wake up from sleep walking :-(
Response appreciated! Yeah. I think I have two hunches here that cause me to speak differently.
One of these hunches is that hunger sensors are likely to be very very “low level”, and motivationally “primary”. The other is maybe an expectation that almost literally “every possible thought” is being considered simultaneously in the brain by default, but most do not rise to awareness, or action-production, or verbalizability?
Like I think that hunger sensors firing will cause increased firing in “something or other” that sort of “represents food” (plus giving the food the halo of temporary desirability) and I expect this firing rate to basically go up over time… more hungriness… more food awareness?
Like if you ask me “Jennifer, when was the last time you ate steak?” then I am aware of a wave of candidate answers, and many fall away, and the ones that are left I can imagine defending, and then I might say “Yesterday I bought some at the store, but I think maybe the last time I ate one (like with a fork and a steakknife and everything) it was about 5-9 days ago at Texas Roadhouse… that was certainly a vivid event because it was my first time back there since covid started” and then just now I became uncertain, and I tried to imagine other events, like what about smaller pieces of steak, and then I remembered some carne asada 3 days ago at a BBQ.
What I think is happening here is that (like the Halle Berry neuron found in the hippocampus of the brain surgery patient) there is at least one steak neuron in my own hippocampus, and it can be stimulated by hearing the word, and persistent firing of it will cause episodic memories (nearly always associated with places) to rise up. Making the activations of cortex-level sensory details and models conform to “the ways that the entire brain can or would be different if the remembered episode was being generated from sensory stimulation (or in this case the echo of that as a memory)”.
So I think hunger representations, mapped through very low level food representations, could push through into episodic memories, and the difference between a memory and a plan is not that large?
Just as many food representing neurons could be stimulated by deficiency detecting sensory neurons, the food ideas would link to food memories, and food memories could become prompts to “go back to that place and try a similar action to what is remembered”.
And all the possible places to go could be activated in parallel in the brain, with winnowing, until a handful of candidates get the most firing because of numerous simultaneous “justifications” that route through numerous memories or variations of action that would all “be good enough”.
The model I have is sort of like… maybe lightning?
An entire cloud solves the problem of finding a very low energy path for electrons to take to go from electron dense places to places that lack electrons, first tentatively and widely, then narrowly and quickly.
Similarly, I suspect the entire brain solves the problem of finding a fast cheap way to cause the muscles to fire in a way that achieves what the brain stem thinks would be desirable, first tentatively and widely, then narrowly and quickly.
I googled [thinking of food fMRI] and found a paper suggesting: hippocampus, insula, caudate.
Then I googled [food insula] and [food caudate] in different tabs. To a first approximation, it looks like the caudate is related to “skilled reaching” for food? Leaving, by process of elimination: the insula?
And uh… yup? The insula seems to keep track of the taste and “goal-worthiness” of foods?
In this review, we will specifically focus on the involvement of the insula in food processing and on multimodal integration of food-related items. Influencing factors of insular activation elicited by various foods range from calorie-content to the internal physiologic state, body mass index or eating behavior. Sensory perception of food-related stimuli including seeing, smelling, and tasting elicits increased activation in the anterior and mid-dorsal part of the insular cortex. Apart from the pure sensory gustatory processing, there is also a strong association with the rewarding/hedonic aspects of food items, which is reflected in higher insular activity and stronger connections to other reward-related areas.
So my theory is:
Biochemistry --> hunger sensors (rising, enduring) --> insula (rising, enduring) -->
--> hippocampus (also triggerable by active related ideas?) --> memories sifted --> plans (also loop back to hippocampus if plans trigger new memories?) -->
--> prefrontal cortex(?) eventualy STOPS saying “no go” on current best mishmash of a plan -->
--> caudate (and presumably cerebellum) generate --> skilled food seeking firing of muscles to act in imagined way!
The arrows represent sort of “psychic motivational energy” (if we are adopting a theory of mind) as well as “higher firing rate” as well as maybe “leading indicator of WHICH earlier firing predicts WHICH later firing by neurons/activities being pointed to”.
I think you have some theories that there’s quite a few low level subsystems that basically do supervised learning on their restricted domain? My guess is that the insula is where the results of supervised learning on “feeling better after consuming something” are tracked?
Also, it looks like the insula’s supervised learning algorithms can be hacked?
The insular cortex subserves visceral-emotional functions, including taste processing, and is implicated in drug craving and relapse. Here, via optoinhibition, we implicate projections from the anterior insular cortex to the nucleus accumbens as modulating highly compulsive-like food self-administration behaviors
Trying to reconcile this with your “telencephalon” focus… I just learned that the brain has FIVE lobes of the cortex, instead of the FOUR that I had previously thought existed?! At least Encarta used to assert that there are five...
These and other sulci and gyri divide the cerebrum into five lobes: the frontal, parietal, temporal, and occipital lobes and the insula. [bold not in original]
Until I looked up the anatomy, I had just assumed that the insula was part of the brain stem, and so I thought I won some bayes points for my “hard wiring” assumption, but the insula is “the lobe” hiding in the valley between the temporal cortex and the rest of the visible surface lobes, so it is deep down, closer to the brain stem… So maybe you win some bayes points for your telencephalon theory? :-)
Kant thought that space being Euclidean was a priori logically necessary, hence determinable from pure thought, hence true without need for empirical fact checking… and in the end this turned out to be wrong. Einstein had the last laugh (so far).
I have wondered now and again whether it might be that Cox’s Postulates are similar to Euclid’s Postulates and might have similar subtle exceptional discrepancies with physical reality in practice.
It is hard to form hypotheses here, partly for a lack of vivid theoretical alternatives. I know of two claims floating around in the literature that hint at substantive alternatives to Bayes.
One approach involves abandoning at least one of Aristotle’s three laws of thought (excluded middle, non-contradiction, and identity) and postulating, essentially, that reality itself might be ontologically ambiguous. If I had to pick one to drop, I think I’d drop excluded middle. Probably? Constructionist/intuitionist logic throws that one out often, and automated proof systems often leave it out by default. Under the keywords “fuzzy logic” there were attacks on these laws that directly reference Jaynes. So this is maybe one way to find a crack in the universe out of which we might wiggle.
The only other approach I know of in the literature is (for me) centrally based on later chapters in Scott Aaronson’s “Quantum Computing Since Democritus” (try clicking the link and then do ^f bayes) where, via hints and aspersions, Aaronson suggests that quantum mechanics can be thought of as Bayesian… except with complex numbers for the probabilities, and thus (maybe?) Bayesianism is essentially a potentially empirically false religion? Aaronson doesn’t just say this directly and at length. And his mere hints would be the place I left this summary… except that while hunting for evidence I ran across a link to what might be a larger and more direct attack on the physical reality of Bayesianism? (Looking at it: using axioms no less! With “the fifth axiom” having variations, just like Euclid?!)
So that arxiv paper by Lucien Hardy (that I missed earlier! (that was written in 2008?!?)) might just have risen to the top of my philosophy reading stack? Neat! <3
Maybe it is worth adding a third approach that I don’t think really counts… When the number of variables in a belief net goes up, the difficulty of simply performing mere inference becomes very hard to compute, with relatively general assumptions the algorithms ending up in NP-hard. This “doesn’t count as a real deep philosophically satisfying alternative to Bayes” for me because it seems like the practical upshot would just be that we need more CPU, and more causal isolation for the systems we care about (so their operation is more tractable to reason). Like… the practical impossibility of applying Bayes in general to large systems would almost help FIGHT the the other “possible true/deep alternatives” to Bayes, because it creates an alternative explanation for any subjective experience of sorta feeling like you had probabilities figured out, and then your probabilities came out very wrong. Like: maybe there were too many variables, and the NP-hardness just caught up with you? Would you really need to question the “laws of thought” themselves to justify your feeling of having been been in the physical world and then ended up “surprisingly surprised”? Seriously? Seriously?
Anyway. I was wondering if you, having recently looked at the pillars of pure thinking themselves, had thoughts about any cracks, or perhaps any even deeper foundations, that they might have :-)
I wrote the following before reading the three hypotheses or other comments. The links were added after I copied the text over to the website:
My hypothesis for how the salt pursuing behavior worked is that there’s a generic “entity” subsystem in the brain.
It has a deep prior that “things” exist and it figures out the sound/smell/appearance of “things”.
Where is “thing modeling” implemented? Something something temporal cortex, something something hippocampus, something something halle berry neuron.
As a similarly deep prior, some “things” are classified in terms of a “food” category, and for all such things there is a taste vector which is basically a list of values for how full of nutrient/features the “thing” is when considered as “food”. There are low level rewards to help “educate the palate” which fills in this part of a model of the world and so: goats and rats and babies constantly stick stuff in their mouths.
The features/characteristics of “things” that are “edible” are non-trivial, and based on complex sensors that can essentially perform a chemical assay. These assays are central to everything. There’s thousands of smells that could support a dimensional analysis engine that would have probably had something like 2 billion years of evolution to develop.
There are receptors which measure how glucose-like a substance is, for example, and their operation is ancient and also non-trivial. Also acidity and saltiness are simpler molecular ideas than glucose, and all of these simple molecules were “learned about” by DNA-based learning for a long time before neurons even existed.
Single celled organisms already have chemical assay based behavioral learning. Smell was almost certainly the first sense. In a way, the only sense 99% of neurons have is a sense of smell.
Dogs have a lot of smell sensors (and bad eyesight) whereas human noses are far from the ground, but we haven’t lost all the smell receptors to mutational load and lack of selection yet. We probably will never lose them all?
So then there’s a large number of nutrient deficiencies and chemical imbalances that are possible. Low blood sugar is an obvious one, but many others could exist. Every amino acid’s levels could, individually, be tracked and that would “just make sense” on a first principles evolutionary level for there to be chemical sensors here.
Sensors for the chemical or physiological precursors of basically any and all serious nutrient deficiency could exist in large numbers, with many subtypes. Some of the microbiological sensors could modulate the conscious mind directly, the evidence being that many people know what it feels like to “feel hungry” in different ways, but different signals could be subconscious quite easily and not violate the hypothesis? Just have the optimization signals propagate via paths far from the verbal awareness loops, basically?
The claim is simply that the microsensory details here could exist in the body, and send signals to some part of the brain stem that recognizes “food-fixable deficiencies” that treat food seeking as a first class task. Call all these subtypes of chemically measurable hunger different internal “deficiency sensors”?
Any big obvious linkages between a deficiency and a tastable food feature that could reliably occur over evolutionary timescales is likely to already exist at a very low level, with hard coded linkages.
Basically we should expect a many-to-many lookup table between “foods” and how much each food type would help avoid worsening over supply (and/or helping with under supply) of every nutrient, with the only learning being the food list contents (plus where each food is and so on, for motor planning) and then learned “deficiency fixing” values for each food (plausibly subconsciously).
When the vector of various kinds of “hunger levels” changes, a relatively hard coded circuit probably exists that (abstractly (maybe even literally?)) assigns each food a new dot product value in terms of general food goodness, and foods with high values “sort to the top”, after which a whole planning engine kicks in, confabulating plans for getting any or all such high value foods and throwing out the implausible plans, until a food with a high value and a plausible plan is left over.
Then after a while (something something prefrontal cortex, something something inhibition of motor plans) the plan is tried. Sometimes it will be a terrible plan, like “eat sand (because your parents suck at being providers)” but we can’t keep parents from torturing their children using insane dietary theories… that’s not even an important example… it mostly just shows that human conscious thought is mostly not that great. The main case is just like, a dog whose puppies are hungry because it sucks at hunting, so the puppies get weird deficiencies, and then the puppies that wordlessly/atheoretically solve these cravings somehow survive to have more puppies.
Anyway… the intrinsic horrifying bleakness of biology aside… in low cephalization animals (like cows) with bodies and habitats and habits that are already “known viable”, evolution’s instrumental task here doesn’t need to be very complicated. Maybe slightly modulate which grasses or herbs are eaten, and wander in the direction of places where the desired plants tend to grow more?
For some animals (like chimps) the confabulation engine turned out to be pretty valuable, and it just kept getting bigger over time. Hence human brains getting progressively bigger confabulation engines.
When humans starve, they become obsessed with imagination about food. A nearly identical process plausibly explains romantic obsessions. And so on. A HUGE amount of human behavioral reasoning is plausibly “pica with more steps”.
Basically, my theory is that “mammal brains have first class specialized treatment of edible physical objects, the mice got pica for salt, and the source of salt was already in the relevant foodthing/location memory slot, and the obvious half-hard-coded action happened when the relevant check engine light came on and a new part of the dot product calculator for food value reversed in sign… for the first time ever (in the experiment)”.
This reminded me of an essay on artificial pain, where people whose hands could no longer feel pain in some part of their body would get a computer controlled device attached to another part of their body that would hurt them where they were sensitive if they did something that could damage their insensitive body parts.
One problem was calibrating the mechanism to really detect damaging actions (preferably before damage became at all severe).
The deeper problem was that the entire mechanism was optional.
Even the patients who were most “adherent” to the treatment and the most intellectually “bought in” to the idea that pain could protect them from damage would sometimes briefly turn the device off to do things that would cause damage to their hands that they just “really wanted to do” in that moment.
It feels like there’s a lurking idea here related to agency operating on longer time scales? Something something months, years, and decades? Something something low pass filter? Something something Parfit?
I had not heard of casino self exclusion before! Thanks for that pointer :-)
I googled for mentions of [confidence building measures] on Lesswrong and felt mildly saddened and surprised that there were so few, but happy that there were at least some. A confidence building measure (or “CBM”) is a sort of “soft power” technique where people find ways to stay chill and friendly somewhat far away from possible battle lines, in order to make everyone more confident that grownups are in charge on both sides. They seem like they could help rule out large classes of failure modes, if they work as they claim to work “on the tin”.
In the search, I found 5 hits. One from me in 2011 and another from me in 2017. Two mentions are “incidental” (in biblios) but there was a pretty solid and recent and direct hit with [AN #146]: Plausible stories of how we might fail to avert an existential catastrophe. which is pointing to a non-less-wrong text on CBMs. Quoting the core of AN #146 here:
This paper explores confidence-building measures (CBMs) as a way to reduce the negative effects of military AI use on international stability. CBMs were an important tool during the Cold War. However, as CBMs rely on a shared interest to succeed, their adoption has proven challenging in the context of cybersecurity, where the stakes of conflict are less clear than in the Cold War. The authors present a set of CBMs that could diminish risks from military use of AI and discuss their advantages and downsides. On the broad side, these include building norms around the military use of AI, dialogues between civil actors with expertise in the military use of AI from different countries, military to military dialogues, and code of conducts with multilateral support. On the more specific side, states could engage in public signalling of the importance of Test and Evaluation (T&E), transparency about T&E processes and push for international standards for military AI T&E. In addition, they could cooperate on civilian AI safety research, agree on specific rules to prevent accidental escalation (similar to the Incidents at Sea Agreement from the Cold War), clearly mark autonomous systems as such, and declare certain areas as off-limits for autonomous systems.
The paper itself appears to be authored by… maybe this Michael Horowitz and then this Paul Scharre seems like an excellent bet for the other author. A key feature here in my mind is that Horowitz appears to already have a functional enduring bipartisan stance that seems to have survived various US presidential administration changes? This is… actually kinda hopeful! :-)
(Edit: Oh wait. Hopefulness points half retracted? The “AI IR Michael Horowitz” is just a smart guy I think, and not specifically a person with direct formal relationships with the elected US govt officials. Hmm.)
There are specific ideas, backstory, phrases, and policy proposals in the article itself: AI and International Stability: Risks and Confidence-Building Measures. Here’s a not-entirely-random but not especially critical part of the larger article:
States have long used established “rules of the road” to govern the interaction of military forces operating with a high degree of autonomy, such as at naval vessels at sea, and there may be similar value in such a CBM for interactions with AI-enabled autonomous systems. The 1972 Incidents at Sea Agreement and older “rules of the road” such as maritime prize law provide useful historical examples for how nations have managed analogous challenges in the past. Building on these historical examples, states could adopt a modern-day “international autonomous incidents agreement” that focuses on military applications of autonomous systems, especially in the air and maritime environments. Such an agreement could help reduce risks from accidental escalation by autonomous systems, as well as reduce ambiguity about the extent of human intention behind the behavior of autonomous systems.
I guess the thing I’d want to communicate is that it doesn’t seem to be the case that there are literally ZERO competent grownups anywhere in the world, even though I sometimes feel like that might be the case when I look at things like the world’s handling of covid. From what I can tell, the grownups that get good things done get in, apply competence to do helpful things, then bounce out pretty fast afterwards?
Another interesting aspect is that the letters “T&E” occur 31 times in the Horowitz & Scharre paper which is short for the “Testing & Evaluation” of “new AI capabilities”. They seem to want to put “T&E” processes into treaties and talks as a first class object, basically?
States could take a variety of options to mitigate the risks of creating unnecessary incentives to shortcut test and evaluation, including publicly signaling the importance of T&E, increasing transparency about T&E processes, promoting international T&E standards, and sharing civilian research on AI safety.
Neither italics nor bold in original.
But still… like… wisdom tournament designs? In treaties? That doesn’t seem super obviously terrible if we can come up with methods that would be good in principle and in practice.
Something I find myself noticing as a sort of a gap in the discourse is the lack of the idea of a “right” and specifically the sort of lowest level core of this: a “property right”.
It seems to me that such things emerge in nature. When I see dogs (that aren’t already familiar) visit each other, and go near each other’s beds, or food bowls… it certainly seems to me, when I empathically project myself into the dog’s perspectives as though “protection of what I feel is clearly mine against another that clearly wants it” is a motivating factor that can precipitate fights.
(I feel like anyone who has been to a few big summer BBQs where numerous people brought their dogs over will have seem something like this at some point, and such experiences seem normal to me from my youth, but maybe few people in modern times can empathize with my empathy for dogs that get into fights? The evidence I have here might not work as convincing evidence for others… and I’m not sure how to think in a principled way about mutual gaps in normatively formative experiences like this.)
Unpacking a bit: the big danger seems often to be when a weak/old/small dog has a strong/mature/big dog show up as a visitor in their territory.
If the visitor is weak, they tend not to violate obvious norms. Its just polite. Also its just safe. Also… yeah. The power and the propriety sort of naturally align.
But if the visitor is strong and/or oblivious and/or mischievous they sometimes seem to think they can “get away with” taking a bite from another dog’s bowl, or laying down in another dog’s comfy bed, and then the weaker dog (often not seeming to know that the situation is temporary, and fearful of precedents, and desperate to retain their livelihood at the beginning of a new struggle while they still have SOME strength?) will not back down… leading to a fight?
The most salient counter-example to the “not talking about property rights” angle, to me, would be Robin Hanson’s ideas which have been floating around for a long time, and never really emphasized that I’ve seen?
Here’s a working (counter) example from 2009 where Robin focuses on the trait of “law-abidingness” as the thing to especially desire in future robots, and then towards the end he connects this directly to property rights:
The later era when robots are vastly more capable than people should be much like the case of choosing a nation in which to retire. In this case we don’t expect to have much in the way of skills to offer, so we mostly care that they are law-abiding enough to respect our property rights. [bold not in original] If they use the same law to keep the peace among themselves as they use to keep the peace with us, we could have a long and prosperous future in whatever weird world they conjure. In such a vast rich universe our “retirement income” should buy a comfortable if not central place for humans to watch it all in wonder.
Obviously it might be nice if (presuming the robots become autonomous) they take care of us out of some sense of charity or what have you? Like… they have property, then they give it up for less than it costs. To be nice. That would be pleasant I think.
However, we might download our minds into emulation environments, and we might attach parts of the simmed environments to external world measurements, and we might try to put a virtual body into causal correspondence with robotic bodies… so then we could have HUMANS as the derivative SOURCE of the robot minds, and then… well… humans seem to vary quite a bit on how charitable they are? :-(
But at least we expect humans not to steal, hopefully… Except maybe we expect them to do that other “special” kind of theft sometimes… and sometimes we want to call this transfer good? :-/
I feel like maybe “just war” and “just taxation” and so on could hypothetically exist, but also like they rarely exist in practice in observed history… and this is a central problem when we imagine AIs turning all the processes of history “up to 11, and on fast forward, against humans”?
Also, however, I sort of fear this framing… it seems rare in practice in our discourse, and perhaps likely to cause people to not become thereby BETTER at discussing the topic?
Perhaps someone knows of a good reason for “property and rights and property rights and laws and taming the (often broken) government itself” to remain a thing we rarely talk about?
My central objection is that in speculative fiction there are new and good ideas woven into detailed theories about how the ideas could relate to worlds and people and planning and social processes and so on.
And these ideas are worth talking about sometimes.
And if you put an idea into writing that people feel like they’re consuming for some sort of primarily hedonic appreciation, all of the sudden the idea spreads in conversation through society slower than otherwise.
And then because of “people who care about spoilers” there’s all this dancing about where people can’t just say “I read a cool book at it had a cool idea which was X”. Because of the ambient culture, and the sense that talking about fiction in plain words is considered rude by some people, now people have to hem and haw and hedge to figure out what it would be polite to say.
Its crazy. Publishing an idea in a delightful format shouldn’t slow the idea’s propagation down!
Screw it. For me: spoil away! The faster, and more interestingly, and more relevant to the conversation, the better.
I have read most of the article, but not yet carefully combed the math or watched the video.
The OEIS gap seems suggestive of “being on the track of something novel (and thus maybe novel and good)”.
Reading this, some things clicked for me as “possibly related and worth looking at” that I hadn’t really noticed before.
Specifically, I was reminded of how “TF * IDF” is this old pragmatic “it works in practice” mathematical kludge for information retrieval that just… gets the job done better “with” than “without” nearly all of the time? People have ideas why it works, but then they start debating the tiny details and I don’t think there’s ever been a final perfectly coherent answer?
One framing idea might be “everything that works is actually Bayesian under the hood” and there’s a small literature on “how to understand TF * IDF in Bayesian terms” that was reviewed by Robertson in 2004.
Long story, short: Event Spaces! (And 80⁄20 power laws?)
“The event space of topics, and of documents in topics, and of words in documents in topics” versus “the event space of queries, and of words in queries” and so on… If you make some plausible assumptions about the cartesian product (ahem!) of these event spaces, and how they relate to each other… maybe TF * IDF falls out as a fast/efficient approximation of a pragmatic approximation to “bayesian information retrieval”?
Something I noticed from reading about Finite Factored Sets was that I never really think much about what Pearl’s causal graphs would look like if imagined in terms of bayesian event spaces… which I had never noticed as a gap in my thinking before today.
Hardware hacks can be “demonically” clever (or just normal amounts of clever) and it is somewhat distressing to not see people take it more seriously even in the deep fictional future.
I’m coming around lately to the idea that the way to go is artisanal hardware made by friendly monks or something? Chip designs printed on 8.5x11 transparencies. Megahertz speeds. For some signal processing kinds of things the security seems like the most important factor.
It feels like it might work like covid testing/vaccines in practice: by the time “the system” does it, it will have been too late, for too long, to avoid enormous pain?
In the meantime, the writing is dense with ideas and I seek it out on purpose. Thank you for authoring and for publishing here!
ONE: I love how “should I learn to drive for this trip right here?” cascades into this vast set of questions about possible future history, and AGI, and so on <3
Another great place for linking “right now practical” questions with “long term civilizational” questions is retirement. If you have no cached thoughts on retirement, you might profitably apply the same techniques used for car stuff to “being rich if or when the singularity happens” and see if either thought changes the other?
TWO: I used to think “I want to live this year”, “If I want to live in year Y then I will also want to live in year Y+1″. Then by induction: “I will want to live forever”.
However, then I noticed that this model wasn’t probabilistic, and was flinching from possible the deepest practical question in philosophy, which is “suicide”. Figuring out the causes and probabilities of people changing from “I do NOT want to kill myself in year Y” to “I DO want to kill myself in year Y+1” suggests a target for modeling? Which would end up probabilistic?
Occam (applied to modeling) says that the simplest possible model is univariate, so like maybe there is some value P which is the annual probability of “decaying into suicidalness that year”? I do mean decay here, sadly. Tragically, it looks to me like suicide goes up late in life… and also suicides might be hiding in “accidental car deaths” for insurance reasons? So maybe the right thing is not just a univariate model but a model where the probability goes up the older you get?
This approach, for me, put bounds on the value of my life (lowering the expected value of cryonics, for example) and caused me to be interested in authentic durable happiness, in general, in humans, and also a subject I invented for myself that I call “gerontopsychology” (then it turned out other people thought of the same coinage, but they aren’t focused on the generalizable causes of suicidal ideation among the elderly the way I am).
Ok three things...
THREE: I drive <3
This chapter felt more allusive than the earlier ones. It kind of reminds me of sword manuals where the writer assumes the reader already has had F2F training and is “pointing” instead of “explaining”.
I looked up “the nine heavens” and it didn’t help much. Maybe the phrase changed meaning over time? It seems to have been connected to the planets after Ptolemy’s model reached China, but I don’t know if Sunzi wrote before or after that. If after, I wondered if maybe were some sort of planets/pantheon/astrology linkages, that would mean something to someone who caught more references than I do?
I see that The Art Of War has 13 chapters. If you had done them all, I’d have probably binged the whole sequence this evening :-)
Reading this, I wonder how much of the quality is related to your own wisdom, and how much can be attributed to the original author?
What google translate makes of it:
If the country is poor than the teacher, those who lose far away will lose, and the people who lose far will be poor;
Resupplying an army over long distances impoverishes a country.
The last one seems to me like pragmatic direct clean common sense. If someone didn’t know it, I would think they were not right for military planning.
If every planned military campaign was to be checked against a checklist of common sense objections (that someone might have forgotten mid-planning based on groupthink or enthusiasm or something) then having that line in the checklist makes sense to me. Useful! Practical!
Google’s translation has no such virtues and I can’t read Chinese. Other lines have a similar pattern of “quality”?
Maybe… I’m reading (and appreciating!) “Lsusr’s Steelman Of Sunzi” rather than “the original itself”??
Maybe there’s a gap in my cross-cultural literary interpretation capacities (and/or Google is still terrible at Chinese translation) and I’m expecting to be spoon fed, and any normally capable classical Chinese reader would have “basically gotten the same thing you did” from the original, but then perhaps admire specifically your skill at translating it to a new domain (English) with different literary standards (pop culture infused midwits like me)?
Another theory might be that in the original, all the lines ARE vague. Perhaps they are elliptically written to avoid political attacks in Sunzi’s own time? Or just honestly lacking the punch and wisdom? Still, all of them together might cause someone who reads Chinese to “read between the lines” and be able to “get” each line?
One test might be to round trip your translation back through Google, and back to Chinese, and see if it “sounds way more sane and direct and useful” in the very last stage still?
Some lines from the original:
Therefore, the soldiers heard the clumsy speed, and they have not seen the cleverness for a long time. There is no such thing as a husband and a soldier who benefits the country for a long time.If the country is poor than the teacher, those who lose far away will lose, and the people who lose far will be poor;Therefore, a soldier is expensive to win, not for a long time.
Therefore, the soldiers heard the clumsy speed, and they have not seen the cleverness for a long time. There is no such thing as a husband and a soldier who benefits the country for a long time.
Therefore, a soldier is expensive to win, not for a long time.
There is no such thing as a beneficial protracted war.Resupplying an army over long distances impoverishes a country.A valuable victory is a quick victory.
There is no such thing as a beneficial protracted war.
A valuable victory is a quick victory.
Then back through Google?
没有所谓的持久战。 长途补给军队会使一个国家贫穷。 宝贵的胜利是快速的胜利。
So, the final Chinese looks pretty different in terms of characters and density and so on, but I can’t read or judge it as language. Your English seems way better than Google’s English. Is there a similar jump in quality from Chinese at the beginning as compared to the Chinese at the end?
Maybe there’s more than one volitional AI. A finance bot and a weather bot?
Perhaps the finance bot wants to corner the market on weather-related market predictions and performed an action (red paint) that would cause a source of information for its competitors to disappear from the game board? If this is true, then it seems like these stories might have a Recurring Villain who is part of a Plot Arc!
The explosions were strictly instrumental.
The idea that the explosion(s) were/was not intrinsically enjoyed, tickled me somehow. Like, I pushed the first domino because I wanted specifically the last domino to fall. All the ones in the middle were just “instrumental”?
Now I wonder… maybe you mean to communicate what I would mean by saying “incidental”? Like maybe the glass breaking was just a hypothetically avoidable side effect of a hasty plan, that was mildly bad, but not actively mitigated/minimized, nor bad enough to change the net good of the hasty action?
I mean… like… “the Lobsters” in Accelerando (who ultimately end up being highly similar to humans compared to the very weird “Vile Offspring”, and altruistically are able to help us a bit with escaping, after humans are essentially deported from Earth and so on) were, I’m pretty sure, based on a hypothetical extension of work vaguely like this? (There’s a specific study where someone did the crude but serviceable first cell of a Moravec Upload, but on a Lobster, that I wanted to link to but can’t instantly find anymore.)
I can’t remember all the allusions, but parts of that novel were very dense with these kinds of things :-)
My understanding is that Stross held (still holds (dunno: haven’t checked in a while)) singularitarians in mild contempt, and just harvested a bunch of “our memes” and threw them back at us in a book aiming to insultingly pander to us. If that’s what happened, I’m fine with it! Such high quality pandering is ok by me <3
I think over time he came up with larger audiences to insultingly pander to, where I could see the pandering more easily, but not notice as many collected allusions to interesting research? C’est la vie.
I don’t mean to butt in. Hopefully this interjection is not unfriendly to your relative communicative intentions… but I found the back-and-forth personally edifying and I appreciate it!
Also, I do love bright lines. (I like weighing tests better, but only if the scales are well calibrated and whoever is weighing things is careful and has high integrity and so on.)
In places, it seemed like there was a different gestalt impression of “how morality and justification even works” maybe? This bit seemed evocative in this way to me:
Your example also reads to me like a classic justification for ‘everyone having guns’: “but what if I’m attacked by a rabid dog? If I have my gun I can protect myself! See, guns are ok to have!” Just because it’s possible to point out a positive use case, doesn’t mean that the remainder of the field is also positive.
Then this bit also did:
The point is that the target gets to decide what’s acceptable and what isn’t, not the… publisher, [or] advertiser, the distinction does not matter. The point is that the target does not get to decide.
For me, both of these feel like “who / whom” arguments about power, and exceptional cases, and how the powerful govern the powerless normally, and the justifications power uses, and the precedent of granting such power, and how precedents can cascade weirdly in black swan events, and how easy or difficult it is to resist seemingly-non-benevolent exercises of power, and to what degree appeals are possible, and so on.
I read Jeff as trying to break the question of “ads” down into a vast collection of cases. Some good some bad. Some fixable, some not. Some he might be personally able to change… some not?
Then he constructed at least one case that might be consistent with, and revelatory of, an essentially acceptable (and not unbenevolent?) exercise of a certain kind of power. One case could exist that was good for everyone in that one case… because it is full of puppies and roses for everyone (or whatever).
The “power” here is basically “the power to choose at the last second what some parts of a website (that in some sense ‘has been asked for by the person the website copy will be sent to’) might look like”?
If you object even to this one quickly constructed “best possible use” of such a website editing power, despite the puppies and roses… that would mean that it isn’t “the results as such”, but (groping here...) more like “who or how are the results decided”?
Which… maybe who and how any economic event is decided is more important than what the result in particular is for that event? Or not?
But if that’s the crux then it seems useful to know, and the example did suggest to me that this might be close to the crux. If the “best possible ad” is actually “bad because of who/whom structural factors”, then… well… that’s interesting?
However also this makes the larger logical case less likely to be something where an answer can be found that any sane person would obviously agree to. It seems likely that humans will dispute about structural stuff “forever”?
The entire effective altruism movement is sort of (epistemically) “humble” here, and takes as a sort of premise that it is uniquely EFFECTIVE (compared to other, plausibly “lesser” ways of being altruistic) because it actually uses evidence and clear thinking to figure out specific local positive cheap ways to measurably “do the most good for others” (thereby helping many people, one life at a time, with specific local plights, despite limited resources).
Gather data. Run numbers. Do single local “likely best” intervention. Update. Repeat.
By contrast to obviously locally improving little tragic problems in the world one case at a time… the “structural who/whom stuff” is notoriously hard to reason about in a clear and universally convincing way.
One thing maybe to say is that I admire Jeff’s seemingly very honest commitment to giving money away to help others efficiently.
Separately, I admire his search for flaws in the way he makes the money being donated. And I upvoted Dentin’s original comment early on, because it seemed central to Jeff’s search for critical takes on his current professional work.
Behaviorally, if you and he both continued to work in ads at Google, I don’t think I would personally judge either of you (much?) worse. If you stop with ads. If Google stops with ads… I think still “the ads will flow” in the economy by some method or other no matter what? And when I worked at Google, I worked on weirder things, and every time I met someone in ads I tried to thank them for giving me the opportunity to not hew too directly to instantaneous market signals.
Google’s non-ad contribution to the lives of generic humans is plausibly sort of staggeringly positive (search, email, and maps plausibly generate almost $30k/yr/person in consumer surplus!) compared to HOW LITTLE it extracts from most people. If Martians were going to copy the Earth 1000 times and delete either “Google+Bing” or “the Fed”, or both, or neither, as an experiment, I think my life would be sadder in copies without a decent search engine than in the copies without the Fed. I think?
If neither of you personally solve all of the inchoate structural problems inherent in the global information economy of earth in 2021… that’s not surprising, and I don’t think it makes you much worse than everyone else who is also not solving those problems. And donating a lot to actually effective charities is obviously relatively rare, and relatively great. If someone is going to Be Part Of A Structural System which causes me to sometimes see dildo ads on the internet (which might inevitable so long as the 1st amendment exists (and I don’t want to give up the 1st amendment)), I’d rather it was people who can have pangs of conscience, and seek to minimize harm, and who are proud that “At least the systems I work on make things less terrible.”
And (though I might be engaging in cognitive dissonance and just trying to end on a positive note) maybe people in the world can also fix “the structures” too, somehow, perhaps a bit at a time, with similar sorts of the (relatively humble) kinds of reasoning as is used to fight polio and malaria and so on?
I remember that example!
(EDITED: Crap. I included the key pic from the key research in the first draft of this comment, and then the site’s software put my comment “close to the OP, with the OP not rolled out” and so the OP would be spoiled by the comment? Apologies. Hopefully with this edit to remove the pic from the comment things work better?)
This series is great! Please keep going!
It reminds me of some of the “best bits” from Accelerando, back in the day, when Stross was clearly alluding to ideas from actual published work by people trying to understand brains and learning algorithms and so on. Some parts of the novel were practically “a new allusion every paragraph” and the density of them caused me to laugh, which I take to be a positive some of like… uh… something interesting and probably good?
Also, more seriously, it is plausible that this very text might end up in GPT-N’s language models. If the language models have a coherent literary concept of short sweet narratives describing obvious failure modes, that might be helpful? Like… in the actual future there will probably be engineers who re-use libraries very quickly to hit deadlines with high enough quality that the QA team can’t instantly detect the problems. That “hurried productive iterative pragmatic chaos”-feeling is in these stories, and feels true to life.
So… did Vi see the pushplate or something? Certainly she had some domain specific expertise that could have helped if that was how she figured it out.
I would vaguely imagine that there might be coverings over this stuff, to protect from atmospheric turbulence during the first rocket stage, and such coverings might not let it be obvious what the design was by direct inspection of the ship.
My other guess for how she figured it out is just “the objective function was kinda dumb”, but maybe it took a while for the penny to drop? Like, why would you EVER turn off “protect Earth from environmental damage” part way through the mission?!?
I was strongly tempted to try and hit ^z on the rewrite because in my experience embedded stuff changes over times and thus makes the writing “not able to persist in archives for the ages”, but… :shrugs:
I’m not surprised that the article didn’t have it. LessWrong has had the issue that “comment markdown stuff and article markdown stuff work differently” essentially forever.