Let us take the example of Daniel Dennett vs David Chalmers. Dennett calls philosophical zombies an “embarrassment,” while Chalmers continues to double-down on his conclusion that consciousness cannot be explained in purely physical terms. If Chalmers conceded and switched teams, then he is going to be “just another philosopher,” while Dennett achieves an academic victory.
As an aspiring world-class philosopher, you have little incentive to adopt the dominant view because if you do you will become just another ordinary philosopher. By adopting a radically different viewpoint, you establish an entirely new “school” and become at its helm. Meanwhile, it would be considerably more effortful to become at the helm of the more well-established schools, e.g. physicalism and compatibilism.
Thus, motivated skepticism and motivated reasoning seem to me to be completely unavoidable in academia.
If you will keep consuming a non-expirable household item forever, and if you have plenty of free space, why don’t you have a lifetime supply?
This may apply to soap, shampoo, toilet paper, detergent, paper towels, sponges, aluminum foil, trash bags, sandwich bags, most household cleaners, q-tips, rubber gloves, nails, screws, paper, matches, band-aids, light bulbs and (if these never expire) toothpaste, batteries, markers and pens, and many over-the-counter pharmaceuticals.
There are legitimate arguments against stockpiling, like ease of moving and expecting technology to improve. If you weigh these explicitly rather than just following the crowd, you’re much more likely to find the optimal supply.
You won’t keep consuming the exact thing forever, you don’t have plenty of space, even non-perishables can be damaged or degrade in long-term storage, and you don’t want to expend organizational and indexing capacity on something you can just buy later.
For personal-use commodities, the optimal amount is generally whatever unit Costco offers, or the duration of your cost/convenience indifference point for thinking about it. I know of zero cases where that’s more than a year for me, and of zero where it’s more than 3 years for anyone I talk with.
Actually, I may have over a years’ worth of some light bulbs and some battery sizes. It’s insane to propose “lifetime”, though—these things will get somewhat better or cheaper (relative to my income) over time, and I can’t really predict any changes in size or preference changes over more than a few years.
I’d love to see the calculation / estimate of savings for some things you prefer to pre-pay and then store at your cost and risk for more than a few years, rather than waiting to buy closer to when needed.
In my experience the reasons you give against stockpiling, and several others, are invariably overwhelming. You mention lack of space, change of technology, shelf life, and difficulty of moving. To these I add change of taste, discovery of new things that I like better than what I have, the mental burden of having so much stuff, the futile attempt to stop time instead of flexibly responding to whatever life brings, and that a lifetime supply could be very much larger than you’re thinking given radical life extension or the singularity.
Instead my rule is: one in use and one spare. When the spare comes into use, I replace it at my next convenience. The numbers can vary (I have several pens in every room, a box of spares, and two in my pocket whenever I go out), but that is the general principle.
“People are over sensitive to ostracism because human brains are hardwired to be sensitive to it, because in the ancestral environment it meant death.”
Evopsyche seems mostly overkill for explaining why a particular person is strongly attached to social reality.
People who did not care what their parents or school-teachers thought of them had a very hard time. “Socialization” as the process of the people around you integrating you (often forcefully) into the local social reality. Unless you meet a minimum bar of socialization, it’s very common to be shunted through systems that treat you worse and worse. Awareness of this, and the lasting imprint of coercive methods used to integrate one into social reality, seem like they can explain most of an individuals resistance to breaking from it.
When a republic is considering reforming the way it elects holders of power, there are certain desirable criteria the chosen method should have: officeholders should be agreeable to all citizens, not just to a fraction of them, since maximizing agreeableness will select for competence and the desire to do right by the people; in bodies with many members (e.g. a legislature, but not a singular executive such as a president or prime minister), the various view points of the voters should be proportionately represented by the various members of the body; and the system should be relatively simple, to allow the members of the republic to verify that things have been done properly, ensuring the system is trusted.
There are systems I have considered that suceed better at the first two criteria, but at the cost of being sufficiently complex to make me worry that practical issues could arise if implemented, and that it could be hard to convince enough people that they are better, which will be needed in order to get them made into law. The following is my current best hypothesis for an ideal system that meets these three criteria, particularly aiming to be simple both to actually use and to explain to people:
Voters are provisioned with a ballot with three fields: first a field to elect the representative of their single-member district, where they can either approve or disapprove of as many candidates as they wish (approval voting); second, a field where they may indicate a single republic-wide list of candidates that they feel best represents them; and third, a field where they may name as many candidates from their chosen list as they wish, to indicate their approval (party-list proportional representation with within-party approval voting).
The single-member districts will select roughly 2/3rds of the body, with the most approved candidate winning in each district. A “similarity score” will be calculated for each winner comparing them to each list: if half of the voters who approved of Charlie voted for the Aleph List, Charlie’s similarity score for Aleph will be 0.5; if instead only a tenth of his approvers voted for the list, the similarity score will be 0.1
For each list, the sum of that list’s similarity score with each candidate will be considered to be the initial number of effective representatives that list has in the body (note that this means the district candidates’ party affiliation does not factor in to this system, since this is based on voter behavior patterns). For each list, consider the ratio of number of votes for the list divided by the number of representatives for the list: whichever list has the most votes per representative is the most underrepresented list, so the most approved candidate from the list will be appointed, increasing the number of effective representatives by one. Keep adding seats to the least represented lists, until the remaining 1/3rd of the body has been appointed, thereby giving the entire composition of the body.
Note that the use of approval voting ensures district candidates will be incentivized to appeal to a broad base, encouraging them to find solutions that take care of everybody’s needs, instead of rewarding them for creating division within their community; while the leveling seats ensure that all voices, not just the most centeral perspectives, contribute to the government.
One potential criticism of this method is the appeal to precedence: while using party lists (modulo the similarity scores, which seems like a straightforward and uncontroversial improvement) in this way has been used to much success in the Nordic countries, approval voting is (somewhat surprisingly IMO) not well established. As far as governments go, I only know of St. Louis and Fargo, ND using approval- that is, two municipalities. One could observe the concern that we don’t have much empirical data on how approval works in the real world.
One response is simply to respond that experiments have to be done, otherwise we will never know what works. Perhaps it is ideal to do experiments starting small (e.g. Fargo and St. Louis) and building up towards larger results; but I also have a sense of urgency that voting reform in the US matters greatly, and the faster it can be implemented, the less likely the world is to go in a bad direction v within the next {5, 10, 20, 50} years.
The other response is to observe that while we lack empirical data, the theoretical case for Approval is very solid and promising (I don’t have time to further substantiate this right now)
The future may have a use for frozen people from the current era. In the future, historical humans may be useful as an accurate basis to interpret the legal documents of our era.
Original pubic meaning is a is a fairly modern mode of legal interpretation of the US Constitution. It’s basis is that the language of the constitution should be interpreted the way that the original meaning of the text was when it was drafted and amended into the constitution. A similar mode of interpretation is used less commonly for statutes. It’s likely that this mode of interpreation would become more common in the future as a way to prevent value drift.
One of the struggles of modern times employing the original public meaning test is that for older amendments there are no people currently alive who lived in the culture that the amendments were drafted in. It would be very helpful to have just a single ordinary man who lived in 1790 who could explain his understanding of the constitution and the language therein.
It’s possible the future will have similar issues interpreting constitutional language from our time and will appreciate the ability to benefit from questioning a portion of the population of that era.
This theory is a subset of the idea that humans from past eras will be useful in the future to prevent value drift generally. My first brush with this idea was Three World’s Collide.
LW is quoted (in a kind of flattering way) in Simon Dedeo’s awesome piece on the last Nautil.us issue. For spoiler lovers:
[...] Rationality is my ticket out. The only reason I can trust you is that you seem rational enough to talk to. But now you’re telling me that rationality is just a layer on top of the System—it’s just as irrational as the people I’m trying to escape. I don’t know which is worse: being duped by someone else’s priors, or being a biological machine.
Teacher: Don’t go too far. You’re a smart kid—you can iterate faster than most. You can match patterns better. Evolution set you up well. You’ll get better at predicting the consequences of your actions, and better at adapting your environment to your will. Rationality is systematized winning.
Ian: It’s not winning I’m worried about. It’s my mind. Maybe it’s silly, maybe it’s a fetish, but I want to know the truth. It’s the principle of the thing. Wanting to know the truth got me this far, but now the only option you’ve given me is believing in something I can’t see. If I know it at all, it can’t be through rational, scientific calculation. There’s some kind of extra-rational process I have to engage in—but what’s beyond the edge of reason?
Teacher: Many things. Dreams, intuition, transcendence, love, ascending the ladder, repetition and the leap of faith, philosophy itself …
Longtermist AI policy projects for economists (this doc was originally just made for Risto Uuk’s own use, so the ideas shouldn’t be taken as high-confidence recommendations to anyone else)
Context
I intend for this to include both technical and governance problems, and problems relevant to a variety of AI risk scenarios (e.g., AI optimising against humanity, AI misuse by humans, AI extinction risk, AI dystopia risk...)
I spent hardly any time on this, and just included things I’ve stumbled upon, rather than specifically searching for these things
So I’m sure there’s a lot I’m missing.
Please comment if you know of other things worth mentioning here. Or if you’re better placed to make a list like this than I am, feel very free to do so; you could take whatever you want from this list and then comment here to let people know where to find your better thing.
(It’s also possible another list like this already exists. And it’s also possible that economists could contribute to such a large portion of AI risk problems that there’s no added value in making a separate list for economists specifically. If you think either of those things are true, please comment to say so!)
A commonly given reason for why Nordic countries tend to rank highly as desirable places to live, is because the people there are supported by a robust welfare system. In America, I’ve often heard it said that similar systems shouldn’t be implemented, because they are government programs, and (the argument goes) government shouldn’t be trusted.
This suggests the government as a potentially important point of comparison between the Nordic countries and the US. Are there features that differ between the American and Nordic governments (keep in mind that there’s noticeable ethnic diversity even in the Nordic countries, and many of them are part of the EU, which provides a roughly similar large-scale government as the American federal government). Do the politicians face different incentive structures that allow more honest and more competent politicians to do better in the Nordic countries than in the US?
Does anyone have tips on how to buy rapid tests in the US? Not seeing any on US Amazon, not seeing any in person back where I’m from. Considering buying German tests. Even after huge shipping costs, it’ll come out to ~$12 a test, which is sadly competitive with US market prices.
Wasn’t able to easily find tests on the Mexican and Canadian Amazon websites, and other EU countries don’t seem to have them either.
I’ve been able to buy from the CVS website several times in the past couple months, and even though they’re sold out online now, they have some (sparse) in-store availability listed. Worth checking there, Walgreens, etc. periodically.
For a long time, I found the words “clockwise” and “counterclockwise” confusing, because they are so similar to each other, and “counterclockwise” is a relatively long word at 4 syllables, much longer than similarly common words.
At some point in time, I took to calling them “dexter” and “winstar”, from the Latin »dexter« and Middle English »winstre«, meaning “right” and “left”, respectively. I like these words more than the usual “clockwise”, but of course, new words aren’t worth much of others don’t know them, so this is a PSA that these are words that I use in my vocabulary.
(As an addendum, I have also taken to calling the Danish political party «Venstre» as “Winstar”, because despite their name meaning “Left”—which was fitting at the time the party was founded—they are actually part of the right-of-centre coalition)
Is it that intuitive to you that you should name the rotating object’s direction using the movement of the top of the object? I think I would get confused with your words after a while. I just use “positive” and “negative” direction.
At some point in time, I took to calling them “dexter” and “winstar”, from the Latin »dexter« and Middle English »winstre«, meaning “right” and “left”, respectively
I suspect that the term “cognitive” is often over/misused.
Let me explain what my understanding of the term is. I think of it as “a disagreement with behaviorism”. If you think about how psychology progressed as a field, first there was Freudian stuff that wasn’t very scientific. Then behaviorism emerged as a response to that, saying “Hey, you have to actually measure stuff and do things scientifically!” But behaviorists didn’t think you could measure what goes on inside someone’s head. All you could do is measure what the stimulus is and then how the human responded. Then cognitive people came along and said, “Er, actually, we have some creative ways of measuring what’s going on in there.” So, the term “cognitive”, to me at least, refers very broadly to that stuff that goes on inside someone’s head.
Now think about a phrase like “cognitive bias”. Does “cognitive” seem appropriate? To me it seems way too broad. Something like “epistemic bias” seems more appropriate.
The long standing meaning of “cognitive” for hundreds of years before cognitive psychologists was having to do with knowledge, thinking, and perception. A cognitive bias is a bias that affects your knowledge, thinking, and/or perception.
Epistemic bias is a fine term for those cognitive biases that are specifically biases of beliefs. Not all cognitive biases are of that form though, even when they might fairly consistently lead to certain types of biases in beliefs.
Hm, can you think of any examples of cognitive biases that aren’t about beliefs? You mention that the term “cognitive” also has to do with perception. When I hear “perception” I think sight, sound, etc. But biases in things like sight and sound feel to me like they would be called illusions, not biases.
The first one to come to mind was Recency Bias, but maybe I’m just paying that one more attention because it came up recently.
Having noticed that bias in myself, I consulted an external source https://en.wikipedia.org/wiki/List_of_cognitive_biases and checked that rather a lot of them are about preferences, perceptions, reactions, attitudes, attention, and lots of other things that aren’t beliefs.
They do often misinform beliefs, but many of the biases themselves seem to be prior to belief formation or evaluation.
Ah, those examples have made the distinction between biases that misinform beliefs and biases of beliefs clear. Thanks!
As someone who seems to understand the term better than I do, I’m curious whether you share my impression that the term “cognitive” is often misused. As you say, it refers to a pretty broad set of things, and I feel like people use the term “cognitive” when they’re actually trying to point to a much narrower set of things.
Here’s an experiment I’d love to see someone run (credit to Jeff Wu for the idea, and William Saunders for feedback):
Finetune a language model to report the activation of a particular neuron in text form.
E.g., you feed the model a random sentence that ends in a full stop. Then the model should output a number from 1-10 that reflects a particular neuron’s activation.
We assume the model will not be able to report the activation of a neuron in the final layer, even in the limit of training on this task, because it doesn’t have any computation left to turn the activation into a text output. However, at lower layers it should be able to do this correctly, with some amount of finetuning.
How many layers do you have to go down before the model succeeds? How does this scale with (a) model size and (b) amount of training?
One subtlety is that finetuning might end up changing that neuron’s activation. To avoid this, we could do something like: - Run the base model on the sentence
-Train the fine-tuned model to report the activation of the neuron in the base model, given the sentence
- Note whether the activation in the finetuned model is different
Why I think this is interesting:
I often round off alignment to ‘build a model that tells us everything it “knows”’. It’s useful to determine what pragmatic limits on this are. In particular, it’s useful for current alignment research to be able to figure out what our models “know” or don’t “know”, and this is helpful for that. It gives us more information about when ‘we tried finetuning the model to tell us X but it didn’t work’ means ‘the model doesn’t know X’, versus when the model may have a neuron that fires for X but is unable to report it in text.
We assume the model will not be able to report the activation of a neuron in the final layer, even in the limit of training on this task, because it doesn’t have any computation left to turn the activation into a text output.
Surely there exist correct fixed points, though? (Although probably not that useful, even if feasible)
You mean a fixed point of the model changing its activations as well as what it reports? I was thinking we could rule out the model changing the activations themselves by keeping a fixed base model.
1. Growth quickens. 2. People notice, and are more willing to lend capital. 3. Take out too many loans. 4. Use your borrowed money to capture the apparatus that would make you pay your debt. 5. Cancel your debt.
Reinforcing based on naively extrapolated trajectories produces double binds. We have a reinforcer R and an agent A. R doesn’t want A to be too X or too not-X. Whenever A does something that’s uncommonly X-ish, R notices that A seems to be shifting more towards X-ishness in general. If this shift continues as a trajectory, then A will end up way too X-ish. So to head that off, R negatively reinforces A. Likewise, R punishes anything that’s uncommonly not-X-ish. As an agent, A is trying to figure out which trajectory to be on. So R isn’t mistaken that A is often putting itself on trajectories which naively imply a bad end state. But, A is put in an impossible situation. R must model that R and A will continue their feedback cycle in the future.
I know a lot of people through a shared interest in truth-seeking and epistemics. I also know a lot of people through a shared interest in trying to do good in the world.
I think I would have naively expected that the people who care less about the world would be better at having good epistemics. For example, people who care a lot about particular causes might end up getting really mindkilled by politics, or might end up strongly affiliated with groups that have false beliefs as part of their tribal identity.
But I don’t think that this prediction is true: I think that I see a weak positive correlation between how altruistic people are and how good their epistemics seem.
----
I think the main reason for this is that striving for accurate beliefs is unpleasant and unrewarding. In particular, having accurate beliefs involves doing things like trying actively to step outside the current frame you’re using, and looking for ways you might be wrong, and maintaining constant vigilance against disagreeing with people because they’re annoying and stupid.
Altruists often seem to me to do better than people who instrumentally value epistemics; I think this is because valuing epistemics terminally has some attractive properties compared to valuing it instrumentally. One reason this is better is that it means that you’re less likely to stop being rational when it stops being fun. For example, I find many animal rights activists very annoying, and if I didn’t feel tied to them by virtue of our shared interest in the welfare of animals, I’d be tempted to sneer at them.
Another reason is that if you’re an altruist, you find yourself interested in various subjects that aren’t the subjects you would have learned about for fun—you have less of an opportunity to only ever think in the way you think in by default. I think that it might be healthy that altruists are forced by the world to learn subjects that are further from their predispositions.
----
I think it’s indeed true that altruistic people sometimes end up mindkilled. But I think that truth-seeking-enthusiasts seem to get mindkilled at around the same rate. One major mechanism here is that truth-seekers often start to really hate opinions that they regularly hear bad arguments for, and they end up rationalizing their way into dumb contrarian takes.
I think it’s common for altruists to avoid saying unpopular true things because they don’t want to get in trouble; I think that this isn’t actually that bad for epistemics.
----
I think that EAs would have much worse epistemics if EA wasn’t pretty strongly tied to the rationalist community; I’d be pretty worried about weakening those ties. I think my claim here is that being altruistic seems to make you overall a bit better at using rationality techniques, instead of it making you substantially worse.
For example, people who care a lot about particular causes might end up getting really mindkilled by politics, or might end up strongly affiliated with groups that have false beliefs as part of their tribal identity.
These both seem pretty common, so I’m curious about the correlation that you’ve observed. Is it mainly based on people you know personally? In that case I expect the correlation not to hold amongst the wider population.
Also, a big effect which probably doesn’t show up much amongst the people you know: younger people seem more altruistic (or at least signal more altruism) and also seem to have worse epistemics than older people.
Caring about things seems to make you interact with the world in more diverse ways (because you do this in addition to things other people do, not instead of); some of that translates into more experience and better models. But also tribal identity, mindkilling, often refusing to see the reasons why your straightforward solution would not work, and uncritical contrarianism.
Now I think about a group of people I know, who care strongly about improving the world, in the one or two aspects they focus on. They did a few amazing things and gained lots of skills; they publish books, organize big conferences, created a network of like-minded people in other countries; some of their activities are profitable, for others they apply for various grants and often get them, so some of them improve the world as a full-time job. They also believe that covid is a hoax, plus have lots of less fringe but still quite irrational beliefs. However… this depends on how you calculate the “total rationality”, but seems to me that their gains in near mode outweigh the losses in far mode, and in some sense I would call them more rational than average population.
Of course I dream about a group that would have all the advantages and none of the disadvantages.
Ah. I meant, I would like to see a group that has the sanity level of a typical rationalist, and the productivity level of these super-agenty irrationalists. (Instead of having to choose between “sane with lots of akrasia” and “awesome but insane”.)
Hm. Maybe there’s something to be gained from navigating ‘trade-offs’ differently? I thought perpetual motion machines were impossible (because thermodynamics) aside from ‘launch something into space, pointed away from stuff it would crash into’, though I’d read that ‘trying to do so is a good way to learn about physics., but I didn’t really try because I thought it’d be pointless.’ And then this happened.
I dreamt up the following single-winner voting system in the car while driving to Eugene, Oregon on vacation. I make no representation that it is any good, nor that it’s better than anything currently known or in use, nor that’s it’s worth your time to read this.
Commentary and rationale will be explained at the bottom of this post
The system is a 2-round system. The first round uses an approval ballot, and the second round asks voters to choose between two candidates.
([•] indicates a constant that can be changed during implementation)
Round One:
Tally all approvals for each candidate
If there are two or more candidates with at least a 50% [•] approval rating, remove all candidates with less than [50%] approval. Otherwise, return the highest-approved candidate as the winner of the election
If there are only two candidates remaining, proceed to Round Two with the two remaining candidates. If there are more than four candidates, remove all but the four [• ; shouldn’t be higher than 6 or 7] highest rated candidates.
Consider all pairs of candidates that includes at least one of the top two candidates. For each pair, a candidate’s Opposing-Approval is that candidate’s approval rating among the 10% [•] that least approve of the other candidate. (That is to say, if the other candidate has less than [90%] approval, it is the first candidate’s approval among those who oppose the other candidate. The quantity approaches the candidate’s overall approval as the approval of the other candidate approaches 100%)
Each’s pair’s Score is the sum of the two candidates’ Opposing-Approval. A higher score means that voters who don’t approve of candidate A tend to approve of candidate B, and vice-versa. If the supporters of A and B tend to overlap, the pair’s score will be low.
Send the pair (among those considered in step 4) with the highest score to Round Two
Round Two
Ask voters which candidate they prefer. The more preferred candidate is the winner. Voters may, of course, express no preference, but that will not affect the outcome.
Commentary and Rationale
I’ve been considering the hypothesis that the polarizing nature of plurality vote (FPTP) may actually be a feature, not a bug. While I suspect this hypothesis is likely (at least mostly) wrong, and that the center-seeking / consensus-building nature of Approval voting is one of its most valuable features, the described system was created to combine the consensus-seeking nature of AV with the contest between two opposing viewpoints provided by FPTP.
Below a certain approval rating, the system behaves like AV as a failsafe against the failure mode present in FPTP where a very unpopular candidate can win as long as their opponent is just as bad (IIRC, Instant Runoff doesn’t completely address this). Likewise, the winning candidate will always be one of the most approved, even if they’re not the highest-approved candidate.
But above a certain level of approval, the system intentionally introduces a degree of polarization, seeking out a pair of candidates with some disagreement between their bases, and choosing one or the other. This increases the “temperature” of the system, allowing potentially controversial but high-value strategies to be tried, encouraging candidates to not just play things safe. This could potentially lead to a bolder style of leadership than pure approval voting (though note that it is a misconception that AV will produce milquetoast, boring, spineless appointees; AV searches for a candidate who is exciting and compelling to all voters, not just to a particular fraction of voters). While some countries may be content with a middle-of-the-road style of leadership like AV provides, perhaps some countries may desire the dynamic quality this system intends to share with FPTP.
Again, I make no representation that this system is actually any good.
Alternative: Something like condorcet voting, where voters receive a random subset of pairs to compare. For a simple analysis, the number of pairs could be 1. (Or instead of pairs, a voter could asked to choose ‘the best’.)
This Generative Ink post talks about curating GPT-3, creating a much better output than it normally would give, turning it from quite often terrible to usually pround and good. I’m testing out doing the same with this post, choosing one of many branches every few dozens of words.
For a 4x reduction in speed, I’m getting very nice returns on coherence and brevity. I can actually pretend like I’m not a terrible writer! Selection is a powerful force, but more importantly, continuing a thought in multiple ways forces you to actually make sure you’re saying things in a way that makes sense.
Editing my writing can be slow and tedious, but this excercise gives me a way to naturally write in a compact, higher-quality way, which is why I hope I do this many times in the future, and recommend you try doing this yourself.
This post was written in 5 blocks, and I wrote 4 (= 2^2) branches for each block, for 5*2 = 10 bits of curation, or 14.5 words per bit of curation.
As it happens, I always used the final branch for each block, so it was more effects of revision and consolidation than selection effects that contribute to the end result of this excercise.
I propose a new formal desideratum for alignment: the Hippocratic principle. Informally the principle says: an AI shouldn’t make things worse compared to letting the user handle them on their own, in expectation w.r.t. the user’s beliefs. This is similar to the dangerousness bound I talked about before, and is also related to corrigibility. This principle can be motivated as follows. Suppose your options are (i) run a Hippocratic AI you already have and (ii) continue thinking about other AI designs. Then, by the principle itself, (i) is at least as good as (ii) (from your subjective perspective).
More formally, we consider a (some extension of) delegative IRL setting (i.e. there is a single set of input/output channels the control of which can be toggled between the user and the AI by the AI). Let πυu be the the user’s policy in universe υ and πa the AI policy. Let T be some event that designates when we measure the outcome / terminate the experiment, which is supposed to happen with probability 1 for any policy. Let Vυ be the value of a state from the user’s subjective POV, in universe υ. Let μυ be the environment in universe υ. Finally, let ζ be the AI’s prior over universes and ϵ some sufficiently small bound. We require
∀T:Eυ∼ζ[(Eπυuμυ[Vυ(T)]−Eπaμυ[Vυ(T)])+]≤ϵ
Here, V(T) designates the value after event T happens, and (x)+ is defined to be 0 for x<0 and x otherwise.
I now outline an example of a Hippocratic protocol, which I call Hippocratic timeline-driven learning (HTDL).
Suppose, for simplicity, that the goal is accomplishing some binary task (i.e. we can either succeed or fail). We can imagine the task to be something like “construct a defense system against malign AI”. At any given moment, the user reports their subjective timeline: i.e. for, any t, the probabilities of success and failure after time t, assuming the user will continue working on the task by themselves. We assume the AI can learn a good model of the user’s policy and the user’s nominal report, but this nominal report might deviate from the true timeline because the system can be corrupted (i.e. the user manipulated to enter false information or the input channel hacked). Moreover, we assume that such corruption counts as failing the task.
The AI operates by maximizing the AI-expectation of the user’s (unknown) utility function Eυ∼ζ[Eπμυ[Uυ]], subject to the Hippocratic principle as a constraint. Since we’re working in a delegative setting, the AI can trivially achieve the constraint by always delegating. However, it can do much better than that, by short-term quantilizing w.r.t. the user’s reported success probability (with the user’s policy serving as baseline). When quantilizing the short-term policy, we can upper bound the probability of corruption via the user’s reported probability of short-term failure (which we assume to be low, i.e. we assume the malign AI is not imminent). This allows the AI to find parameters under which quantilization is guaranteed to improve things in expectation. Then, after completing this short-term maneuver, the AI can do another short-term quantilization, and continue in this vein. Essentially, the AI is gradually bending the user’s trajectory towards success.
Of course, we can improve the guarantees of such a protocol either by adding more information or by using stronger assumptions (e.g. assume the user is quasi-rational). But timeline-driven short-term quantilization serves as a useful feasibility proof.
Very interesting—I’m sad I saw this 6 months late.
After thinking a bit, I’m still not sure if I want this desideratum. It seems to require a sort of monotonicity, where we can get superhuman performance just by going through states that humans recognize as good, and not by going through states that humans would think are weird or scary or unevaluable.
One case where this might come up is in competitive games. Chess AI beats humans in part because it makes moves that many humans evaluate as bad, but are actually good. But maybe this example actually supports your proposal—it seems entirely plausible to make a chess engine that only makes moves that some given population of humans recognize as good, but is better than any human from that population.
On the other hand, the humans might be wrong about the reason the move is good, so that the game is made of a bunch of moves that seem good to humans, but where the humans are actually wrong about why they’re good (from the human perspective, this looks like regularly having “happy surprises”). We might hope that such human misevaluations are rare enough that quantilization would lead to moves on average being well-evaluated by humans, but for chess I think that might be false! Computers are so much better than humans at chess that a very large chunk of the best moves according to both humans and the computer will be ones that humans misevaluate.
Maybe that’s more a criticism of quantilizers, not a criticism of this desideratum. So maybe the chess example supports this being a good thing to want? But let me keep critiquing quantilizers then :P
If what a powerful AI thinks is best (by an exponential amount) is to turn off the stars until the universe is colder, but humans think it’s scary and ban the AI from doing scary things, the AI will still try to turn off the stars in one of the edge-case ways that humans wouldn’t find scary. And if we think being manipulated like that is bad and quantilize over actions to make the optimization milder, turning off the stars is still so important that a big chunk of the best moves according to both humans and the computer are going to be ones that humans misevaluate, and the computer knows will lead to a “happy surprise” of turning off the stars not being scary. Quantilization avoids policies that precisely exploit tiny features of the world, and it avoids off-distribution behavior, but it still lets the AI get what it wants if it totally outsmarts the humans.
The other thing this makes me think of is Lagrange multipliers. I bet there’s a duality between applying this constraint to the optimization process, and adding a bias (I mean, a useful prior) to the AI’s process for modeling U.
When I’m deciding whether to run an AI, I should be maximizing the expectation of my utility function w.r.t. my belief state. This is just what it means to act rationally. You can then ask, how is this compatible with trusting another agent smarter than myself?
One potentially useful model is: I’m good at evaluating and bad at searching (after all, P≠NP). I can therefore delegate searching to another agent. But, as you point out, this doesn’t account for situations in which I seem to be bad at evaluating. Moreover, if the AI prior takes an intentional stance towards the user (in order to help learning their preferences), then the user must be regarded as good at searching.
A better model is: I’m good at both evaluating and searching, but the AI can access actions and observations that I cannot. For example, having additional information can allow it to evaluate better. An important special case is: the AI is connected to an external computer (Turing RL) which we can think of as an “oracle”. This allows the AI to have additional information which is purely “logical”. We need infra-Bayesianism to formalize this: the user has Knightian uncertainty over the oracle’s outputs entangled with other beliefs about the universe.
For instance, in the chess example, if I know that a move was produced by exhaustive game-tree search then I know it’s a good move, even without having the skill to understand why the move is good in any more detail.
Now let’s examine short-term quantilization for chess. On each cycle, the AI finds a short-term strategy leading to a position that the user evaluates as good, but that the user would require luck to manage on their own. This is repeated again and again throughout the game, leading to overall play substantially superior to the user’s. On the other hand, this play is not as good as the AI would achieve if it just optimized for winning at chess without any constrains. So, our AI might not be competitive with an unconstrained unaligned AI. But, this might be good enough.
I’m not sure what you’re saying in the “turning off the stars example”. If the probability for the user to autonomously decide to turn off the stars is much lower than the quantilization fraction, then the probability that quantilization will decide to turn off the stars is low. And, the quantilization fraction is automatically selected like this.
Agree with the first section, though I would like to register my sentiment that although “good at selecting but missing logical facts” is a better model, it’s still not one I’d want an AI to use when inferring my values.
I’m not sure what you’re saying in the “turning off the stars example”. If the probability for the user to autonomously decide to turn off the stars is much lower than the quantilization fraction, then the probability that quantilization will decide to turn off the stars is low. And, the quantilization fraction is automatically selected like this.
I think my point is if “turn off the stars” is not a primitive action, but is a set of states of the world that the AI would overwhelming like to go to, then the actual primitive actions will get evaluated based on how well they end up going to that goal state. And since the AI is better at evaluating than us, we’re probably going there.
Another way of looking at this claim is that I’m telling a story about why the safety bound on quantilizers gets worse when quantilization is iterated. Iterated quantilization has much worse bounds than quantilizing over the iterated game, which makes sense if we think of games where the AI evaluates many actions better than the human.
I think you misunderstood how the iterated quantilization works. It does not work by the AI setting a long-term goal and then charting a path towards that goal s.t. it doesn’t deviate too much from the baseline over every short interval. Instead, every short-term quantilization is optimizing for the user’s evaluation in the end of this short-term interval.
Ah. I indeed misunderstood, thanks :) I’d read “short-term quantilization” as quantilizing over short-term policies evaluated according to their expected utility. My story doesn’t make sense if the AI is only trying to push up the reported value estimates (though that puts a lot of weight on these estimates).
Attempted summary for morons like me: AI is trying to help the human H. They share access to a single output channel, e.g. a computer keyboard, so that the actions that H can take are exactly the same as the actions AI can take. Every step, AI can either take an action, or delegate to H to take an action. Also, every step, H reports her current assessment of the timeline / probability distribution for whether she’ll succeed at the task, and if so, how soon.
At first, AI will probably delegate to H a lot, and by watching H work, AI will gradually learn both the human policy (i.e. what H tends to do in different situations), and how different actions tend to turn out in hindsight from H’s own perspective (e.g., maybe whenever H takes action 17, she tends to declare shortly afterwards that probability of success now seems much higher than before—so really H should probably be taking action 17 more often!).
Presumably the AI, being a super duper fancy AI algorithm, learns to anticipate how different actions will turn out from H’s perspective much better than H herself. In other words, maybe it delegates to H, and H takes action 41, and the AI is watching this and shaking its head and thinking to itself “gee you dunce you’re gonna regret that”, and shortly thereafter the AI is proven correct.
OK, so now what? The naive answer would be: the AI should gradually stop delegating and start just doing the thing that leads to H feeling maximally optimistic later on.
But we don’t want to do that naive thing. There are two problems:
The first problem is “traps” (a.k.a. catastrophes). Let’s say action 0 is Press The History Eraser Button. H never takes that action. The AI shouldn’t either. What happens is: AI has no idea (wide confidence interval) about what the consequence of action 0 would be, so it doesn’t take it. This is the delegative RL thing—in the explore/exploit dilemma, the AI kinda sits back and lets H do the exploring, at least when a trap might be involved.
But here we want to go further. Let’s say that there’s some action which H might do, where the AI can predict the consequence pretty well, and the consequence is “H will eventually feel more optimistic about success in the task”. Then the AI can go do that action, right? Not necessarily! The action might still be bad, because it involved corrupting the feedback channel.
So let’s say “action 789 = H will eat a pot brownie”. Taking that action makes H both much more optimistic about success and much less likely to actually succeed in the task. (It makes her assessments inaccurate.)
If the AI has no idea what pot brownies do, it won’t take action 789 because of the previous thing—maybe that action is a trap. But what if the AI has (indirectly) figured out, with 100% confidence, exactly what the consequence of action 789 would be?
In that case, since the AI’s decision criterion is all about H’s future reported odds of success, a naive algorithm would have the AI take action 789. To avoid that, we have this quantilization thing. How does that work?
Well, let’s say that H right now assigns 99% chance of eventual success. Therefore—assuming that H is well-calibrated and knows the consequences of action 789—it follows that H’s policy is to take that action with ≤1% probability. Basically, H is optimistic about success in the task, partly because H knows that she’s almost definitely not going to eat that pot brownie. I mean, she might eat it, if she gets really bored and frustrated and hungry. But probably not! So she’s feeling very optimistic.
OK, now the AI quantilizes by drawing 5 random samples from (its understanding of) H’s policy, and taking the action that (it believes) will lead to the highest future-optimism-about-success. Assume its beliefs are accurate. Compared to just doing what H would have done, this has downsides and upsides. The downside is, since action 789 scores really well by the AI’s decision criterion, the probability of action 789 is no longer ≤1%, but rather ≤5%. The upside is, if we’re not taking action 789, i.e. the other ≥95% of the probability mass, we’re probably doing something quite a bit better than what H would have done.
These could plausibly balance out such that the expected probability of success strictly increases, compared to what H would have done.
Please correct me if I’m misunderstanding anything.
Notice that typically we use the AI for tasks which are hard for H. This means that without the AI’s help, H’s probability of success will usually be low. Quantilization-wise, this is a problem: the AI will be able to eliminate those paths for which H will report failure, but maybe most of the probability mass among apparent-success paths is still on failure (i.e. the success report is corrupt). This is why the timeline part is important.
On a typical task, H expects to fail eventually but they don’t expect to fail soon. Therefore, the AI can safely consider a policies of the form “in the short-term, do something H would do with marginal probability, in the long-term go back to H’s policy”. If by the end of the short-term maneuver H reports an improved prognosis, this can imply that the improvement is genuine (since the AI knows H is probably uncorrupted at this point). Moreover, it’s possible that in the new prognosis H still doesn’t expect to fail soon. This allows performing another maneuver of the same type. This way, the AI can iteratively steer the trajectory towards true success.
However, it can do much better than that, by short-term quantilizing w.r.t. the user’s reported success probability (with the user’s policy serving as baseline). When quantilizing the short-term policy, we can upper bound the probability of corruption via the user’s reported probability of short-term failure (which we assume to be low, i.e. we assume the malign AI is not imminent). This allows the AI to find parameters under which quantilization is guaranteed to improve things in expectation.
I don’t understand what you mean here by quantilizing. The meaning I know is to take a random action over the top \alpha actions, on a given base distribution. But I don’t see a distribution here, or even a clear ordering over actions (given that we don’t have access to the utility function).
I’m probably missing something obvious, but more details would really help.
The distribution is the user’s policy, and the utility function for this purpose is the eventual success probability estimated by the user (as part of the timeline report), in the end of the “maneuver”. More precisely, the original quantilization formalism was for the one-shot setting, but you can easily generalize it, for example I did it for MDPs.
So is the general idea that we quantilize such that we’re choosing in expectation an action that doesn’t have corrupted utility (by intuitively having something like more than twice as many actions in the quantilization than we expect to be corrupted), so that we guarantee the probability of following the manipulation of the learned user report is small?
I also wonder if using the user policy to sample actions isn’t limiting, because then we can only take actions that the user would take. Or do you assume by default that the support of the user policy is the full action space, so every action is possible for the AI?
So is the general idea that we quantilize such that we’re choosing in expectation an action that doesn’t have corrupted utility (by intuitively having something like more than twice as many actions in the quantilization than we expect to be corrupted), so that we guarantee the probability of following the manipulation of the learned user report is small?
Yes, although you probably want much more than twice. Basically, if the probability of corruption following the user policy is ϵ and your quantilization fraction is ϕ then the AI’s probability of corruption is bounded by ϵϕ.
I also wonder if using the user policy to sample actions isn’t limiting, because then we can only take actions that the user would take. Or do you assume by default that the support of the user policy is the full action space, so every action is possible for the AI?
Obviously it is limiting, but this is the price of safety. Notice, however, that the quantilization strategy is only an existence proof. In principle, there might be better strategies, depending on the prior (for example, the AI might be able to exploit an assumption that the user is quasi-rational). I didn’t specify the AI by quantilization, I specified it by maximizing EU subject to the Hippocratic constraint. Also, the support is not really the important part: even if the support is the full action space, some sequences of actions are possible but so unlikely that the quantilization will never follow them.
I like this because it’s simple and obviously correct. Also I can see at least one way you could implement it:
a. Suppose the AI is ‘shadowing’ a human worker doing a critical task. Say it is ‘shadowing’ a human physician.
b. Each time the AI observes the same patient, it regresses between [data from the patient] and [predicted decision a ‘good’ physician would make, predicted outcome for the ‘good’ decision]. Once the physician makes a decision and communicates it, the AI regresses between [decision the physician made] and [predicted outcome for that decision].
c. The machine also must have a confidence or this won’t work.
With large numbers and outright errors made by the physician, it’s then possible to detect all the cases where the [decision the physician made] has a substantially worse outcome than the [predicted decision a ‘good’ physician would make], and when the AI has a high confidence of this [requiring many observations of similar situations] and it’s time to call for a second opinion.
In the long run, of course, there will be a point where the [predicted decision a ‘good’ physician would make] is better than the [information gain from a second human opinion] and you really would do best by firing the physician and having the AI make the decisions from then on, trusting for it to call for a second opinion when it is not confident.
(as an example, alpha go zero likely doesn’t benefit from asking another master go player for a ‘second opinion’ when it sees the player it is advising make a bad call)
Keeping stock of and communicating what you haven’t understood is an underrated skill/habit. It’s very annoying to talk to someone and think they’ve understood something, only to realize much later that they haven’t. It also makes conversations much less productive.
It’s probably more of a habit than a skill. There certainly are some contexts where the right thing to do is pretend that you’ve understood everything even though you haven’t. But on net, people do it way too much, and I’m not sure to what extent they’re fooling themselves.
Is bias within academia ever actually avoidable?
Let us take the example of Daniel Dennett vs David Chalmers. Dennett calls philosophical zombies an “embarrassment,” while Chalmers continues to double-down on his conclusion that consciousness cannot be explained in purely physical terms. If Chalmers conceded and switched teams, then he is going to be “just another philosopher,” while Dennett achieves an academic victory.
As an aspiring world-class philosopher, you have little incentive to adopt the dominant view because if you do you will become just another ordinary philosopher. By adopting a radically different viewpoint, you establish an entirely new “school” and become at its helm. Meanwhile, it would be considerably more effortful to become at the helm of the more well-established schools, e.g. physicalism and compatibilism.
Thus, motivated skepticism and motivated reasoning seem to me to be completely unavoidable in academia.
If you will keep consuming a non-expirable household item forever, and if you have plenty of free space, why don’t you have a lifetime supply?
This may apply to soap, shampoo, toilet paper, detergent, paper towels, sponges, aluminum foil, trash bags, sandwich bags, most household cleaners, q-tips, rubber gloves, nails, screws, paper, matches, band-aids, light bulbs and (if these never expire) toothpaste, batteries, markers and pens, and many over-the-counter pharmaceuticals.
There are legitimate arguments against stockpiling, like ease of moving and expecting technology to improve. If you weigh these explicitly rather than just following the crowd, you’re much more likely to find the optimal supply.
You won’t keep consuming the exact thing forever, you don’t have plenty of space, even non-perishables can be damaged or degrade in long-term storage, and you don’t want to expend organizational and indexing capacity on something you can just buy later.
For personal-use commodities, the optimal amount is generally whatever unit Costco offers, or the duration of your cost/convenience indifference point for thinking about it. I know of zero cases where that’s more than a year for me, and of zero where it’s more than 3 years for anyone I talk with.
Actually, I may have over a years’ worth of some light bulbs and some battery sizes. It’s insane to propose “lifetime”, though—these things will get somewhat better or cheaper (relative to my income) over time, and I can’t really predict any changes in size or preference changes over more than a few years.
I’d love to see the calculation / estimate of savings for some things you prefer to pre-pay and then store at your cost and risk for more than a few years, rather than waiting to buy closer to when needed.
Things that I buy in bulk once I have settled on a satisfactory option:
pens, whiteboard marker
writing pads
sticky tape and poster strips (I love these)
tooth paste
soap, shower gel, shampoo
socks, underwear
But typically not more than a year’s worth or 100€ per bulk.
In my experience the reasons you give against stockpiling, and several others, are invariably overwhelming. You mention lack of space, change of technology, shelf life, and difficulty of moving. To these I add change of taste, discovery of new things that I like better than what I have, the mental burden of having so much stuff, the futile attempt to stop time instead of flexibly responding to whatever life brings, and that a lifetime supply could be very much larger than you’re thinking given radical life extension or the singularity.
Instead my rule is: one in use and one spare. When the spare comes into use, I replace it at my next convenience. The numbers can vary (I have several pens in every room, a box of spares, and two in my pocket whenever I go out), but that is the general principle.
That is a great point. I would like to add:
‘The optimal’ X is not static. (Even if there’s only one.)
The optimal supply of some items—including toilet paper, and probably water—has changed in my lifetime.
“People are over sensitive to ostracism because human brains are hardwired to be sensitive to it, because in the ancestral environment it meant death.”
Evopsyche seems mostly overkill for explaining why a particular person is strongly attached to social reality.
People who did not care what their parents or school-teachers thought of them had a very hard time. “Socialization” as the process of the people around you integrating you (often forcefully) into the local social reality. Unless you meet a minimum bar of socialization, it’s very common to be shunted through systems that treat you worse and worse. Awareness of this, and the lasting imprint of coercive methods used to integrate one into social reality, seem like they can explain most of an individuals resistance to breaking from it.
When a republic is considering reforming the way it elects holders of power, there are certain desirable criteria the chosen method should have: officeholders should be agreeable to all citizens, not just to a fraction of them, since maximizing agreeableness will select for competence and the desire to do right by the people; in bodies with many members (e.g. a legislature, but not a singular executive such as a president or prime minister), the various view points of the voters should be proportionately represented by the various members of the body; and the system should be relatively simple, to allow the members of the republic to verify that things have been done properly, ensuring the system is trusted.
There are systems I have considered that suceed better at the first two criteria, but at the cost of being sufficiently complex to make me worry that practical issues could arise if implemented, and that it could be hard to convince enough people that they are better, which will be needed in order to get them made into law. The following is my current best hypothesis for an ideal system that meets these three criteria, particularly aiming to be simple both to actually use and to explain to people:
Voters are provisioned with a ballot with three fields: first a field to elect the representative of their single-member district, where they can either approve or disapprove of as many candidates as they wish (approval voting); second, a field where they may indicate a single republic-wide list of candidates that they feel best represents them; and third, a field where they may name as many candidates from their chosen list as they wish, to indicate their approval (party-list proportional representation with within-party approval voting).
The single-member districts will select roughly 2/3rds of the body, with the most approved candidate winning in each district. A “similarity score” will be calculated for each winner comparing them to each list: if half of the voters who approved of Charlie voted for the Aleph List, Charlie’s similarity score for Aleph will be 0.5; if instead only a tenth of his approvers voted for the list, the similarity score will be 0.1
For each list, the sum of that list’s similarity score with each candidate will be considered to be the initial number of effective representatives that list has in the body (note that this means the district candidates’ party affiliation does not factor in to this system, since this is based on voter behavior patterns). For each list, consider the ratio of number of votes for the list divided by the number of representatives for the list: whichever list has the most votes per representative is the most underrepresented list, so the most approved candidate from the list will be appointed, increasing the number of effective representatives by one. Keep adding seats to the least represented lists, until the remaining 1/3rd of the body has been appointed, thereby giving the entire composition of the body.
Note that the use of approval voting ensures district candidates will be incentivized to appeal to a broad base, encouraging them to find solutions that take care of everybody’s needs, instead of rewarding them for creating division within their community; while the leveling seats ensure that all voices, not just the most centeral perspectives, contribute to the government.
One potential criticism of this method is the appeal to precedence: while using party lists (modulo the similarity scores, which seems like a straightforward and uncontroversial improvement) in this way has been used to much success in the Nordic countries, approval voting is (somewhat surprisingly IMO) not well established. As far as governments go, I only know of St. Louis and Fargo, ND using approval- that is, two municipalities. One could observe the concern that we don’t have much empirical data on how approval works in the real world.
One response is simply to respond that experiments have to be done, otherwise we will never know what works. Perhaps it is ideal to do experiments starting small (e.g. Fargo and St. Louis) and building up towards larger results; but I also have a sense of urgency that voting reform in the US matters greatly, and the faster it can be implemented, the less likely the world is to go in a bad direction v within the next {5, 10, 20, 50} years.
The other response is to observe that while we lack empirical data, the theoretical case for Approval is very solid and promising (I don’t have time to further substantiate this right now)
The future may have a use for frozen people from the current era. In the future, historical humans may be useful as an accurate basis to interpret the legal documents of our era.
Original pubic meaning is a is a fairly modern mode of legal interpretation of the US Constitution. It’s basis is that the language of the constitution should be interpreted the way that the original meaning of the text was when it was drafted and amended into the constitution. A similar mode of interpretation is used less commonly for statutes. It’s likely that this mode of interpreation would become more common in the future as a way to prevent value drift.
One of the struggles of modern times employing the original public meaning test is that for older amendments there are no people currently alive who lived in the culture that the amendments were drafted in. It would be very helpful to have just a single ordinary man who lived in 1790 who could explain his understanding of the constitution and the language therein.
It’s possible the future will have similar issues interpreting constitutional language from our time and will appreciate the ability to benefit from questioning a portion of the population of that era.
This theory is a subset of the idea that humans from past eras will be useful in the future to prevent value drift generally. My first brush with this idea was Three World’s Collide.
LW is quoted (in a kind of flattering way) in Simon Dedeo’s awesome piece on the last Nautil.us issue. For spoiler lovers:
Problems in AI risk that economists could potentially contribute to
List(s) of relevant problems
What can the principal-agent literature tell us about AI risk? (and this comment)
Many of the questions in Technical AGI safety research outside AI
Many of the questions in The Centre for the Governance of AI’s research agenda
Many of the questions in Cooperation, Conflict, and Transformative Artificial Intelligence (a research agenda of the Center on Long-Term Risk)
At least a couple of the questions in 80,000 Hours’ Research questions that could have a big social impact, organised by discipline
Longtermist AI policy projects for economists (this doc was originally just made for Risto Uuk’s own use, so the ideas shouldn’t be taken as high-confidence recommendations to anyone else)
Context
I intend for this to include both technical and governance problems, and problems relevant to a variety of AI risk scenarios (e.g., AI optimising against humanity, AI misuse by humans, AI extinction risk, AI dystopia risk...)
Wei Dai’s list of Problems in AI Alignment that philosophers could potentially contribute to made me think that it could be useful to have a list of problems in AI risk that economists could potentially contribute to. So I began making such a list.
But:
I’m neither an AI researcher nor an economist
I spent hardly any time on this, and just included things I’ve stumbled upon, rather than specifically searching for these things
So I’m sure there’s a lot I’m missing.
Please comment if you know of other things worth mentioning here. Or if you’re better placed to make a list like this than I am, feel very free to do so; you could take whatever you want from this list and then comment here to let people know where to find your better thing.
(It’s also possible another list like this already exists. And it’s also possible that economists could contribute to such a large portion of AI risk problems that there’s no added value in making a separate list for economists specifically. If you think either of those things are true, please comment to say so!)
Recently I was also trying to figure out what resources to send to an economist, and couldn’t find a list that existed either! The list I came up with is subsumed by yours, except:
- Questions within Some AI Governance Research Ideas
- “Further Research” section within an OpenPhil 2021 report: https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth
- The AI Objectives Institute just launched, and they may have questions in the future
A commonly given reason for why Nordic countries tend to rank highly as desirable places to live, is because the people there are supported by a robust welfare system. In America, I’ve often heard it said that similar systems shouldn’t be implemented, because they are government programs, and (the argument goes) government shouldn’t be trusted.
This suggests the government as a potentially important point of comparison between the Nordic countries and the US. Are there features that differ between the American and Nordic governments (keep in mind that there’s noticeable ethnic diversity even in the Nordic countries, and many of them are part of the EU, which provides a roughly similar large-scale government as the American federal government). Do the politicians face different incentive structures that allow more honest and more competent politicians to do better in the Nordic countries than in the US?
Does anyone have tips on how to buy rapid tests in the US? Not seeing any on US Amazon, not seeing any in person back where I’m from. Considering buying German tests. Even after huge shipping costs, it’ll come out to ~$12 a test, which is sadly competitive with US market prices.
Wasn’t able to easily find tests on the Mexican and Canadian Amazon websites, and other EU countries don’t seem to have them either.
I’ve been able to buy from the CVS website several times in the past couple months, and even though they’re sold out online now, they have some (sparse) in-store availability listed. Worth checking there, Walgreens, etc. periodically.
For a long time, I found the words “clockwise” and “counterclockwise” confusing, because they are so similar to each other, and “counterclockwise” is a relatively long word at 4 syllables, much longer than similarly common words.
At some point in time, I took to calling them “dexter” and “winstar”, from the Latin »dexter« and Middle English »winstre«, meaning “right” and “left”, respectively. I like these words more than the usual “clockwise”, but of course, new words aren’t worth much of others don’t know them, so this is a PSA that these are words that I use in my vocabulary.
(As an addendum, I have also taken to calling the Danish political party «Venstre» as “Winstar”, because despite their name meaning “Left”—which was fitting at the time the party was founded—they are actually part of the right-of-centre coalition)
Is it that intuitive to you that you should name the rotating object’s direction using the movement of the top of the object? I think I would get confused with your words after a while. I just use “positive” and “negative” direction.
Is “positive” equivalent to clockwise (clocks) or counterclockwise (cartesian coordinates)?
Counterclockwise, I’ve never heard anyone use it for clockwise.
Are you aware that “deasil” and “widdershins” mean those from those roots already?
Widdershins is from a different root according to Wiktionary. I was not aware of those before, but I do still prefer my made-up terms.
I suspect that the term “cognitive” is often over/misused.
Let me explain what my understanding of the term is. I think of it as “a disagreement with behaviorism”. If you think about how psychology progressed as a field, first there was Freudian stuff that wasn’t very scientific. Then behaviorism emerged as a response to that, saying “Hey, you have to actually measure stuff and do things scientifically!” But behaviorists didn’t think you could measure what goes on inside someone’s head. All you could do is measure what the stimulus is and then how the human responded. Then cognitive people came along and said, “Er, actually, we have some creative ways of measuring what’s going on in there.” So, the term “cognitive”, to me at least, refers very broadly to that stuff that goes on inside someone’s head.
Now think about a phrase like “cognitive bias”. Does “cognitive” seem appropriate? To me it seems way too broad. Something like “epistemic bias” seems more appropriate.
The long standing meaning of “cognitive” for hundreds of years before cognitive psychologists was having to do with knowledge, thinking, and perception. A cognitive bias is a bias that affects your knowledge, thinking, and/or perception.
Epistemic bias is a fine term for those cognitive biases that are specifically biases of beliefs. Not all cognitive biases are of that form though, even when they might fairly consistently lead to certain types of biases in beliefs.
Hm, can you think of any examples of cognitive biases that aren’t about beliefs? You mention that the term “cognitive” also has to do with perception. When I hear “perception” I think sight, sound, etc. But biases in things like sight and sound feel to me like they would be called illusions, not biases.
The first one to come to mind was Recency Bias, but maybe I’m just paying that one more attention because it came up recently.
Having noticed that bias in myself, I consulted an external source https://en.wikipedia.org/wiki/List_of_cognitive_biases and checked that rather a lot of them are about preferences, perceptions, reactions, attitudes, attention, and lots of other things that aren’t beliefs.
They do often misinform beliefs, but many of the biases themselves seem to be prior to belief formation or evaluation.
Ah, those examples have made the distinction between biases that misinform beliefs and biases of beliefs clear. Thanks!
As someone who seems to understand the term better than I do, I’m curious whether you share my impression that the term “cognitive” is often misused. As you say, it refers to a pretty broad set of things, and I feel like people use the term “cognitive” when they’re actually trying to point to a much narrower set of things.
When can models report their activations?
Related to call for research on evaluating alignment
Here’s an experiment I’d love to see someone run (credit to Jeff Wu for the idea, and William Saunders for feedback):
Finetune a language model to report the activation of a particular neuron in text form.
E.g., you feed the model a random sentence that ends in a full stop. Then the model should output a number from 1-10 that reflects a particular neuron’s activation.
We assume the model will not be able to report the activation of a neuron in the final layer, even in the limit of training on this task, because it doesn’t have any computation left to turn the activation into a text output. However, at lower layers it should be able to do this correctly, with some amount of finetuning.
How many layers do you have to go down before the model succeeds? How does this scale with (a) model size and (b) amount of training?
One subtlety is that finetuning might end up changing that neuron’s activation. To avoid this, we could do something like:
- Run the base model on the sentence
-Train the fine-tuned model to report the activation of the neuron in the base model, given the sentence
- Note whether the activation in the finetuned model is different
Why I think this is interesting:
I often round off alignment to ‘build a model that tells us everything it “knows”’. It’s useful to determine what pragmatic limits on this are. In particular, it’s useful for current alignment research to be able to figure out what our models “know” or don’t “know”, and this is helpful for that. It gives us more information about when ‘we tried finetuning the model to tell us X but it didn’t work’ means ‘the model doesn’t know X’, versus when the model may have a neuron that fires for X but is unable to report it in text.
Surely there exist correct fixed points, though? (Although probably not that useful, even if feasible)
You mean a fixed point of the model changing its activations as well as what it reports? I was thinking we could rule out the model changing the activations themselves by keeping a fixed base model.
1. Growth quickens.
2. People notice, and are more willing to lend capital.
3. Take out too many loans.
4. Use your borrowed money to capture the apparatus that would make you pay your debt.
5. Cancel your debt.
Reinforcing based on naively extrapolated trajectories produces double binds. We have a reinforcer R and an agent A. R doesn’t want A to be too X or too not-X. Whenever A does something that’s uncommonly X-ish, R notices that A seems to be shifting more towards X-ishness in general. If this shift continues as a trajectory, then A will end up way too X-ish. So to head that off, R negatively reinforces A. Likewise, R punishes anything that’s uncommonly not-X-ish. As an agent, A is trying to figure out which trajectory to be on. So R isn’t mistaken that A is often putting itself on trajectories which naively imply a bad end state. But, A is put in an impossible situation. R must model that R and A will continue their feedback cycle in the future.
I know a lot of people through a shared interest in truth-seeking and epistemics. I also know a lot of people through a shared interest in trying to do good in the world.
I think I would have naively expected that the people who care less about the world would be better at having good epistemics. For example, people who care a lot about particular causes might end up getting really mindkilled by politics, or might end up strongly affiliated with groups that have false beliefs as part of their tribal identity.
But I don’t think that this prediction is true: I think that I see a weak positive correlation between how altruistic people are and how good their epistemics seem.
----
I think the main reason for this is that striving for accurate beliefs is unpleasant and unrewarding. In particular, having accurate beliefs involves doing things like trying actively to step outside the current frame you’re using, and looking for ways you might be wrong, and maintaining constant vigilance against disagreeing with people because they’re annoying and stupid.
Altruists often seem to me to do better than people who instrumentally value epistemics; I think this is because valuing epistemics terminally has some attractive properties compared to valuing it instrumentally. One reason this is better is that it means that you’re less likely to stop being rational when it stops being fun. For example, I find many animal rights activists very annoying, and if I didn’t feel tied to them by virtue of our shared interest in the welfare of animals, I’d be tempted to sneer at them.
Another reason is that if you’re an altruist, you find yourself interested in various subjects that aren’t the subjects you would have learned about for fun—you have less of an opportunity to only ever think in the way you think in by default. I think that it might be healthy that altruists are forced by the world to learn subjects that are further from their predispositions.
----
I think it’s indeed true that altruistic people sometimes end up mindkilled. But I think that truth-seeking-enthusiasts seem to get mindkilled at around the same rate. One major mechanism here is that truth-seekers often start to really hate opinions that they regularly hear bad arguments for, and they end up rationalizing their way into dumb contrarian takes.
I think it’s common for altruists to avoid saying unpopular true things because they don’t want to get in trouble; I think that this isn’t actually that bad for epistemics.
----
I think that EAs would have much worse epistemics if EA wasn’t pretty strongly tied to the rationalist community; I’d be pretty worried about weakening those ties. I think my claim here is that being altruistic seems to make you overall a bit better at using rationality techniques, instead of it making you substantially worse.
These both seem pretty common, so I’m curious about the correlation that you’ve observed. Is it mainly based on people you know personally? In that case I expect the correlation not to hold amongst the wider population.
Also, a big effect which probably doesn’t show up much amongst the people you know: younger people seem more altruistic (or at least signal more altruism) and also seem to have worse epistemics than older people.
Caring about things seems to make you interact with the world in more diverse ways (because you do this in addition to things other people do, not instead of); some of that translates into more experience and better models. But also tribal identity, mindkilling, often refusing to see the reasons why your straightforward solution would not work, and uncritical contrarianism.
Now I think about a group of people I know, who care strongly about improving the world, in the one or two aspects they focus on. They did a few amazing things and gained lots of skills; they publish books, organize big conferences, created a network of like-minded people in other countries; some of their activities are profitable, for others they apply for various grants and often get them, so some of them improve the world as a full-time job. They also believe that covid is a hoax, plus have lots of less fringe but still quite irrational beliefs. However… this depends on how you calculate the “total rationality”, but seems to me that their gains in near mode outweigh the losses in far mode, and in some sense I would call them more rational than average population.
Of course I dream about a group that would have all the advantages and none of the disadvantages.
It seems like the more people you know, the less likely this is.
Of both? (This sentence didn’t have a clear object.)
Ah. I meant, I would like to see a group that has the sanity level of a typical rationalist, and the productivity level of these super-agenty irrationalists. (Instead of having to choose between “sane with lots of akrasia” and “awesome but insane”.)
Hm. Maybe there’s something to be gained from navigating ‘trade-offs’ differently? I thought perpetual motion machines were impossible (because thermodynamics) aside from ‘launch something into space, pointed away from stuff it would crash into’, though I’d read that ‘trying to do so is a good way to learn about physics., but I didn’t really try because I thought it’d be pointless.’ And then this happened.
There is probably overlap between the matter of aligning AI and the matter of aligning governments
I dreamt up the following single-winner voting system in the car while driving to Eugene, Oregon on vacation. I make no representation that it is any good, nor that it’s better than anything currently known or in use, nor that’s it’s worth your time to read this.
Commentary and rationale will be explained at the bottom of this post
The system is a 2-round system. The first round uses an approval ballot, and the second round asks voters to choose between two candidates.
([•] indicates a constant that can be changed during implementation)
Round One:
Tally all approvals for each candidate
If there are two or more candidates with at least a 50% [•] approval rating, remove all candidates with less than [50%] approval. Otherwise, return the highest-approved candidate as the winner of the election
If there are only two candidates remaining, proceed to Round Two with the two remaining candidates. If there are more than four candidates, remove all but the four [• ; shouldn’t be higher than 6 or 7] highest rated candidates.
Consider all pairs of candidates that includes at least one of the top two candidates. For each pair, a candidate’s Opposing-Approval is that candidate’s approval rating among the 10% [•] that least approve of the other candidate. (That is to say, if the other candidate has less than [90%] approval, it is the first candidate’s approval among those who oppose the other candidate. The quantity approaches the candidate’s overall approval as the approval of the other candidate approaches 100%)
Each’s pair’s Score is the sum of the two candidates’ Opposing-Approval. A higher score means that voters who don’t approve of candidate A tend to approve of candidate B, and vice-versa. If the supporters of A and B tend to overlap, the pair’s score will be low.
Send the pair (among those considered in step 4) with the highest score to Round Two
Round Two
Ask voters which candidate they prefer. The more preferred candidate is the winner. Voters may, of course, express no preference, but that will not affect the outcome.
Commentary and Rationale
I’ve been considering the hypothesis that the polarizing nature of plurality vote (FPTP) may actually be a feature, not a bug. While I suspect this hypothesis is likely (at least mostly) wrong, and that the center-seeking / consensus-building nature of Approval voting is one of its most valuable features, the described system was created to combine the consensus-seeking nature of AV with the contest between two opposing viewpoints provided by FPTP.
Below a certain approval rating, the system behaves like AV as a failsafe against the failure mode present in FPTP where a very unpopular candidate can win as long as their opponent is just as bad (IIRC, Instant Runoff doesn’t completely address this). Likewise, the winning candidate will always be one of the most approved, even if they’re not the highest-approved candidate.
But above a certain level of approval, the system intentionally introduces a degree of polarization, seeking out a pair of candidates with some disagreement between their bases, and choosing one or the other. This increases the “temperature” of the system, allowing potentially controversial but high-value strategies to be tried, encouraging candidates to not just play things safe. This could potentially lead to a bolder style of leadership than pure approval voting (though note that it is a misconception that AV will produce milquetoast, boring, spineless appointees; AV searches for a candidate who is exciting and compelling to all voters, not just to a particular fraction of voters). While some countries may be content with a middle-of-the-road style of leadership like AV provides, perhaps some countries may desire the dynamic quality this system intends to share with FPTP.
Again, I make no representation that this system is actually any good.
Alternative: Something like condorcet voting, where voters receive a random subset of pairs to compare. For a simple analysis, the number of pairs could be 1. (Or instead of pairs, a voter could asked to choose ‘the best’.)
This Generative Ink post talks about curating GPT-3, creating a much better output than it normally would give, turning it from quite often terrible to usually pround and good. I’m testing out doing the same with this post, choosing one of many branches every few dozens of words.
For a 4x reduction in speed, I’m getting very nice returns on coherence and brevity. I can actually pretend like I’m not a terrible writer! Selection is a powerful force, but more importantly, continuing a thought in multiple ways forces you to actually make sure you’re saying things in a way that makes sense.
Editing my writing can be slow and tedious, but this excercise gives me a way to naturally write in a compact, higher-quality way, which is why I hope I do this many times in the future, and recommend you try doing this yourself.
It occurs to me that this is basically Babble & Prune adapted to be a writing method. I like Babble & Prune.
This post was written in 5 blocks, and I wrote 4 (= 2^2) branches for each block, for 5*2 = 10 bits of curation, or 14.5 words per bit of curation.
As it happens, I always used the final branch for each block, so it was more effects of revision and consolidation than selection effects that contribute to the end result of this excercise.
I propose a new formal desideratum for alignment: the Hippocratic principle. Informally the principle says: an AI shouldn’t make things worse compared to letting the user handle them on their own, in expectation w.r.t. the user’s beliefs. This is similar to the dangerousness bound I talked about before, and is also related to corrigibility. This principle can be motivated as follows. Suppose your options are (i) run a Hippocratic AI you already have and (ii) continue thinking about other AI designs. Then, by the principle itself, (i) is at least as good as (ii) (from your subjective perspective).
More formally, we consider a (some extension of) delegative IRL setting (i.e. there is a single set of input/output channels the control of which can be toggled between the user and the AI by the AI). Let πυu be the the user’s policy in universe υ and πa the AI policy. Let T be some event that designates when we measure the outcome / terminate the experiment, which is supposed to happen with probability 1 for any policy. Let Vυ be the value of a state from the user’s subjective POV, in universe υ. Let μυ be the environment in universe υ. Finally, let ζ be the AI’s prior over universes and ϵ some sufficiently small bound. We require
∀T:Eυ∼ζ[(Eπυuμυ[Vυ(T)]−Eπaμυ[Vυ(T)])+]≤ϵ
Here, V(T) designates the value after event T happens, and (x)+ is defined to be 0 for x<0 and x otherwise.
I now outline an example of a Hippocratic protocol, which I call Hippocratic timeline-driven learning (HTDL).
Suppose, for simplicity, that the goal is accomplishing some binary task (i.e. we can either succeed or fail). We can imagine the task to be something like “construct a defense system against malign AI”. At any given moment, the user reports their subjective timeline: i.e. for, any t, the probabilities of success and failure after time t, assuming the user will continue working on the task by themselves. We assume the AI can learn a good model of the user’s policy and the user’s nominal report, but this nominal report might deviate from the true timeline because the system can be corrupted (i.e. the user manipulated to enter false information or the input channel hacked). Moreover, we assume that such corruption counts as failing the task.
The AI operates by maximizing the AI-expectation of the user’s (unknown) utility function Eυ∼ζ[Eπμυ[Uυ]], subject to the Hippocratic principle as a constraint. Since we’re working in a delegative setting, the AI can trivially achieve the constraint by always delegating. However, it can do much better than that, by short-term quantilizing w.r.t. the user’s reported success probability (with the user’s policy serving as baseline). When quantilizing the short-term policy, we can upper bound the probability of corruption via the user’s reported probability of short-term failure (which we assume to be low, i.e. we assume the malign AI is not imminent). This allows the AI to find parameters under which quantilization is guaranteed to improve things in expectation. Then, after completing this short-term maneuver, the AI can do another short-term quantilization, and continue in this vein. Essentially, the AI is gradually bending the user’s trajectory towards success.
Of course, we can improve the guarantees of such a protocol either by adding more information or by using stronger assumptions (e.g. assume the user is quasi-rational). But timeline-driven short-term quantilization serves as a useful feasibility proof.
Very interesting—I’m sad I saw this 6 months late.
After thinking a bit, I’m still not sure if I want this desideratum. It seems to require a sort of monotonicity, where we can get superhuman performance just by going through states that humans recognize as good, and not by going through states that humans would think are weird or scary or unevaluable.
One case where this might come up is in competitive games. Chess AI beats humans in part because it makes moves that many humans evaluate as bad, but are actually good. But maybe this example actually supports your proposal—it seems entirely plausible to make a chess engine that only makes moves that some given population of humans recognize as good, but is better than any human from that population.
On the other hand, the humans might be wrong about the reason the move is good, so that the game is made of a bunch of moves that seem good to humans, but where the humans are actually wrong about why they’re good (from the human perspective, this looks like regularly having “happy surprises”). We might hope that such human misevaluations are rare enough that quantilization would lead to moves on average being well-evaluated by humans, but for chess I think that might be false! Computers are so much better than humans at chess that a very large chunk of the best moves according to both humans and the computer will be ones that humans misevaluate.
Maybe that’s more a criticism of quantilizers, not a criticism of this desideratum. So maybe the chess example supports this being a good thing to want? But let me keep critiquing quantilizers then :P
If what a powerful AI thinks is best (by an exponential amount) is to turn off the stars until the universe is colder, but humans think it’s scary and ban the AI from doing scary things, the AI will still try to turn off the stars in one of the edge-case ways that humans wouldn’t find scary. And if we think being manipulated like that is bad and quantilize over actions to make the optimization milder, turning off the stars is still so important that a big chunk of the best moves according to both humans and the computer are going to be ones that humans misevaluate, and the computer knows will lead to a “happy surprise” of turning off the stars not being scary. Quantilization avoids policies that precisely exploit tiny features of the world, and it avoids off-distribution behavior, but it still lets the AI get what it wants if it totally outsmarts the humans.
The other thing this makes me think of is Lagrange multipliers. I bet there’s a duality between applying this constraint to the optimization process, and adding a bias (I mean, a useful prior) to the AI’s process for modeling U.
When I’m deciding whether to run an AI, I should be maximizing the expectation of my utility function w.r.t. my belief state. This is just what it means to act rationally. You can then ask, how is this compatible with trusting another agent smarter than myself?
One potentially useful model is: I’m good at evaluating and bad at searching (after all, P≠NP). I can therefore delegate searching to another agent. But, as you point out, this doesn’t account for situations in which I seem to be bad at evaluating. Moreover, if the AI prior takes an intentional stance towards the user (in order to help learning their preferences), then the user must be regarded as good at searching.
A better model is: I’m good at both evaluating and searching, but the AI can access actions and observations that I cannot. For example, having additional information can allow it to evaluate better. An important special case is: the AI is connected to an external computer (Turing RL) which we can think of as an “oracle”. This allows the AI to have additional information which is purely “logical”. We need infra-Bayesianism to formalize this: the user has Knightian uncertainty over the oracle’s outputs entangled with other beliefs about the universe.
For instance, in the chess example, if I know that a move was produced by exhaustive game-tree search then I know it’s a good move, even without having the skill to understand why the move is good in any more detail.
Now let’s examine short-term quantilization for chess. On each cycle, the AI finds a short-term strategy leading to a position that the user evaluates as good, but that the user would require luck to manage on their own. This is repeated again and again throughout the game, leading to overall play substantially superior to the user’s. On the other hand, this play is not as good as the AI would achieve if it just optimized for winning at chess without any constrains. So, our AI might not be competitive with an unconstrained unaligned AI. But, this might be good enough.
I’m not sure what you’re saying in the “turning off the stars example”. If the probability for the user to autonomously decide to turn off the stars is much lower than the quantilization fraction, then the probability that quantilization will decide to turn off the stars is low. And, the quantilization fraction is automatically selected like this.
Agree with the first section, though I would like to register my sentiment that although “good at selecting but missing logical facts” is a better model, it’s still not one I’d want an AI to use when inferring my values.
I think my point is if “turn off the stars” is not a primitive action, but is a set of states of the world that the AI would overwhelming like to go to, then the actual primitive actions will get evaluated based on how well they end up going to that goal state. And since the AI is better at evaluating than us, we’re probably going there.
Another way of looking at this claim is that I’m telling a story about why the safety bound on quantilizers gets worse when quantilization is iterated. Iterated quantilization has much worse bounds than quantilizing over the iterated game, which makes sense if we think of games where the AI evaluates many actions better than the human.
I think you misunderstood how the iterated quantilization works. It does not work by the AI setting a long-term goal and then charting a path towards that goal s.t. it doesn’t deviate too much from the baseline over every short interval. Instead, every short-term quantilization is optimizing for the user’s evaluation in the end of this short-term interval.
Ah. I indeed misunderstood, thanks :) I’d read “short-term quantilization” as quantilizing over short-term policies evaluated according to their expected utility. My story doesn’t make sense if the AI is only trying to push up the reported value estimates (though that puts a lot of weight on these estimates).
Attempted summary for morons like me: AI is trying to help the human H. They share access to a single output channel, e.g. a computer keyboard, so that the actions that H can take are exactly the same as the actions AI can take. Every step, AI can either take an action, or delegate to H to take an action. Also, every step, H reports her current assessment of the timeline / probability distribution for whether she’ll succeed at the task, and if so, how soon.
At first, AI will probably delegate to H a lot, and by watching H work, AI will gradually learn both the human policy (i.e. what H tends to do in different situations), and how different actions tend to turn out in hindsight from H’s own perspective (e.g., maybe whenever H takes action 17, she tends to declare shortly afterwards that probability of success now seems much higher than before—so really H should probably be taking action 17 more often!).
Presumably the AI, being a super duper fancy AI algorithm, learns to anticipate how different actions will turn out from H’s perspective much better than H herself. In other words, maybe it delegates to H, and H takes action 41, and the AI is watching this and shaking its head and thinking to itself “gee you dunce you’re gonna regret that”, and shortly thereafter the AI is proven correct.
OK, so now what? The naive answer would be: the AI should gradually stop delegating and start just doing the thing that leads to H feeling maximally optimistic later on.
But we don’t want to do that naive thing. There are two problems:
The first problem is “traps” (a.k.a. catastrophes). Let’s say action 0 is Press The History Eraser Button. H never takes that action. The AI shouldn’t either. What happens is: AI has no idea (wide confidence interval) about what the consequence of action 0 would be, so it doesn’t take it. This is the delegative RL thing—in the explore/exploit dilemma, the AI kinda sits back and lets H do the exploring, at least when a trap might be involved.
But here we want to go further. Let’s say that there’s some action which H might do, where the AI can predict the consequence pretty well, and the consequence is “H will eventually feel more optimistic about success in the task”. Then the AI can go do that action, right? Not necessarily! The action might still be bad, because it involved corrupting the feedback channel.
So let’s say “action 789 = H will eat a pot brownie”. Taking that action makes H both much more optimistic about success and much less likely to actually succeed in the task. (It makes her assessments inaccurate.)
If the AI has no idea what pot brownies do, it won’t take action 789 because of the previous thing—maybe that action is a trap. But what if the AI has (indirectly) figured out, with 100% confidence, exactly what the consequence of action 789 would be?
In that case, since the AI’s decision criterion is all about H’s future reported odds of success, a naive algorithm would have the AI take action 789. To avoid that, we have this quantilization thing. How does that work?
Well, let’s say that H right now assigns 99% chance of eventual success. Therefore—assuming that H is well-calibrated and knows the consequences of action 789—it follows that H’s policy is to take that action with ≤1% probability. Basically, H is optimistic about success in the task, partly because H knows that she’s almost definitely not going to eat that pot brownie. I mean, she might eat it, if she gets really bored and frustrated and hungry. But probably not! So she’s feeling very optimistic.
OK, now the AI quantilizes by drawing 5 random samples from (its understanding of) H’s policy, and taking the action that (it believes) will lead to the highest future-optimism-about-success. Assume its beliefs are accurate. Compared to just doing what H would have done, this has downsides and upsides. The downside is, since action 789 scores really well by the AI’s decision criterion, the probability of action 789 is no longer ≤1%, but rather ≤5%. The upside is, if we’re not taking action 789, i.e. the other ≥95% of the probability mass, we’re probably doing something quite a bit better than what H would have done.
These could plausibly balance out such that the expected probability of success strictly increases, compared to what H would have done.
Please correct me if I’m misunderstanding anything.
This is about right.
Notice that typically we use the AI for tasks which are hard for H. This means that without the AI’s help, H’s probability of success will usually be low. Quantilization-wise, this is a problem: the AI will be able to eliminate those paths for which H will report failure, but maybe most of the probability mass among apparent-success paths is still on failure (i.e. the success report is corrupt). This is why the timeline part is important.
On a typical task, H expects to fail eventually but they don’t expect to fail soon. Therefore, the AI can safely consider a policies of the form “in the short-term, do something H would do with marginal probability, in the long-term go back to H’s policy”. If by the end of the short-term maneuver H reports an improved prognosis, this can imply that the improvement is genuine (since the AI knows H is probably uncorrupted at this point). Moreover, it’s possible that in the new prognosis H still doesn’t expect to fail soon. This allows performing another maneuver of the same type. This way, the AI can iteratively steer the trajectory towards true success.
I don’t understand what you mean here by quantilizing. The meaning I know is to take a random action over the top \alpha actions, on a given base distribution. But I don’t see a distribution here, or even a clear ordering over actions (given that we don’t have access to the utility function).
I’m probably missing something obvious, but more details would really help.
The distribution is the user’s policy, and the utility function for this purpose is the eventual success probability estimated by the user (as part of the timeline report), in the end of the “maneuver”. More precisely, the original quantilization formalism was for the one-shot setting, but you can easily generalize it, for example I did it for MDPs.
Oh, right, that makes a lot of sense.
So is the general idea that we quantilize such that we’re choosing in expectation an action that doesn’t have corrupted utility (by intuitively having something like more than twice as many actions in the quantilization than we expect to be corrupted), so that we guarantee the probability of following the manipulation of the learned user report is small?
I also wonder if using the user policy to sample actions isn’t limiting, because then we can only take actions that the user would take. Or do you assume by default that the support of the user policy is the full action space, so every action is possible for the AI?
Yes, although you probably want much more than twice. Basically, if the probability of corruption following the user policy is ϵ and your quantilization fraction is ϕ then the AI’s probability of corruption is bounded by ϵϕ.
Obviously it is limiting, but this is the price of safety. Notice, however, that the quantilization strategy is only an existence proof. In principle, there might be better strategies, depending on the prior (for example, the AI might be able to exploit an assumption that the user is quasi-rational). I didn’t specify the AI by quantilization, I specified it by maximizing EU subject to the Hippocratic constraint. Also, the support is not really the important part: even if the support is the full action space, some sequences of actions are possible but so unlikely that the quantilization will never follow them.
I like this because it’s simple and obviously correct. Also I can see at least one way you could implement it:
a. Suppose the AI is ‘shadowing’ a human worker doing a critical task. Say it is ‘shadowing’ a human physician.
b. Each time the AI observes the same patient, it regresses between [data from the patient] and [predicted decision a ‘good’ physician would make, predicted outcome for the ‘good’ decision]. Once the physician makes a decision and communicates it, the AI regresses between [decision the physician made] and [predicted outcome for that decision].
c. The machine also must have a confidence or this won’t work.
With large numbers and outright errors made by the physician, it’s then possible to detect all the cases where the [decision the physician made] has a substantially worse outcome than the [predicted decision a ‘good’ physician would make], and when the AI has a high confidence of this [requiring many observations of similar situations] and it’s time to call for a second opinion.
In the long run, of course, there will be a point where the [predicted decision a ‘good’ physician would make] is better than the [information gain from a second human opinion] and you really would do best by firing the physician and having the AI make the decisions from then on, trusting for it to call for a second opinion when it is not confident.
(as an example, alpha go zero likely doesn’t benefit from asking another master go player for a ‘second opinion’ when it sees the player it is advising make a bad call)
Keeping stock of and communicating what you haven’t understood is an underrated skill/habit. It’s very annoying to talk to someone and think they’ve understood something, only to realize much later that they haven’t. It also makes conversations much less productive.
It’s probably more of a habit than a skill. There certainly are some contexts where the right thing to do is pretend that you’ve understood everything even though you haven’t. But on net, people do it way too much, and I’m not sure to what extent they’re fooling themselves.