Host of the technical ai safety podcast https://technical-ai-safety.libsyn.com/
Streams linear algebra in coq on sundays twitch.tv/quinndougherty92
Quinn
Karma: 323
[linkpost] The Psychological Economy of Inaction by William Gillis
Is logical time at all like cost semantics? also this—cost semantics of a programming language are sort of like “number of beta reductions” from lambda calculus.
New discord server dedicated to multi-multi delegation research
DM me for invite if you’re at all interested in multipolar scenarios, cooperative AI, ARCHES, social applications & governance, computational social choice, heterogeneous takeoff, etc.
(side note I’m also working on figuring out what unipolar worlds and/or homogeneous takeoff worlds imply for MMD research).
I think both sets of bullets (multi-multi (eco?)systems either replicating cooperation-etc-as-we-know-it or making new forms of cooperation etc) are important, I think I’ll call them prosaic cooperation and nonprosaic cooperation, respectively, going forward. When I say “cooperation etc.” I mean cooperation, coordination, competition, negotiation, compromise.
You’ve provided crisp scenarios, so thanks for that!
In some sense “most” of that work will presumably be done by AI systems, but doing the work ourselves may unlock those benefits much earlier.
But if the AI does that work there will be an interpretability problem, an inferential distance. I’m imagining people ask a somewhat single-single aligned AI for solutions to multi-multi problems and the black box returns something inscrutable. Putting ourselves in a position where we can grok what its recommendations are seems aligned with researching it for ourselves so we won’t have to ask the black box in the first place, though this probably only applies to prosaic cooperation.
I wrote out the 2x2 grid you suggested in MS paint
I’m not sure I’m catching how multi-inner is game theory. Except that I think “GT is the mesa- of SCT” is an interesting, reasonable (to me) claim that is sort of blowing my mind as I contemplate it, so far.
Thanks! Trust, compromise, and communication are all items in Dafoe et. al. 2020, if you’re interested in exploring. I agree that primitive forms of these issues are present in multi-single and single-multi, it’s not clear to me whether we should think of solving these primitive forms then solving some sort of extension to multi-multi or if we should think of attacking problems that are unique to multi-multi directly. It’s just not clear to me which of those better reflects the nature of what’s going on.
[Question] What are some claims or opinions about multi-multi delegation you’ve seen in the memeplex that you think deserve scrutiny?
Cliffnotes to Craft of Research parts I, II, and III
No you’re right. I think I’m updating toward thinking there’s a region of nonprosaic short-timelines universes. Overall it still seems like that region is relatively much smaller than prosaic short-timelines and nonprosaic long-timelines, though.
That’s totally fair, but I have a wild guess that the pipeline from google brain to google products is pretty nontrivial to traverse, and not wholly unlike the pipeline from arxiv to product.
I should’ve mentioned in OP but I was lowkey thinking upper bound on “short” would be 10 years.
I think developer ecosystems are incredibly slow (longer than ten years for a new PL to gain penetration, for instance). I guess under a singleton “one company drives TAI on its own” scenario this doesn’t matter, because tooling tailored for a few teams internal to the same company is enough which can move faster than a proper developer ecosystem. But under a CAIS-like scenario there would need to be a mature developer ecosystem, so that there could be competition.
nonprosaic ai will not be on short timelines
I think a property of my theory of change is that academic and commercial speed is a bottleneck. I recently realized that my mass assignment for timelines synchronized with my mass assignment for the prosaic/nonprosaic axis. The basic idea is that let’s say a radical new paper that blows up and supplants the entire optimization literature gets pushed to the arxiv tomorrow, signaling the start of some paradigm that we would call nonprosaic. The lag time for academics and industry to figure out what’s going on, figure out how to build on that result, for developer ecosystems to form, would all compound to take us outside of what we would call “short timelines”.
How flawed is this reasoning?
You might check out Donald Braben’s view, it says “transformative research” (i.e. fundamental results that create new fields and industries) is critical for the survival of civilization. He does not worry that transformative results might end civilization.
Question your argument as your readers will—thoughts on chapter 10 of Craft of Research
Three predictable disagreements are
There are causes in addition to the one you claim
What about these counterexamples?
I don’t define X as you do, to me X means...
There are roughly two kinds of queries readers will have about your argument
intrinsic soundness—“challenging the clarity of a claim, relevance of reasons, or quality of evidence”
extrinsic soundness—“different ways of framing the problem, evidence you’ve overlooked, or what others have written on the topic.” The idea is to anticipate, acknowledge, and respond to both kinds of questions. This is the path to making an argument that readers will trust and accept.
Voicing too many hypothetical objections up front can paralyze you. Instead, what you should do before anything else is focus on what you want to say. Give that some structure, some meat, some life. Then, an important exercise is to imagine readers’ responses to it.
I think cleaving these into two highly separated steps is an interesting idea, doing this with intention may be a valuable exercise next time I’m writing something.
View your argument through the eyes of someone who has a stake in a different outcome, someone who wants you to be wrong.
The authors provide some questions about your problem from a possible reader:
Why do you think there’s a problem at all?
Have you properly defined the problem?
Is your solution practical or conceptual?
Have you stated your claim too strongly?
Why is your practical/conceptual solution better than others?
Then, they provide some questions about your support from a possible reader.
“I want to see a different kind of evidence” i.e. hard numbers over anecdotes / real people over cold numbers
“It isn’t accurate”
“It isn’t precise enough”
“It isn’t current”
“It isn’t representative”
“It isn’t authoritative”
“You need more evidence”
It builds credibility to play defense: to recognize your own argument’s limitations. It builds even more credibility to play offense: to explore alternatives to your argument and bring them into your reasoning. If you can, you might develop those alternatives in your own imagination, but more likely you’d like to find alternatives in your sources.
Often your readers will be likeyour sources’ authors; sometimes they may even include them.
What is the perfect amount of objections to acknowledge? Acknowledging too many can distract readers from the core of your argument, while acknowledging too few is a signal of laziness or even disrespect. You need to narrow your list of alternatives or objections by subjecting them to the following priorities
plausible charges of weaknesses that you can rebut
alternative lines of argument important in your field
alternative conclusions that readers want to be true
alternative evidence that readers know
important counterexamples that youu have to address. What if your argument is flawed? The best thing to do is candidly acknowledge the issue and respond that...
the rest of your argument more than balances the flaw
while the flaw is serious, more research will show a way around it
while the flaw makes it impossible to accept your claim fully, your argument offers important insight into the question and suggests what a better answer would need.
It is wise to build up good faith by acknowledging questions you can’t answer. Concessions are often interpreted as positive signals by the reader.
It is important for your responses to acknowledgments to be subordinate to your main point, or else the reader will miss the forest for the trees.
Remember to make an intentional decision about how much credence to give to an objection or alternative. Weaker ones imply weaker credences, imply less effort in your acknowledgment and response.
Excellence and adequacy
I asked a friend whether I should TA for a codeschool called ${{codeschool}}.
You shouldn’t hang around ${{codeschool}}. People at ${{codeschool}} are not pursuing excellence.
A hidden claim there that I would soak up the pursuit of non-excellence by proximity or osmosis isn’t what’s interesting (though I could see that turning out either way). What’s interesting is the value of non-excellence, which I’ll call adequacy.
${{codeschool}} in this case is effective and impactful at putting butts in seats at companies, and is thereby responsible for some negligible slice of economic growth. It’s students and instructors are plentiful with the virtue of getting things done, do they really need the virtue of high-craftsmanship? The student who reads SICP and TAPL because they’re pursuing mastery over the very nature of computation is strictly less valuable to the economy than the student who reads react tutorials because they’re pursuing some cash.
Obviously, my friend who was telling me this was of the SICP/TAPL type. In software, this is problematic: lisp and type theory will increase your thinking about the nature of computation, but will it increase your thinking about the social problem of steering a team? From an employer’s perspective, it is naive to prefer excellence over adequacy, it is much wiser to saddle the excellent person with the burden of proving that they won’t get bored easily.
Hufflepuffs can go far, and the fuel is adequacy. Enough competence to get it done, any more is egotistical, a sunk cost.
But what if it’s not about industry/markets, what if it’s about the world’s biggest problems? Don’t we want people who are more competent than strictly necessary to be working on them? Maybe, maybe not.
Related: explore/exploit, become great/become useful
For a long time I’ve operated in the excellence mindset: more energy for struggling with textbooks than for exploiting the skills I already have to ship projects and participate in the real world. Thinking it might be good to shift gears and flex my hufflepuff virtues more.
thoughts on chapter 9 of Craft of Research
Getting the easy things right shows respect for your readers and is the best training for dealing with the hard things.
If they don’t believe the evidence, they’ll reject the reasons and, with them, your claim.
We saw previously that claims ought to be supported with reasons, and reasons ought to be based on evidence. Now we will look closer at reasons and evidence.
Reasons must be in a clear, logical order. Atomically, readers need to buy each of your reasons, but compositionally they need to buy your logic. Storyboarding is a useful technique for arranging reasons into a logical order: physical arrangements of index cards, or some DAG-like syntax. Here, you can list evidence you have for each reason or, if you’re speculating, list the kind of evidence you would need.
When storyboarding, you want to read out the top level reasons as a composite entity without looking at the details (evidence), because you want to make sure the high-level logic makes sense.
Readers will not accept a reason until they see it anchored in what they consider to be a bedrock of established fact. … To count as evidence, a statement must report something that readers agree not to question, at least for the purposes of the argument. But if they do question it, what you think is hard factual evidence is for them only a reason, and you have not yet reached that bedrock of evidence on which your argument must rest.
I think there is a contract between you and the reader. You must agree to cite sources that are plausibly truthful, and your reader must agree to accept that these sources are reliable. A diligent and well-meaning reader can always second-guess whether, for instance, the beureau of subject matter statistics is collecting and reporting data correctly, but at a certain point this violates the social contract. If they’re genuinely curious or concerned, it may fall on them to investigate the source, not on you. The bar you need to meet is that your sources are plausibly trustworthy. The book doesn’t talk much about this contract, so there’s little I can say about what “plausible” means.
Sometimes you have to be extra careful to distinguish reasons from evidence, a
(<claim>, <reason>, <evidence>)tuple is subject to regress in the latter two components,(A, B, C)may need to be justified by(B, C, D)and so on. The example given of this regress is if I told you(american higher education must curb escalating tuition costs, because the price of college is becoming an impediment to the american dream, today a majority of students leave college with a crushing debt burden). In the context of this sentence, “a majority of students...” is evidence, but it would be reasonable to ask for more specifics. In principle, any time information is compressed it may be reasonable to ask for more specifics. A new tuple might look like(the price of college is becoming an impediment to the american dream, because today a majority of students leave college with a crushing debt burden, in 2013 nearly 70% of students borrowed money for college with loans averaging $30000...). The third component is still compressing information, but it’s not in the contract between you and the reader for the reader to demand the raw spreadsheet, so this second tuple might be a reasonable stopping point of the regress.If you can imagine readers plausibly asking, not once but many times, how do you know that? What facts make it true?, you have not yet reached what readers want—a bedrock of uncontested evidence.
Sometimes you have to be careful to distinguish evidence from reports of it. Again, because we are necessarily dealing with compressed information, we can’t often point directly to evidence. Even a spreadsheet, rather than summary statistics of it, is a compression of the phenomena in base reality that it tracks.
data you take from a source have invariably been shaped by that source, not to misrepresent them, but to put them in a form that serves that source’s ends. … when you in turn report those data as your own evidence, you cannot avoid manipulating them once again, at least by putting them in a new context.
There is a criteria you want to screen your evidence with respect to.
sufficient
representative
accurate
precise
authoritative
Being honest about the reliability and prospective accuracy of evidence is always a positive signal. Evidence can be either too precise or not precise enough. The women in one or two of Shakespeare’s plays do not represent all his women, they are not representative. Figure out what sorts of authority signals are considered credible in your community, and seek to emulate them.
Claims—thoughts on chapter eight of Craft of Research
Broadly, the two kinds of claims are conceptual and practical.
Conceptual claims ask readers not to ask, but to understand. The flavors of conceptual claim are as follows:
Claims of fact or existence
Claims of definition and classification
Claims of cause and consequence
Claims of evaluation or appraisal
There’s essentially one flavor of practical claim
Claims of action or policy.
If you read between the lines, you might notice that a kind of claim of fact or cause/consequence is that a policy works or doesn’t work to bring about some end. In this case, we see that practical claims deal in ought or should. There is a difference, perhaps subtle perhaps not, between “X brings about Y” and “to get Y we ought to X”.
Readers expect a claim to be specific and significant. You can evaluate your claim along these two axes.
To make a claim specific, you can use precise language and explicit logic. Usually, precision comes at the cost of a higher word count. To gain explicitness, use words like “although” and “because”. Note some fields might differ in norms.
You can think of significance of a claim as the quantity it asks readers to change their mind, or I suppose even behavior.
While we can’t quantify significance, we can roughly estimate it: if readers accept a claim, how many other beliefs must they change?
Avoid arrogance.
As paradoxical as it seems, you make your argument stronger and more credible by modestly acknowledging its limits.
Two ways of avoiding arrogance are acknowledging limiting conditions and using hedges to limit certainty.
Don’t run aground: there are innumerable caveats that you could think of, so it’s important to limit yourself only to the most relevant ones or the ones that readers would most plausibly think of. Limiting certainty with hedging is given by example of Watson and Crick, publishing what would become a high-impact result, “We wish to suggest … in our opinion … we believe … Some … appear”
without the hedges, Crick and Watson would be more concise but more aggressive.
In most fields, readers distrust flatfooted certainty
It is not obvious how to walk the line between hedging too little and hedging too much.
Good arguments—notes on Craft of Research chapter 7
Arguments take place in 5 parts.
Claim: What do you want me to believe?
Reasons: Why should I agree?
Evidence: How do you know? Can you back it up?
Acknowledgment and Response: But what about … ?
Warrant: How does that follow?
This can be modeled as a conversation with readers, where the reader prompts the writer to taking the next step on the list.
Claim ought to be supported with reasons. Reasons ought to be based on evidence. Arguments are recursive: a part of an argument is an acknowledgment of an anticipated response, and another argument addresses that response. Finally, when the distance between a claim and a reason grows large, we draw connections with something called warrants.
The logic of warrants proceeds in generalities and instances. A general circumstance predictably leads to a general consequence, and if you have an instance of the circumstance you can infer an instance of the consequence.
Arguing in real life papers is complexified from the 5 steps, because
Claims should be supported by two or more reasons
A writer can anticipate and address numerous responses. As I mentioned, arguments are recursive, especially in the anticipated response stage, but also each reason and warrant can necessitate a subargument.
You might embrace a claim too early, perhaps even before you have done much research, because you “know” you can prove it. But falling back on that kind of certainty will just keep you from doing your best thinking.
Sources—notes on Craft of Research chapters 5 and 6
Primary, secondary, and tertiary sources
Primary sources provide you with the “raw data” or evidence you will use to develop, test, and ultimately justify your hypothesis or claim. Secondary sources are books, articles, or reports that are based on primary sources and are intended for scholarly or professional audiences. Tertiary sources are books and articles that synthesize and report on secondary sources for general readers, such as textbooks, articles in encyclopedias, and articles in mass-circulation publications.
The distinction between primary and secondary sources comes from 19th century historians, and the idea of tertiary sources came later. The boundaries can be fuzzy, and are certainly dependent on the task at hand.
I want to reason about what these distinctions look like in the alignment community, and whether or not they’re important.
The rest of chapter five is about how to use libraries and information technologies, and evaluating sources for relevance and reliability.
Chapter 6 starts off with the kind of thing you should be looking for while you read
Look for creative agreement
Offer additional support. You can offer new evidence to support a source’s claim.
Confirm unsupported claims. You can prove something that a source only assumes or speculates about.
Apply a claim more widely. You can extend a position.
Look for creative disagreement
Contradictions of kind. A source says something is one kind of thing, but it’s another.
Part-whole contradictions. You can show that a source mistakes how the parts of something are related.
Developmental or historical contradictions. You can show that a source mistakes the origin or development of a topic.
External cause-effect contradictions. You can show that a source mistakes a causal relationship.
Contradictions of perspective. Most contradictions don’t change a conceptual framework, but when you contradict a “standard” view of things, you urge others to think in a new way.
The rest of chapter 6 is a few more notes about what you’re looking for while reading (evidence, reasons), how to take notes, and how to stay organized while doing this.
The alignment community
I think I see the creative agreement modes and the creative disagreement modes floating around in posts. Would it be more helpful if writers decided on one or two of these modes before sitting down to write?
Moreover, what is a primary source in the alignment community? Surely if one is writing about inner alignment, a primary source is the Risks from Learned Optimization paper. But what are Risks’ primary, secondary, tertiary sources? Does it matter?
Now look at Arbital. Arbital started off to be a tertiary source, but articles that seemed more like primary sources started appearing there. I remember distinctively thinking “what’s up with that?” it struck me as awkward for Arbital to change it’s identity like that, but I end up thinking about and citing the articles that seem more like primary sources.
There’s also the problem of stuff in the memeplex not written down is the real “primary” source while the first person who happens to write it down looks like they’re writing a primary source when in fact what they’re doing is really more like writing a secondary or even tertiary source.
capabilities-prone research.
I come to you with a dollar I want to spend on AI. You can allocate
ppennies to go to capabilities and100-ppennies to go to alignment, but only if you know of a project that realizes that allocation. For example, we might think that GAN research setsp = 98(providing 2 cents to alignment) while interpretability research setsp = 10(providing 90 cents to alignment).Is this remotely useful? This is a really rough model (you might think it’s more of a venn diagram and that this model doesn’t provide a way of reasoning about the double counting problem).
a task: rate research areas, even whole agendas, with such value
p. Many people may disagree about my example assignments to GANs and interpretability, or think both of those are too broad.What are some alternatives to the splitting a dollar intuition?
To say something is capabilities-prone is less to say a dollar has been cleanly split, and more to say that there are some dynamics that sort of tend toward or get pushed toward different directions. Perhaps I want some sort of fluid metaphor instead.