A Ray

Karma: 961

A Ray 4 Mar 2023 21:20 UTC
7 points
0
on: Taboo “compute overhang”
I like pointing out this confusion. Here’s a grab-bag of some of the things I use it for, to try to pull them apart:
- actors/institutions far away from the compute frontier produce breakthroughs in AI/AGI tech (juxtaposing “only the top 100 labs” vs “a couple hackers in a garage”)
- once a sufficient AI/AGI capability is reached, that it will be quickly optimized to use much less compute
- amount of “optimization pressure” (in terms of research effort) pursuing AI/AGI tech, and the likelihood that they missed low-hanging fruit
- how far AI/AGI research/products are away from the highest-value-marginal-use of compute, and how changes making AI/AGI the biggest marginal profit of compute would change things
- the legibility of AI/AGI research progress (e.g. in a high-overhang world, small/illegible labs can make lots of progress)
- the likelihood of compute-control interventions to change the trajectory of AI/AGI research
- the asymmetry between compute-to-build (~=training) and compute-to-run (~=inference) of advanced AI/AGI technology
probably also others im forgetting

A Ray 25 Feb 2023 18:06 UTC
4 points
on: Alex Ray’s Shortform
Comparing AI Safety-Capabilities Dilemmas to Jervis’ Cooperation Under the Security Dilemma
I’ve been skimming some things about the Security Dilemma (specifically Offense-Defense Theory) while looking for analogies for strategic dilemmas in the AI landscape.
I want to describe a simple comparison here, lightly held (and only lightly studied)
- “AI Capabilities”—roughly, the ability to use AI systems to take (strategically) powerful actions—as “Offense”
- “AI Safety”—roughly, that AI systems under control and use do not present a catastrophic/existential threat—as “Defense”
Now re-creating the two values (each) of the two variables of Offense-Defense Theory:
- “Capabilities Advantaged”—Organizations who disproportionately invest in capabilities get strategic advantage
- “Safety Advantaged”—Organizations who disproportionately invest in safety get strategic advantage.^[1]
- “Capabilities Posture Not Distinguishable from Safety Posture”—from the outside, you can’t credibly determine how an organization is investing Capabilities-vs-Safety. (This is the case if very similar research methods/experiments are used to study both, or that they are not technologically separable. My whole interest in this is in cultivating and clarifying this one point. This exists in the security dilemma as well inasmuch as defensive missile interceptors can be used to offensively strike ranged targets, etc.)
- “Capabilities Posture Distinguishable from Safety Posture”—it’s credibly visible from the outside whether an organization is disproportionately invested in Safety
Finally, we can sketch out the Four Worlds of Offense-Defense Theory
1. Capabilities Advantaged / Not-Distinguishable : Doubly dangerous (Jervis’ words, but I strongly agree here). I also think this is the world we are in.
2. Safety Advantaged / Not-Distinguishable : There exists a strategic dilemma, but it’s possible that non-conflict equilibrium can be reached.
3. Capabilities Advantaged / Distinguishable : No security dilemma, and other strategic responses can be made to obviously-capabilities-pursuing organizations. We can have warnings here which trigger or allow negotiations or other forms of strategic resolution.
4. Safety Advantaged / Distinguishable : Doubly stable (Jervis’ words, also strongly agree).
From the paper linked in the title:
My Overall Take
- If this analogy fits, I think we’re in World 1.
- The analogy feels weak here, there’s a bunch of mis-fits from the original (very deep) theory
- I think this is neat and interesting but not really a valuable strategic insight
- I hope this is food for thought
@Cullen_OKeefe
1. ^
  This one is weird, and hard for me to make convincing stories about the real version of this, as opposed to seeming to have a Safety posture—things like “recruiting”, “public support”, “partnerships”, etc all can come from merely seeming to adopt the Safety Posture. (Though, this is actually a feature of the security dilemma and offense-defense theory, too)

A Ray 18 Feb 2023 1:27 UTC
15 points
0
on: Recommendation: Bug Bounties and Responsible Disclosure for Advanced ML Systems
I largely agree with the above, but commenting with my own version.
What I think companies with AI services should do:
Can be done in under a week:
1. Have a monitored communication channel for people, primarily security researchers, to responsibly disclose potential issues (“Potential Flaw”)
  1. Creating an email (ml-disclosures@) which forwards to an appropriate team
  2. Submissions are promptly responded to with a positive receipt (“Vendor Receipt”)
2. Have clear guidance (even just a blog post or similar) about what constitutes an issue worth reporting (“Vulnerability”)
  1. Even just a simple paragraph giving a high level overview could be evolved with time
3. Have a internal procedure/playbook for triaging and responding to potential issues. Here’s some options I think you could have, with heuristics to pick:
  1. Non-Vulnerability: reply that the reported behavior is safe to publicly disclose
  2. Investigation: reply that more investigation is needed, and give a timeline for an updated response
  3. Vulnerability: reply that the reported behavior is a vulnerability, and give a timeline for resolution and release of a fix, as well as updates as those timelines change
Longer term
1. Have a public bounty program to incentivize responsible disclosure
  1. There even exist products to help companies deploy these sorts of programs
2. Coordinate with other organizations with similar systems
  1. Confidential communication channels to share potentially critical severity vulnerabilities.
  2. Possibly eventually a central coordinating organization (analogous to MITRE) that de-duplicates work handling broad vulnerabilities—I think this will be more important when there are many more service providers than today.
  3. Coordinate as a field to develop shared understanding of what does and does not constitute a vulnerability. As these systems are still nascent, lots of work needs to be done to define this, and this work is better done with a broad set of perspectives and inputs.
3. Cultivate positive relationships with responsible researchers and responsible research behaviors
  1. Don’t stifle this kind of research by just outright banning it—which is how you get the only researchers breaking your system are black hats
  2. Create programs and procedures specifically to encourage and enable this kind of research in ways that are mutually beneficial
  3. Reward and publicly credit researchers that do good work
References:
- Responsible Vulnerability Disclosure Process (draft IETF report)
- Guidelines for Security Vulnerability Reporting and Response (Organization for Internet Safety)
- Probably better more updated ones but I’ve been out of this field for a long time
What links here?
- Ideas for AI labs: Reading list by Zach Stein-Perlman (EA Forum; 24 Apr 2023 19:00 UTC; 28 points)
- Ideas for AI labs: Reading list by Zach Stein-Perlman (24 Apr 2023 19:00 UTC; 11 points)

A Ray 15 Jan 2023 5:39 UTC
13 points
on: Coase’s “Nature of the Firm” on Polyamory
Weakly positive on this one overall. I like Coase’s theory of the firm, and like making analogies with it to other things. I don’t think this application felt like it quite worked to me, and trying to write up why.
One thing is I think feels off is an incomplete understanding of the Coase paper. What I think the article gets correct: Coase looks at the difference between markets (economists preferred efficient mechanism) and firms / corporation, and observes that transaction costs (for people these would be contracts, but in general all transaction costs are included) are avoided in firms. What I think it misses: A primary question explored in the paper is what factors govern the size of firms, and this leads to a mechanistic model that the transaction costs internal to the firm increase with the size of the firm until they reach a limit of the same as transaction costs for the open market (and thus the expected maximum efficient size of a non-monopoly firm). A second, smaller, missed point I think is that the price mechanism works for transactions outside the firm, but does not for transactions inside the firm.
Given these, I think the metaphor presented here seems incomplete. It’s drawing connections to some of the parts of the paper, but not all of the central parts, and not enough to connect to the central question of size.
I’m confused exactly what parts of the metaphor map to the paper’s concept of market and firm. Is monogamy the market, since it doesn’t require high-order coordination? Is polyamory the market since everyone can be a free-ish actor in an unbundled way? Is monogamy the firm since it’s not using price-like mechanisms to negotiate individual unbundled goods? Is polyamory the firm since its subject to the transaction cost scaling limit of size?
I do think that it seems to use the ‘transaction costs matter’ pretty solidly from the paper, so there is that bit.
I don’t really have much I can say about the polyamory bits outside of the economics bits.
What links here?
- Raemon's comment on The 2021 Review Phase by Raemon (20 Jan 2023 20:54 UTC; 8 points)
- Raemon's comment on Highlights and Prizes from the 2021 Review Phase by Raemon (27 Jan 2023 23:35 UTC; 2 points)

A Ray 15 Jan 2023 0:00 UTC
4 points
on: Rationalism before the Sequences
This post was personally meaningful to me, and I’ll try to cover that in my review while still analyzing it in the context of lesswrong articles.
I don’t have much to add about the ‘history of rationality’ or the description of interactions of specific people.
Most of my value from this post wasn’t directly from the content, but how the content connected to things outside of rationality and lesswrong. So, basically, i loved the citations.
Lesswrong is very dense in self-links and self-citations, and to a lesser degree does still have a good number of links to other websites.
However it has a dearth of connections to things that aren’t blog posts—books, essays from before the internet, etc. Especially older writings.
I found this posts citation section to be a treasure trove of things I might not have found otherwise.
I have picked up and skimmed/started at least a dozen of the books on the list.
I still come back to this list sometimes when I’m looking for older books to read.
I really want more things like this on lesswrong.
What links here?

A Ray 14 Jan 2023 23:50 UTC
15 points
on: Cryonics signup guide #1: Overview
I read this sequence and then went through the whole thing. Without this sequence I’d probably still be procrastinating / putting it off. I think everything else I could write in review is less important than how directly this impacted me.
Still, a review: (of the whole sequence, not just this post)
First off, it signposts well what it is and who it’s for. I really appreciate when posts do that, and this clearly gives the top level focus and whats in/out.
This sequence is “How to do a thing”—a pretty big thing, with a lot of steps and branches, but a single thing with a clear goal.
The post is addressing a real need in the community (and it was a personal need for me as well) -- which I think are the best kinds of “how to do a thing” posts.
It was detailed and informative while still keeping the individual points brief and organized.
It specifically calls out decision points and options, how much they matter, what the choices are, and information relevant to choosing. This is a huge energy-saver in terms of actually getting people to do this process.
When I went through it, it was accurate, and I ran into the decision points and choices as expected.
Extra appreciation for the first post which also includes a concrete call to action for a smaller/achievable-right-now thing for people to do (sign a declaration of intent to be cryopreserved). Which I did! I also think that a “thing you can do right now” is a great feature to have in “how to do a thing” posts.
I’m in the USA, so I don’t have much evaluation or feedback on how valuable this is to non-USA folks. I really do appreciate that a bunch of extra information was added for non-USA cases, and it’s organized such that it’s easy to read/skim past if not needed.
I know that this caused me personally to sign up for cryonics, and I hope others as well. Inasmuch as the authors goal was for more people in our community to sign up for cryonics—I think that’s a great goal and I think they succeeded.
What links here?
- Raemon's comment on The 2021 Review Phase by Raemon (20 Jan 2023 20:54 UTC; 8 points)
- Raemon's comment on Highlights and Prizes from the 2021 Review Phase by Raemon (27 Jan 2023 23:35 UTC; 2 points)

A Ray 14 Jan 2023 23:22 UTC
12 points
0
on: Politics is way too meta
Summary
- public discourse of politics is too focused on meta and not enough focused on object level
- the downsides are primarily in insufficient exploration of possibility space
Definitions
- “politics” is topics related to government, especially candidates for elected positions, and policy proposals
- opposite of meta is object level—specific policies, or specific impacts of specific actions, etc
- “meta” is focused on intangibles that are an abstraction away from some object-level feature, X, e.g. someones beliefs about X, or incentives around X, or media coverage vibes about X
- Currently public discourse of politics is too much meta and not enough object level
Key ideas
- self-censorship based on others’ predicted models of self-censorship stifles thought
- worrying about meta-issues around a policy proposal can stifle the ability to analyze the object-level implications
- public discourse seems to be a lot of confabulating supporting ideas for the pre-concluded position
- downsides of too much meta are like distraction in terms of attention and cognition
- author has changed political beliefs based on repeated object-level examples why their beliefs were wrong
Review Summary
- overall i agree with the author’s ideas and vibe
- the piece feels like its expressing frustration / exasperation
- i think it’s incomplete, or a half-step, instead of what i’d consider a full article
- I give examples of things that would make it feel full-step to me
Review
Overall I think I agree with the observations and concepts presented, as well as the frustration/exasperation sense at the way it seems we’re collectively doing it wrong.
However I think this piece feels incomplete to me in a number of ways, and I’ll try to point it out by giving examples of things that would make it feel complete to me.
One thing that would make it feel complete to me is a better organized set of definitions/taxonomy around the key ideas. I think ‘politics’ can be split into object level things around politicians vs policies. I think even the ‘object level’ can be split into things like actions (vote for person X or not) vs modeling (what is predicted impact on Y). When I try to do this kind of detail-generation, I think I find that my desire for object-level is actually a desire for specific kinds of object level focus (and not object-level in the generic).
Another way of making things more precise is to try to make some kind of measure or metric out of the meta<->object dimension. Questions like ‘how would it be measured’ or even ‘what are the units of measurement’ would be great for building intuitions and models around this. Relatedly—describing what ‘critical meta-ness’ or the correct balance point would also be useful here. Assuming we had a way to decrease-meta/increase-object, how would we know when to stop? I think these are the sorts of things that would make a more complete meta-object theory.
A gears-level model of what’s going on would also make this feel complete to me. Here’s me trying to come up with one on the spot:
Discourse around politics is a pretty charged and emotionally difficult thing for most people, in ways that can subvert our normally-functioning mechanisms of updating our beliefs. When we encounter object-level evidence that contradicts our beliefs, we feel a negative emotion/experience (in a quick/flash). One palliative to this feeling is to “go meta”—hop up the ladder of abstraction to a place where the belief is not in danger. We habituate ourselves to it by seeing others do similarly and imitation is enough to propagate this without anyone doing it intentionally. This model implicitly makes predictions about how to make spaces/contexts where more object level discussions happen (less negative experience, more emotional safety) as well as what kinds of internal changes would facilitate more object level discussions (train people to notice these fast emotional reactions and their corresponding mental moves).
Another thing that would make this article feel ‘complete’ to me would be to compare the ‘politics’ domain to other domains familiar to folks here on lesswrong (candidates are: effective altruism, AI alignment, rationality, etc). Is the other domain too meta in the way politics is? Is it too object level? It seems like the downsides (insufficient exploration, self-censorship, distraction) could apply to a much bigger domain of thought.
What links here?
- Raemon's comment on The 2021 Review Phase by Raemon (20 Jan 2023 20:54 UTC; 8 points)
- Raemon's comment on Highlights and Prizes from the 2021 Review Phase by Raemon (27 Jan 2023 23:35 UTC; 2 points)

A Ray 21 Dec 2022 7:38 UTC
6 points
0
on: Performing an SVD on a time-series matrix of gradient updates on an MNIST network produces 92.5 singular values
Thoughts, mostly on an alternative set of next experiments:
I find interpolations of effects to be a more intuitive way to study treatment effects, especially if I can modulate the treatment down to zero in a way that smoothly and predictably approaches the null case. It’s not exactly clear to me what the “nothing going on case is”, but here’s some possible experiments to interpolate between it and your treatment case.
- alpha interpolation noise: A * noise + (A − 1) * MNIST, where the null case is the all-noise case. Worth probably trying out a bunch of different noise models since mnist doesn’t really look at all gaussian.
- shuffle noise: Also worth looking at pixel/row/column shuffles, within an example or across dataset, as a way of preserving some per-pixel statistics while still reducing the structure of the dataset to basically noise. Here the null case is again that “fully noised” data should be the “nothing interesting” case, but we don’t have to do work to keep per-pixel-statistics constant
- data class interpolation: I think the simplest version of this is dropping numbers, and maybe just looking at structurally similar numbers (e.g. 1,7 vs 1,7,9). This doesn’t smoothly interpolate, but still having a ton of different comparisons with different subsets of the numbers. The null case here is that more digits adds more structure
- data size interpolation: downscaling the images, with or without noise, should reduce the structure such that the small / less data an example has, the closer it resembles the null case
- suboptimal initializations: neural networks are pretty hard to train (and can often degenerate) if initialized incorrectly. I think as you move away from optimal initialization (both of model parameters and optimizer parameters), it should approach the null / nothing interesting case.
- model dimensionality reduction: similar to intrinsic dimensionality, you can artificially reduce the (linear) degrees of freedom of the model without significant decrease to its expressivity by projecting into a smaller subspace. I think you’d need to get clever about this, because i think the naive version would just be linear projection before your linear operation (and then basically a no-op).
I mostly say all this because I think it’s hard to evaluate “something is up” (predictions dont match empirical results) in ML that look like single experiments or A-B tests. It’s too easy (IMO) to get bugs/etc. Smoothly interpolating effects, with one side as a well established null case / prior case, and another different case; which vary smoothly with whatever treatment, is IMO strong evidence that “something is up”.
Hope there’s something in those that’s interesting and/or useful. If you haven’t already, I strongly recommend checking out the intrinsic dimensionality paper—you might get some mileage by swapping your cutoff point for their phase change measurement point.

A Ray 9 Aug 2022 6:56 UTC
LW: 1 AF: 1
1
AF
in reply to: Quintin Pope’s comment on: Steganography in Chain of Thought Reasoning
I think at this point these feel like empirical questions, which I think would be much more clearly answered by demonstrations or experiments.
Trying to encode an additional penalty on changing non-semantic information is an interesting idea.
However I think you’re missing that you don’t have the ability to directly compare to a reference LM in cases where you’re training to improve on some performance benchmark. During training the model will change its predictions on everything to some degree (both semantic and nonsemantic content).
So your proposed detection will always show some amount of disagreement between the prior and the trained model on weird grammatical patterns as well as conceptual tokens. The question is: “is the difference merely due to the changes to improve performance, or is it also transmitting hidden information”

A Ray 9 Aug 2022 3:57 UTC
LW: 2 AF: 2
1
AF
in reply to: Quintin Pope’s comment on: Steganography in Chain of Thought Reasoning
If what you’re saying is “any change to the distribution will change KL” — I think that’s just correct.

This also applies to changes during training where the model is learning to perform better on the objective task.

So we are expecting some amount of KL divergence already.

My claims are:
1. The cheapest place to hide information (due to KL) are places where the model already has high entropy (ie it is uncertain between many possible outputs)
2. optimization pressure will try to push this extra information into the cheapest places to hide
3. the increase in KL won’t be clearly distinguishable from the increase due to increased performance on the task

A Ray 9 Aug 2022 3:50 UTC
1 point
in reply to: Ivan Vendrov’s comment on: Alex Ray’s Shortform
I think that’s right but also the moral maze will be mediating the information and decision making support that’s available to the leadership, so they’re not totally immune from the influences

A Ray 8 Aug 2022 17:09 UTC
LW: 2 AF: 2
0
AF
in reply to: Quintin Pope’s comment on: Steganography in Chain of Thought Reasoning
I think I understand what you’re saying, but I want to double check and try laying it out explicitly.
- The “null space” is actually the null space of the prior model, which is human-like, but not exactly lined up to human perception of language
- Because of this, the information is “hidden” w.r.t. human perceptions, it’s “hidden” w.r.t. the language model prior
I think I agree with both of these points, but here’s my thinking for why I still expect to see this phenomena (and why the article above was simplified to just be “human”)
- Empirically drawing many samples results in lots of “contains similar high level contents according to a human (me)”
  - It might be worth studying this explicitly, such as setting up a bunch of chain of thought reasoning like scenarios, and sampling a lot of generated reasoning, then asking humans to rate how similar the generations are
  - My prediction is that people will think that they are largely similar
  - We could also use this same method to study human-measured relevance of other non-trivial sampling methods, like beam search, etc
- “Hiding” w.r.t. language model prior is sufficient, in the limit of good language models
  - One of the things I think my reasoning relies on is that we will continue to produce better language models, which means the prior over time more and more closely resembles human priors
  - This suggests that one way to avoid this phenomena is to only use smaller / less capable models—or at least they will do a worse job at hiding information. I currently expect that many research labs will continue to do chain of thought reasoning with state of the art models, though.
- There’s a lot of room in the model distribution
  - Chain of thought reasoning is not currently, nor do I expect it will ever be, overconstrained.
  - Instead, it can be seen as a mechanism for dramatically increasing the space of possible outputs, both those seen during training, and those encountered at test time
- Human preference effects
  - Research Campaign Effects: Human researchers will want human-like outputs, and thus will be expected to tweak/retrain/etc in order to get human-like reasoning in the chain of thought. This introduces a separate optimization pressure from the KL penalty towards reasoning being human-like
  - Human preference models: I think an easy-to-predict future direction here is training reward models to match subjective human preferences and then optimizing with them or sampling with them (e.g. best of N) -- these preference models could also introduce additional optimization pressure towards human-like
I think the most important point in all of this is that I expect the model prior to be sufficiently close to the human prior—and I’d be interested in learning more about evidence or research to the contrary.

A Ray 8 Aug 2022 16:46 UTC
LW: 4 AF: 3
0
AF
in reply to: Vaniver’s comment on: Steganography in Chain of Thought Reasoning
I think this is worth considering even in cases of things like recursive prompting for a couple of reasons that I haven’t organized:
- In general additional compute time lets you improve your answer, so by default I’d expect answers to any significant question to take multiple steps
  - I agree for trivial questions it might not do any of this. My expectation is that AI deployments impacts will mostly be on things like non-trivial questions or tasks
- So far current research hasn’t done a great deal of regularizing or penalizing models for using a lot of thinking
  - Mostly they’ve optimized for “maximize the chance of correct answer”
  - This could change in the future, especially in cases where the model is expected to be deployed / otherwise sensitive to runtime costs
- Steganography is rewarded over openness if the openness is somehow far from the prior for a given piece of information it wants to transmit
  - Caveat this point is more speculative than the above document
  - One of my points is that due to null space + optimization pressure, it’s very easy and convenient for gradient descent to put information in these hidden dimensions
  - I think one of the things this can do well is handle concepts that don’t yet exist—and so reasoning moves like “invent a new term” or “give an illustrative example” or “invent a thought experiment” are lower likelihood under the prior
  - In these cases, it can be more likely for steganography to transmit certain information, if it is less expensive in terms of KL penalties
  - I do agree that for straightforward reasoning on straightforward evidence, where the reasoning is human-like and close to the prior, we can expect it to mostly reason out loud in a way that is “open”
- Myopia—though less important than it was earlier in RFLO research—seemed like a good alignment criterion to aim for, and still does today somewhat
  - The original starting point for this direction of thinking was about Myopia / Non-Myopia in chain of thought reasoning systems
  - The sorts of guarantees I was hoping to get for chain of thought systems included myopia guarantees
  - Steganography came up as an example of how it could break myopia (other more complicated examples include side effects and situational awareness)
  - The concepts in this document came up by asking “how might steganography arise naturally given current research techniques and directions”

A Ray 8 Aug 2022 16:35 UTC
LW: 3 AF: 3
0
AF
in reply to: Hoagy’s comment on: Steganography in Chain of Thought Reasoning
Putting what I think is the most important part of my reply first: I think research into mitigations is premature and instead demonstrating/measuring the phenomena should take priority.
However given that, I think I agree that these are all possible mitigations to the phenomena, in particular (rephrasing your points):
- Reducing the channel capacity, and thus the amount of information that can be transmitted
  - Constraining the outputs, and thus reducing the degrees of freedom of the null space
  - Adding noise (via dropout or otherwise) also reduces the channel capacity, and probably also has some desirable side effects (like increasing the diversity or creativity of reasoning)
- Introducing “barriers” past which optimization for hidden information becomes difficult (in your example, you use different models; I also think “stop gradients” in end-to-end training has similar effects, if weaker)
- If you can detect/localize this phenomena, you might be able to use interpretability like techniques to limit or constrain it
  - I’m less certain about this, but possible future interpretability research will unlock this ability.
Also I think an important point here is that we can expect to see it with fairly simple techniques—merely including sampled/generated data in the training set is sufficient, as opposed to it requiring a complicated reinforcement learning algorithm like MuZero.
Did you publish your proposal? I’d be interested in reading it.

A Ray 8 Aug 2022 5:55 UTC
LW: 6 AF: 5
AF
in reply to: Ivan Vendrov’s comment on: Alex Ray’s Shortform
Agree that founders are a bit of an exception. Actually that’s a bit in the longer version of this when I talk about it in person.

Basically: “The only people who at the very top of large tech companies are either founders or those who were able to climb to the tops of moral mazes”.
So my strategic corollary to this is that it’s probably weakly better for AI alignment for founders to be in charge of companies longer, and to get replaced less often.
In the case of facebook, even in the face of all of their history of actions, I think on the margin I’d prefer the founder to the median replacement to be leading the company.
(Edit: I don’t think founders remaining at the head of a company isn’t evidence that the company isn’t a moral maze. Also I’m not certain I agree that facebook’s pivot couldn’t have been done by a moral maze.)

A Ray 6 Aug 2022 5:42 UTC
3 points
2
in reply to: green_leaf’s comment on: My advice on finding your own path
Thanks, fixed the link in the article. Should have pointed here: https://www.lesswrong.com/posts/dhj9dhiwhq3DX6W8z/hero-licensing

A Ray 5 Aug 2022 19:08 UTC
LW: 12 AF: 7
AF
on: Alex Ray’s Shortform
I think there should be a norm about adding the big-bench canary string to any document describing AI evaluations in detail, where you wouldn’t want it to be inside that AI’s training data.
Maybe in the future we’ll have a better tag for “dont train on me”, but for now the big bench canary string is the best we have.
This is in addition to things like “maybe don’t post it to the public internet” or “maybe don’t link to it from public posts” or other ways of ensuring it doesn’t end up in training corpora.
I think this is a situation for defense-in-depth.

A Ray 5 Jul 2022 18:55 UTC
1 point
on: Alex Ray’s Shortform
More Ideas or More Consensus?
I think one aspect you can examine about a scientific field is it’s “spread”-ness of ideas and resources.
High energy particle physics is an interesting extrema here—there’s broad agreement in the field about building higher energy accelerators, and this means there can be lots of consensus about supporting a shared collaborative high energy accelerator.
I think a feature of mature scientific fields that “more consensus” can unlock more progress. Perhaps if there had been more consensus, the otherwise ill-fated superconducting super collider would have worked out. (I don’t know if other extenuating circumstances would still prevent it.)
I think a feature of less mature scientific fields that “more ideas” (and less consensus) would unlock more progress. In this case, we’re more limited about generating and validating new good ideas. One way this looks is that there’s not a lot of confidence with what to do with large sums of research funding, and instead we think our best bet is making lots of small bets.
My field (AI alignment) is a less mature scientific field in this way, I think. We don’t have a “grand plan” for alignment, which we just need to get funding. Instead we have a fractal of philanthropic organizations empowering individual grantmakers to try to get small and early ideas off the ground with small research grants.
A couple thoughts, if this model does indeed fit:
There’s a lot more we could do to orienting as a field with “the most important problem is increasing the rate of coming up with good research ideas”. In addition to being willing to fund lots of small and early stage research, I think we could factorize and interrogate the skills and mindsets needed to do this kind of work. It’s possible that this is one of the most important meta-skills we need to improve as a field.
I also think this could be more of a priority when “field building”. When recruiting or trying to raise awareness of the field, it would be good to consider more focus or priority on places where we expect to find people who are likely to be good generators of new ideas. I think one of the ways this looks is to focus on more diverse and underrepresented groups.
Finally, at some point it seems like we’ll transition to “more mature” as a field, and it’s good to spend some time thinking about what would help that go better. Understanding the history of other fields making this transition, and trying to prepare for predicted problems/issues would be good here.

A Ray 29 Jun 2022 17:27 UTC
LW: 16 AF: 8
AF
on: Alex Ray’s Shortform
AGI will probably be deployed by a Moral Maze
Moral Mazes is my favorite management book ever, because instead of “how to be a good manager” it’s about “empirical observations of large-scale organizational dynamics involving management”.
I wish someone would write an updated version—a lot has changed (though a lot has stayed the same) since the research for the book was done in the early 1980s.
My take (and the author’s take) is that any company of nontrivial size begins to take on the characteristics of a moral maze. It seems to be a pretty good null hypothesis—any company saying “we aren’t/won’t become a moral maze” has a pretty huge evidential burden to cross.
I keep this point in mind when thinking about strategy around when it comes time to make deployment decisions about AGI, and deploy AGI. These decisions are going to be made within the context of a moral maze.
To me, this means that some strategies (“everyone in the company has a thorough and complete understanding of AGI risks”) will almost certainly fail. I think the only strategies that work well inside of moral mazes will work at all.
To sum up my takes here:
- basically every company eventually becomes a moral maze
- AGI deployment decisions will be made in the context of a moral maze
- understanding moral maze dynamics is important to AGI deployment strategy

A Ray 25 Jun 2022 21:58 UTC
8 points
on: AI Training Should Allow Opt-Out
(Caveat: I ran the first big code scrape and worked on the code generating models which later became codex.)
My one line response: I think opt-out is obviously useful and good and should happen.
AFAIK there are various orgs/bodies working on this but kinda blanking what/where. (In particular there’s a FOSS mailing list that’s been discussing how ML training relates to FOSS license rights that seems relevant)
Opt-out strings exist today, in an insufficient form. The most well known and well respected one is probably the big-bench canary string: https://github.com/google/BIG-bench/blob/main/docs/doc.md—but this is just intended to protect data used for evaluating text models.
Mimicking the structure to comment on each point:
Simplicity
I think simplicity is points in favor of cheapness, but not points (directly) in favor of why something “should be done”. I see this as “technical cost to implement are low”, and agree.
Competitiveness
I think this also is points in favor of cheapness, but again not why it “should be done”. I see this as “expected reduction in ML perf is small”, and agree.
Ethics
I think this makes the point that we currently don’t have settled understanding on what the ethics of various options are here. People being upset at the state of things is pretty strong evidence that it’s not settled, but seems to be less strong evidence that it’s unethical. I can’t tell the point you’re trying to make here is that “we should figure out the ethics of opt-out” (which I agree with) or that “opt-out is ethically required” (which I don’t think you’ve sufficiently supported here for me to agree with).
Risk
I see this as making the point “opt-out would (very minorly) reduce AI risk”. I think this is both well supported by the arguments and technically valid. I’m personally skeptical about the amount of protection this gets us, and am mostly optimistic in applying it to non-software domains (e.g. nanotech, gain of function, virology, etc).
A personal technical prediction I can add: I think that in the software domain, it will be inexpensive for a capable system to compose any non-allowed concepts out of allowed concepts. I think this is non-obvious to traditional ML experts. In traditional ML, removing a domain from the dataset usually robustly removes it from the model—but things like the large-scale generative models mentioned in the top of the post have generalized very well across domains. (They’re still not very capable in-domain, but are similarly not-capable in domains that didn’t exist in training.) I think this “optimism about generalization” is the root of a bunch of my skepticism about domain-restriction/data-censoring as a method of restricting model capabilities.
Precedent
I think the robots.txt example is great and basically this is the one that is most directly applicable. (Other precedents exist but IMO none are as good.) I totally agree with this precedent.
Separately, there’s a lot of precedent for people circumventing or ignoring these—and I think it’s important to look at those precedents, too!
Risk Compensation
This is an interesting point. I personally don’t weigh this highly, and feel like a lot of my intuition here is attached to gut-level stuff.
As far as I know, the literature on risk compensation is almost entirely about things that are direct personal risk to someone. I don’t know of any cases of risk compensation where the risk was indirect or otherwise largely separated from the person. (At some point of indirectness this seems to reduce more to a “principal-agent problem” than a risk-compensation problem)
What’s Missing
I think it’s easy to focus on the technical implementation costs and less on the “what happens next” costs. Figuring out the legal status of this opt-out (and possibly pushing for legislation to change this) is difficult and expensive. Figuring out standards for evaluation will be similarly hard, especially as the tech itself changes rapidly.
Personal Conclusion
I think opt-out is obviously good and useful and should be done. It think its a pretty clear positive direction for ML/AI policy and regulatory development—and also I’m optimistic that this is the sort of thing that will happen largely on its own (i.e. no drastic action is required).