Former AI safety research engineer, now PhD student in philosophy of ML at Cambridge. I’m originally from New Zealand but have lived in the UK for 6 years, where I did my undergrad and masters degrees (in Computer Science, Philosophy, and Machine Learning). Blog: thinkingcomplete.blogspot.com
Richard_Ngo
Karma: 4,711
I notice that you didn’t give any arguments, in this post, for why people should try to write on the pareto frontier. Intuitively it seems like something that you should sometimes aim for, and sometimes not. But it seems like you find it a more compelling goal than I do, if you’ve made it a rule for yourself. Mind explaining briefly what the main reasons you think this is a good goal are?
Also, do you intend this to apply to fiction as well as nonfiction?
I discuss the possibility of it going in some other direction when I say “The two most salient options to me”. But the bit of Evan’s post that this contradicts is:
Now, if the model gets to the point where it’s actually just failing because of this, then gradient descent will probably just remove that check—but the trick is never to actually get there.
Interesting report :) One quibble:
For one, our AIs can only use “things like SPI” if we actually formalize the approach
I don’t see why this is the case. If it’s possible for humans to start using things like SPI without a formalisation, why couldn’t AIs too? (I agree it’s more likely that we can get them to do so if we formalise it, though.)
Thoughts on gradient hacking
A short introduction to machine learning
It needs to be higher than what the competitor is willing to pay since they can always counter offer.
Agreed. My previous comment was incorrect; it should have been “it just needs to be more than what it’s sustainable for the competitor to pay”. The underlying problem is still that the cost of disruption may be far higher than the sustainable price to rent each piece of land. So if I gain a billion dollars from Apple’s supply chains being cut, then I can bid up to a billion dollars for each piece of land used in their supply chain, and the only way Apple can stop me is paying a billion dollars of rent per piece of land.
You could also require that a bid to take over a piece of land stands for at least a length of time equal to K * (total value of current property / bid rental price) for some KHow would we determine the total value of the current property?
The thing is, it doesn’t have to be a “ridiculous” price, it just needs to be more than what the competitor
is currently payingEDIT: can sustainably pay. Imagine the cost to Apple of not being able to produce any smartphones for a year because a competitor had snatched a crucial factory—it’s probably orders of magnitude greater than the rental cost of all the land they use. And remember that the attacker only has to remove one piece of the supply chain, whereas the defender has to ensure that they maintain control over every piece of infrastructure needed.Given this disparity, you wouldn’t even need malicious destruction, you could just snatch pieces of land from under them and wait it out.
(Re tit-for-tat, see patent wars—these things do happen!)
Suppose my competitor has just built a factory on land they are renting for $X. I offer $X+1 rent for a day, then demolish their factory, then give the land back to them to rent for $X.
You mention that the exact details will need to be hashed out separately, but I don’t think that they can prevent this problem, because it derives from asymmetry between renting being temporary but demolition being permanent.
For example, you might say that the original owner can just outbid me. But I only need the land for the shortest available unit of time in order to destroy their factory and drive them out of business. Suppose that if this happens, I’ll gain 20% of the profits my competitor would have earned. So they’ll need to pay 20% of their profits for each time period that I can bid on the land.
Insurance doesn’t solve this either, because my competitor will still be paying the same cost in expectation, just to the insurance company.
[Question] SL4 in more legible format
I will also never withhold information that [pronoun] would in hindsight prefer to know
This phrasing seems a little strange. Both because the only thing you can actually commit to is never withholding information which you believe that [pronoun] will end up wishing they’d known; but also because the optimal communication policy has some non-zero rate of false negatives.
I’d actually be fine with a solution where we all agree to stop using the terms “long timelines” and “short timelines” and just use numbers instead. How does that sound?
Yeah, that sounds very reasonable; let’s do that.
Ajeya’s report says median 2050, no?
I just checked, and it’s at 2055 now. Idk what changed.
The most comprehensive (perhaps only comprehensive?) investigation into this says median +35 years, with 35% credence on 50+, and surveys of experts in ML give even higher numbers. I don’t know who you’re counting as a timelines expert, but I haven’t seen any summaries/surveys of their opinions which justifies less than 20 years being the default option.
I’m not saying that this makes your view unreasonable. But presenting your view as consensus without presenting any legible evidence for that consensus is the sort of thing which makes me concerned about information cascades—particularly given the almost-total lack of solid public (or even private, to my knowledge) defences of that view.
I want to push back on calling 20+ years “long timelines”. It’s a linguistic shift which implicitly privileges a pretty radical position, in a way which is likely to give people a mistaken impression of general consensus.
Take a big language model like GPT-3, and then train it via RL on tasks where it gets given a language instruction from a human, and then it gets reward if the human thinks it’s done the task successfully.
A short complaint (which I hope to expand upon at some later point): there are a lot of definitions floating around which refer to outcomes rather than processes. In most cases I think that the corresponding concepts would be much better understood if we worked in terms of process definitions.
Some examples: Legg’s definition of intelligence; Karnofsky’s definition of “transformative AI”; Critch and Krueger’s definition of misalignment (from ARCHES).Sure, these definitions pin down what you’re talking about more clearly—but that comes at the cost of understanding how and why it might come about.
E.g. when we hypothesise that AGI will be built, we know roughly what the key variables are. Whereas transformative AI could refer to all sorts of things, and what counts as transformative could depend on many different political, economic, and societal factors.
Half-formed musing: what’s the relationship between being a nerd and trusting high-level abstractions? In some sense they seem to be the opposite of each other—nerds focus obsessively on a domain until they understand it deeply, not just at high levels of abstraction. But if I were to give a very brief summary of the rationalist community, it might be: nerds who take very high-level abstractions (such as moloch, optimisation power, the future of humanity) very seriously.
Deep limitations? Examining expert disagreement over deep learning
Whether this is a point for the advocate or the skeptic depends on whether advances in RL from human feedback unlock other alignment work more than they unlock other capabilities work. I think there’s room for reasonable disagreement on this question, although I favour the former.
Here are three less cynical ways of thinking about this:
Firstly, other people have different preferences than you do. Most people in Berlin worry much less about the personal consequences of catching covid than most people on LessWrong do. And sure, wearing masks more carefully helps other people too. But so does recycling, or helping old ladies across the street. It’s reasonable for people to make different choices about what types of altruistic behaviour they should prioritise (at least in cases where the size of the benefit is pretty unclear, like it is here).
Secondly, thinking things through for yourself is costly in time and effort. It’s reasonable for most people to spend very little time reasoning from first principles about how to reduce their covid risk, and to instead just listen to what the government/everyone else says, e.g. by following “socially approved rituals against COVID-19”. (Indeed, I think many rationalists would also have benefited from spending far less time thinking about covid.)
Thirdly, policymakers are dealing with many complex tradeoffs under a great deal of uncertainty. Mandating masks is a step which helps reduce the spread of covid. Ensuring that every individual step in implementing the mask mandate is carried out competently is a huge deal of effort, and so even if the highest-level decisionmakers were totally rational, you’d expect to see a bunch of local inefficiencies like security guards wearing suboptimal masks.
These both seem pretty common, so I’m curious about the correlation that you’ve observed. Is it mainly based on people you know personally? In that case I expect the correlation not to hold amongst the wider population.
Also, a big effect which probably doesn’t show up much amongst the people you know: younger people seem more altruistic (or at least signal more altruism) and also seem to have worse epistemics than older people.