@SaferAI
simeon_c
Karma: 929
You said that naive questions were tolerated so here’s a scenario I can’t figure out why it wouldn’t work.
It seems to me that the fact that an AI fails to predict the truth (because it predicts as humans would) is due to the fact that the AI has built an internal model of how humans understand things and predict based on that understanding. So if we assume that an AI is able to build such an internal model, why wouldn’t we train an AI to predict what a (benevolent) human would say given an amount of information and a capacity to process information ? Doing so, it could develop an understanding of why humans predict badly and then understand that given a huge amount of information and a huge capacity to process information, the true answer is the right one.
A concrete training procedure could use the fact that even among humans, there’s a lot of variance in :
what they know (i.e the amount of information they have)
their capacity to process information (so for instance in the case of the vault, it could be the capacity to infer what happened based on partial information / partial images and based on a certain capacity to process images (no more that x images per second)
So we could use the variance among humans capacity to understand what happened to try to make the AI understanding that benevolent humans predict badly whether the diamond has been stolen only because they lack information or capacity to process information. There’s a lot of fields and questions where the majority of humans are wrong and only a small subset is right, becase there is either a big gap in the information they have or in their ability to process information.
Once we would have trained that AI on humans, we would like to do inference specifying the true capabilities of the AI to ensure that it tells us the truth. The remaining question might be whether such an out-of-sample prediction would work. My guess is that if we also included examples with the human bayes net to add more variance in the training dataset, it would probably reduce the chances that it fails.
Finally, the concrete problem of specifying the information humans have access to is not trivial but I think it’s feasible.
I don’t understand why it wouldn’t work, so I’d be happy to have an answer to better understand the problem!
EDIT : Here’s an update after a discussion about my proposal. After having read Learning The Prior (https://ai-alignment.com/learning-the-prior-48f61b445c04), I think that the key novelty of my proposal, if there’s any, is to give to the model input information about the capacity to reason of the person / entity that predicts an outcome. So here are a few relevant features that we could give it :In the case of the vault, we could give an estimate of the number of images that a human is able to see and understand per seconds.
IQ (or equivalent) when the task involves reasoning
Accuracy on a few benchmark datasets of the AI assistant who’s helping the human to think (human’s Bayes net)
That said, I feel like the main problem is to know whether such a model would do well out-of-distribution (i.e on problems no human is able to resolve). I feel like using the approach I suggested, we should able to use the great variations of capacities among humans and algorithms to increase the chances that our algorithm do well when it’s much better.
In particular, I thought about 2 approaches that could enable us to go far beyond human capacity. We could:
leverage the fact that we are able to verify a solution to some problems that we can’t solve. There might be algorithms very good at solving some problems that no humans can solve, but we can still verify (and thus label) these results. There might be mathematical problems such that we can use algorithms that would do mathematical reasoning that no human would be able to understand and still be able to verify its solution. If we gave to our predictor’s Bayes net the characteristics of the algorithm which is much better than us at solving some problems and the labels of the answers, that would be a way to leverage narrowly superhumans algorithms to generate more variance on the training dataset and decrease the chances of failures in inference. I feel like this approach is very interesting because that would enable to expose the predictor’s Bayes net to algorithms who are both superhuman and have a lot of information (more than any human) which is the situation in which the predictor’s would have to do inferences.
use more information than a more (narrowly) clever entity than us to verify its prediction. In the case of the vault for instance, there could be situations such that no humans would be able to predict whether the vault was stolen or not given partial information, but we could then just use better cameras for ourselves to be able to verify accurately the predictions of a good algorithm. That way, we would also be able to communicate to the predictor’s Bayes net how is it like to be incredibly smart (more than any human) and do good predictions when you are like that. I feel like using that approach of hiding information to extremely accurate narrow algorithms could be a way to still be able to verify their predictions, and expose the predictor’s Bayes net to very smart entities in multiple domains. The problem of this approach though is that it doesn’t give examples of both “maximal information” and “maximal intelligence” to the predictor because it relies on the fact that it always hides a part of the information to the algorithm to enable us to still verify its claims.
Despite that, I don’t know whether asymptotically, I’d expect the algorithm to still be truthful. But it could greatly increase the distribution on which it’s truthful.
Thanks for the answer! The post you mentioned indeed is quite similar!
Technically, the strategies I suggested in my two last paragraphs (Leverage the fact that we’re able to verify solutions to problems we can’t solve + give partial information to an algorithm and use more information to verify) should enable to go far beyond human intelligence / human knowledge using a lot of different narrowly accurate algorithms.
And thus if the predictor has seen many extremely (narrowly) smart algorithms, it would be much more likely to know what is it like to be much smarter than a human on a variety of tasks. It probably still requires some optimism on generalization. So technically the counterexample could be happening on the gap between the capability of the predictor and the capability of the reporter. I feel like one question is : do we expect some narrow algorithms to be much better on very precise tasks than general-purpose algorithms (such as the predictor for instance) ? Because if it were the case, then the generalization that the reporter would have to do from training data (humans + narrowly accurate algorithms capabilities) to inference data (predictor’s capabilities) could be small. We could even have data on the predictor’s capability in the training dataset using the second approach I mentioned (i.e giving partial information to the predictor (e.g one camera in SuperVault) and using more information (i.e more cameras for humans) than him to verify its prediction). We could give some training examples and show the AI how the human fails much more often than the predictor on the exact same sample of examples. That way, we could greatly reduce the gap of generalization which is required.
The advantage of this approach is that the bulk of the additionnal cost of training that the reporter requires is due to the generation of the dataset which is a fixed cost that no user has to repay. So that could slightly decrease the competitivity issues as compared with approaches where we affect the training procedure.
Despite all that, thanks to your comment and the report, I see why the approach I mention might have some intrinsic limitations in its ability to elicit latent knowledge though. The problem is that even if it understands roughly that it has incentives to use most of what it knows when we ask him simulating the prediction of someone with its own characteristics (or 1400 IQ), given that with ELK we look for an global maximum (we want that it uses ALL its knowledge), there’s always an uncertainty on whether it did understand that point or not for extreme intelligence / examples or whether it tries to fit to the training data as much as possible and thus still doesn’t use something it knows.
Are there existing models for which we’re pretty sure we know all their latent knowledge ? For instance small language models or something like that.
New GPT3 Impressive Capabilities—InstructGPT3 [1/2]
Thanks for the feedback! I will think about it and maybe try to do something along those lines!
Cost: You have basically 3 months free with GPT3 Davinci (175B) (under a given limit but which is sufficient for personal use) and then you pay as you go. Even if you use it a lot, you’re likely to pay less than 5$ or 10$ per months.
And if you have some tasks that need a lot of tokens but that are not too hard (e.g hard reading comprehension), Curie (GPT3 6B) is often enough and is much cheaper to use!In few-shot settings (i.e a setting in which you show examples of something so that it reproduces it), Curie is often very good so it’s worth trying it!
Merits: It’s just a matter of cost and inference speed that you need. The biggest models are almost always better so taking the biggest thing that you can afford, both in terms of speed and of cost is a good heuristicUse: It’s very easy to use with the new Instruct models. You just put your prompt and it completes it. The only parameter you have to care about are token uses (which is basically the max size of the completion you want) / temperature (it’s a parameter that affects how “creative” is the answer ; the higher the more creative)
Is GPT3 a Good Rationalist? - InstructGPT3 [2/2]
Thanks for trying! I don’t think that’s much evidence against GPT3 being a good oracle though, bc to me it’s pretty normal that without fine-tuning he’s not able to forecast. He’d need to be extremely sample efficient to be able to do that. Does anyone want to try fine-tuning?
Likewise!
Do you think we could use grokking/current existing generalization phenomena (e.g induction heads) to test your theory? Or do you expect the generalizations that would lead to the sharp left turn to be greater/more significant than those that occurred earlier in the training?
It would be worth paying someone to do this in a centralized way:
Reach out to authors
Convert to LaTeX, edit
Publish
If someone is interested in doing this, reach out to me (campos.simeon @gmail.com)
To get a better sense of people’s standards’ on “cut at the hard core of alignment”, I’d be curious to hear examples of work that has done so.
AGI Timelines in Governance: Different Strategies for Different Timeframes
[Cross-posting my answer]
Thanks for your comment!
That’s an important point that you’re bringing up.My sense is that at the movement level, the consideration you bring up is super important. Indeed, even though I have fairly short timelines, I would like funders to hedge for long timelines (e.g. fund stuff for China AI Safety). Thus I think that big actors should have in mind their full distribution to optimize their resource allocation.
That said, despite that, I have two disagreements:
I feel like at the individual level (i.e. people working in governance for instance, or even organizations), it’s too expensive to optimize over a distribution and thus you should probably optimize with a strategy of “I want to have solved my part of the problem by 20XX”. And for that purpose, identifying the main characteristics of the strategic landscape at that point (which this post is trying to do) is useful.
“the EV gained in the worlds where things move quickly is not worth the expected cost in worlds where they don’t.” I disagree with this statement, even at the movement level. For instance I think that the trade-off of “should we fund this project which is not the ideal one but still quite good?” is one that funders often encounter and I would expect that funders have more risk adverseness than necessary because when you’re not highly time-constrained, it’s probably the best strategy (i.e. in every fields except in AI safety, it’s probably a way better strategy to trade-off a couple of years against better founders).
Finally, I agree that “the best strategies will have more variance” is not a good advice for everyone. The reason I decided to write it rather than not is because I think that the AI governance community tends to have a too high degree of risk adverseness (which is a good feature in their daily job) which penalizes mechanically a decent amount of actions that are way more useful under shorter timelines.
Thanks for your comment!
First, you have to have in mind that when people are talking about “AI” in industry and policymaking, they usually have mostly non-deep learning or vision deep learning techniques in mind simply because they mostly don’t know the ML academic field but they have heard that “AI” was becoming important in industry. So this sentence is little evidence that Russia (or any other country) is trying to build AGI, and I’m at ~60% Putin wasn’t thinking about AGI when he said that.
If anyone who could play any role at all in developing AGI (or uncontrollable AI as I prefer to call it) isn’t trying to develop it by now, I doubt very much that any amount of public communication will change that.
I think that you’re deeply wrong about this. Policymakers and people in industry, at least till ChatGPT had no idea what was going on (e.g at the AI World Summit, 2 months ago very few people even knew about GPT-3). SOTA large language models are not really properly deployed, so nobody cared about them or even knew about them (till ChatGPT at least). The level of investment right now in top training runs probably doesn’t go beyond $200M. The GDP of the US is 20 trillion. Likewise for China. Even a country like France could unilaterally put $50 billion in AGI development and accelerate timelines quite a lot within a couple of years.
Even post ChatGPT, people are very bad at projecting what it means for next years and still have a prior on the fact that human intelligence is very specific and can’t be beaten which prevents them from realizing all the power of this technology.
I really strongly encourage you to go talk to actual people from industry and policy to get a sense of their knowledge on the topic. And I would strongly recommend not publishing your book as long as you haven’t done that. I also hope that a lot of people who have thought about these issues have proofread your book because it’s the kind of thing that could really increase P(doom) substantially.
I think that to make your point, it would be easier to defend the line that “even if more governments got involved, that wouldn’t change much”. I don’t think that’s right because if you gave $10B more to some labs, it’s likely they’d move way faster. But I think that it’s less clear.a common, clear understanding of the dangers
I agree that it would be something good to have. But the question is: is it even possible to have such a thing?
I think that within the scientific community, it’s roughly possible (but then your book/outreach medium must be highly targeted towards that community). Within the general public, I think that it’s ~impossible. Climate change, which is a problem which is much easier to understand and explain is already way too complex for the general public to have a good idea of what are the risks and what are the promising solutions to these risks (e.g. a lot people’s top priorities is to eat organic food, recycle and decrease plastic consumption).
I agree that communicating with the scientific community is good, which is why I said that you should avoid publicizing only among “the general public”. If you really want to publish a book, I’d recommend targeting the scientific community, which is not at all the same public as the general public.
“On the other hand, if most people think that strong AI poses a significant risk to their future and that of their kids, this might change how AI capabilities researchers are seen, and how they see themselves”
I agree with this theory of change and I think that it points a lot more towards “communicate in the ML community” than “communicate towards the general public”. Publishing great AI capabilities is mostly cool for other AI researchers and not that much for the general public. People in San Francisco (where most of the AGI labs are) also don’t care much about the general public and whatever it thinks ; the subculture there and what is considered to be “cool” is really different from what the general public thinks is cool. As a consequence, I think they mostly care about what their peers are thinking about them. So if you want to change the incentives, I’d recommend focusing your efforts on the scientific & the tech community.
I hesitated on decreasing the likelihood on that one based on your consideration to be honest, but I still think that 30% of having strong effects is quite a lot because as you mentioned it requires the intersection of many conditions.
In particular, you don’t mention which intervention you expect from them. If you take the intervention I took as a reference class (“Constrain labs to airgap and box their SOTA models while they train them”), do you think there are things that are as much or more “extreme” than this and that are likely?
What might be misleading in my statement is that it could be understood as “let’s drop national government policy” while it’s more “I think that currently too many people are focused on national government policy and not enough are focused on corporate governance, and it puts us in a fairly bad position for pre-2030 timelines”.
Have you read note 2? If note 2 was made more visible, would you still think that my claims imply a too high certainty?
The points you make are good, especially in the second paragraph. My model is that if scale is all you need, then it’s likely that indeed smaller startups are also worrying. I also think that there could be visible events in the future that would make some of these startups very serious contenders (happy to DM about that).
Having a clear map of who works in corporate governance and who works more towards policy would be very helpful. Is there anything like a “map/post of who does what in AI governance” or anything like that?
Thanks for your comment!
I see your point on fear spreading causing governments to regulate. I basically agree that if it’s what happens, it’s good to be in a position to shape the regulation in a positive way or at least try to. I still think that I’m more optimistic about corporate governance which seems more tractable than policy governance to me.
I think that “There are many talented people who want to work on AI alignment, but are doing something else instead.” is likely to be true. I met at least 2 talented people who tried to get into AI Safety but who weren’t able to because open positions / internships were too scarce. One of them at least tried hard (i.e applied for many positions and couldn’t find one (scarcity), despite the fact that he was one of the top french students in ML). If there was money / positions, I think that there are chances that he would work on AI alignment independently.
Connor Leahy in one of his podcasts mentions something similar aswell.
That’s the impression I have.