AI Safety Contractor and Masters student in Physics at the University of Queensland.
I am interested in Quantum Computing, physical AI Safety guarantees and alignment techniques that will work beyond contemporary models.
Views expressed here are my own.
What is the sample size?
I think any theory of impact that relied on model architectures remaining static was likely doomed. I hope that in studying models without a residual stream, interpretability researchers find techniques that will generalise further.
“invent and reason in an alien language”
That step isn’t necessary for the argument. The point is just that you’re encouraging extra compression rather than incentivising the model to invent a language from scratch.
“On the logic that reinforcement learning can accomplish more than was understood.”
Could you explain this? It seemed obvious that RL results in difficult to parse CoT.
This seems to be long list of several augments and I’m not in a position to address every single one.
“On the Impossibility of Supersized Machines”
I am confused because the paper with this name I assume you are referring to is an April Fools prank.
The obvious difference was that it was acceptable for mistakes to be made. When there was an explosion in Alfred Nobel’s lab it only killed 5 people.
Can you explain how you are envisioning trial and error working for an out of control general intelligence?
It seems correct to refer to the justification for worst case MO as “hope” that it will be strictly harder than real alignment. It seems like patently bad security mindset.
I view MOs as being able to demonstrate the existance of a failure mode, but never prove that a failure mode doesn’t exist.
It’s worth flagging that 1899 is extremely old and I wouldn’t expect European authors to do a good job providing an unbiased description of First Nations culture.
Indigenous Australians only received equal voting rights at all levels of government across the country in 1966.
I wanted to give some literature links in case there was anyone who hadn’t previously encountered the argument “if every physical object can with equal justice to be said to implement every computation.”
My understanding is that you’re describing the Simple Mapping Account of what makes something a computer. (As a side note: I’m not quite sure how much confidence I have that it is actually true, given that it is a completely unhelpful way to identify “computers” in any practical sense.)
I’d suggest a curious reader starts with Picinini’s overview:
Picinini (2010) “Computation in Physical Systems”
Chalmers (1996) “Does a rock implement every finite state automaton?”
(Does Ants mean Anthropic here?)
I’m not sure the situation in which LLMs are not good at crystallised intelligence is good for AI Safety. Or at least, it is good in the sense that timelines are delayed by a few years but makes our odds of surviving an AGI worse.
A core component of modern technical AI safety is studying LLMs. Major figures in the field seem to be under the impression that their experiments will extrapolate to future models, at least to some extent. As a concrete example, the team at Google DeepMind endorses a safety approach in which “there will not be large discontinuous jumps in general AI capabilities”(pg 3). Authors on this paper include Rohin Shah, Neel Nanda and Victoria Krakovna.*
My concern is that, should LLMs be terrible at fluid intelligence and are fundamentally inefficient learners, then we’ll see a temporary slow in progress. But this leads to a strong economic incentive to explore alternate architectures and at some point one is found which learns faster with less data and energy. (This is Yann LeCun’s current agenda.)
The end result is we will see a sudden jump in AI capabilities that we are unprepared for.
*This isn’t to single out DeepMind’s team, their paper just came to mind.
I agree one should focus on counterfactual impact, but simply because your work would eventually be done anyway does not mean the counterfactual impact is identical.
Specific research being done earlier may shorten timelines and/or encourage others to do more work on the same topic.
3 unrelated points:
I’d love to see this argument expanded further but also appreciate what you’ve written here.
You sort of mention this, but it strikes me that the argument doesn’t need to be “are prediction markets useful for doing good” but just needs to be “does the improvements to prediction markets and infrastructure made by EA money and resources actually meaningfully increase the amount of good prediction markets do?”
Lastly, may I suggest cross-posting this to the EA forum?
I would suggest a larger barrier is that individual tech companies in the Bay Area do not have the political or moral authority to negotiate on behalf of their domestic government or humanity as a whole.
There are substantial national security implications and I’d expect any government that takes AI risk seriously will take an extremely grim view of individuals attempting to negotiate with known misaligned systems.
“we’ve had so much evidence roll in… we could have more evidence about intelligence and its steerability than any other point in human history”
Anthropic staff perform experiments on modern LLMs and see this as generalisable evidence about future models.
Another interpretation is that these experiments are simply evidence about the behaviour of modern LLMs and will not generalise cleanly to AGI.
I am inclined to believe the latter argument.
The security techniques needed to secure chimpanzees in a zoo are different to the techniques you need to secure an intelligent and healthy adult.
Notice that this holds despite the similar “architecture” of the chimp and human brains. Our neuron count is likewise within an order of magnitude.
What useful heuristics you use for quickly estimating probabilities do you use?
Mine is: try to actively think about picking the probability such that if you would be impartial between taking either side of the bet.
This is (tautologically) the correct approach but I am often estimating probabilities in an “adversarial” context (i.e. informally with friends where we both have fairly different models). In this context you’re incentivised to negotiate the best possible odds for yourself, even if you aren’t actively trying to do this.
Rather than estimating the actual probability, I found my brain was estimating “what odds would I be happy to place a bet on” which ends up estimating lower odds for the side you expect to bet on.
(Obviously, the best tip is “practice” and “keep a record of your predictions and examine them”.)
I believe 4 to be bad advice for most children. If you’re from a rich, stable family and recieve a good education yet still struggle at a particular subject then maybe, yes, it does reflect some kind of inherit weakness.
But the majority of children who are behind in an area simply haven’t spent enough time working on it. Regardless of a childs genuine abilities, they’ll do worse if you tell them they should consider themselves to be “less capable”.
How much extra money and compensation will you be recieiving from OpenAI compared to METR?
Would you be willing to donate the entirety of it?