AI Safety Contractor and Masters student in Physics at the University of Queensland.
I am interested in Quantum Computing, physical AI Safety guarantees and alignment techniques that will work beyond contemporary models.
Views expressed here are my own.
Can you explain how you are envisioning trial and error working for an out of control general intelligence?
It seems correct to refer to the justification for worst case MO as “hope” that it will be strictly harder than real alignment. It seems like patently bad security mindset.
I view MOs as being able to demonstrate the existance of a failure mode, but never prove that a failure mode doesn’t exist.
It’s worth flagging that 1899 is extremely old and I wouldn’t expect European authors to do a good job providing an unbiased description of First Nations culture.
Indigenous Australians only received equal voting rights at all levels of government across the country in 1966.
I wanted to give some literature links in case there was anyone who hadn’t previously encountered the argument “if every physical object can with equal justice to be said to implement every computation.”
My understanding is that you’re describing the Simple Mapping Account of what makes something a computer. (As a side note: I’m not quite sure how much confidence I have that it is actually true, given that it is a completely unhelpful way to identify “computers” in any practical sense.)
I’d suggest a curious reader starts with Picinini’s overview:
Picinini (2010) “Computation in Physical Systems”
Chalmers (1996) “Does a rock implement every finite state automaton?”
(Does Ants mean Anthropic here?)
I’m not sure the situation in which LLMs are not good at crystallised intelligence is good for AI Safety. Or at least, it is good in the sense that timelines are delayed by a few years but makes our odds of surviving an AGI worse.
A core component of modern technical AI safety is studying LLMs. Major figures in the field seem to be under the impression that their experiments will extrapolate to future models, at least to some extent. As a concrete example, the team at Google DeepMind endorses a safety approach in which “there will not be large discontinuous jumps in general AI capabilities”(pg 3). Authors on this paper include Rohin Shah, Neel Nanda and Victoria Krakovna.*
My concern is that, should LLMs be terrible at fluid intelligence and are fundamentally inefficient learners, then we’ll see a temporary slow in progress. But this leads to a strong economic incentive to explore alternate architectures and at some point one is found which learns faster with less data and energy. (This is Yann LeCun’s current agenda.)
The end result is we will see a sudden jump in AI capabilities that we are unprepared for.
*This isn’t to single out DeepMind’s team, their paper just came to mind.
I agree one should focus on counterfactual impact, but simply because your work would eventually be done anyway does not mean the counterfactual impact is identical.
Specific research being done earlier may shorten timelines and/or encourage others to do more work on the same topic.
3 unrelated points:
I’d love to see this argument expanded further but also appreciate what you’ve written here.
You sort of mention this, but it strikes me that the argument doesn’t need to be “are prediction markets useful for doing good” but just needs to be “does the improvements to prediction markets and infrastructure made by EA money and resources actually meaningfully increase the amount of good prediction markets do?”
Lastly, may I suggest cross-posting this to the EA forum?
I would suggest a larger barrier is that individual tech companies in the Bay Area do not have the political or moral authority to negotiate on behalf of their domestic government or humanity as a whole.
There are substantial national security implications and I’d expect any government that takes AI risk seriously will take an extremely grim view of individuals attempting to negotiate with known misaligned systems.
“we’ve had so much evidence roll in… we could have more evidence about intelligence and its steerability than any other point in human history”
Anthropic staff perform experiments on modern LLMs and see this as generalisable evidence about future models.
Another interpretation is that these experiments are simply evidence about the behaviour of modern LLMs and will not generalise cleanly to AGI.
I am inclined to believe the latter argument.
The security techniques needed to secure chimpanzees in a zoo are different to the techniques you need to secure an intelligent and healthy adult.
Notice that this holds despite the similar “architecture” of the chimp and human brains. Our neuron count is likewise within an order of magnitude.
What useful heuristics you use for quickly estimating probabilities do you use?
Mine is: try to actively think about picking the probability such that if you would be impartial between taking either side of the bet.
This is (tautologically) the correct approach but I am often estimating probabilities in an “adversarial” context (i.e. informally with friends where we both have fairly different models). In this context you’re incentivised to negotiate the best possible odds for yourself, even if you aren’t actively trying to do this.
Rather than estimating the actual probability, I found my brain was estimating “what odds would I be happy to place a bet on” which ends up estimating lower odds for the side you expect to bet on.
(Obviously, the best tip is “practice” and “keep a record of your predictions and examine them”.)
I believe 4 to be bad advice for most children. If you’re from a rich, stable family and recieve a good education yet still struggle at a particular subject then maybe, yes, it does reflect some kind of inherit weakness.
But the majority of children who are behind in an area simply haven’t spent enough time working on it. Regardless of a childs genuine abilities, they’ll do worse if you tell them they should consider themselves to be “less capable”.
What predictions does this model make?
Why would you expect the American government not to pursue America first policies?
My prior is that almost any decision which is not explictly absurd can be provided a cohesive and somewhat defensible justification when written by intelligent people.
I think a lot of your confusion is stemming from the fact that you are treating PR statments from Anthropic as if they were being made in good faith.
For example:
“Maybe Anthropic should’ve been more clear about what “behind” and “ahead” mean, and when or when not they’re giving themselves the option/soft obligation to pause”
They will try to avoid doing this because it is very embarassing when your previous statements contradict your actions.
“Are Anthropic employees not reacting to this?”
Anthropic employees are paid large amounts of money and get to talk about their concerns with other people in the organisation. They have a direct financial incentive to avoid speaking publicly against the company.
“On a personal note, many of us are much more nervous about working for Anthropic and are much more nervous about the strategic decision-making of its leadership during the critical period.”
The good news is that having a cool job and earning a huge amount of money is enough to quell any moral concerns you might have.
I’d previously worked through a dozen or so chapters of the same Woit textbook you’ve linked as context for Representation Theory.
Given some group , a (limear) “representation” is a homomorphism from G into the GL(V) the general linear group of some vector space.
That is, a map is a representation iff for all elements ,.
Does “preferences between deals dependant on unknown world states” have a group structure? If not it cannot be a representation in the sense meant by Woit.
Is anyone been able to do this right now?
After a quick check, I’m unable to replicate the behaviour shown in this thread using Google Translate on Chrome android mobile browser.
I think it is extremely difficult to predict how a community of intelligences would react to that information.
The obvious difference was that it was acceptable for mistakes to be made. When there was an explosion in Alfred Nobel’s lab it only killed 5 people.