Can we get a live counter somewhere of how many people have signed up?
Eli Tyre
Karma: 9,946
I believe I might have been the literal first person, outside of the lightcone team, to sign up for this. Also the second person, depending on their data validation, since I entered my name and email twice.
(Though it’s possible that what lies at the end will be a powerful-enough non-AGI AI tool that it’ll make it very easy for the frontier labs to then use it to R&D an actual AGI, or take over the world, or whatever. This is a subtly different cluster of scenarios, though.)
Why do you say that? They seem functionally very similar, leaving aside that that second possibility suggests we’re in for a terrifying fast takeoff, for the reasons outlined above about lumpiness of innovations.
Observing early hominids from the outside, aliens might have said that the humans are “acting as though they want to maximize their inclusive genetic fitness”
I’m skeptical that this is true for aliens that are paying close attention.
Many ancient cultures had (attempts at) contraception. Claude tells me that Kahun Papyrus (~1850 BCE), the oldest surviving medical text, contains contraceptive recipes.
I bet that humans have been trying to have sex without getting pregnant as early as they realized the connection between the two, and first noticed that there were things to try. I would not be surprised if the aliens sometimes observed infanticide.
It’s posted on my personal blog, for whatever that is worth to you.
Are you eluding to some specific failure mode without wanting to state it outright?
Why would it have bad consequences, or worse consequences than any other post that didn’t depend on a literature review?
The only one of these criteria that any of the labs are plausibly meeting is:
The company has a broadly good track record of deploying current AIs safely and responsibly, including owning up to and correcting mistakes.
Not having followed super closely, I think Anthropic and GDM reasonably meet this standard. All of xAI, OpenAI, and Meta definitively do not.
I guess that none of them have operational security adequate to protect against nationstate actors, but I don’t know enough to say one way or the other. (Also, some of them might have realistic plans and policies in place to scale up their operational security as their capabilities increase, which I would count, if security professionals agreed that their plans and policies were sound.)
I’m trying to outline the minimum criteria that make it deontologically ok to be a scaling lab. Of course, you can do better or worse without hitting that bar, but you don’t get partial deontology points by getting closer.
Stealing a billion dollars is worse than stealing a million dollars, but either way, you’re a criminal.
Totally. You can pause frontier capability development without pausing applications and commercialization, or non-general capability developments.
But, realistically, a company saying “we’ve crossed the line, we think it’s irresponsible to scale further”, especially if other companies don’t respond in kind, will cause the stock price to fall?
When frontier labs are pausing, they will be the people who will have the most momentum towards rushing forwards with AGI development. They will have created a culture of scaling, have ready-made deals that would allow them to immediately become extremely powerful and rich if they pushed the frontier, and be most psychologically attached to building extremely powerful AI systems in the near future.
This post is about a hypothetical different lab that has a notably different corporate culture, in which some notable effort was taken to improve the incentives of the decision-makers?
trying to get a frontier lab that both facilitates a pause by being in the room where it happens, and then just pivots seamlessly to using all their resources on alignment successfully, is asking for too much, and trying to get two things that are very hard to get at the same time.
This seems like a plausible take to me. I’m pretty open to “the get-ready-to-pause scaling lab should have one job, which is to get ready to pause and get the world to pause.”
But also, do you imagine the people who work there are just going to retire the day that the initial 6 month pause (with the possibility of renewal) goes into effect? Many of those people will be world class ML researchers who were in this position specifically because of the existential stakes. Definitely lots of them are going to pivot to trying to make progress on the problem (just as many of them are going to keep up the work of maintaining and extending the pause).
Some evidence about this: Eliezer was deliberating holding off on publishing TDT to use it as a test of philosophical / FAI research competence. He dropped some hints on LW (I think mostly that it had to do with Newcomb or cooperating in one-shot PD, and of course people knew that it had to do with AI) and also assigned MIRI (then SIAI) people to try to guess/reproduce his advance, and none of the then-SIAI people figured out what he had in mind or got very close until I posted about UDT (which combined my guess of Eliezer’s idea with some of my own and other discussions on LW at the time, mainly from Vladmir Nesov).
Very interested to hear this history!
we have to fully solve DT before building AGI/ASI, otherwise it could be catastrophic due to something like the AI falling prey to an acausal threat or commitment races, or can’t cooperate with other AIs.
These seem to me like much lower priority problems than ensuring that our AI agent don’t stage a takeover. In comparison, this seems like an exotic failure mode. Further, this is a problem that a self-modifying AGI might very well be able to solve on it’s own before being extorted.
Which isn’t to say that there are not catastrophic possibilities here, but I’m surprised that these were the reasons given at the time for Decision Theory getting top billing.
Am I missing something or is this indeed much lower priority than subproblems that are more directly about preventing AI takeover?
I meant to give you a friendly ribbing, not to actually call you out. You’re good. No need to apologize.
If you right click on the text in comment you’re writing an option box will pop up. One of those options looks like a chain, that’s he one you want.
Isn’t that a completely unrealistic possibility?
I doubt there’s any physical artifact that has that property?
Maybe there are mathematical objects that can only be discovered by brute search, even in the limit of technological maturity, that have values that have terrible scaling properties (trillions and trillions of times more input for a 2x improvement). But even that seems like a stretch.
Noting that you said that you’re unsure.
You tell me about a secret Eliezer post, and say that that you found useful, but you don’t even link to it?!
If we decide utility requires many information round-trips with the rest of the colonized universe (for cultural exchange or something), the supply of valuable galaxies will be lower, and so the price will be higher.
Why would that be? Can’t we just simulate them?
or just shoot down the galaxy owners’ space probes.
Won’t those probes be heading off toward their galaxies at a noticeable fraction of the speed of light?
I guess you mean shoot them down with high intensity lasers?
...and I think this is the positive update. It feels very plausible, in a visceral way, that the first economically transformative AI systems could be, in many ways, really dumb.
I think this is right, but I think is misleadingly encouraging.
I keep having to remind myself: Most of the risk does not come from the early transformative AIs. Most of the risk comes from the overwhelming superintelligences that come only a few years later.
Maybe we can leverage our merely transformative capabilities into a way to stick the landing with the overwhelming superintelligences, but that’s definitely not a forgone conclusion.
Someone should make a post saying “hi, I’m a human, posing via the API”. I want to know how the (presumed) AIs respond to that.
But a key point is that 3 is, fundamentally, a skill-issue.
And maybe, given our human-capability levels, it’s functionally a skill-issue for everyone.
But it’s a pretty common pattern for someone to try to do [something] (using a basically hopeless methodology, often), fail, and then declare that [something] is beyond the ken of man, and we must be humble in the face of our ultimate ignorance and impotence.
See for instance the way many (in my opinion, midwit) people declare that there is irreducible mystery, or that no one worldview and accommodate all of the problems of society.