Love the title
lc
Karma: 10,213
One large part of the AI 2027 piece is contigent on the inability of nation state actors to steal model weights. The authors’ take is that while China is going to be able to steal trade secrets, they aren’t going to be able to directly pilfer the model weights more than once or twice. I haven’t studied the topic as deeply as the authors but this strikes me as naive, especially when you consider side-channel[1] methods of ‘distilling’ the models based on API output.
@Daniel Kokotajlo @ryan_greenblatt Did you guys consider these in writing the post? Is there some reason to believe these will be ineffective, or not provide the necessary access that raw weights lifted off the GPUs would give?
- ^
I would normally consider “side channel attacks work” to be the naive position, but in the AI 2027 post, the thesis is that there exists a determined, well resourced attacker (China) that already has insiders who can inform them on relevant details about OpenAI infrastructure, and already understands how the models were developed in the first place.
- ^
are… are you sure you read the post you’re responding to?
We definitely read the same words!
I think Cremieux is an honest[1], truthseeking, and intelligent guest speaker, and I would be extraordinarily disappointed in the organizers if they disinvited him. I also have a very high opinion of LessOnline’s organizers, so I’m not particularly worried about them cowtowing to attempts to chill speech.
- ^
(In the sense of e.g. his work output being factually correct, not speaking to his character personally)
- ^
The full article sort of explains the bizarre kafkaesque academic dance that went on from 2020-2022, and how the field talked about these changes.
My hypothesis there is that we have systematized VC-backed YC-style founders. The rules are a lot easier to discover and follow, the track record there makes it a career path one can essentially plan on in a way that it wasn’t before, and the people who gate progress with money are there to reward those who internalize and follow those principles.
But a paved road would create more outsized successes, not fewer.
I think a better theory is that corp dev has systematized acquiring companies-that-look-like-Google very early. Early multiples on revenue for YC startups are insane. 23 year old founders coming out of the gate today with Google’s growth circa 2000 get a buyout offer 10-100x what Larry & Sergei did and sell immediately.
I loved the MASK benchmarks. Does anybody here have any other ideas for benchmarks people could make that measure LLM honesty or sycophancy? I am quite interested in the idea of building an LLM that you can trust to give the right answer to things like political questions, or a way to identify such an LLM.
Wow, I didn’t realize.
The two guys from Epoch on the recent Dwarkesh Patel podcast repeatedly made the argument that we shouldn’t fear AI catastrophe, because even if our successor AIs wanted to pave our cities with datacenters, they would negotiate a treaty with us instead of killing us. It’s a ridiculous argument for many reasons but one of them is that they use abstract game theoretic and economic terms to hide nasty implementation details
Formatting is off for most of this post.
“Treaties” and “settlements” between two parties can be arbitrarily bad. “We’ll kill you quickly and painlessly” is a settlement.
The doubling down is delusional but I think you’re simplifying the failure of projection a bit. The inability of markets and forecasters to predict Trump’s second term is quite interesting. A lot of different models of politics failed.
The indexes above seem to be concerned only with state restrictions on speech. But even if they weren’t, I would be surprised if the private situation was any better in the UK than it is here.
Very hard to take an index about “freedom of expression” seriously when the United States, a country with constitutional guarantees of freedom of speech and freedom of the press, is ranked lower than the United Kingdom, which prosecutes hundreds of people for political speech every year.
The future is hard to predict, especially under constrained cognition.
AGI is still 30 years away—but also, we’re going to fully automate the economy, TAM 80 trillion
I find it a little suspicious that the recent OpenAI model releases became leaders on the MASK dataset (when previous iterations didn’t seem to trend better at all), but I’m hopeful this represents deeper alignment successes and not simple tuning on the same or a similar set of tasks.
Works now for me
If anybody here knows someone from CAIS they need to setup their non-www domain name. Going to https://safe.ai shows a github landing page
I think the main problem is that the second cover looks really rushed.