AI control is supposed to defend against human-level AI, but the line between a human red team and an AI-augmented one has started to blur. And we’re still optimizing our proxy task of safety-usefulness frontiers as if that line were clean.
What in the control setup relies on a non-AI-augmented red team, and what breaks when you relax this assumption?
Don’t you feel ashamed to spend so much time with AIs, given that you think they’ll likely put an end to humanity ? How can you justify that morally? Your behaviour is inconsistent with your beliefs. You’re encouraging AI, if not with your money, then with your time and attention.
AGI labs are currently compute constrained, not funding constrained, so your use of AI by itself will neither speed up nor slow down the end of humanity. There are cases which I would consider positive uses of AI (e.g. speeding up alignment research), and negative uses (e.g. cheating on homework), but I think it very much depends on specifics.
Data centers are draining all available energy and accelerating global warming more than ever, while nobody pays attention anymore. The doom scenario has already begun. You should just stop !
Epoch AI estimates that datacenters will account for about 5% of US energy demand by 2030. Personally this is not the doom scenario I’m worried about.
The most interesting part of the first half were his comments about Anthropic, and in particular how Anthropic ended up primarily training and running on Tritium and TPUs.
Running on Tritium would be an interesting plot twist, but I think they are running on Trainium. Perhaps one day they will run Trainium on Tritium though!
In 1930, John Maynard Keynes—one of the most famous economists of all time—predicted that his grandkids would work just 15 hours a week. Why aren’t we doing that?
His estimate may have been wrong, but the trend seems roughly correct:
Given the Mythos news, could we have (optional) 2 factor authentication for Lesswrong?
Stockfish, relying heavily on deep search over subtle long term evaluation, does not see this. It thinks that white is clearly winning, and doesn’t realise that this is not the case until you play out the moves up to the point where it sees the “50 move rule” draw arriving.
My understanding is that there are multiple different methods for detecting fortresses, but these positions are fairly rare and this hasn’t been a focus of the developers[1]. One reason is that all Stockfish commits must pass though fishtest, a series of self-play games to prevent regressions, and the added strength from fortress detection historically hasn’t been worth the additional complexity. I don’t think it would be very difficult to make a version of Stockfish which was slightly weaker overall but fixed this issue.
I think that fine-tuning evals are a good idea, and there has been some important related work done on this here. To quote from that post:
Why is this useful?
Soft prompts offer a simple way to quantify the conditional distance between a model and a target behavior. If many tokens are required, then lots of information-as-conditions is needed to define the behavior and it’s unlikely that it will happen by chance. If no number of tokens suffices, the behavior is likely inaccessible by inputs alone.
It can be used as a part of evaluation suites to monitor the change in distance to concerning behaviors across models and scales and the degree to which those behaviors can be achieved in the first place.
It is also difficult for the model to hide a capability that is otherwise accessible to some input. If the original model is an end-to-end differentiable product of gradient descent, it has already been shown to be transparent to the same type of optimizer that will be tasked with adversarially optimizing the input soft prompts against the machinery of the model.
Compared to manual attempts at elicitation or automated techniques restricted to discrete tokens or black box optimization, it dramatically improves the chances that an evaluation’s attempt to elicit concerning behavior will succeed.
It’s also dumb simple.
I want to draw your attention to some AI doom markets I created on manifold.
Since traders in many of these markets[1] can’t benefit from their shares resolving YES, these markets effectively serve as mana-weighted polls. This may be what you were aiming for, and I think it is interesting, but I would be careful to interpret the results accordingly.
e.g. “If AI kills everyone, how will it do it?”
I agree that there are many interesting parts of the system card and encourage everyone to read it in full!
IMO xAI is no longer a frontier lab. All 11 of Musk’s cofounders have left, along with much of the technical talent, and they haven’t produced a frontier model in some time. If someone took over xAI, it might speed up the race, since you’d be giving over their vast compute to a more competent actor. That being said, I agree that they have historically been a worse actor for AI Safety than the big three labs.
That’s interesting, do you have a link?
It’s not exactly the same, but Zvi has strongly argued against persuasion risk being taken out of the OpenAI preparedness framework.
This makes me sad, but I’m not sure there’s a real solution here.
Once the AI is doing most of the work, there are AI debate and formal verification schemes which might help. This of course assumes we’ve solved alignment and many other issues.
Google TPUs will keep their advantage in hypothetical models with tens of trillions of parameters until 2028, and we might soon observe from Anthropic’s rumored larger-than-Opus Claude 5 model whether that’s likely to become an important class of models before 2028.
Do you think this is likely to be trained using TPUs or Trainium?
I’ve heard this before.
Why would this be useful for interpretability? You can already represent INT4 networks with logic gates. I’d also guess that DLG networks break many of our interp tools; e.g. it’s harder to do linear probes or look at the residual stream or activations.