I work primarily on AI Alignment. Scroll down to my pinned Shortform for an idea of my current work and who I’d like to collaborate with.
Website: https://jacquesthibodeau.com
Twitter: https://twitter.com/JacquesThibs
GitHub: https://github.com/JayThibs
LinkedIn: https://www.linkedin.com/in/jacques-thibodeau/
If true, I think part of the change can be attributed to a more dramatic increase in those who are technically competent and have various accomplishments yet don’t care so much for safety (though would love a role at Anthropic). If the sample of those folks has increased, it becomes more challenging to turn down those who are highly competent but maybe care a bit less about safety. Especially if you are a mentor and you just want someone to execute exceptionally well on your project.
Last time I mentored (for SPAR), I had to turn down a lot of folks who were more accomplished and likely to succeed (e.g. professors) so that I could accept those I felt would have more long-term potential to contribute to safety.
Did you try feeding it examples of texts you’ve written (or that you like the style of) and then having the model generate a style guide to loosely follow?
I’ve done this in the past and found it somewhat valuable (+ telling it to avoid the usual AI slop crimes), though definitely not enough.
I definitely also found that forcing a new model session for each axis is worth it despite the increased cost.
I wrote this take on Twitter. I wish I had time to refine it more and make it easier to read, but hopefully better than nothing:
Good nuggets in there. Agree with his notes on how to impact research in the labs if you’re an external researcher.
One thing I struggle with though is that Rohin seems to not be concerned about what ASI alignment will be like in the future. Like, I feel there are competing thoughts that make me think he is not making the next logical inference on the end state. It’s like he is pointing at all the correct things, but then arriving at a different conclusion than I’d expect.
For example, here’s what he’s considering all at once:
RL was obviously going to be the future, but currently pre-training is most of capabilities and RL is just reinforcing existing circuits
LLMs are currently not that smart (compared to what we imagine AI will be), which also means current alignment work means very little about the alignment of future AIs
We’ll only RL AIs on 1 week time horizon (???)
Chain of thought monitoring will probably stick around for another ~4 years given the performance issues we expect within this paradigm and the transformer-like architecture (which allows for parallelization required), but acknowledges that it likely eventually won’t stick around
Believes in longer timelines
Believes that current trends point to a more continuous takeoff
I think the inference one could make from this is that LLMs and the current method of developing intelligence is limited in fundamental ways that can give us a false comfort in future developments.
In other words, it feels to me like Rohin is pointing to many of the correct things, but instead of making the more pessimistic prediction, he is extrapolating from the current paradigm being positive signal.
Ultimately, I’d like to know why he seemingly puts less weight on the current paradigm potentially being a bearish signal for alignment.
I guess my point is that it seems that he implicitly believes that AI will need to fundamentally change in order to have the capabilities we focus on in the alignment literature, but then maybe he doesn’t believe that change is not so fundamental that it we are talking about a different form of cognition.
Like, LLMs model the statistical distribution of pretraining but are largely developing context-specific heuristics (also generalize poorly OOD), which is much different from the consequentialist reasoner we’d expect would be required in order to capably carry our worldly transformative actions. In other words, alignment worries may only show up after that change, meaning current work may be largely irrelevant to that future.
There would be a whole world of other countries to keep in line via carrot and stick—Russia, Israel, India—but they might be able to do it—for a while.
Those aren’t really the countries I’m thinking about, but yes I would imagine US, China, and all other countries would punish any country going against the treaty.
I do think the clock would eventually run out, however—by which I mean, that the resources needed to approach the threshold of superintelligence would diminish year by year, thanks to algorithmic advances, and probably cloud computing on the dark web.
I worry about that, but:
In that world we may be screwed regardless if misaligned AIs are so compute efficient.
I’m not sure I believe a misaligned AI from one country could succeed in a takeover. I’d expect that they would need to be so entangled in our critical infrastructure and maintain the supply chain process for its further development. So, seems like it could cause damage, but imo would not succeed in a takeover (or maybe even attempt one for a long time).
You may get higher odds of a US and China agreement if those other countries have enough will to push for it (so, a bit of a chicken and egg).
Why is this [less dependence on the US] good (overall)?
There is likely a sweet spot between being completely independent of the US and being more willing to impact US supply chains as a consequence of the US carrying on. For example, it’s way less likely that the Netherlands will leverage ASML as a bargaining chip in a treaty if it’s clear that it will suffer unacceptable economic harm (by not leveraging AI to benefit itself).
This post is worth reading:
All this even opens a potential door back to the development focus – if there is some way to bring middle powers into a position of strength, they might once again affect AI development, serve as a real third center of gravity on the question of advanced AI. But the path to this runs through the effective and uncompromising pursuit of their national interest. It will not be reached from a position of economic and strategic weakness they are headed for on their current trajectories.
What do you mean by this [better governance for inference]?
If more data centers for inference are not in the US, other countries that don’t have as strong incentives to cut corners in an AI race can enforce stronger governance if deployed in their country. In addition, places like the EU have historically been stricter on this sort of thing.
But compute is pretty fungible. If OpenAI has more compute for inference abroad, they can allocate more of their US compute to training to reach whatever allocations they find ideal. It would only be if a majority of compute is located abroad that this would become an issue, I think?
We are in a compute-constrained world. Anthropic is a prime example of a company dealing with these constraints. Nvidia limits how many GPUs a single company can get (because it wants diversity and competition), but even it is limited by how much TSMC can produce. The compute abroad doesn’t necessarily have to go to OAI. It could just as easily go to open-source models running on sensitive applications within the country, where they would rather not be subject to the CLOUD Act.
What do you mean by this [sovereignty]?
Not subject to the CLOUD Act, building capability within their own country so that they can use it to enrich themselves, being less at the whim of what the US does, post-training AIs in critical infrastructure domains that run on-prem, etc.
It’s not clear that this would make international treaties easier? If you have compute more distributed, and therefore more relevant actors, it may be harder to reach a deal than if only two parties need to come to terms (the US and China). Or maybe I’m misunderstanding your point.
As I mentioned in my other comments:
I am saying that the odds of other countries building ASI in a world where the US and China have signed a treaty to pause are incredibly low. They would either get attacked, suffer from insane tariff, no longer get any new chips, be cyberhacked to hell, etc. So, in such a world they would have to bend the knee.
The US will not be against signing a treaty with other nations because Spain doesn’t want to sign it. If they decide this is an important decision to make internationally, the US and other countries will cripple Spain.
I am saying that the odds of other countries building ASI in a world where the US and China have signed a treaty to pause are incredibly low. They would either get attacked, suffer from insane tariff, no longer get any new chips, be cyberhacked to hell, etc. So, in such a world they would have to bend the knee.
So, they will be more likely to be forced to sign for that reason + they know deep down that they will get crushed if they attempt to race with the US or China. At best they can leverage AI to avoid making their country poor as hell in comparison to other countries not benefiting from AI.
I get the feeling you are misunderstanding my point. Which is:
If you are worried about misaligned ASI, you are likely worried about it being developed in the US or China (and thus it does not matter much if other countries are attempting to build ASI). Therefore, if there are data centers outside of US and China, it does not directly impact ASI development in those two countries. However, if you are trying to arrive at an international treaty across countries, the main countries that matter are the US and China. Having data centers outside of those countries likely increases the leverage of those other countries, despite them being unlikely to build ASI. So, in practice you just allow them to benefit from AI without really advancing ASI development, and instead potentially causing a slowdown. If other countries are highly dependent on the US (due to not investing in benefiting from AI), I expect them to have nearly zero leverage in pushing for a treaty.
So, is building data centers outside of the US just good overall?
Less dependence on the US
AGI labs likely won’t use data centers outside of the US for training or internal deployments
Likely better governance for inference
Middle powers benefit from AI
Takes training GPUs away from AGI labs
Data and compute sovereignty for regulated industries and defense
Ideal for a post-AGI world (e.g. reduces likelihood of vassal states and massive inequality)
All likely point to more leverage for international treaties
If you largely believe that no non-US or non-Chinese company/state will build AGI, then the threat of misaligned ASI is not really an issue
Perhaps the main bad thing could be that you are trying to do an international treaty and one country does not want to sign it and continues to build towards AGI. I suspect that if the US and China agree to a treaty, that lone country would be royally fucked (besides being much less likely to have the talent, infra and perhaps total compute to do so).
Note for additional clarity:
I am not saying this means AIs will never become “ASI”. I am not saying that timelines can’t be as short as 1-2 years. All of that is within the realm of possibility and despite me poking at the problems with current models, I still put decent weight on them being solved fairly soon. And even if it does take a number of years to solve that variety of capability problems, I still think AIs can be highly transformative in the next 2-3 years.
In anticipation of people using that example as a response, I added a follow-up tweet to the tweet version of this post:
yes, dear reader, the new erdos unit distance problem solved by OpenAI’s model still fits within this story. in fact, i expected these kinds of results.
My understanding is that the main difficulty in arriving at this result is performing sufficient inference over the entire search space, far beyond what any human can do. And so, it is indeed superhuman in that specific capability.
But I suspect that every step to get there was within distribution, and ICL was enough to arrive at the result. However, I also suspect that there are many other types of problems that require integrating knowledge into the weights and can’t be done with long chains of “within-distribution” reasoning that sum to a new result.
So, I suspect the bottleneck in solving the problem in the past was more about human inability to search through an extremely large number of paths.
This is not to say what the model did isn’t impressive, but imo this is within the realm of problems I’d expect it to solve (which is a lot!). There are other types of problems I expect they would need to consolidate information (in the weights) while solving the problem. Though who is to say they aren’t doing that, I don’t know what OAI is doing for sure.
FYI, I’m running on the assumption that proofs will be cheap/free in the coming years (for most applied math, likely sooner) and partially betting my startup on this, which is another reason I expected this kind of result.
Let me clarify that despite me not having a perfectly precise definition here, part of my goal is to point out that most of the community seem to 1) fail at being precise about what they mean and consider to be generalization, 2) overstate the novelty generated by the models.
I wanted to at least highlight a greater separation between the interpolated generalization and the OOD generalization that seems more separated than people let on.
Please read my other comments in the thread for more context, particularly the one about Mythos. They largely contain my takes on your questions.
I agree it is possible and fits within my model.
However, I think it is important to separate what can be repeatable at 1000x and what is actual increased productivity.
For example, I can generate so many plots now! More than I used to! So much code too. +1000x in fact! But is it actually providing more value to the world at that rate? No!
As Terrence Tao said during the recent Dwarkesh interview:
Dwarkesh Patel
So let’s see if you can continue this streak. You personally are 2x more productive as a result of AI. What year would you say that?
Terence Tao
Productivity, I think, is not quite a one-dimensional quantity. I’m definitely noticing that the style in which I do mathematics is changing quite a bit, and the type of things I do. For example, my papers now have a lot more code, a lot more pictures, because it’s so easy to generate these things now. Some plot which would have taken me hours to do, now I can do in minutes. But in the past, I just wouldn’t have put the plot in my paper in the first place. I would just talk about it in words. So it’s hard to measure what 2x means.
On the one hand, I think the type of papers that I would write today, if I had to do them without AI assistance, would definitely take five times longer. But I would not write my papers that way.
Dwarkesh Patel
5x?
Terence Tao
Yeah, but these are auxiliary tasks. Things like doing a much deeper literature search or supplying a lot more numerics. They enrich the paper. The core of what I do, actually solving the most difficult part of a math problem, hasn’t changed too much. I still use pen and paper for that.
But there’s lots of silly things. I use an AI agent now to reformat. Sometimes if all my parentheses are not quite the right size, I used to manually change them by hand, and now I can get an AI agent to do all that quite nicely in the background.
They’ve really sped up lots of secondary tasks. They haven’t yet sped up the core thing that I do, but it’s allowed me to add more things to my papers. By the same token, if I were to write a paper I wrote in 2020 again—and not add all these extra features, but just have something of the same level of functionality—it actually hasn’t saved that much time, to be honest. It’s made the papers richer and broader, but not necessarily deeper.
All of that seems within-distribution to me.
Isn’t zero-day discovery the sort of process that is necessarily OOD?
In many cases, lots of security bugs that haven’t been found are simply a case of not enough effort being put into finding them. In this case, I think you could just as reasonably say that Mythos is becoming better at modeling the data distribution due to scale, and therefore ends up being better at finding these vulnerabilities.
On a related note, I’ve started to distrust Anthropic’s judgement on these things. Particularly, I believe that they oversold the C compiler experiment as being OOD, but I think this is false.
From the Jeremy Howard podcast link I shared:
So for example, I was talking to Chris Lattner yesterday about how Anthropic had got Claude to write a C compiler. And they were like, “oh, this is a clean-room C compiler. You can tell it’s clean-room because it was created in Rust.” So, Chris created the, I guess it’s probably the top most widely used C / C++ compiler nowadays, Clang, on top of LLVM, which is the most widely used kind of foundation for compilers. They’re like: “Chris didn’t use rust. And we didn’t give it access to any compiler source code. So it’s a clean-room implementation.”
But that misunderstands how LLMs work. Right? Which is: all of Chris’s work was in the training data. Many many times. LLVM is used widely and lots and lots of things are built on it, including lots of C and C++ compilers. Converting it to Rust is an interpolation between parts of the training data. It’s a style transfer problem. So it’s definitely compositional creativity at most, if you can call it creative at all. And you actually see it when you look at the repo that it created. It’s copied parts of the LLVM code, which today Chris says like, “oh, I made a mistake. I shouldn’t have done it that way. Nobody else does it that way.” Oh, wow. Look. The Claude C compiler is the only other one that did it that way. That doesn’t happen accidentally. That happens because you’re not actually being creative. You’re actually just finding the kind of nonlinear average point in your training data between, like, Rust things and building compiler things.
Which OOD cyber capabilities? How do you know it’s OOD?
I’ll try to make this clearer if I turn it into a more serious top-level post. My intent here was to just push this out since it’s been bothering me, but I have other things to do.
TLDR: Lots of researchers seem to be banking on the idea that LLMs are generalizing OOD or that scale will just solve this (whether through scale alone or scale + using the scaled model to come up with a research breakthrough that does). Lots of research and funding seem to hinge on this idea, which, imo, is underappreciated. If taken seriously, it may mean that 1) timelines are longer, 2) we should expect fundamental reshaping of AI cognition due to the LLM inability to generalize OOD, 3) we shouldn’t update much on alignment progress based on current safety research.
I shared this in the post, but more thoughts here.
This post by @Hyperion describes another natural consequence of the above with respect to RSI (that the field seems to be understating):
Some takes about RSI from discussions with many smart researchers & thinkers:
1. Many RSI (or automated AI R&D) debates converge to similar cruxes: is a 1000x sample efficiency improvement possible, can you just simulate reality and train on it with no sim2real gap, can we easily make models good at “fuzzy” tasks? People like to assume that automated research agents will find such breakthroughs specifically *because* without them, progress could be heavily bottlenecked on data or continued compute scale-ups.
2. The Yudkowsky “genius brain in a box” framing of ASI has latent influence on many researcher views even though people may not be aware of it. A common move is to “flip” predictions, as they go further out, from assuming LLM or deep learning-specific properties of future AI to assuming “von Neumann x1000″, human brain-like properties. I’d like to see more thought-out reasoning of why this flip should occur at any particular point (eg pre or post automated AI R&D)—this question is a crux behind many predictions like AI 2027.
3. There are some cracks in this worldview beginning to show: predictions from a few years ago that models would be less jagged now than they are, or that they would be more deceptive, synthetic data would work better, etc. Many of these seem like prediction errors from imagining future models as a “human brain in a box”, but LLMs are empirically a different kind of intelligence. Most models of software-only intelligence explosion are also coarse enough to mostly ignore properties of LLMs.
4. Views about fast RSI progress seem to be correlated with (a) belief that synthetic data is all you need (b) belief in very high GDP growth and an industrial explosion because of automated firms (c) having worked only in AI research or in small organizations.
5. Key technical things to track over the next 1-2 years: does RL increase in its generalization, AI lab data spend, can we automate synthetic RL env construction, best practices for FDEs deploying AI into large enterprises, coherency of AI personas, how powerful will multi-agent scaling of test-time compute be, and continual learning.
6. Overall I think the “RSI leading to *fast* takeoff” frame had huge alpha in 2022, moderate in 2024, and potentially is of neutral usefulness in 2026 for predicting the future.
I would probably consider “novel” puns to be within-distribution, even if not memorized puns.
But honestly, I think these examples are just generally hard to make sense of, since we don’t have access to their training setup or data (is it a type of pun interpolated across many languages? How much does it relate to true novelty in complex, long-horizon domains?). I could see scale being useful for interpolating these new puns while not necessarily being relevant to what is needed for ASI. Or, scale could actually be making progress towards these sorts of capabilities! It just seems overstated (at least pre-Mythos, which I can’t test), and I feel like it poisons research selection and experiment interpretation.
Scale is obviously helpful, but imo there is more nuance to it than lots of folks consider properly. I’m asking that we try to be more precise about all of this.
For example, I think Talkie-1930 (model trained pre-1930s) is a great example of generalization research (though yes, it does not say much about frontier scaling)! It helps us better understand generalization. But I saw implied claims that the model was able to ICL solve a Python problem, but when you look at the details of the experiment, the OOD generalization coding example feels dubious. From @Steven Byrnes (link to his post / my take):
I was surprised and puzzled by this, because I’m a general skeptic of (so-called) “in-context learning”—I generally say that LLMs have decent “understanding” of what’s in their weights, but quite sketchy & superficial “understanding” of stuff in the context window but NOT the weights. The context window can really only support “recognition” of things that the weights already “understand”. Or at least, that’s what I’ve been saying for years.
So how is Talkie-1930 doing any Python at all? I was puzzled.
…But then I looked at the example. All that Talkie did was exactly copy the example in context but switch “+ 5” to “– 5”. (And it got it right at least once given 100 tries!)
I can definitely imagine that a person who has read every pre-1930 book on symbolic logic, cryptography, and the rest of math (& jacquard looms etc.) could guess that answer, at a glance, given 100 tries, while remaining deeply deeply confused about wtf was going on in the code snippet, and while “understanding” zero Python (and zero code) in any real sense.
So I don’t think this one example gives me any new reason to change my mind about the (lack of) power of (so-called) in-context learning, or to be suspicious of data leakage, subliminal learning, etc. in Talkie-1930.
(Cool project, kudos to the authors.)
I feel like I see examples like this all the time! Often, I expect it because there’s some sort of bias towards trying to ‘warn the world about what is coming’, which leads people in AI safety to overstate such results and muddy our comprehension of what is happening.
It’s incredibly easy to be fooled by the capabilities of the current top-performing tech (LLM agents). It’s easy because they have a vast amount of training data to interpolate from.
This works fine to acquire capabilities within our existing data distribution of the world (one that is also easy to verify), but what happens when they go out of distribution?
LLMs perform poorly! Yet, people seem to think they can actually generalize to new problems. Why is that?
It’s, again, the vastness of their training data. It makes it hard to distinguish between interpolation and extrapolation (or hyperpolation, if you want to add a third dimension).
For example, a Typescript app is within-distribution! AI research in the existing body of research is within-distribution, and companies are paying millions to build RL environments to make them *specifically* good at some of those things!
Related and great post from Beren, “Most Algorithmic Progress is Data Progress”:
In a way, this is like a large-scale reprise of the expert systems era, where instead of paying experts to directly program their thinking as code, they provide numerous examples of their reasoning and process formalized and tracked, and then we distill this into models through behavioural cloning. This has updated me slightly towards longer AI timelines since given we need such effort to design extremely high quality human trajectories and environments for frontier systems implies that they still lack the critical core of learning that an actual AGI must possess. Simply grinding to AGI by getting experts to exhaustively cover every possible bit of human knowledge and skill and hand-coding (albeit with AI assistance) every single possible task into an RL-gym seems likely to both be inordinately expensive, take a very long time, and seems unlikely to suddenly bootstrap to superintelligence.
It might still be impressive, but models are largely remixing many things it has seen in great detail during training (many impressive headline results have even been determined to be the model re-using existing implementations/PRs via search instead of coming up with actually-new ones!). This is not about LLMs not doing impressive things! This is about precisely describing their capability profile, where it comes from, and whether more of the same (e.g., scale) gets you a whole new set of impressive outcomes (e.g., novel R&D that isn’t just remixing existing research).
And yes, I know you can make a ton of discoveries by interpolating existing research (e.g., interdisciplinary research and automating research pipelines to run more experiments). I also think that people are overly confident that it means LLMs will be capable of novel R&D breakthroughs, and what that means is needed from future AIs.
Even if you consider “researchers can come up with novel ideas and give them to the AIs”, that likely involves longer timelines. But, just as importantly, LLMs may be exceptional at automating within-paradigm research, disproportionately better than at automating out-of-paradigm research. Therefore, you end up accelerating research that may largely be irrelevant for ‘True’ AGI (yes, you still accelerate many coding parts, but the speed-up is still bottlenecked in ways that it’s not easy to just say the entire process of arriving at these research breakthroughs is now 1000x faster than before).
“But the models are still capable and growing more capable! Why does this matter? Scale will just solve this!”
It matters because:
1. The whole point of alignment has always been about generalizing ‘human values’ out-of-distribution. So, if alignment and capabilities are tied, it means models are capable of modeling the existing within-distribution ‘values’, but things may pull apart once we undergo the distributional shift of a post-AGI deployment world.
An example you can test right now is LLMs lacking a sense of how to engage with the world in this post-agent era. You have to keep reminding them about the current state of the world. The closer you get to novel R&D that the labs haven’t paid millions in RL envs for (e.g. AI R&D), the starker this becomes.
You can point to continual learning ‘solving’ this, but that is kind of my point. These capability unlocks will fundamentally change the AI and its relationship with itself. Related, “You can’t imitation-learn how to continual-learn”.
Also, from “Training AI agents to solve hard problems could lead to Scheming”:
Future AI models will be asked to solve hard tasks. We expect that solving hard tasks requires some sort of goal-directed, self-guided, outcome-based, online learning procedure, which we call the “science loop”, where the AI makes incremental progress toward its high-level goal. We think this “science loop” encourages goal-directedness, instrumental reasoning, instrumental goals, beyond-episode goals, operational non-myopia, and indifference to stated preferences, which we jointly call “Consequentialism”. We then argue that consequentialist agents that are situationally aware are likely to become schemers (absent countermeasures) and sketch three concrete example scenarios.
[...]
Self-guided online learning: There is an online learning component to it, i.e. the model has to condense the new knowledge it learned from iterations. For example, the model could run thousands of different trajectories in parallel. Then, it could select the trajectories that it expects to make the most progress toward its goal and fine-tune itself on them. The decisions about which data to select for fine-tuning are made by the model itself with little human correction, e.g. in some form of self-play fashion. Since the problem is hard, humans perform worse than the model at selecting different rollouts, and since there is a lot of data to sift through, humans couldn’t read it all in time anyway.
2. It also matters because it means that the existing paradigm may be missing something so foundational that much of the safety research as it exists today will simply not generalize (off-distribution). They are testing the shallow within-distribution heuristic mimicking and generalization of LLMs.
It’s like doing evals on a brain that regurgitates what it’s seen, but hasn’t actually gone through a thoughtful, reflective process to bring coherence to it all. The training data might let it mimic what we’ve fed it, but it still hasn’t gone through the process of evolving its own beliefs as it engages with the world.
To me, all of this is consistent with the experiments and behaviour we see from LLMs, yet my interpretation of the results of experiments seems to be different from lots of the safety community. They seem to be looking for “scheming” and other such things, but the incoherent behaviour of LLMs seems much shallower than that, imo! (Relevant posts: The Case Against AI Control Research and Current AIs seem pretty misaligned to me).
The type of thing they are missing might mean that they don’t really understand things. And the requirement for ‘understanding’ is also so interwoven with alignment, novel R&D, pursuing long-term complex goals in changing environments, etc that existing (empirical) safety research gets itself fundamentally confused.
An LLM that is behaving ‘nice’ may be so shallow and heuristic-driven that it is effectively in a system 1-like mode despite the appearance of ‘reasoning’ and ‘thinking’. In pursuit of complex, long-term goals, we might expect that an autonomously self-trained AI would systematically remove these weak heuristics as a necessary step to succeed at these goals.
Just imagine an AI starting a complex company where it needs to maximize shareholder value and is competing with an entire economy of other AIs. The world is changing; they all have similar heuristics. The change in behaviour needs to be more fundamental for it to win.
Ultimately, I think we need to provide further clarity on the above, as I believe it has led folks to misapply their vague understanding of traditional alignment research (which many new researchers should engage with more) to existing AI models, and it may be leading AI safety research of superintelligence astray.
Further reading:
I’d like to have conversations with people who work or are knowledgeable about energy and security. Whether that’s with respect to energy grids, nuclear power plants, solar panels, etc. I’m exploring a startup idea to harden the world’s critical infrastructure against powerful AI. (I am also building a system to make formal verification more deployable at scale so that it may reduce loss of control and misuse scenarios.)
I’ve given workshops on using AIs for productivity/research to various research organizations like MATS. I’m happy to offer a bit of my time to share my expertise on that if that would make the meeting more interesting for you (or any other topics you’d like to hear my perspective on).
Context about me: I’m Jacques. I started working on technical AI safety research in January 2022. Before that, I had been engaging with AI ethics in a more personal capacity, worked as a data scientist at the Canada Energy Regulator, and earned a BSc/master’s in Physics. I’m currently based in Montreal.
Please schedule a meeting if interested (or DM if you know someone I should talk to): https://calendly.com/jacquesthibodeau/45-minute-meeting
Would there be an AI safety case for building a 100 GW data center?
Assuming it was possible to build a 100 GW data center in a non-US/China country, could there be a reasonable AI safety case for this? For example, the data center would:
Be part of an international effort
Have restrictions on frontier model training
Have mandatory independent evals
Need a world-class model-weight and cyber security
Have a high-ish % of compute reserved for safety
Compatible with pausing (frontier training)
Have contracts that enforce specific behaviour from AI labs using it
I’m asking this question because I was looking into whether it would be possible to get +100 GWs of energy into a data center by ~2030, and it seems like an option here may be offshore wind turbines in Atlantic Canada. The wind power density in that area could probably meet the demand, and an ambitious founder could probably make it happen.
Regardless of whether this specific case is viable, I wanted to keep an open mind and explore whether there is a situation where it could actually be beneficial for safety despite seeming unlikely from the outset.
In the specific situation I’m referencing, no. My impression is that they were just curious about automated research scaffolds, agent frameworks and applying interpretability for capabilities. They did not seem all that interested in alignment.
I could imagine that in practice many professors end up optimizing for prestige and more paper publications (that have nothing to do with superintelligence), whereas a younger person may not be locked into that mindset yet.