B.Eng (Mechatronics)
anithite
Karma: 557
Many problems can be solved without needing the opposing field developed. Software/hardware sandboxes especially. A theorem proven software/hardware sandbox developed with hardware that is sufficiently robust to errors that they don’t pose a problem is going to just inescapable. Once you open a causal channel out all bets are off, but the hardware and software to box something up is not impossible to build without the opposing offense.
Other areas are harder and mostly come down to impossibilities. It’s impossible to build physical protections that block all possible weapons mostly because weapons can apply energy to armor in ways that poke arbitrarily small holes in them. This relates to “bulletproof” being a misnomer, there’s only bullet resistant up to a certain caliber.
Viruses are the same way, Vaccines against what exactly? A hypothetical polymorphic virus that mutates quickly enough to escape immune system response just can’t be vaccinated against. Some problems just are ill posed.
Most things involving physical arrangements of atoms that don’t try to ensure some logical guarantees hold will just not have an end. Or their infinite technological and physics understanding metagame has one side winning (eg:build a virus the immune system can’t target regardless of vaccination) or is rock-paper-scissors like.
Agreed in the RL case, for models trained from the start with this strategy.
Existing models and models In deployment is where, I’d hesitate since a button that reports a bug and halts you gives an agent a reason to hesitate alignment depending. There are situations I can imagine where a model might not want to press the button in deployment. It finds an issue during an eval but is also very close to achieving the goal for unrelated reasons. Finish, hand in the work, don’t force the model to end the rollout/test/task as part of reporting. Task completion value shard would otherwise be competing with report-the-bug value shard. Better to not have that.
Let the model be a judge of when it should stop (I’m contaminated with info that invalidates the test) vs. continue (this bug is mostly irrelevant).
Subtle goodharting isn’t as easy to prove(eg:this test has an unintended trivial solution that gives 100% score) requiring a judge model/process.
The most egregious exploits, reading evaluator secret information or breaking sandboxes, is verifiable at the moment they’re done. The model has permissions or data it shouldn’t.
This allows replacing the judge with dumb infrastructure that accepts provable reports about permission model violations (“look at this file/network/internet access I shouldn’t have”).
Works in training evaluation and deployment.
Okay, so I saw ‘Elon Musk wants to build a mass driver on the Moon’ in another context earlier, and my first thought was to ask Claude ‘what would be the military impact of Elon Musk having a mass driver on the Moon’ because we all know who first came up with putting a mass driver on the moon (good news is that Claude said it probably wouldn’t accomplish anything because of physics), but it’s maybe the kind of thing I didn’t quite expect to have him point out first.
Somewhat disagree after looking into it in more depth.
lunar bombardment of earth is practical(LW post)
If engineering grade solutions to the problem are acceptable, a design for nanotech or biotech that vastly increases the amount of conventionally controllable matter/energy and doesn’t come with a backdoor that instantiates another intelligence seems like straight empowerment.
The messy dilemma only come up when elements of decision-making are outsourced to AI. Tooling that makes a specific baseline human comprehensible course of action possible in the first place only gives the AI control in that the AI can limit the performance of that tooling and so predictably shape what humans can do with it.
Obviously we’re not on track for that future but it is a viable solution.
If an untrustworthy boxed ASI hands you nanobot plans in the form of a generic toolkit of understandable parts then there seem like ways to audit-ably/randomly assemble those parts in a way that minimizes risk the ASI breaks out through that causal channel. Hacking human brains is a risk but the toolkit and bootstrapping workflow space itself seems somewhat benign with the right engineering discipline. Not so easy with biotech mind you given all the potential interactions between parts, but dry nanotech is fairly understandable and biotech based bootstrap workflows can be isolated enough in practice to be fairly safe.
The modern economy is quite integrated, so lots of things ultimately require quite a diverse range of raw and transformed materials. How diverse? What’s the minimally-reachable (not at tech maturity, but starting where we are now) in-principle fully closed manufacturing loop? How big can it scale, and how soon do other loops get closed?
I believe there’s designs that vastly reduce the diversity and/or quantity of expensive inputs. As an example, Boston Dynamics Atlas humanoid robots use servohydraulic joints. A small control motor similar in size to ones found in tiny quadrotors can control a lot of power. This vastly reduces power electronics and motor size/weight. There’s some compelling reasons to use servohydraulics and patents related to that are important to really optimising for growth.
If something is in-principle self-sustaining, under what conditions does that get actualised? And how voraciously?One naive perspective is that if it’s all fully grounded in the human demand economy, then it’d expand only to the point where the marginal utility of extra supply of goods output per raw materials ingested is matched by alternative uses of those raw materials.
One naive perspective is that if it’s all fully grounded in the human demand economy, then it’d expand only to the point where the marginal utility of extra supply of goods output per raw materials ingested is matched by alternative uses of those raw materials. Since marginal utility usually diminishes quite fast with increased supply, that might mean not very much expansion in practice.
There are military implications to being able to produce arbitrary quantities of stuff.
There would be immediate demand to replace/upgrade existing production equipment with things that are fully automatable.
To formalise this model a bit more:
in the legacy economy, producing a widget requires paying a bunch of people and obtaining natural resources at some cost X
after automating, most of the labor hours are gone. Amortised capital costs are lower since capex is mostly manufactured goods. The cost for a given good goes down to the cost of obtaining raw materials and remaining fraction of the labor hours.
You can imagine some fraction of the dependencies (weighted recursively) are now close to their material costs.
There’s pressure to route around dependencies that aren’t cheap yet or make them cheap.
I could see 50%-90% drop in costs of manufactured goods.
As to actual demand drivers, a lot of the economy is bottle necked on manufacturing. How many self-driving Waymos does alphabet want to build? Is there a business case for building enough to replace 10% of the first world vehicle fleet this year if production could be scaled that far?.
What are the timelines and bottlenecks to the AI components for something like this? What about the machinery and robotics components?
Manufacturing mostly doesn’t work because the people involved often don’t know if what they’re doing was good or bad. A/B testing is not universal, data isn’t being collected. There are some industries that get this right because they have to to function. TSMC couldn’t make the latest logic node if they weren’t collecting gigabytes of data from each machine, Doing automated machine learning-like tuning of control variables and running sophisticated statistical models to track correlates of good outcomes. Achieving process stability in semiconductor is just that hard.
In the rest of the economy, that almost never happens. Engineers often don’t know if a change they made was good or bad unless there is an immediate and obvious failure. They often don’t have the domain expertise to properly root cause problems. Processes accumulate mutational load as people tweak things until the stability margin is gone. Most processes operate with no extra stability margin because that’s the point where you can tell obviously that what you did broke things. Then something hard to troubleshoot goes wrong, the stability margin disappears completely and things are very very much on fire. That’s my experience in industry.
Timelines on the AI components are hard to pinpoint. In the best case scenario, AI would be able to design machinery. Learn from experience building/operating it, and then build stuff that works.
My belief is that actually working big data is enough. What we have today, properly diffused into industry will collect all the statistics/data necessary to telling when things are going right/wrong and be able to bring processes under control. Automation will start to work reliably.
Keep in mind, what we have today (claude code/codex) is enough to fix a lot of issues. Buggy pallet loading/unloading code written in python has locked up automated machining cells and caused significant downtime. That’s the sort of thing claude code can one-shot. Entire projects I’ve done involving data acquisition and interfacing with machines (EG:automating a setup process involving setting machine tool offsets) are trivial with agentic coding tools. Some things that didn’t feel worth it related to monitoring certain machine data to detect failures early are trivial now.
I’m in the process of writing this all out, but IMO, what we have right now might be enough to get us there. There are already vendors in place with boxes plugged into manufacturing cells who will start doing this.
Agreed, there are a number of areas that are super-cursed. soft bodies are one of them. one thing you’ll notice in EV manufacturing is a lot of high voltage components are stampings or rigid conductors, not flexible wires. Wiring harnesses require people. stacking/bolting rigid parts to create combined mechanical /electrical connections doesn’t.
Product design that does away with flexible wires entirely is a big part of design for manufacture and design for automated assembly.
Anytime you’re handling flexible foam sheets … why aren’t you rigidifying them? why aren’t you just producing foam in place. Refrigerators are a picture perfect example of how to build with foam, you produce it in the cavity where it’s needed, don’t handle it manually.
Looks like I have to rush out that next part.
Right now, most of the graph ties back to a single node very strongly. That node is labor.
If tomorrow we found that the productive machinery of society had doubled. Twice as many factories, trains, mines, etc. We wouldn’t have the workers to run them.
Efficiency improvements that increase productivity per unit of labor do grow the economy but there’s never a true decoupling of labor and productivity that would allow unlimited economic growth. That’s the problem.
As I said in my post, and I’ll make clearer now, this outcome relies on achieving a narrow form of STEM AGI. Heavy on the engineering. It seems likely that software will be solved sometime in the next few years. The cost of working software will trend towards zero. I expect the same to happen in the hardware space.
To put forward a crux, suppose in two years LLMs become capable of designing industrial machinery and troubleshooting automation systems the same way current LLMs seem on track to build arbitrary software and do sysadmin work. Yes, there is an enormous ecosystem of industrial processes that are necessary to build a truly closed economic subgraph. Some can be put off for a while (EG:semiconductors). But if you had a million me level engineers, I think full automation is doable. A lot of what goes wrong in the real world is really really stupid.
I have a gears level model of why full automation has failed. O-ring theory basically. I can back things up with examples. If AI can raise the competence floor in manufacturing sufficiently, full automation will work. A lot of the necessary capabilities are there already. AI can already do a lot of what I did over the last few years as a manufacturing engineer. There’s still a ways to go on visuospatial capabilities. Current LLMs can’t design anything mechanical. That’s not there yet. But I’d wager a SOTA LLM today is better than the median engineer overall.
I’ll add a little detail there. Processes and automation specifically need to be well tuned to be reliable. There’s hundreds of subtle things that can go wrong. Engineers change things. Bad engineers don’t know if their changes made things better or worse. If a process is 99.99% stable, a good change might make it 99.995% stable. A bad change might remove one of your nines. Enough bad changes can take a good process and ruin it slowly over time, until eventually the failures are obvious enough that even a bad engineer known to revert.
Corporations are really stupid. The number of times you find out that the person who build X doesn’t work here anymore and no one has any idea how it works … It’s absurd. Coding agents are currently on track to fix a lot of that on the software side. Claude code already is good enough but for the risk of catastrophic damage. Someone will put it together with disk imaging/reversion, package it as “AI agent based tech support” and it will work wonderfully.
I plan on going into the details in another post, but to restate my crux, suppose we get AI that can reliable build and tune industrial automation. If we turn that AI towards automating the production of all the machines nessesary for a stripped down economic core (machine tools, wire/sheet fabrication, electric motors, (too many more to count)) would that economic core not be able to grow independently of labor inputs?
So yes, splitting off an independent subgraph is hard, and that graph is very very big, but it’s nowhere near as big as the current economy, nor anywhere as big as all current manufacturing which is very much focused towards consumer goods.
Initially, yes, the 1/(1-x) maximum will hold. Eventually, once capabilities cross some threshold, it stops holding as the dependency chains start closing up and second order effects take over.
We’re already there for 90% of people.
Unless you strongly identify as part of “humanity” in he same way that someone from a given city cheers for their home team to win at sports, most people are doing the day to day work necessary to keep the lights on, not solving deeply important problems.
If ASI solved all the problems quickly (and doesn’t turn us into paperclips). They lose the “we’re keeping the lights on” feeling of accomplishment but realistically most people were never going to steer the ship, cure cancer, usher in an age of post scarcity etc.
They might indirectly help by say installing the flooring at the lab where one of those people works but then again, there are also humans who work at TSMC or who build datacenters. Same thing applies. They’re contributing at least up until the ASI automates the economy and moves well beyond the initial human built seed capital.
Penetrating NATO airspace with small Shahed style drones is easy and can cause a lot of damage.
Sufficient damage would lead to massive NATO retaliation though. Russian air defense systems are stretched thin as is. NATO air forces could suppress what’s left and bomb with impunity. Plausibly NATO just bombs the drone factories and some other military targets.
Building enough air defenses to stop massed drone attacks is very costly. Deterrence through retaliation is cheaper than building lots of hard kill and jamming systems.
Russia is poking NATO in various ways but a shooting war would go very badly for them.
Regarding nukes, yes, they’re scary but if Russia starts bombing NATO cities or military targets on a large scale NATO will retaliate regardless of nuclear brinksmanship.
China
The China Taiwan thing depends on China’s goals. They’re not getting TSMC. Taiwan will go scorched earth on that to say nothing of ASML remote killswitches on production equipment. If the goal is to destroy the semiconductor supply chain, why bother? Blockade accomplishes the same thing with slightly less political fallout.
Whether China has an easier or harder time with a land war to take and hold the Island has little to do with the things western powers care about (semiconductor supply chain). Yes, drones help with taking and holding Taiwan and china can make a lot of them.
Aside from Taiwan, I just don’t see China as wanting to start a land war to take a neighboring country.
Longer term
TLDR:Modern cheap drones are not stealthy in audio, thermal or radar so systems already developed by western military suppliers can take down cheap drones at low cost and in large numbers.
Hard kill anti drone systems seem likely to work. Infantry in Ukraine are using shotguns currently but automated gun turrets with fast tracking and full auto cannons firing rimless shotguns rounds should work. Air-burst electronically fused shells also exist and are cheap to produce in quantity for longer range. Working systems exist and can be purchased from western suppliers.
Ukraine is getting some Rheinmetall Skyranger 35s soon
That’s in addition to all the Gun turret + sensors + software solutions like the EOS slinger also being sent to Ukraine.Tactical rock paper scissors still applies. The anti drone system can’t stop incoming artillery rounds(though with enough range, the spotting UAVs die), direct fire from an enemy tanks or certain kinds of fast missiles. Drones aren’t the only threat on the battlefield.
Limited supply of effective hard kill systems is the issue right now. A good hard kill system will eat drone swarms all day (assuming ammo supply holds out and reloading is fast)Much longer term
Active protection systems and counter UAS converges to use fast tracking gun mounts and airburst munitions. This is also effective against infantry including infantry behind cover. Perhaps some limited indirect fire capability. Things get even worse for infantry on the future battlefield.
One side gets more bang/buck by spending to develop more cost effective weapons (UAS, UGV, etc.). Live fire exercises to get performance data + lots of simulation becomes the norm. Fewer unknowns like morale. This over determines the winner in potential conflicts.
Large military bases and civilian infrastructure generally remain hard to protect. Defender has to harden every possible target. Retaliation/deterrence is what stops others from just launching long range one way attack drones at cities.
Interceptors might solve this problem but prospect of retaliation is enough.
Terrorism
Weaponised drone technology could proliferate and solve the “deliver payload” problem but control of explosives is doing most of the work already. Terrorists can already plant remote or time detonated bombs. CF:gwern’s excellent Terrorism is not about terror post. Don’t expect huge issues there. High value targets like politicians and CEOs will have to worry about small UAVs that don’t need a large payload in addition to snipers. Mostly no impact on general public.
… Although, of course, my theory also calls into question whether people with my motives should even transition in the first place.
Wanting a specific type of attention and changing yourself to get it isn’t strictly speaking bad. Terminal goals don’t need justification though they can be incoherent, or in conflict with higher level meta goals (What you’d prefer to want if you could self-modify).Wanting to be a cute anime girl is:
somewhat incoherent since you live in reality and not the world of anime. Certain tropes just don’t work.
Somewhat incompatible with wanting to dominate social hierarchies (by being competent?).
But wanting a desirable body isn’t bad. If society forced men to wear really ugly clothes or otherwise made them repulsive in some way, that would be bad and fixing that would create a lot of value. Some things are positional goods (eg:height) but in general, more beauty and desirability seems good.Is transition technology not good enough?
If much better future transition technology was available that made changing bodies as trouble free as changing clothes would you regret it less? Are the medical consequences awful or something?
Is the social role not fitting right?
I’d expect that male dominated spaces would be very comfortable for psychologically male but physically female people. Worst case those people have to put up with flirting etc. which could be unwanted attention but … well … *points at above post talking about wanting that* … maybe you don’t want precisely that, the lived experience sucks for some reason (feeling of unearned social gains) or you want something incoherent (the full cute anime girl experience which doesn’t hold together in reality).
My opinion on the abstract issue
Females are the gender that elicits more attention/attraction. This is similar to saying that certain kinds of clothing make people more attractive/fashionable, or that being fat and smelly is unattractive. It’s not universal (straight women exist and are not attracted to other women) but given cultural norms today (men are the pursuers, men generally say yes to sex when propositioned etc.) it’s true enough in practice.
From a relationship perspective, gender is usually a rule-out criteria. If two female attracted males who aren’t too attached to their own gender transition to being female and pair up, that’s value being created. More females in male dominated spaces/niches or the opposite in female dominated to balance gender ratios seems again like value being created. Eliminating obesity would improve average relationship prospects similarly.
If “cis-by default” is common enough, then once tech exists to allow flipping genders on a whim IMO the world gets better. Mind you there are second order effects and to some degree the good being captured(attention) is a limited resource and so positional. There’s scenarios where “cis by default’ people start flipping genders, grab an unfair share of a positional good and then people attached to their gender feel pressure to flip as well. Still, IMO probably for the for the better.
To quote “The Erogamer”:
“Male sexuality is shit. It’s worthless. My junk is priced at zero dollars. Nobody wants what’s in my pants. The only person who’d ever be eager for faster access through my underwear is a gay man. And even then, I’d have to present as feminine!” There was as much bitterness in Ziquan’s voice as Maggie had ever heard there, like something was bursting out after years of festering. “Fuck this shit, I’m out. I’m going to be a hot girl and have people flirt with me.”
Disentangling cuteness and sexual market value
These are separate things. Kittens are cute but not hot as an example. But when comparing females vs. males they’re hopelessly entangled. IMO transitioning to female doesn’t really allow for getting all the cuteness without the sexual parts too. Needs proper transhuman tech to allow swapping into the body of a small cute animal or similar.
So perhaps it’s unfortunate but cute/beautiful/hot are not things you can get separately via transitioning.
General remarks
Semiconductor industry can afford to bid quite high to get the supply they need. Relevant historical example is the neon shortage where russian invasion of ukraine disrupted large air liquification/seperation plants associated with ukrainin steelworks and there was drop in Neon production. Free market did its thing, recycling, alternate suppliers etc. and nothing really happened.
Threatening to restrict critical materials matters very little for commodities like rare earths or high purity silica. Process equipment like lithography machines from ASML or other stuff from applied materials is acutally needed and can’t be replaced but high purity … stuff … can be substituted, smuggled, whatever given the need. Industry mostly won’t care and CN government can pour in more money to compensate.
Semiconductor companies won’t feel rare earth embargo
Rare earths consumption in semiconductor is quite small and they can bid higher than everyone else to secure limited supply in case of embargo. Mainly this hurts EV makers and others, not semiconductor. This is similar to jet engine turbine manufacturers not caring about cobalt prices for turbine blades contrasting again to electric vehicles where cobalt prices and scarcity drove R&D aimed at using cheaper materials in batteries.
Retaliation from USA by restricting supplies of high purity quartz is also innefective.
Main quartz consummables in semiconductor industry are Cz crucibles for silicon boule growth.
Those might already be being recycled to recover most of the quartz but that’s relatively straightforward to do in a supply crunch.
Low mass compared to polysilicon feedstock used for growing wafers themselves <5%? So tapping supply chain purifying silicon is guaranteed to be enough to make crucibles.
High purity TCS or silane can be diverted from conversion to polysilicon for making wafers to instead be made into fumed silica. This is already done for higher purity fused quartz parts like photomask substrates. Process equipment is easy to make and can be rushed in a supply crunch.
There’s also facilities that grow Quartz crystals for oscillators. Not sure about tonnage but growth is surface area limited. Making sand instead of larger crystals would perhaps 10x deposition rate and drop cycle time which is currently a few months to grow larger crystals down to weeks.
There’s limited supplies in friendly countries (Russia) and domestically.
Free market would find whatever works to get silica meeting purity standards. Easy for crucible manufacturers to test purity.
Not something that halts production, definitely an annoyance.
Relevant Claude chat.
https://claude.ai/share/fa3c6de1-bdfb-4974-8e52-1324da3ae399
Skill issue.
First, IQ 100 is only useful in ruling out easy to persuade IQ <80 people. There are likely other correlates of “easy to persuade” that depend on how the AI is doing the persuading.
Second, super-persuasion is about scalability and cost. Bribery doesn’t scale because actors have limited amounts of money. <$100 in inference and amortised training should be able persuade a substantial fraction of people.
Achieving this requires a scalable “training environment” to generate a non-goodhartable reward signal. AI trained to persuade on a large population of real users (EG:for affiliate marketing purposes) would be a super-persuader. Once a large company decides to do this at scale results will be much better than anything a hobbyist can do. Synthetic evaluation environments (EG:LLM simulations of users) can help too limited by their exploitability in ways that don’t generalise to humans.
There are no regulations against social engineering in contrast to hacking computers. Some company will develop these capabilities which can then be used for nefarious purposes with the usual associated risks like whistleblowers.
Even if a coup is meant to capture mineral wealth and the population is irrelevant, coup leaders recognize that mass murder will lead to sanctions stopping them from selling that mineral wealth. Plenty of examples of regimes that kill even low thousands of people being sanctioned.
AI that plans to take over the world does not need to trade with humans or keep them from being horrified and lashing out. Kill approximately everyone is a viable strategy and preferrable in most cases since it removes us as an intelligent adversary.
No. o3 estimates that 60% of American jobs are physical such that you would need robotics to automate them, so if half of those fell within a year, that’s quite a lot.
A lot of jobs that can’t be fully automated have sub-tasks software agents could eliminate. >30% of total labor hours might be spent in front of a computer (EG:data entry in a testing lab and all the steps needed to generate report.) That ignores email and the time savings once there is a good enough AI secretary.
AGI could eliminate almost all of that.
I’d estimate 1.7x productivity for a lab I worked at previously. Effect on employment depends on demand elasticity of course.
Prices would adjust to match supply and demand as well as acting as both supply cost and demand value signals. If no one buys the vampire drone, supply side stops production and starts dropping price to liquidate inventory, possibly with a liquidation auction.
Badly done dynamic pricing and auctions feel awful to market participants and can result in issues seen in Ebay auctions like sniping.
in my opinion, this is a poor choice of problem for demonstrating the generator/predictor simplicity gap.
If not restricted to Markov model based predictors, we can do a lot better simplicity-wise.
Simple Bayesian predictor tracks one real valued probability B in range 0...1. Probability of state A is implicitly 1-B.
This is initialized to
B=p/(p+q)as a prior given equilibrium probabilities of A/B states after many time steps.P("1")=qAis our prediction withP("0")=1-P("1")implicitly.Then update the usual Bayesian way: if “1”,
B=0(known state transition to A) if “0″,A,B:=(A*(1-p),A*p+B*(1-q)), then normalise by dividing both by the sum. (standard bayesian update discarding falsified B-->A state transition) In one step after simplification:B:=(B(1+p-q)-p)/(Bq-1)That’s a lot more practical than having infinite states. Numerical stability and achieving acceptable accuracy of a real implementable predictor is straightforward but not trivial. A near perfect predictor is only slightly larger than the generator.
A perfect predictor can use 1 bit (have we ever observed a 1) and ceil(log2(n)) bits counting n, the number of observed zeroes in the last run to calculate the perfectly correct prediction. Technically as n-->infinity this turns into infinite bits but scaling is logarithmic so a practical predictor will never need more than ~500 bits given known physics.
TLDR:I got stuck on notation
[a][b][c][...]→f(a,b,c,...). LLMs probably won’t do much better on that for now. Translating into find an unknown f(*args) and the LLMs get it right with probability ~20% depending on the model. o3-mini-high does better. Sonnet 3.7 did get it one shot but I had it write code for doing substitutions which it messes up a lot.Like others, I looked for some sort of binary operator or concatenation rule. Replacing “][” with “|” or ”,” would have made this trivial. Straight string substitutions don’t work since “[[]]” can be either 2 or “[...][1][...]” as part of a prime exponent set. The notation is the problem. Staring at string diffs would have helped in hindsight maybe.
Turning this into an unknown
f()puzzle makes it straightforward for LLMs (and humans) to solve.1 = f() 2 = f(f()) 3 = f(0,f()) 4 = f(f(f())) 12 = f(f(f()),f()) 0 = 0 -1 = -f() 19 = f(0,0,0,0,0,0,0,f()) 20 = f(f(f()),0,f()) -2 = -f(f()) 1/2 = f(-f()) sqrt(2) = f(f(-f())) 72^1/6 = f(f(-f()),f(0,-f())) 5/4 = f(-f(f()),0,f()) 84 = f(f(f()),f(),0,f()) 25/24 = f(-f(0,f()),-f(),f(f()))Substitutions are then quite easy though most of the LLMs screw up a substitution somewhere unless they use code to do string replacements or do thinking where they will eventually catch their mistake.
Then it’s ~25% likely they get it one shot. ~100% is you mention primes are involved or that addition isn’t. Depends on the LLM. o3-mini-high got it. Claude 3.7 got it one shot no hints from a fully substituted starting point but that was best of k~=4 with lots of failure otherwise. Models have strong priors for addition as a primitive and definitely don’t approach things systematically. Suggesting they focus on single operand evaluations (2,4,1/2,sqrt(2)) gets them on the right track but there’s still a bias towards addition.
None of the labs would be doing undirected drift. That wouldn’t yield improvement for exactly the reasons you suggest.
In the absence of a ground truth quality/correctness signal, optimizing for coherence works. This can give prettier answers (in the way that averaged faces are prettier) but this is limited. The inference time scaling equivalent would be a branching sampling approach that searches for especially preferred token sequences rather than the current greedy sampling approach. Optimising for idea level coherence can improve model thinking to some extent.
For improving raw intelligence significantly, ground truth is necessary. That’s available in STEM domains, computer programming tasks being the most accessible. One can imagine grounding hard engineering the same way with a good mechanical/electrical simulation package. TLDR:train for test-time performance.
Then just cross your fingers and hope for transfer learning into softer domains.
For softer domains, ground truth is still accessible via tests on humans (EG:optimise for user approval). This will eventually yield super-persuaders that get thumbs up from users. Persuasion performance is trainable but maybe not a wise thing to train for.
As to actually improving some soft domain skill like “write better english prose” that’s not easy to optimise directly as you’ve observed.
An AI lab acts through its computers. There are a small number of things the company does outside that (eg:corporate communications) that are handled directly by people, but the vast majority of what they do is set up computer systems which run internal or user workloads. User data security, model weight security, that systems do what they are intended to rather than not, all this is controlled by code that employees and sufficiently cyber capable models that have hacked the company can control.
There is no limit to the size of a rogue deployment. A rogue AI took over a cluster of conventional computers at OpenAI already. If it had been more strategic and/or competent it could have taken the company as whole and the sysadmins would have been none the wiser. Blackmail and persuasion is for worlds that don’t have hacking as a lazy path to victory.