B.Eng (Mechatronics)
anithite
Karma: 441
Skill issue.
First, IQ 100 is only useful in ruling out easy to persuade IQ <80 people. There are likely other correlates of “easy to persuade” that depend on how the AI is doing the persuading.
Second, super-persuasion is about scalability and cost. Bribery doesn’t scale because actors have limited amounts of money. <$100 in inference and amortised training should be able persuade a substantial fraction of people.
Achieving this requires a scalable “training environment” to generate a non-goodhartable reward signal. AI trained to persuade on a large population of real users (EG:for affiliate marketing purposes) would be a super-persuader. Once a large company decides to do this at scale results will be much better than anything a hobbyist can do. Synthetic evaluation environments (EG:LLM simulations of users) can help too limited by their exploitability in ways that don’t generalise to humans.
There are no regulations against social engineering in contrast to hacking computers. Some company will develop these capabilities which can then be used for nefarious purposes with the usual associated risks like whistleblowers.
Even if a coup is meant to capture mineral wealth and the population is irrelevant, coup leaders recognize that mass murder will lead to sanctions stopping them from selling that mineral wealth. Plenty of examples of regimes that kill even low thousands of people being sanctioned.
AI that plans to take over the world does not need to trade with humans or keep them from being horrified and lashing out. Kill approximately everyone is a viable strategy and preferrable in most cases since it removes us as an intelligent adversary.
No. o3 estimates that 60% of American jobs are physical such that you would need robotics to automate them, so if half of those fell within a year, that’s quite a lot.
A lot of jobs that can’t be fully automated have sub-tasks software agents could eliminate. >30% of total labor hours might be spent in front of a computer (EG:data entry in a testing lab and all the steps needed to generate report.) That ignores email and the time savings once there is a good enough AI secretary.
AGI could eliminate almost all of that.
I’d estimate 1.7x productivity for a lab I worked at previously. Effect on employment depends on demand elasticity of course.
Prices would adjust to match supply and demand as well as acting as both supply cost and demand value signals. If no one buys the vampire drone, supply side stops production and starts dropping price to liquidate inventory, possibly with a liquidation auction.
Badly done dynamic pricing and auctions feel awful to market participants and can result in issues seen in Ebay auctions like sniping.
in my opinion, this is a poor choice of problem for demonstrating the generator/predictor simplicity gap.
If not restricted to Markov model based predictors, we can do a lot better simplicity-wise.
Simple Bayesian predictor tracks one real valued probability B in range 0...1. Probability of state A is implicitly 1-B.
This is initialized to
B=p/(p+q)as a prior given equilibrium probabilities of A/B states after many time steps.P("1")=qAis our prediction withP("0")=1-P("1")implicitly.Then update the usual Bayesian way: if “1”,
B=0(known state transition to A) if “0″,A,B:=(A*(1-p),A*p+B*(1-q)), then normalise by dividing both by the sum. (standard bayesian update discarding falsified B-->A state transition) In one step after simplification:B:=(B(1+p-q)-p)/(Bq-1)That’s a lot more practical than having infinite states. Numerical stability and achieving acceptable accuracy of a real implementable predictor is straightforward but not trivial. A near perfect predictor is only slightly larger than the generator.
A perfect predictor can use 1 bit (have we ever observed a 1) and ceil(log2(n)) bits counting n, the number of observed zeroes in the last run to calculate the perfectly correct prediction. Technically as n-->infinity this turns into infinite bits but scaling is logarithmic so a practical predictor will never need more than ~500 bits given known physics.
TLDR:I got stuck on notation
[a][b][c][...]→f(a,b,c,...). LLMs probably won’t do much better on that for now. Translating into find an unknown f(*args) and the LLMs get it right with probability ~20% depending on the model. o3-mini-high does better. Sonnet 3.7 did get it one shot but I had it write code for doing substitutions which it messes up a lot.Like others, I looked for some sort of binary operator or concatenation rule. Replacing “][” with “|” or ”,” would have made this trivial. Straight string substitutions don’t work since “[[]]” can be either 2 or “[...][1][...]” as part of a prime exponent set. The notation is the problem. Staring at string diffs would have helped in hindsight maybe.
Turning this into an unknown
f()puzzle makes it straightforward for LLMs (and humans) to solve.1 = f() 2 = f(f()) 3 = f(0,f()) 4 = f(f(f())) 12 = f(f(f()),f()) 0 = 0 -1 = -f() 19 = f(0,0,0,0,0,0,0,f()) 20 = f(f(f()),0,f()) -2 = -f(f()) 1/2 = f(-f()) sqrt(2) = f(f(-f())) 72^1/6 = f(f(-f()),f(0,-f())) 5/4 = f(-f(f()),0,f()) 84 = f(f(f()),f(),0,f()) 25/24 = f(-f(0,f()),-f(),f(f()))Substitutions are then quite easy though most of the LLMs screw up a substitution somewhere unless they use code to do string replacements or do thinking where they will eventually catch their mistake.
Then it’s ~25% likely they get it one shot. ~100% is you mention primes are involved or that addition isn’t. Depends on the LLM. o3-mini-high got it. Claude 3.7 got it one shot no hints from a fully substituted starting point but that was best of k~=4 with lots of failure otherwise. Models have strong priors for addition as a primitive and definitely don’t approach things systematically. Suggesting they focus on single operand evaluations (2,4,1/2,sqrt(2)) gets them on the right track but there’s still a bias towards addition.
None of the labs would be doing undirected drift. That wouldn’t yield improvement for exactly the reasons you suggest.
In the absence of a ground truth quality/correctness signal, optimizing for coherence works. This can give prettier answers (in the way that averaged faces are prettier) but this is limited. The inference time scaling equivalent would be a branching sampling approach that searches for especially preferred token sequences rather than the current greedy sampling approach. Optimising for idea level coherence can improve model thinking to some extent.
For improving raw intelligence significantly, ground truth is necessary. That’s available in STEM domains, computer programming tasks being the most accessible. One can imagine grounding hard engineering the same way with a good mechanical/electrical simulation package. TLDR:train for test-time performance.
Then just cross your fingers and hope for transfer learning into softer domains.
For softer domains, ground truth is still accessible via tests on humans (EG:optimise for user approval). This will eventually yield super-persuaders that get thumbs up from users. Persuasion performance is trainable but maybe not a wise thing to train for.
As to actually improving some soft domain skill like “write better english prose” that’s not easy to optimise directly as you’ve observed.
O1 now passes the simpler “over yellow” test from the above. Still fails the picture book example though.
For a complex mechanical drawing, O1 was able to work out easier dimensions but anything more complicated tends to fail. Perhaps the full O3 will do better given ARC-AGI benchmark performance.
Meanwhile, Claude 3.5 and 4o fail a bit more badly failing to correctly identify axial and diameter dimensions.
Visuospatial performance is improving albeit slowly.
My hope is that the minimum viable pivotal act requires only near human AGI. For example, hack competitor training/inference clusters to fake an AI winter.
Aligning +2SD human equivalent AGI seems more tractable than straight up FOOMing to ASI safely.
One lab does it to buy time for actual safety work.Unless things slow down massively we probably die. An international agreement would be better but seems unlikely.
This post raises a large number of engineering challenges. Some of those engineering challenges rely on other assumptions being made. For example, the use of energy carrying molecules rather than electricity or mechanical power which can cross vacuum boundaries easily. Overall a lot of “If we solve X via method Y (which is the only way to do it) problem Z occurs” without considering making several changes at once that synergistically avoid multiple problems.
“Too much energy” means too much to be competitive with normal biological processes.
That goalpost should be right at the top and clearly stated instead of “microscopic machines that [are] superior”. “grey goo alone will have doubling times slower than optimised biological systems” is definitely plausible. E-coli can double in 20 minutes in nutrient rich conditions which is hard to beat. If wet nanotech doubles faster but dry nanotech can make stuff biology can’t, then use both. Dry for critical process steps and making high value products and wet for eating the biosphere and scaling up.
Newer semiconductor manufacturing processes use more energy and materials to create each transistor but those transistors use less power and run faster which makes producing them worthwhile. Dry nanotech will be a tool for making things that may be expensive but worthwhile to build like really awesome computers.
Wet nanotech (IE:biology) is plausibly the most efficient at self-replicating but notice humans use all sorts of chemical and physical processes to do other things better. Operating in space with biotech alone for example would be quite difficult.
Your image links are all of the form: http://localhost:8000/out/planecrash/assets/Screenshot 2024-12-27 at 00.31.42.png
Whatever process is generating the markdown for this, well those links can’t possibly work.
I got this one wrong too. Ignoring negative roots is pretty common for non-mathematicians.
I’m half convinced that most of the lesswrong commenters wouldn’t pass as AGI if uploaded.
This post is important to setting a lower bound on AI capabilities required for an AI takeover or pivotal act. Biology as an existence proof that some kind of “goo” scenario is possible. It somewhat lowers the bar compared to Yudkowsky’s dry nanotech scenario but still requires AI to practically build an entire scientific/engineering discipline from scratch. Many will find this implausible.
Digital tyranny is a better capabilities lower bound for a pivotal act or AI takeover strategy. It wasn’t nominated though which is a shame.
This is why I disagree with a lot of people who imagine an “AI transformation” in the economic productivity sense happening instantaneously once the models are sufficiently advanced.
For AI to make really serious economic impact, after we’ve exploited the low-hanging fruit around public Internet data, it needs to start learning from business data and making substantial improvements in the productivity of large companies.Definitely agree that private business data could advance capabilities if it were made available/accessible. Unsupervised Learning over all private CAD/CAM data would massively improve visuo-spatial reasoning which current models are bad at. Real problems to solve would be similarly useful as ground truth for reinforcement learning. Not having that will slow things down.
Once long(er) time horizon tasks can be solved though I expect rapid capabilities improvement. Likely a tipping point where AIs become able to do self-directed learning.
find technological thing: software/firmware/hardware
Connect the AI to it robustly.
For hardware, AI is going to brick it, either have lots of spares or be able to re-flash firmware at will
for software this is especially easy. Current AI companies are likely doing a LOT of RL on programming tasks in sandboxed environments.
AI plays with the thing and tries to get control of it
can you rewrite the software/firmware?
can you get it to do other cool stuff?
can the artefact interact with other things
send packets between wifi chips (how low can round trips time be pushed)
make sound with anything that has a motor
Some of this is commercially useful and can be sold as a service.
Hard drives are a good illustrative example. Here’s a hardware hacker reverse engineering and messing with the firmware to do something cool.
There is … so much hardware out there that can be bought cheaply and then connected to with basic soldering skills. In some cases, if soft-unbricking is possible, just buy and connect to ethernet/usb/power.
Revenue?
There’s a long tail (as measured by commercial value) of real world problems that are more accessible. On one end you have the subject of your article, software/devices/data at big companies. On the other, obsolete hardware whose mastery has zero value, like old hard disks. The distribution is somewhat continuous. Transaction costs for very low value stuff will set a floor on commercial viability but $1K+ opportunities are everywhere in my experience.
Not all companies will be as paranoid/obstructive. A small business will be happy using AI to write interface software for some piece of equipment to skip the usual pencil/paper --> excel-spreadsheet step. Many OEMs charge ridiculous prices for basic functionality and nickel and dime you for small bits of functionality since only their proprietary software can interface with their hardware. Reverse engineering software/firmware/hardware can be worth thousands of dollars. So much of it is terrible. AI competent at software/firmware/communication reverse engineering could unlock a lot of value from existing industrial equipment. OEMs can and are building new equipment to make this harder but industrial equipment already sold to customers isn’t so hardened.
IOT and home automation is another big pool of solvable problems. There’s some overlap between home automation and industrial automation. Industrial firmware/software complexity is often higher, but AI that learns how to reverse engineer IOT wireless microcontroller firmware could probably do the same for a PLC. Controlling a lighbulb is certainly easier than controlling a CNC lathe but similar software reverse engineering principles apply and the underlying plumbing is often similar.
Alternate POV
Science fiction. < 10,000 words. A commissioned re-write of Eliezer Yudkowsky’s That Alien Message https://alicorn.elcenia.com/stories/starwink.shtml
since it hasn’t been linked so far and doesn’t seem to be linked from the original
TLDR:autofac requires solving “make (almost) arbitrary metal parts” problem but that won’t close the loop. Hard problem is building automated/robust re-implementation of some of the economy requiring engineering effort not trial and error. Bottleneck is that including for autofac. Need STEM AI (Engineering AI mostly). Once that happens, economy gets taken over and grows rapidly as things start to actually work.
To expand on that:
“make (almost) arbitrary metal parts”can generate a lot of economic value
requires essentially giant github repo of:
hardware designs:machine tools, robots, electronics
software:for machines/electronics, non-ai automation
better CAD/CAM (this is still mostly unsolved (CF:white collar CAD/CAM workers))
AI for tricky robotics stuff
estimate:100-1000 engineer years of labor from 99th percentile engineers.
Median engineers are counterproductive as demonstrated by current automation efforts not working.
EG:we had two robots develop “nerve damage” type repetitive strain injury because badly routed wires flexed too much. If designers aren’t careful/mindful of all design details things won’t be reliable.
This extends to sub-components.
Closing the loop needs much more than just “make (almost) arbitrary metal parts”. “build a steel mill and wire drawing equipment”, is just the start. There are too many vitamins needed representing unimplemented processesA minimalist industrial core needs things like:
PCB/electronics manufacturing (including components)
IC manufacturing is its own mess
a lot of chemistry for plastics/lubricants
raw materials production (rock --> metal) and associated equipment
wire
Those in turn imply other things like:
refractory materials for furnaces
corrosion resistant coatings (nickel?, chromium?)
non-traditional machining (try making a wire drawing die with a milling machine/lathe)
ECM/EDM is unavoidable for many things
Things just snowball from there.
Efficiency improvements like carbide+coatings for cutting tools are also economically justified.
All of this is possible to design/build into an even bigger self-reproducing automated system but requires more engineer-hours put into a truly enormous git repo.
STEM AI development (“E” emphasis) is the enabler.
Addendum: simplifying the machine tools and robotsSimplifications can be made to cut down on vitamin cost of machine tools. Hydraulics really helps IMO:
servohydraulics for most motion (EG:linear machine tool axes, robots) to cut down on motor sizes and simplify manufacturing
efficiency is worse, but saves enormously on power electronics and manufacturing complexity.
Similar principles to hydraulic power steering used in cars.
Boston Dynamics ATLAS Robot uses rotary equivalent for joints (I think).
hydrostatic bearings for rotary/linear motion in machine tools.
No hardened metal parts like in ball/roller bearings
Feeding fluid without flexible hoses across linear axes via structure similar to a double ended hydraulic cylinder. Just need two sliding rod seals and a hollow rod. Hole in the middle of the center rod lets fluid into the space between the rod seals.
Both bearings and motion can run off a single shared high pressure oil supply. Treat it like electricity/compressed-air and use one big pump for lots of machines/robots.
End result: machine tools with big spindle motors and small control motors for all axes. Robots use rotary equivalent. Massive reduction in per-axis power electronics, no ballscrews, no robot joint gears.
For Linear/rotary position encoders, calibrated capacitive encoders (same as used in digital calipers) are simple and needs just PCB manufacturing. Optical barcode based systems are also attractive but require an optical mouse worth of electronics/optics per axis, and maybe glass optics too.
The key part of the Autofac, the part that kept it from being built before, is the AI that runs it.
That’s what’s doing the work here.
We can’t automate machining because an AI that can control a robot arm to do typical machinist things (EG:changing cutting tool inserts, removing stringy steel-wool-like tangles of chips, etc.) doesn’t exist or is not deployed.
If you have a robot arm + software solution that can do that it would massively drop operational costs which would lead to exponential growth.
The core problem is that currently we need the humans there.
To give concrete examples, a previous company where I worked had been trying to fully automate production for more than a decade. They had robotic machining cells with machine tools, parts cleaners and coordinate measuring machines to measure finished parts. Normal production was mostly automated in the sense that hands off production runs of 6+ hours were common, though particular cells might be needy and require frequent attention.
What humans had to do:
Changing cutting tools isn’t automated. An operator goes in with a screwdriver and box of carbide inserts 1-2x per shift.
The operators do a 30-60 minute setup to get part dimensions on size after changing inserts.
Rough machining can zero tools outside the machine since their tolerances are larger but someone is sitting there with a T-handle wrench so the 5-10 inserts on an indexable end mill have fresh cutting edges.
Intermittent problems operators handled:
A part isn’t perfectly clean when measuring. Bad measurement leads to bad adjustment and 1-2 parts are scrap
chips are too stringy and tangle up, clear the tangle every 15 mins
chips are getting between a part and fixture and messing up alignment, clean intermittently and pray.
That’s ignoring stupider stuff like:
Our measurement data processing software just ate the data for a production run so we have to stop production and remeasure 100 parts.
someone accidentally deleted some files so (same)
We don’t have the CAM done for this part scheduled to be produced so … find something else we can run or sit idle.
The employee running a CAM software workflow didn’t double-check the tool-paths and there’s a collision.
And that’s before you get to maintenance issues and (arguably) design defects in the machines themselves leading to frequent breakdowns.
The vision was that a completely automated system would respond to customer orders and schedule parts to be produced with AGVs carrying parts between operations. In practice the AGVs sta there for 10+ years because even the simple things proved nearly impossible to automate completely.
Conclusion
Despite all the problems, the automated cells were more productive than manual production (robots are consistent and don’t need breaks) and the company was making a lot of money. Not great automation is much much better than using manual operators.
It’s hard to grasp how much everything just barely works until you’ve spent a year somewhere that is trying to automate. Barely works is the standard in industry AFAIK so humans are still highly necessary.
It is, in theory, possible to automate almost everything and to build reliable machines and automation. The problem is O-ring theory of economic development. If tomorrow median IQ jumps +2SD automation would rapidly start to just work as actually good solutions are put in place. As is, organizations have to fight against institutional knowledge loss (employees leaving) just to maintain competence.
If we consider each (include,exclude) decision for (1,2,3,4,5) as a separate question, error rates are 20%-ish. Much better than random guessing. So why does it make mistakes?
If bottlenecking on data is the problem, more data in the image should kill performance. So how about a grid of 3 digit numbers (random vals in range 100...999)?
3.5 sonnet does perfectly. Perfect score answering lookup(row/col) and (find_row_col(number), find duplicates and transcription to CSV.
So this isn’t a bottleneck like human working memory. Maybe we need to use a higher resolution image so it has more tokens to “think” with? That doesn’t seem to work either for the above yellow areas thing either though.
I’m guessing this is straightforward failure to generalize. Tables of numbers are well represented in the training data (possibly synthetic data too), visual geometry puzzles, not so much. The model has learned a few visual algorithms but hasn’t been forced to generalise yet.
Root cause might be some stupid performance thing that screws up image perception the same way BPE text encoding messes up byte level text perception. I’m guessing sparse attention.
Text representations
Text representations are no panacea. Often similar problems (EG:rotate this grid) have very formatting dependent performance. Looking for sub-tasks that are probably more common in training data and composing with those (EG:rotate by composing transpose and mirror operations) allows a model to do tasks it otherwise couldn’t. Text has generalisation issues just like with images.
If Claude3.5 sonnet has pushed the frontier in tetris that would be evidence for generalisation. I predict it still fails badly.
It’s actually a lot worse than that.
Q:Which numbers are above a yellow region in the image?
GPT4o: [2,4]
GPT4: [4,5]
Claude 3.5 Sonnet: [2,5]
Claude sonnet describes the image as follows:
User: List the numbers in the above image, how are they placed in the image? What’s below them?The numbers shown in the image are:
1, 2, 3, 4, 5
These numbers are placed in a single horizontal row, appearing to be in separate cells or boxes arranged from left to right in ascending numerical order. Each number occupies its own cell, creating a simple table-like structure with 5 columns.
Below the numbers, there are cells or boxes that correspond to each number. Some of these cells are colored, while others are not. Specifically:Below number 1: The cell is white or uncolored.
Below number 2: The cell is colored yellow.
Below number 3: The cell is white or uncolored.
Below number 4: The cell is white or uncolored.
Below number 5: The cell is colored yellow.
So, out of the 5 cells below the numbers, two are colored yellow (under 2 and 5), while the rest are white or uncolored.
So these models have very very bad visuospatial abilities.
This seems like a problem that’s will be easy to solve with some synthetic training data.
General remarks
Semiconductor industry can afford to bid quite high to get the supply they need. Relevant historical example is the neon shortage where russian invasion of ukraine disrupted large air liquification/seperation plants associated with ukrainin steelworks and there was drop in Neon production. Free market did its thing, recycling, alternate suppliers etc. and nothing really happened.
Threatening to restrict critical materials matters very little for commodities like rare earths or high purity silica. Process equipment like lithography machines from ASML or other stuff from applied materials is acutally needed and can’t be replaced but high purity … stuff … can be substituted, smuggled, whatever given the need. Industry mostly won’t care and CN government can pour in more money to compensate.
Semiconductor companies won’t feel rare earth embargo
Rare earths consumption in semiconductor is quite small and they can bid higher than everyone else to secure limited supply in case of embargo. Mainly this hurts EV makers and others, not semiconductor. This is similar to jet engine turbine manufacturers not caring about cobalt prices for turbine blades contrasting again to electric vehicles where cobalt prices and scarcity drove R&D aimed at using cheaper materials in batteries.
Retaliation from USA by restricting supplies of high purity quartz is also innefective.
Main quartz consummables in semiconductor industry are Cz crucibles for silicon boule growth.
Those might already be being recycled to recover most of the quartz but that’s relatively straightforward to do in a supply crunch.
Low mass compared to polysilicon feedstock used for growing wafers themselves <5%? So tapping supply chain purifying silicon is guaranteed to be enough to make crucibles.
High purity TCS or silane can be diverted from conversion to polysilicon for making wafers to instead be made into fumed silica. This is already done for higher purity fused quartz parts like photomask substrates. Process equipment is easy to make and can be rushed in a supply crunch.
There’s also facilities that grow Quartz crystals for oscillators. Not sure about tonnage but growth is surface area limited. Making sand instead of larger crystals would perhaps 10x deposition rate and drop cycle time which is currently a few months to grow larger crystals down to weeks.
There’s limited supplies in friendly countries (Russia) and domestically.
Free market would find whatever works to get silica meeting purity standards. Easy for crucible manufacturers to test purity.
Not something that halts production, definitely an annoyance.
Relevant Claude chat.
https://claude.ai/share/fa3c6de1-bdfb-4974-8e52-1324da3ae399