anonymousaisafety

• anonymousaisafety 22 Jun 2022 23:12 UTC

  83 points

  0

  on: Air Con­di­tioner Test Re­sults & Discussion

  I take fault with your primary conclusion, for the same reasons I gave in the first thread:

  1. You claim how little adding a 2nd hose would impact the system, without analyzing the actual constraints that apply to engineers building a product that must be shipped & distributed

  2. You still neglect the existence of insulating wraps for the hose which do improve efficiency, but are also not sold with the single-hose AC system, which lends evidence to my first point—companies are aware of small cost items that improve AC system efficiency, but do not include them with the AC by default, suggesting that there is an actual price point /​ consumer market /​ confounding issue at play that prevents them doing so

  The full posts, quoted here for convenience

  I think one reason that this error occurs is that there’s a mistaken assumption that the available literature captures all institutional knowledge on a topic, so if one simply spends enough time reading the literature, they’ll have all requisite knowledge needed for policy recommendations. I realize that this statement could apply equally to your own claims here, but in my experience I see it happen most often when someone reads a handful of the most recently released research papers and from just that small sample of work tries to draw conclusions applicable that are broadly applicable to the entire field.

  Engineering claims are particularly suspect because institutional knowledge (often in the form of proprietary or confidential information held by companies and their employees) is where the difference between what is theoretically efficient and what is practically more efficient is found. It doesn’t even need to be protected information though—it can also just be that due to manufacturing reasons, or marketing reasons, or some type of incredibly aggravating constraint like “two hoses require a larger box and the larger box pushes you into a shipping size with much higher per-volume /​ mass costs so the overall cost of the product needs to be non-linearly higher than what you’d expect would be needed for a single hose unit, and that final per-unit cost is outside of what people would like to pay for an AC unit, unless you then also make drastic improvements to the motor efficiency, thermal efficiency, and reduce the sound level, at which point the price is now even higher than before, but you have more competitive reasons to justify it which will be accepted by a large enough % of the market to make up for the increased costs elsewhere, except the remaining % of the market can’t afford that higher per-unit cost at all, so we’re back to still making and selling a one-hose unit for them”.

  Concrete example while we’re on the AC unit debate—there’s a very simple way to increase efficiency of portable AC units, and it’s to wrap the hot exhaust hose with insulating duct wrap so that less of the heat on that very hot hose radiates directly back into the room you’re trying to cool. Why do companies not sell their units with that wrap? Probably for one of any of the following reasons—A.) takes up a lot of space, B.) requires a time investment to apply to the unit which would dissuade buyers who think they can’t handle that complexity, C.) would cost more money to sell and no longer be profitable at the market’s price point, D.) has to be applied once the AC unit is in place, and generally is thick enough that the unit is no longer “portable” which during market testing was viewed as a negative by a large % of surveyed people, or E.) some other equally trivial sounding reason that nonetheless means it’s more cost effective for companies to NOT sell insulating duct wrap in the same box as the portable AC unit.

  Example of an AC company that does sell an insulating wrap as an optional add-on: https://​​www.amazon.com/​​DeLonghi-DLSA003-Conditioner-Insulated-Universal/​​dp/​​B07X85CTPX

  EDIT: I want to make a meta point here, which is that I have not personally worked on ACs, but I have built & shipped multiple products to consumers, and the type of stupid examples I gave in the first AC post are not just made-up for fun. Engineers argue extensively in meetings about “how can we make product A better”, and ideas get shot down for seemingly trivial reasons that basically come down to—yes, in a vacuum, that would be better, but unfortunately, there’s a ton of existing context like how large a truck is or what parts can actually be bought off the shelf that kneecap those ideas before they leave the design room. The engineers who designed the AC were not idiots, or morons, or clowns who don’t understand thermodynamic efficiency. Engineering is about working around limitations. Those limitations do not have to be rooted in physics; society or infrastructure or consumer behavior around critical price points can all be just as real in terms of what it is feasible for a company to create. Just look at how many startups fail and the founder claims in a postmortem, “Yeah, our tech was way better, but unfortunately people wouldn’t pay 10% more for it, even though it was AMAZING compared to our competitor. We just couldn’t get them to switch.”

  EDIT 2: I’m pretty annoyed that you doubled-down on your conclusion even after admitting the actual efficiency difference was significantly less than expected, and then chose a different analysis to let you defend your original point anyway, so these edits might keep coming. Regarding market pressures, two-hose AC units do exist. Companies do sell them, and if consumers want to buy a two-hose AC unit, they can do so. But the presence of both one-hose AC units and two-hose AC units in the market tells us it is not winner-take-all and there is consumer behavior, e.g. around price or complexity, that prevents two-hose units from acquiring literally all market share. So until that changes, it will always be more rational for companies to sell one-hose AC units in addition to their two-hose AC unit, because otherwise they’d be leaving money on the floor by only servicing part of the consumer market. (EDIT 5: see also this post, which was itself a reply to AllAmericanBreakfast’s reply on this thread here)

  EDIT 3: Let’s look at your math. Outdoor temp is 85-88 F, let’s just take the average and call it 86.5 F. That’s pretty hot. I’d definitely be uncomfortable in that scenario. How cold did the AC cool the rooms? You say on low fan it was 20.6 F degrees with one hose, 22.7 F with two hoses, and then on high fan, 18.3 F with one hose, and 22.2 F with two hoses. The control was 13.1 F. Looking at the control, that gives a room temperature of ~73.4 F. That is uncomfortably hot in my opinion. I keep my room temperature around 68-70 F ish. The internet tells me that this is within the window of a “comfortable room temperature” defined as 67-75 F^[1], so I’m just a normal human, I guess. How well did the ACs accomplish that? With one hose, you got it down to ~66 F, and with two hoses, you had it down to about ~64 F. That is pretty cold in my mind—I would not set my AC that low if it actually reached that temperature. What does this mean? The one hose unit literally did the job it was designed to do. With an incredibly hot outside temperature, that resulted in an uncomfortable indoor “control” temperature, the one-hose AC was able to lower the temperature to a comfortable, ideal range, and then go below that, showing it even has margin left over. But now you’re saying that they should make the thing more expensive and optimize it for even greater efficiency because … why!? It works!

  EDIT 4: I will die on this hill. This is the problem with how the rationalist community approaches the concept of what it means to “make a rational decision” perfectly demonstrated in a single debate. You do not make a “rational decision” in the real world by reasoning in a vacuum. That is how you arrive at a hypothetically good action, but it is not necessarily feasible or possible to perform, so you always need to check your analysis by looking at real world constraints and then pick the action that is 1.) actually possible in the real world, and 2.) still has the highest expected value. Failing to do that is not more clever or more rational, it is just a bad, broken model for how an ideal, optimal agent would behave. An optimal agent doesn’t ignore their surroundings—they play to them, exploit them, use them.

  1. ^

     I averaged the following lower /​ upper temperatures.

     Wikipedia: 64-75
     www.cielowigle.com: 68-72
     www.vivint.com: 68-76
     www.provicincialheating.ca: 68-76

• anonymousaisafety 19 Jun 2022 22:53 UTC

  62 points

  in reply to: lc’s comment on: [Link-post] On Defer­ence and Yud­kowsky’s AI Risk Estimates

  Only if we pretend that it’s an unknowable question and that there’s no way to look at the limitations of a 286 by asking about how much data it can reasonably process over a timescale that is relevant to some hypothetical human-capable task.

  http://​​datasheets.chipdb.org/​​Intel/​​x86/​​286/​​datashts/​​intel-80286.pdf

  The relevant question here is about data transfers (bus speed) and arithmetic operations (instruction sets). Let’s assume the fastest 286 listed in this datasheet -- 12.5 MHz.

  Let’s consider a very basic task—say, catching a ball thrown from 10-15 feet away.

  To simplify this analysis, we are going to pretend that if we can analyze 3 image frames, in close proximity, then we can do a curve fit and calculate the ball’s trajectory, so that we don’t need need to look at any other images.

  Let’s also assume that a 320x240 image is sufficient, and that it’s sufficient for the image to be in 1-byte-per-pixel grayscale. For reference, that looks like this:

  

  With the 12.5 MHz system clock, we’re looking at 80 nanoseconds per clock cycle.

  

  We’ve got 76800 bytes per image and that’s the same as 38400 processor words (it is 16-bit).

  

  Because data transfers are one word per two processor clock cycles, and a processor clock cycle is two system clock cycles, we’ve got one word for every 4 system clock cycles. That’s 220 nanoseconds per word. Multiply that through and we’ve got ~8.5 ms to transfer each image into memory, or 25 milliseconds total for the 3 images we need.

  We can fit all 3 images into the 1 megabyte of address space allowed in real addressing mode, so we don’t need to consider virtual addresses. This is good because virtual addresses would be slower.

  In order to calculate the ball’s trajectory, we’ll need to find it in each image, and then do a curve fit. Let’s start with just the first image, because maybe we can do something clever on subsequent images.

  We’ll also assume that we can do depth perception on each image because we know the size of the ball we are trying to catch. If we didn’t have that assumption, we’d want 2 cameras and 2 images per frame.

  Most image processing algorithms are O(N) with respect to the size of the image. The constant factor is normally the number of multiplications and additions per pixel. We can simplify here and assume it’s like a convolution with a 3x3 kernel, so each pixel is just multiplied 9 times and added 9 times. This is a comical simplification because any image processing for “finding a ball” requires significantly more compute than that.

  Let’s also assume we can do this using only integer math. If it was floating point, we’d need to use the 80287^[1], and we’d pay for additional bus transfers to shuffle the memory to that processor. Also, math operations on the 80287 seem to be about 100-200 clock cycles, whereas our integer add and integer multiply are only 7 and 13 clock cycles respectively.

  

  

  So each pixel is 9 multiplies and 9 additions, which at 12.5 MHz system clock gives us 14.4 microseconds per pixel, or 1.1 seconds per image.

  Note that this is incredibly charitable, because I’m ignoring the fact that we only have 8 registers on this processor, so we’ll actually be spending a large amount of time on clock cycles just moving data into and out of the registers.

  Since we have 3 images, and if we can’t do some type of clever reduction after the first image, then we’ll have to spend 1.1 seconds on each of them as well. 1.1 seconds is a long enough period of time that I’m not sure you can make any reasonable estimate about where the ball might be in subsequent frames after a single sample, so we are probably stuck.

  That means we’re looking at 3.3 seconds before we can do that curve fit to avoid the expensive image processing work. Unless the ball is being thrown from very far away (and if it was, it wouldn’t be resolvable with this low image resolution), this system is not going to be able to react quickly enough to catch a ball thrown from 10-15 feet away.

  Conclusion

  Now is the point in this conversation where someone starts suggesting that a superhuman intelligence won’t need to look at pixels, or transfer data into memory, and they’ll somehow use algorithms that side-step basic facts about how computers work like how many clock cycles a multiplication takes. Or someone suggests that intelligence, as an algorithm, is not like looking at pixels, and reasoning about facts & logic & inferences requires far fewer math operations, so it’s not at all comparable, despite the next obvious question being, “what is the time-complexity of general intelligence?”

  1. ^

     http://​​www.bitsavers.org/​​components/​​intel/​​_dataSheets/​​80287_Data_Sheet_Feb83.pdf

  What links here?

  • Thoughts on hard­ware /​ com­pute re­quire­ments for AGI by Steven Byrnes (24 Jan 2023 14:03 UTC; 52 points)

• anonymousaisafety 14 Jun 2022 1:41 UTC

  59 points

  0

  in reply to: Adam Jermyn’s comment on: Con­tra EY: Can AGI de­stroy us with­out trial & er­ror?

  You’re making two separate assumptions here, both found in the field of computational complexity:

  1. That an AGI can simulate the physical systems perfectly, i.e. physical systems are computable processes.

  2. That an AGI can simulate the physical systems efficiently, i.e. either P = NP, or for some reason all of these interesting problems that the AGI is solving are NOT known to be isomorphic to some NP-hard problem.

  For (1), it might suffice to show that some physical system can be approximated by some algorithm, even if the true system is not known to be computable. Computability is a property of formal systems.

  It is an open question if all real world physical processes are computable. Turing machines are described using natural numbers and discrete time steps. Real world phenomena rely on real numbers and continuous time. Arguments that all physical processes are computable are based on discretizing everything down to the Planck time, length, mass, temperature, and then assuming determinism.

  This axiom tends to come as given if you already believe the “computable universe” hypothesis, “digital physics”, or “Church-Turing-Deutsch” principle is true. In those hypotheses, the entire universe can be simulated by a Turing machine.

  There are examples of problems that are currently not known to be computable in the general case. For example, the chaotic N-body problems in physics, or undecidable problems in logic like Hilbert’s tenth problem.

  But in 20 years, an AGI running on a giant compute farm might both write more efficient simulation codes and have enough compute to power them

  For (2), there is a lack of appreciation for the “exponential blowup” often seen in trying to solve many interesting problems optimally. When we say it is not efficient to solve some problem N optimally, there’s a wide range of ways to interpret that statement:

  • It’s not feasible to solve the problem on your desktop within 1 year.

  • It’s not feasible to solve the problem on the local university’s super-computing cluster within 1 year.

  • It’s not feasible to solve the problem on the USA’s best super-computing clusters within 1 year.

  • It’s not feasible to solve the problem on the world’s compute resources within 1 year.

  • It’s not feasible to solve the problem with compute resources on the scale of the solar system within 1 year.

  Here, for instance, is the famous quote by Paul Erdős regarding Ramsey numbers:

  Suppose aliens invade the earth and threaten to obliterate it in a year’s time unless human beings can find the Ramsey number for red five and blue five. We could marshal the world’s best minds and fastest computers, and within a year we could probably calculate the value. If the aliens demanded the Ramsey number for red six and blue six, however, we would have no choice but to launch a preemptive attack.

  If I had to rank the beliefs by most to least suspect, this would be near the top of the “most suspect” section. We are already aware of many problems that are computable, but not efficiently. We can see complexity classes like NP or EXPTIME in many real world problems, such as chess, Go, traveling salesman, and so forth. We also have problems that we think aren’t NP, but we haven’t found an algorithm in P yet, like graph isomorphism or integer factorization.

  Most measures we can use for ranking people by their respective intelligence, e.g. math proofs, skill at competitive games, mental math, finding solutions to problems with tricky constraints, and so forth are examples of solving problems that we know are isomorphic to NP-hard or EXPTIME problems. So it seems to follow that a hypothetical algorithm (let’s call it “general intelligence”) that could solve all of the above in some optimal manner would be problematic for a digital computer.

  When we write a program that can solve those types of problems, we have 2 options.

  The first option is to try and solve it for some optimal solution. This is where the complexity classes bite us. We don’t know how to do this for many real world problems in an efficient manner. If we did, then we’d have a proof for P=NP, and whoever came up with that proof could get a check for 1 million dollars from the Clay Institute.

  The second option, and the one we actually use, is to use an algorithm that can find any solution, and then to terminate that algorithm once a solution is “good” enough. For some algorithms, called “approximation algorithms”, we can even prove a bound on how close our “good” solution is to the optimal solution. This is called “optimality”. When we train neural networks, those neural networks learn heuristics that often find “good” solutions, but those random or probabilistic algorithms don’t have guarantees on optimality. This last point is often confused by stating that it’s possible to show a training process has resulted in an optimal configuration of a neural network. That is a subtle and different point. Saying that a neural network has the optimal configuration to decide A → B is not the same as saying that B is the optimal solution for A.

  This is basically arguing that the concept of “bounded rationality” is a necessary precondition for efficiently running the hypothetical algorithm of “general intelligence”. If true, all of the normal limits and weaknesses we see in human rationality, like biases, heuristics, or fallacies would be expected within any hypothetical AI/​ML system that demonstrates “general intelligence” under reasonable time constraints.

  How much this matters depends on how strongly you believe in the power of simulation-focused “general intelligence”. For example, if you believe that the key distinguishing factor of an AI/​ML system is that it will be able to simulate the universe and thus accurately predict outcomes with near-perfect probability, or do the same for protein folding such that the simulation of proposed protein is 100% accurate when that same protein is synthesized in real life, then you’ve implicitly subscribed to some form of this belief where the AI/​ML system returns results quickly that are not just “good”, but indeed “optimal”.

  We can look at AlphaFold—a monumental advance in protein folding—for an example of what this looks like in practice. I’ll refer to the post^[1] by Dr. AlQuraishi discussing the accuracy and limitations of AlphaFold. The important point is that while AlphaFold can predict proteins with high accuracy, it is not perfect. Sometimes the calculation of how a protein would fold is still wildly different from how it actually folds. This is exactly what we’d expect from fast, efficient, and reasonably good solvers that have no guarantee on optimality. I don’t want this point to be misunderstood. AI/​ML systems are the cause of huge leaps in our understanding and capabilities within various domains. But they have not replaced the need for actual experimentation.

  We can argue that an AI/​ML system doesn’t need to approximate solutions, and it will simply perform the inefficient computations for optimality, because it has the hardware or compute resources to do so. This is fine. We can write optimal solvers for NP-hard problems. We can also calculate how long they will take to run on certain inputs, given certain hardware. We can ask how parallelizable the algorithm is by looking at Amdahl’s law. What we’ll discover if we do these calculations is that the computation of the optimal solution for these NP-hard or EXPTIME problems becomes non-feasible to compute with any realistic amount of compute power. By realistic amount, I mean “a solar system worth of material devoted solely to computing still cannot provide optimal solutions within the time-span of a human life even on distressingly small inputs for NP-hard problems”.

  There is another argument that gets made for this belief, one where we replace the digital computer or Turing machine with a hypothetical quantum computer. I don’t have a problem with this argument either, except to note that if we accept this rewording, many downstream assumptions about the AI/​ML systems ability to scale in hardware go out of the window. The existence of the AWS cloud or billions of desktop home computers networked to the Internet is irrelevant if your AI/​ML system is only efficient when run on a quantum computer. The counter-argument is that a Turing machine can simulate a quantum computer. Yes—but can it do so efficiently? This is the whole reason for the “quantum supremacy” argument. If “quantum supremacy” is true, then a Turing machine, even if it can simulate a quantum computer, cannot do so efficiently. In that world, the AI/​ML system is still bottlenecked to a real quantum computer platform. Furthermore, a quantum computer is not magically capable of solving any arbitrary problems faster than a classical computer. While there are some problems that a quantum computer could provably solve faster than a classical computer if P != NP, such as integer factorization, it is not true that a quantum computer can solve arbitrary NP-hard problems efficiently.

  Another motivating example we can look at here are things like weather prediction or circuit simulation. In both scenarios, we have a very good understanding of the physical laws that the systems obey. We also have very good simulations for these physical laws. The reason why it’s computationally expensive for a digital computer to simulate these systems is because the real processes are continuous, while the simulated process uses some time delta (usually referred to as dT in code) between discrete steps of the simulation.

  For complicated, physical processes, the algorithm that can simulate that process can be made arbitrarily close to the real world measurement of that process, but only by reducing dT. With each reduction in dT, the algorithm takes longer to run. If we want to avoid the increase in wall time, we need to find a way to break up the problem into some type of sub-problem that can be parallelized on independent compute units, and then reduce dT on those sub-problems. If we can’t do that, because the problem has some tricky global behavior like the N-body problem alluded to above, we have to accept limitations on how accurate our results are for the given wall time we can afford.

  In electrical engineering, this is the tradeoff that an electrical engineer makes when they’re designing hardware. They can request quick & fast simulations for sanity checks during iterative development. The fast simulations might run at higher dT values, or they might use simplifying assumptions for circuit behavior. Then, prior to sending a board to be manufactured, they can run a much longer and detailed simulation. The longer solution might take hours or days to run. For this reason, the EE might only run the detailed simulation on the most critical part of a piece of hardware, and assume that the rest will work as predicted by other simulation or design work. This tradeoff means that there is a risk when the real hardware is manufactured that it does not work as intended, e.g. due to an unexpected RF effect.

  For a concrete example of this tradeoff, Nvidia recently announced that they’ve been able to speed up a certain calculation that takes 3 hours with a traditional tool to only 3 seconds using an AI/​ML system. The AI/​ML system is 94% accurate. This is a huge win for day-to-day iterative work, but you’d still want to run the most accurate simulation you can (even if it takes hours or days!) before going to manufacturing, where the lead time on manufacturing might be weeks or months.

  The final point I’d like to discuss is arguments where a superintelligent AI/​ML system can predict far into the future & therefore exploit the actions of an arbitrary number of other actors.

  I have referred to this point in discussion among friends as the “strong Psychohistory assumption”. Psychohistory is a fictional science conceptualized in the novel Foundation by Isaac Asimov. The idea of psychohistory is that a sufficiently intelligent actor can predict the actions of others (tangent: in the context of the novels it only applies to “large groups”) into the future at an accuracy that allows one to run circles around them in a competitive scenario. In Foundation, the fictional stakes play out over thousands of years. This problem, and others like it, are reducible to the same complexity concerns above. Unless “general intelligence” is able to be efficiently simulated, there’s no reason to expect a superintelligent AI/​ML system to be able to make perfect predictions into the future. Without this assumed trait, many takeoff strategies for intelligent AI/​ML systems are not sound. They rely, implicitly or explicitly, on the intelligent AI/​ML system being able to control individual humans and the society around them like a puppeteer and their puppets.

  The conclusion that I’d like for you to consider is that AI/​ML systems will produce astonishingly efficient, reasonably good solutions to arbitrary tasks—but at no point will that invalidate the need for actual, iterative experimentation in the real world, without making an expensive trade between wall time required for simulation vs the accuracy of that simulation.

  Or, to try and phrase this another way, an AI/​ML system that is 99% accurate at throwing a twenty-sided die to achieve a desired result can still roll a critical failure.

  1. ^

     https://​​moalquraishi.wordpress.com/​​2020/​​12/​​08/​​alphafold2-casp14-it-feels-like-ones-child-has-left-home

  What links here?

  • Con­tra EY: Can AGI de­stroy us with­out trial & er­ror? by Nikita Sokolsky (13 Jun 2022 18:26 UTC; 136 points)
  • anonymousaisafety's comment on [Link-post] On Defer­ence and Yud­kowsky’s AI Risk Estimates by bmg (19 Jun 2022 22:53 UTC; 62 points)
  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)
  • anonymousaisafety's comment on I No Longer Believe In­tel­li­gence to be “Mag­i­cal” by DragonGod (18 Jun 2022 0:28 UTC; 11 points)
  • anonymousaisafety's comment on Air Con­di­tioner Test Re­sults & Discussion by johnswentworth (24 Jun 2022 18:28 UTC; 4 points)

• anonymousaisafety 13 Jun 2022 0:23 UTC

  LW: 51 AF: 18

  36

  AF

  on: Godzilla Strategies

  James Mickens is writing comedy. He worked in distributed systems. A “distributed system” is another way to say “a scenario in which you absolutely will have to use software to deal with your broken hardware”. I can 100% guarantee that this was written with his tongue in his cheek.

  The modern world is built on software that works around HW failures.

  • You likely have ECC ram in your computer.

  • There are checksums along every type of data transfer (Ethernet frame check sequences, IP header checksums, UDP datagram checksums, ICMP checksums, eMMC checksums, cryptographic auth for tokens or certificates, etc).

  • An individual SSD or HDD have algorithms for detecting and working around failed blocks /​ sectors in HW.

  • There are fully redundant processors in safety-critical applications using techniques like active-standby, active-active, or some manner of voting for fault tolerance.

  • In anything that involves HW sensors, there’s algorithms like an extended Kalman filter for combining the sensor readings to a single consistent view of reality, and stapled to that are algorithms for determining when sensors are invalid because they’ve railed high, railed low, or otherwise failed in a manner that SW can detect.

  • Your phone’s WiFi works because the algorithm used for the radio is constantly working around dropouts and reconnecting to new sources as needed.

  • We can read this post because it’s sent using TCP and is automatically retransmitted as many times as needed until it’s been ACK’d successfully.

  • We can play multiplayer video games because they implement eventually consistent protocols on top of UDP.

  • Almost all computer applications implement some form of error handling + retry logic for pretty much anything involving I/​O (file operations, network operations, user input) because sometimes things fail, and almost always, retrying the thing that failed will work.

  • Large data centers have hundreds of thousands of SSDs and they are constantly failing—why doesn’t Google fall over? Because SW + HW algorithms like RAID compensate for drives dying all of the time.

  If we use one AI to oversee another AI, and something goes wrong, that’s not a recoverable error; we’re using AI assistance in the first place because we can’t notice the relevant problems without it. If two AIs debate each other in hopes of generating a good plan for a human, and something goes wrong, that’s not a recoverable error; it’s the AIs themselves which we depend on to notice problems. If we use one maybe-somewhat-aligned AI to build another, and something goes wrong, that’s not a recoverable error; if we had better ways to detect misalignment in the child we’d already have used them on the parent.

  Replace “AI” with “computer” in this paragraph and it is obviously wrong because every example here is under-specified. There is a dearth of knowledge on this forum of anything resembling traditional systems engineering or software system safety and it shows in this thread and in the previous thread you made about air conditioners. I commented as such here.

  “If we use one computer to oversee another computer, and something goes wrong, that’s not a recoverable error; we’re using computer assistance in the first place because we can’t notice the relevant problems without it.”

  Here are some examples off the top of my head where we use one computer to oversee another computer:

  1. It’s common to have one computer manage a pool of workers where each worker is another computer and workers may fail. The computer doing the management is able to detect a stalled or crashed worker, power cycle the hardware, and then resubmit the work. Depending on the criticality of this process, the “manager” might actually be multiple computers that work synchronously. The programming language Erlang is designed for this exact use-case—distributed, fault tolerance SW applications in contexts where I/​O is fallible and it’s unacceptable for the program to crash.

  2. We often use one computer program to calculate some type of path or optimal plan and it’s a very complicated program to understand, and then we use a 2nd computer program to validate the outputs from the 1st program—why do we use two programs? Because the first is inscrutable and difficult to explain, but the 2nd reads like straightforward requirements in English. In other words, it is often far easier to check a solution than it is to create a solution. The mathematically inclined will recognize this as a consequence of P vs NP problems, if P != NP.

  3. It’s common in safety-critical applications to have a fail-safe or backup using a significantly less complicated architecture—e.g. you might use <complicated system> to do <some complex task>, but for a fail-safe like “power off everything”, you might have a tiny microprocessor sitting in-line with the main power rail serving as a glorified switch. So normally the <complicated system> is driving the show, but if that starts to go sideways, the tiny microprocessor can shut it down.

  4. Almost all microprocessors have a “watchdog” built into them. A watchdog is a secondary processor that will reset the primary processor if the primary is non-responsive. Have you ever seen your Android phone mysteriously reboot when the UI locks up? That was a watchdog.

  5. The “watchdog” concept is even used in pure SW contexts, e.g. when the Android OS kills an application on your phone because it has frozen, that’s a one computer program (the OS) overseeing another (the application). Ditto for Windows & the task manager.

  6. We often use “voting” where we run the same SW on 3 or more systems and then only output to some downstream hazard if all 3 systems agree. If they don’t agree, we can fail-safe or try to recover, e.g. by power-cycling whichever system was out-of-family, or both—first try to recover, then fail-safe if that didn’t work. This is done by running code in “lockstep” on synchronized inputs, very similar to how old multiplayer RTS games used to do networking.

  7. You can buy self-checking processors that implement lockstep comparison between 2 internal cores so that whenever instructions are executed, you know that the execution occurred identically across both cores.

  These aren’t cherry-picked. This is the bread & butter of systems safety. We build complex, safe systems by identifying failure modes and then using redundant systems to either tolerate faults or to fail-safe. By focusing on the actual system, and the actual failure modes, and by not getting stuck with our head in the clouds considering a set of “all possible hypothetical systems”, it is possible to design & implement robust, reliable solutions in reality.

  To claim it is not just impossible to do that, but that it is foolhardy to even try, is the exact opposite of a safety-critical mindset.

  What links here?

  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)
  • anonymousaisafety's comment on Godzilla Strategies by johnswentworth (13 Jun 2022 0:33 UTC; 2 points)

• anonymousaisafety 21 Apr 2022 3:39 UTC

  48 points

  13

  in reply to: Eliezer Yudkowsky’s comment on: “Pivotal Act” In­ten­tions: Nega­tive Con­se­quences and Fal­la­cious Arguments

  I’m surprised that there’s not more push back in this community on the idea of a “pivotal act” even being feasible on any reasonable timeline that wouldn’t give up the game (i.e.: reveal that you have AGI and immediately get seized by the nearest state power), in the same way that there’s push back on regulation as a feasible approach.

  SUMMARY: Pivotal acts as described here are not constrained by intelligence, they’re constrained by resources and time. Intelligence may provide novel solutions, but it does not immediately reduce the time needed for implementation of hardware/​software systems in a meaningful way. Novel supply chains, factories, and machinery must first be designed by the AGI and then created by supporters before the AGI will have the influence on the world that is expected by proponents of the “pivotal act” philosophy.

  I’m going to structure this post as 2 parts.

  First, I will go through the different ideas posed in this thread as example “pivotal acts” and point out why it won’t work, specifically by looking at exceptions that would cause the “pivotal act” to not reliably eliminate 100% of adversaries.

  Then, I’ll look at the more general statement that my complaints in part 1 are irrelevant because a superhuman AGI is by definition smarter than me, and therefore it’ll do something that I can’t think of, etc.

  Part 1. Pivotal act <x> is not feasible.

  social sabotage: auto-generating mass media campaigns to shut down competitor companies by legal means, or

  This has the same issues as banning gain-of-function research. It’s difficult to imagine a mass media campaign in the US having the ability to persuade or shutdown state-sponsored AI research in another country, e.g. China.

  threats: demonstrating powerful cyber or physical or social threats, and bargaining with competitors to shut down “or else”.

  Nation states (and companies) don’t generally negotiate with terrorists. First, you’d have to be able to convince the states you can follow through on your threat (see arguments below). Second, you’d need to be able to make your threat from a position of security such that you’re not susceptible to a preemptive strike, either delivered in the form of military or law enforcement personal showing up at your facilities with search warrants OR depending on what exactly you threatened and where you are located in the world, a missile instead.

  Destroying all competing AI projects might mean that the AI took a month to find a few bugs in linux and tensorflow and create something that’s basically the next stuxnet. This doesn’t sound like that fast a takeoff to me.

  Stuxnet worked by targeting hardware/​software systems. Specifically, it targeted the hardware (the centrifuges) controlled via software (PLCs) and sent commands that would break the hardware by exceeding the design spec. The Iran network would have been air-gapped, so the virus had to be delivered on site either via a human technician performing the deployment, or via some form of social engineering like leaving a trapped USB device on the premises and hoping that an Iranian engineer would plug it into their network. Even that last vector can be circumvented by not allowing USB storage devices on computers attached to the network which is absolutely a thing that certain networks do for security. By “not allow”, I don’t mean it’s a piece of paper that says “don’t do this”, I mean that the software stack running on the computers don’t allow USB storage devices and/​or the ports are physically inaccessible.

  Let’s assume for the sake of argument that the bug already exists and we just need to exploit it. How are you delivering your exploit? Are you assuming the other AI projects are connected to the public internet? Or do you first have to assume that you can break into their VPN, or worse, somehow get into an air-gapped network? When does your exploit take effect? Is it immediately? Or is it when someone tries to run some type of experiment? If it’s immediately, then you risk early discovery when the exploit is being spread and you give your adversaries time to pull themselves from a network. If it’s triggered on some external command, then you risk the command not being received and certain exploited systems failing to be destroyed as intended. If it’s triggered by the system it has exploited, e.g. in response to some GPU usage threshold, then you run into the same issue where people will start posting online “my GPU melted after I ran a neural network, what’s going on?”

  Even the above discussion ignores the fact that Linux is not a monolithic entity and neither is Windows, or MacOS, so you probably need bugs for each OS or distro, and probably separate versions, and you’re soon looking at hundreds of different exploits all of which need to be orchestrated at the same time to avoid early detection /​ avoidance by your adversaries. Add in the need to target specific libraries and you’ve got even more exploits to deal with, but that’s still assuming that your adversaries use the public versions of libraries, vs using internal forks or private code entirely.

  This isn’t even getting into the most pressing problem of this hypothetical. Stuxnet could destroy the hardware—not just break it until the bug was removed, but actually destroy it—because it was targeting centrifuges which are things that spin at super high rates (think 1000 revolutions per second for a big piece of machinery) and were thus susceptible to oscillations that would cause the system to physically tear itself apart. How are you going to destroy the GPUs in this hypothetical? Exploit some type of bug that bricks them? That isn’t unheard of. The Amazon MMO “New World” reportedly bricked some high end graphics cards. That particular example though was traced to a hardware fault on less than 1% of RTX 3090′s created by the company EVGA, so you’d need a different exploit for the other 99%, plus the other graphics cards, plus the other manufacturers of those cards. If you can’t identity a way for your software exploit to physically break the hardware, actually break it, then at best this is just a minor speed bump. Even if you can 100% of the time nuke the attached graphics card in a computer, companies have stock of extra computers and components in storage. You aren’t destroying those, because they’re not even in a computer right now.

  Compare all of the above to Stuxnet: a single worm, designed to destroy a single hardware/​software system, with the exact target environment (the hardware, and the software) known to the creators, and it still took 2 (?) nation states to pull it off, and crucial to our discussion of pivotal acts, it was not 100% effective. The best estimate is that maybe 10% of the Iran centrifuges were destroyed by Stuxnet.

  cyber sabotage: hacking into competitors’ computer systems and destroy their data;

  See statements above about air-gaps /​ VPNs /​ etc. Pointing to anecdotal hacks of big companies doesn’t work, because for your pivotal act you need to hit 100% of adversaries. You also need to deal with backups, including backups that are off-site or otherwise unconnected to the network, which is standard practice for corporations that don’t want to care about ransomware.

  Part 2. Magic isn’t real.

  • physical sabotage: deploying tiny robotic systems that locate and destroy AI-critical hardware without (directly) harming any humans;

  • e.g., nanotechnology targeting GPUs, drones carrying tiny EMP charges, or similar.

  • “[AI lab] should just melt all the chip fabs as soon as they get AGI”

  • Alternatively, an AGI could produce nanomachines which destroy GPUs, are extremely hard to eradicate, but otherwise don’t do much of anything.

  • upload aligned people who think hard for 1000 subjective years and hopefully figure something out.

  • It seems relatively plausible that you could use a Limited AGI to build a nanotech system capable of uploading a diverse assortment of (non-brain, or maybe only very small brains) living tissue without damaging them, and that this system would learn how to upload tissue in a general way. Then you could use the system (not the AGI) to upload humans (tested on increasingly complex animals). It would be a relatively inefficient emulation, but it doesn’t seem obviously doomed to me.

  • Lightspeed bubble of hedonium. All humans are uploaded into a virtual utopia by femtobots. The sun is fully disassembled for raw materials whithin 10 minutes of you giving the order.

  • Subtly break their AI. A cyberattack that stops their AI from doing anything, and otherwise has no effect.

  These attacks all fall under a bucket I’ll call “wizardry”. In these attacks, we assume that a superintelligence can do things that defy our current understanding of physics, supply chains, factories, chip design, yield rates, etc. I don’t mean that the superintelligence is able to solve problems faster or better than a human, because that trivially follows from the definition of “superintelligence”. What I mean is that the superintelligence in these attacks is able to skip the part of the process that follows “think of a solution”—implementation. For all hardware/​software systems that I’ve worked on, coming up with a solution to a problem was probably less than 1% of the total engineering effort spent on bringing that solution into reality. The rest of the time is on implementation. Specifically, the rest of the time is spent on iterative loops of problem discovery and problem solving until you’ve designed a system that actually works in the real world.

  
  Let’s look at nanotechnology to start with, since it’s a popular example.

  Ok, so the first thing you need to do is develop working nanobots, because nanobots don’t exist. And to do that, the superhuman AGI is going to think really hard, and design the following at a minimum:

  • schematics for nanobots, including novel power supplies

  • verilog /​ vhdl /​ equivalent for processors or other integrated circuits on the nanobots

  • source code for those custom processors (or are you assuming that you’ve got an x86 compatible processor and an environment to run some embedded Linux distro like yocto or buildroot()

  • machinery to create the processors, custom ICs, and power supplies used by the nanobots

  • machinery to assemble the nanobots themselves

  • machinery to processors and nanobots

  • machinery for deploying nanobots, assuming they aren’t capable of just flying over the world to wherever they’re needed

  The nanobots need to be designed to the constraints that make “melt the GPU factory” a realistic goal, so this means the superhuman AGI needs to be considering things like: how are they programmed, what are their actuators (for doing the melting), how do they sense the world (for seeing what to melt), what is their power supply (externally powered? if so, by what? what’s that device? how is it not a failure point in this plan? if they’re internally powered, how is that battery sized or where is the energy derived?), how are they controlled, what is their expected lifetime? When you’re answering these questions, you need to reason about how much power is needed to melt a GPU factory, and then work backwards from that based on the number of nanobots you think you get into the factory, so that you’ve got the right power output + energy requirements per nanobot for the melting.

  Now, you need to actually build that machinery. You can’t use existing fabs because that would give away your novel designs, plus, these aren’t going to be like any design we have in existence since the scale you’re working with here isn’t something we’ve gotten working in our current reality. So you need to get some permits to build a factory, and pay for the factory, and bring in-house all of the knowledge needed to create these types of machines. These aren’t things you can buy. Each one of the machines you need is going to a custom design. It’s not enough to “think” of the designs, you’ll still need an army of contractors and sub-contractors and engineers and technicians to actually build them. It’s also not sufficient to try and avoid the dependency on human support by using robots or drones instead that’s not how industrial robots or drones work either. You’ll have a different, easier bootstrap problem first, but still a bootstrap problem nonetheless.

  If you’re really lucky, the AGI was able to design machinery using standard ICs and you just need to get them in stock so you can put them together in house. Under that timeline, you’re looking at 12-15 week lead times for those components, prior to the chip shortage. Now it’s as high as 52+ week lead times for certain components. This is ignoring the time that it took to build the labs and clean rooms you’ll need to do high-tech electronics work, and the time to stock those labs with equipment, where certain pieces of equipment like room-sized CNC equipment are effectively one-off builds from a handful of suppliers in the world with similar lead times to match.

  If you’re unlucky, the AGI had to invent novel ICs just for the machinery for the assembly itself, and now we get to play a game of Factorio in real life as we ask the AGI to please develop a chain of production lines starting from the standard ICs that we can acquire, up to those we need for our actual production line for the nanobots. Remember that we’ve still got 12-15 week lead times on the standard ICs.

  Tangent: You might look at Amazon and think, well, I can buy a processor and have it here next day, why can’t I get my electronic components that quickly? In a word: scale. You’re not going to need a single processor or a single IC to build this factory. You’re going to need tens of thousands. If you care about quality control on this hardware you’re developing, you might even buy an entire run of a particular hardware component to guarantee that everything you’re using was developed by the same set of machines and processes at a known point in time from a known factory.

  The next problem is that you’ll build all of the above, and then it won’t work. You’ll do a root cause analysis to figure out why, discover something you hadn’t realized about how physics works in that environment, or a flaw in some of your components (bad manufacturer, bad lot, etc), update your designs, and go through the process all over again. This is going to take time. Not weeks, but months, or worse, years. If you have to buy new components, it’s back to that 12-15 week lead time. If you want to try and avoid buying new components by cleverly desoldering the ones you have and reusing them, that is very difficult.

  You can’t avoid this process by just reading up on the existing literature of how Intel or Nvidia or some other company designs chips because that information 1.) isn’t public, and 2.) isn’t the whole story. The dirty secret of engineering documentation is that engineers hate writing documentation and all big hardware/​software engineering projects are carried by the collective knowledge of the organization more so than whatever token documentation got produced as needed for interfacing with an external supplier. Example: Rocketdyne F-1 rocket engines.

  A typical design document for something like the F-1, though, was produced under intense deadline pressure and lacked even the barest forms of computerized design aids. Such a document simply cannot tell the entire story of the hardware. Each F-1 engine was uniquely built by hand, and each has its own undocumented quirks. In addition, the design process used in the 1960s was necessarily iterative: engineers would design a component, fabricate it, test it, and see how it performed. Then they would modify the design, build the new version, and test it again. This would continue until the design was “good enough.” [1]

  During the time that you spend iterating in the real world to get to working designs, people are going to wonder about your massive factory, and your technicians are going to get drunk at a bar and brag about how they’re developing cutting edge nanobots, and your competitors will realize you’re doing something very novel, very odd, and very difficult to explain.

  This is going to cost you literal billions (with a B) of dollars in hardware costs between the supplies you need to buy, the ICs you’ll have on order, the custom machinery you’ll be purchasing to build your own machines, etc. So there’s another prerequisite here of either 1.) you are Apple /​ Google /​ Microsoft, or 2.) you’re going to ask the AGI to make a bunch of money on the stock market. I actually believe option 2 is fairly achievable, e.g. see RenTech and the Medallion fund. That fund has averaged annual returns of 70% from 1994 through 2014. However, that’s still a timescale of years and significant seed money (millions of dollars) before you’ll have enough cash on hand to bankroll all of this R&D, unless you get external investment, but to get this much external investment you’ll have to 1.) find someone with billions of dollars, and 2.) convince them that you have AGI, 3.) swear them to secrecy, and 4.) hope that they don’t do some shenanigans like poaching your employees or having the government rain down regulation on you as a stall tactic while they develop their own AGI or try to steal yours.

  The likely result here is an arms race where your competitors try to poach your employees ($2 million /​ year?) or other “normal” corporate espionage to understand what’s going on. Example: When Google sued Uber for poaching one of their top self-driving car engineers.

  Tangent: If you want the AGI to be robust to government-sponsored intervention like turning off the grid at the connection to your factory, then you’ll need to invest in power sources at the factory itself, e.g. solar /​ wind /​ geothermal /​ whatever. All of these have permit requirements and you’ll get mired in bureaucratic red tape, especially if you try to do a stable on-site power source like oil, natural gas, or worse nuclear. Energy storage isn’t that great at the moment, so maybe that’s another sub-problem or the AGI to solve first as a precursor to all of these other problems, so that it can run on solar power alone.

  You might think that the superhuman AGI is going to avoid that iterative loop by getting it right on the very first time. Maybe we’ll say it’ll simulate reality perfectly, so it can prove that the designs will work before they’re built, and then there’s only a single iteration needed.

  Let’s pretend the AI only needs one attempt to figure out working designs: Ok, the AGI perfectly simulates reality perfectly. It still doesn’t work the first time because your human contractors miswired some pins during assembly, and you still need to spend X many months debugging and troubleshooting and rebuilding things until all of the problems are found and fixed. If you want to avoid this, you need to add a “perfect QA plan for all sub-components and auditing performed at all integration points” to the list of things that the AGI needs to design in advance, and pair it with “humans that can follow a QA plan perfectly without making human mistakes”.

  On the other hand: The AGI can only simulate reality perfectly if we had a theory of physics that could do so, which we don’t. The AGI can develop their own theory, just like you and I could do so, but at some point the theorizing is going to hit a wall where there are multiple possible solutions, and the only way to see which solution is valid in our reality is to run a test, and in our current understanding of physics, the tests we know how to run involve constructing increasing elaborate colliders and smashing together particles to see what pops out. While it is possible that there exists another path that does not have a prerequisite of “run test on collider”, you need to add that to your list of assumptions, and you might as well add “magic is real”. Engineering is about tradeoffs or constraints. Constraints like mass requirements given some locomotion system, or energy usage given some battery storage density and allowed mass, or maximum force an actuator can provide given the allowed size of it to fit inside of the chassis, etc. If you assume that a superhuman AGI is not susceptible to constraints anymore, just by virtue of that superhuman intelligence, then you’re living in a world just as fictional as HPMOR.
  

  What links here?

  • anonymousaisafety's comment on Con­tra EY: Can AGI de­stroy us with­out trial & er­ror? by Nikita Sokolsky (13 Jun 2022 22:01 UTC; 39 points)
  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)

• anonymousaisafety 14 Apr 2023 22:13 UTC

  47 points

  35

  in reply to: [DEACTIVATED] Duncan Sabien’s comment on: Moder­a­tion notes re: re­cent Said/​Dun­can threads

  @Duncan_Sabien I didn’t actually upvote @clone of saturn’s post, but when I read it, I found myself agreeing with it.

  I’ve read a lot of your posts over the past few days because of this disagreement. My most charitable description of what I’ve read would be “spirited” and “passionate”.

  You strongly believe in a particular set of norms and want to teach everyone else. You welcome the feedback from your peers and excitedly embrace it, insofar as the dot product between a high-dimensional vector describing your norms and a similar vector describing the criticism is positive.

  However, I’ve noticed that when someone actually disagrees with you—and I mean disagreement in the sense of “I believe that this claim rests on incorrect priors and is therefore false.”—I have been shocked by the level of animosity you’ve shown in your writing.

  Full disclosure: I originally messaged the moderators in private about your behavior, but I’m now writing this in public because in part because of your continued statements on this thread that you’ve done nothing wrong.

  I think that your responses over the past few days have been needlessly escalatory in a way that Said’s weren’t. If we go with the Socrates metaphor, Said is sitting there asking “why” over and over, but you’ve let emotions rule and leapt for violence (metaphorically, although you then did then publish a post about killing Socrates, so YMMV).

  There will always be people who don’t communicate in a way that you’d prefer. It’s important (for a strong, functioning team) to handle that gracefully. It looks to me that you’ve become so self-convinced that your communication style is “correct” that you’ve taken a war path towards the people who won’t accept it—Zack and Said.

  In a company, this is problematic because some of the things that you’re asking for are actually not possible for certain employees. Employees who have English as a second language, or who come from a different culture, or who may have autism, all might struggle with your requirements. As a concrete example, you wrote at length that saying “This is insane” is inflammatory in a way that “I think that this is insane” wouldn’t be—while I understand and appreciate the subtlety of that distinction, I also know that many people will view the difference between those statements as meaningless filler at best. I wrote some thoughts on that here: https://​​www.lesswrong.com/​​posts/​​9vjEavucqFnfSEvqk/​​on-aiming-for-convergence-on-truth?commentId=rGaKpCSkK6QnYBtD4

  I believe that you are shutting down debates prematurely by casting your peers as antagonist towards you. In a corporate setting, as an engineer acquires more and more seniority, it becomes increasingly important for them to manage their emotions, because they’re a role model for junior engineers.

  I do think that @Said Achmiz can improve their behavior too. In particular, I think Said could recognize that sometimes their posts are met with hostility, and rather than debating this particular point, they could gracefully disengage from a specific conversation when they determine that someone does not appreciate their contributions.

  However, I worry that you, Duncan, are setting an increasingly poor example. I don’t know that I agree with the ability to ban users from posts. I think I lean more towards “ability to hide any posts from a user” as a feature, more than “prevent users from commenting”. That is to say, I think if you’re triggered by Said or Zack, then the site should offer you tools to hide those posts automatically. But I don’t think that you should be able to prevent Said or Zack from commenting on your posts, or prevent other commentators from seeing that criticism. In part, I agree strongly (and upvoted strongly) with @Wei_Dai’s point elsewhere in this thread that blocking posters means we can’t tell the difference between “no one criticized this” and “people who would criticize it couldn’t”, unless they write their own post, as @Zack_M_Davis did.

• anonymousaisafety 13 Jun 2022 22:01 UTC

  39 points

  on: Con­tra EY: Can AGI de­stroy us with­out trial & er­ror?

  Related reading, if you’re interested—I tried to make these same arguments a few months ago:

  • pivotal acts aren’t constrained by intelligence

  • there are long lead times for production of anything

  • building the very first nanobot still needs bootstrapping

• anonymousaisafety 19 Jun 2022 20:33 UTC

  29 points

  in reply to: lc’s comment on: [Link-post] On Defer­ence and Yud­kowsky’s AI Risk Estimates

  the rest of the field has come to regard Eliezer as largely correct

  It seems possible to me that you’re witnessing a selection bias where the part of the field who disagree with Eliezer don’t generally bother to engage with him, or with communities around him.

  It’s possible to agree on ideas like “it is possible to create agent AGI” and “given the right preconditions, AGI could destroy a sizeable fraction of the human race”, while at the same time disagreeing with nearly all of Eliezer’s beliefs or claims on that same topic.

  That in turn would lead to different beliefs for what types of approach will work, which could go a long way towards explaining why so many AI research labs are not pursuing ideas like pivotal acts or other Eliezer-endorsed solutions.

  For example, the linked post didn’t use this quote when discussing Eliezer’s belief that intelligence doesn’t require much compute power, but as recently as 2021 (?) he said

  Well, if you’re a superintelligence, you can probably do human-equivalent human-speed general intelligence on a 286, though it might possibly have less fine motor control, or maybe not, I don’t know. [source]

  “or maybe not, I don’t know” is doing a lot of work in covering that statement.

  What links here?

  • anonymousaisafety's comment on The “Out­side the Box” Box by Eliezer Yudkowsky (23 Mar 2023 21:59 UTC; 2 points)

• anonymousaisafety 16 Jun 2022 22:58 UTC

  27 points

  0

  on: Don’t Over-Op­ti­mize Things

  My reply is focused on this specific statement:^[1]

  sometimes trying to [over] optimize can lead to worse outcomes

  There is something known as the performance /​ robustness stability tradeoff in controls theory. Controls theory^[2] is the study of dynamic (e.g. autonomous) systems, and I have no idea why it is not more commonly cited on this forum.

  The mathematical description of this gets a little bit unwieldy, so I’m going to simplify.
  Note that everything I’m about to say is discussing ideal systems and real systems are actually worse.

  Higher performance systems are less stable than lower performance systems. For an intuitive idea of why this might be the case, consider the example of a system where you want to keep some variable to some setpoint, like temperature in a room. If you slowly control the error as it occurs, you’ll end up with what is called a proportional error response.

  Consider the following picture of a step response.^[3]

  

  You might want to be faster, so you might try to do something clever and add a term to the controller for how quickly the temperature is changing. Now you have a proportional-derivative error. There is a tradeoff. By making the system more responsive, we’ve made it less stable. It is now possible for our controller to oscillate out of control.

  Here is picture of various step responses.^[4] Take note of the unstable and marginally stable states.

  

  The phrases to know are “gain margin” and “phase margin”.

  Gain margin is about how robust your system is in magnitude—if the error is larger or smaller, how well does the system correct that error? You can think of gain margin as you’re trying to keep a bouncing spring in place by hitting it with a hammer, and if you hit it too hard, it’ll oscillate in a way you don’t like.

  Phase margin is about how robust your system is in time. To continue the previous example, the idea of phase margin is capturing the reality that you’re controlling some external actuator, i.e. the hammer, and there’s some delay between when you need to swing and when the swing actually occurs, and if that delay is too large, the system will respond differently. In fact, if that delay is just the right frequency, it’ll actually add energy into the system and drive it unstable.^[5]

  The first controller I described above is called a PID (proportional, integral, derivative) controller and I gave examples of a P controller and PD controller. Normally you use a PI controller because the integral term drives the error to zero over time, which is necessary when your system has friction or some dead-zone or other bias that prevents a pure proportional controller from working.

  There are various fancy controllers you’ll hear about, like feed-forward, or MPC (“model predictive control”). The performance /​ robustness stability tradeoff applies to all of them. It is an iron law. It does not matter how fancy your controller gets. The fancier you make the controller, the more susceptible it is to going unstable. Basically, increasingly complicated controllers get performance by baking in assumptions about the physical world into the control loop. These assumptions are things like, how much bias is in the system, or quickly can an error change, what is the largest step response we might need to achieve. If those assumptions match reality, the controller will have very high performance and seem very stable. But if any of those assumptions are violated, that fancy controller might immediately go unstable. That’s the price you pay for performance.

  One way to think about this is the following thought experiment. You have a tradeoff between how well you can track a setpoint and how well you can reject disturbances. If you make it very difficult to knock a system off a setpoint, it’ll reject disturbances well. However, a change in that setpoint might also look like a disturbance, and the system will be similarly sluggish to respond.

  For real systems, a lot of the design considerations are going to be around giving yourself enough gain and phase margin so that you’ve got an envelope of safety around the testing you’re able to do. Think of it like the factor of safety used in construction. The bridge is built to be say, 5x stronger than it needs to be. For this reason, and contrary to claims made on this forum, real systems are not engineered to the theoretical limit of performance or “efficiency”.
  

  1. ^

     Bob is over-optimizing towards higher performance (“faster arrival”) solutions that have increasingly higher risks of catastrophic failure (“death due to crashes from violating speed limits”).

  2. ^

     https://​​en.wikipedia.org/​​wiki/​​Control_theory

  3. ^

     https://​​www.goddardconsulting.ca/​​pid-control.html

  4. ^

     https://​​www.sciencedirect.com/​​science/​​article/​​pii/​​B9780750646376500137

  5. ^

     Lack of phase margin is also what stops “I will simply control the robot over the network” ideas from working—if the phase margin is insufficient, the delay incurred over the network will make it impossible for the remote actuators to be controlled in response to disturbances with any degree of accuracy.

  What links here?

  • anonymousaisafety's comment on Air Con­di­tioner Test Re­sults & Discussion by johnswentworth (22 Jun 2022 23:12 UTC; 83 points)
  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)
  • Owen Cotton-Barratt's comment on Don’t Over-Op­ti­mize Things by Owen Cotton-Barratt (EA Forum; 17 Jun 2022 9:46 UTC; 2 points)

• anonymousaisafety 23 Apr 2022 6:38 UTC

  27 points

  9

  in reply to: Ricardo Meneghin’s comment on: “Pivotal Act” In­ten­tions: Nega­tive Con­se­quences and Fal­la­cious Arguments

  Tangent: I don’t think I understand the distinction you’ve made between “AI designed nanotechnology” and “human technology”. Human technology already includes “actual nanotechnology”, e.g. nanolithography in semiconductor production.

  I agree that if the AGI gives us a blueprint for the smallest self-replicating nanobot that we’ll need to bootstrap the rest of the nanobot swarm, all we have to do is assemble that first nanobot, and the rest follows. It’s very elegant.

  We still need to build that very first self-replicating nanobot though.

  We can either do so atom-by-atom with some type of molecular assembler like the ones discussed in Nanosystems, or we can synthesize DNA and use clever tricks to get some existing biology to build things we want for us, or maybe we can build it from a process that the AGI gives us that only uses chemical reactions or lab/​industrial production techniques.

  If we go with the molecular assembler approach, we need to build one of those first, so that we can build the first self-replicating nanobot. This is effectively the same argument I made above, so I’m going to skip it.

  If we go with the DNA approach, then the AGI needs to give us that DNA sequence, and we have to hope that we can create it in a reasonable time despite our poor yield rate and time for on DNA synthesis on longer sequences. If the sequence is too long, we might be in a place where we first need to ask the AGI to design new DNA synthesis machines, otherwise we’ll be stuck. In that world, we return to my arguments above. In the world where the AGI gave us a reasonably length DNA sequence, say the size of a very small cell or something, we can continue. The COVID-19 vaccine provides an example of how this goes. We have an intelligent entity (humans) writing code in DNA, synthesizing that DNA, converting it to mRNA, and getting a biological system (human cells) to read that code and produce proteins. Humanity has these tools. I am not sure why we would assume that the company that develops AGI has them. At multiple steps in the chain of what Pfizer and Moderna did to bring mRNA vaccines to market, there are single vendor gatekeepers who hold the only tooling or processes for industrial production. If we assume that you have all of the tooling and processes, we still need to talk about cycle times. I believe Pfizer aimed to get the cycle time (raw materials → synthesized vaccines) for a batch of vaccine down from 15 weeks to 8 weeks. This is an incredibly complex, amazing achievement—we literally wrote a program in DNA, created a way to deliver it to the human body, and it executed successfully in that environment. However, it’s also an example of the current limitations we have. Synthesizing from scratch the mRNA needed to generate a single protein takes >8 weeks, even if you have the full assembly line figured out. This will get faster in time, and we’ll get better at doing it, but I don’t see any reason to think that we’ll have some type of universal /​ programmable assembly line for an AGI to use anytime soon.

  If we go with a series of chemical reactions/​lab/​industrial production techniques, we need to build clean rooms and labs and vacuum chambers and whatever else is used to implement whatever process the AGI gives us for synthesizing the nanobots. Conceptually this is the simplest idea for how you could get something to work quickly. If the AGI gave you a list of chemicals, metals, biological samples and a step-by-step process of how to mix, drain, heat, sift, repeat, and at the end of this process you had self-replicating nanobots, that would be pretty cool. This is basically taking evolution’s random walk from a planetary petri dish to the life we see today and asking, “could an AGI shorten the duration from a billion years of random iterative development into mere weeks of some predetermined process to get the first self-replicating nanobots?” The problem with programming is that interpreting code is hard. Anything that can interpret the nanobot equivalent of machine code, like instructions for how and where to melt GPU factories, is going to be vastly more complex than the current state-of-the-heart R&D being done by any human lab today. I don’t see a way where this doesn’t reduce to the same Factorio problem I’ve been describing. We’ll first need to synthesize A, so that we can synthesize B, so that we can synthesize C, so that we can synthesize D, and each step will require novel setups and production lines and time, and at the end of it we’ll have a sequence of steps that looks an awful lot like a molecular assembly line for the creation of the very first self-replicating nanobots.

  The hypothetical world(s) where these types of constraints aren’t problems for a “pivotal act” are world(s) where the AGI can give us a recipe for the self-replicating nanobots that we can build in our living room at home with a pair of tweezers and materials from Amazon. The progression of human technology over the past ~60 years in the fields of nano-scale engineering or synthetic biology has been increasingly elaborate, complex, time-consuming, and low-yield processes or lab equipment to replicate the simplest structures that life produces ad-hoc. I am certain this limitation will be conquered, and I’m equally certain that AI/​ML systems will be instrumental in doing so, but I have no evidence to rationally conclude that there’s not a mountain of prerequisite tools still remaining for humanity to build before something like “design anything at any scale” capabilities are generally available in a way that an AGI could make use of them.

  Tangent: If we’re concerned about destroying the world, deliberately building self-replicating nanobots that start simple but rapidly assemble into something arbitrarily complex from the whims of an AGI seems like a bad idea, which is why my original post was focused on a top-down hardware/​software systems engineering process where the humans involved could presumably understand the plans, schematics, and programming that the AGI handed to them prior to the construction and deployment of those nanobots.

  What links here?

  • Con­tra EY: Can AGI de­stroy us with­out trial & er­ror? by Nikita Sokolsky (13 Jun 2022 18:26 UTC; 136 points)
  • anonymousaisafety's comment on Con­tra EY: Can AGI de­stroy us with­out trial & er­ror? by Nikita Sokolsky (13 Jun 2022 22:01 UTC; 39 points)
  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)

• anonymousaisafety 22 Apr 2022 18:48 UTC

  26 points

  3

  in reply to: Mitchell_Porter’s comment on: “Pivotal Act” In­ten­tions: Nega­tive Con­se­quences and Fal­la­cious Arguments

  The latter. I don’t see any reason why a superintelligent entity would not be able to take over the world or destroy it or dismantle it into a Dyson swarm. The point I am trying to make is that the tooling and structures that a superintelligent AGI would need to act autonomously in that way do not actually exist in our current world, so before we can be made into paperclips, there is a necessary period of bootstrapping where the superintelligent AGI designs and manufactures new machinery using our current machinery. Whether it’s an unsafe AGI that is trying to go rogue, or an aligned AGI that is trying to execute a “pivotal act”, the same bootstrapping must occur first.

  Case study: a common idea I’ve seen while lurking on LessWrong and SSC/​ACT for the past N years is that an AGI will “just” hack a factory and get it to produce whatever designs it wants. This is not how factories work. There is no 100% autonomous factory on Earth that an AGI could just take over to make some other widget instead. Even highly automated factories are 1.) highly automated to produce a specific set of widgets, 2.) require physical adjustments to make different widgets, and 3.) rely on humans for things like input of raw materials, transferring in-work products between automated lines, and the testing or final assembly of completed products. 3D printers are one of the worst offenders in this regard. The public perception is that a 3D printer can produce anything and everything, but they actually have pretty strong constraints on what types of shapes they can make and what materials they can use, and usually require multi-step processes to avoid those constraints, or post-processing to clean up residual pieces that aren’t intended to be part of the final design, and almost always a 3D printer is producing sub-parts of a larger design that still must be assembled together with bolts or screws or welds or some other fasteners.

  So if an AGI wants to have unilateral control where it can do whatever it wants, the very first prerequisite is that it needs to make a futuristic, fully automated, fully configurable, network-controlled factory exist—which then needs to be built with what we have now, and that’s where you’ll hit the supply constraints I’m describing above for things like lead times on part acquisition. The only way to reduce this bootstrapping time is to have this stuff designed in advance of the AGI, but that’s backwards from how modern product development actually works. We design products, and then we design the automated tooling to build those products. If you asked me to design a factory that would be immediately usable by a future AGI, I wouldn’t know where to even start with that request. I need the AGI to tell me what it wants, and then I can build that, and then the AGI can takeover and do their own thing.

  A related point that I think gets missed is that our automated factories aren’t necessarily “fast” in a way you’d expect. There’s long lead times for complex products. If you have the specialized machinery for creating new chips, you’re still looking at ~14-24 weeks from when raw materials are introduced to when the final products roll off the line. We hide that delay by constantly building the same things all of the time, but it’s very visibly when there’s a sudden demand spike—that’s why it takes so long before the supply can match the demand for products like processors or GPUs. I have no trouble with imagining a superintelligent entity that could optimize this and knock down the cycle time, but there’s going to be physical limits to these processes and the question is can it knock it down to 10 weeks or to 1 week? And when I’m talking about optimization, this isn’t just uploading new software because that isn’t how these machines work. It’s designing new, faster machines or redesigning the assembly line and replacing the existing machines, so there’s a minimum time required for that too before you can benefit from the faster cycle time on actually making things. Once you hit practical limits on cycle time, the only way to get more stuff faster is to scale wide by building more factories or making your current factories even larger.

  If we want to try and avoid the above problems by suggesting that the AGI doesn’t actually hack existing factories, but instead it convinces the factory owners to build the things it wants instead, there’s not a huge difference—instead of the prerequisite here being “build your own factory”, it’s “hostile takeover of existing factory”, where that hostile takeover is either done by manipulation, on the public market, as a private sale, or by outbidding existing customers (e.g. have enough money to convince TSMC to make your stuff instead of Apple’s), or with actual arms and violence. There’s still the other lead times I’ve mentioned for retooling assembly lines and actually building a complete, physical system from one or more automated lines.

  What links here?

  • What does it take to defend the world against out-of-con­trol AGIs? by Steven Byrnes (25 Oct 2022 14:47 UTC; 194 points)
  • Con­tra EY: Can AGI de­stroy us with­out trial & er­ror? by Nikita Sokolsky (13 Jun 2022 18:26 UTC; 136 points)
  • What does it take to defend the world against out-of-con­trol AGIs? by Steven Byrnes (EA Forum; 25 Oct 2022 14:47 UTC; 43 points)
  • anonymousaisafety's comment on Con­tra EY: Can AGI de­stroy us with­out trial & er­ror? by Nikita Sokolsky (13 Jun 2022 22:01 UTC; 39 points)
  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)
  • Steven Byrnes's comment on Trans­for­ma­tive AGI by 2043 is <1% likely by Ted Sanders (8 Jun 2023 14:44 UTC; 6 points)
  • Steven Byrnes's comment on High-level hopes for AI alignment by HoldenKarnofsky (15 Dec 2022 20:13 UTC; 3 points)

• anonymousaisafety 20 Jun 2022 10:37 UTC

  25 points

  4

  on: Let’s See You Write That Cor­rigi­bil­ity Tag

  I worry that the question as posed is already assuming a structure for the solution—“the sort of principles you’d build into a Bounded Thing meant to carry out some single task or task-class and not destroy the world by doing it”.

  When I read that, I understand it to be describing the type of behavior or internal logic that you’d expect from an “aligned” AGI. Since I disagree that the concept of “aligning” an AGI even makes sense, it’s a bit difficult for me to reply on those grounds. But I’ll try to reply anyway, based on what I think is reasonable for AGI development.

  In a world where AGI was developed and deployed safely, I’d expect the following properties:

  1. Controlled environments.
  2. Controlled access to information.
  3. Safety-critical systems engineering.
  4. An emphasis on at-rest encryption and secure-by-default networking.
  5. Extensive logging, monitoring, interpretability, and circuit breakers.
  6. Systems with AGI are assumed to be adversarial.

  Let’s stop on the top of the mountain and talk about (6).

  Generally, the way this discussion goes is we discuss how unaligned AGI can kill everyone, and therefore we need to align the AGI, and then once we figure out how to align the AGI, problem solved, right?

  Except someone then points out that, well, other people might create unaligned AGI, and then that will kill everyone, so that’s awkward.

  Also maybe the team that thought they aligned the AGI actually didn’t, their proof had a mistake, whoops.

  Or they had a formally proven proof, but they deployed it to a general purpose computer, implemented zero hardware redundancy, and then a bit flip caused the system to kill everyone anyway, whoops.

  So normally we don’t discuss the last 2 failure modes, because it’s a bit awkward for the discussion of alignment, and we instead talk about how the deployment of the aligned AGI is going to go totally fine, and then we’ll just do a little pivotal act.

  How does AGI kill everyone in these stories? Almost always, the answer is nanotechnology. If the answer isn’t nanotechnology, it’s hacking into <hackable thing>.

  When you try to suggest that maybe the AGI shouldn’t be able to hack into <hackable thing>, you’re told that’s insane, because that would be boxing the AGI, and we know that boxing doesn’t work.

  The world I am going to describe might not be our world, but it also isn’t Eliezer’s world.
  We are separated by a few critical assumptions—and it starts with hardware.

  1. Controlled environments.

  In a world where we cared about AGI being developed safely, here’s a fun idea: you know all that TPU hardware that Google designed, or the other custom AI/​ML chips that companies are working? It’s all just glorified GPUs, except the Mythic AMP is kind of cool, since it’s an actual analog compute platform for running neural networks, like an analog FPGA. What if AGI wasn’t developed to run on general-purpose consumer hardware? What if it can’t be run on general-purpose hardware? What if the general intelligence algorithm is computationally inefficient enough that the path to AGI actually looks like incredibly specialized, heavily parallelized, wide processors with low clock rates, large caches, and fast memory access.

  In this world, the general intelligence algorithm needs hardware designed for it because no amount of pipelining with a traditional CPU, or even a GPU, is able to run that code efficiently.

  This is a world where the AGI machine code must be flashed to the ROM via an external debugger. There is no ability to flash or update the code from within the hardware/​software system. The AGI cannot update itself. We do not design, deploy, or use self-improving systems.

  This is a world where the hardware used for AGI development uses a memory-protection unit so that memory accesses to the ROM containing the AGI machine code trigger a hardware exception. It is not possible to read the ROM from within the hardware/​software system. The AGI cannot inspect its own source code.

  This is a world where the AGI has RAM, but no accessible non-volatile memory. If it loses power, poof, gone. The AGI cannot maintain, or persist, any type of internal or “custom” state between resets. Anything the AGI needs to know is either baked into the image flashed to the ROM, or it’s transferred into the AGI’s RAM after power-on.

  This is a world where we deliberately leave performance on the floor because what we want from the system is consistency, predictability, and some minimum level of effectiveness above human capabilities—even if we hypothetically could run an AGI system at some 1000x “speed” (whatever that means) relative to a human mind, doesn’t mean that you’d have to do so. If you’re worried about not being able to react to sudden or unexpected changes, then deliberately run the system at arbitrarily low clock rates. There is zero reason why we couldn’t run an AGI on a custom hardware/​software system that has a clock rate so low it thinks in glacial time relative to human perception.

  This is a world where it’s reasonable to regulate the AGI hardware the same way we regulate components for ICBMs, because it is not ubiquitously used, and we don’t need to burn everyone’s GPU—we just need to not sell consumer AGI hardware, in the same way that we don’t sell consumer GPS units that could be used for an ICBM instead of a car. The very concept of “stealing” or “releasing” the AGI doesn’t make sense, because the hardware to run the AGI machine code is as carefully-controlled and rare as the hardware for custom-built spacecraft.

  2. Controlled access to information.

  In a world where we cared about AGI being developed safely, no one would be dumb enough to allow that system unrestricted access to networks. This would be accomplished by external, non-AGI systems. If we don’t trust the AGI, it would be weird to try and use some other AGI system as a watchdog. So instead, use custom network hardware. Build it with an FPGA. Ethernet ports on either side. The custom switch has a simple rule: when a packet is received from the AGI, request approval. If no approval is given, drop the packet. Same for incoming traffic. If the complaint is that this is hugely inefficient because it slows down the AGI’s ability to do ad-hoc discovery & learning, that’s the point. If we want the AGI to solve a problem about optimizing car production, we will give it a data dump of material that we think is relevant for the problem. If it then wants to request additional information, then we have to approve that request. This is how workers are treated on projects with highly proprietary or otherwise silo’d information and that’s how the AGI should be treated too.

  But there’s more. We’re running the AGI on custom hardware, right? Internal to the AGI board, after the Ethernet (let’s assume we’re using one or more gigabit links) port on the PCB, we’ll put a hardware security module and FPGA for decrypting the packets. This is not connected to, or part of, the hardware/​software system running the AGI code. It is located on the PCB, but otherwise it is a separate system entirely. This system has one job: try to decrypt the traffic and drop anything that is invalid. Why are we going to do this? Because outside of this system, on the non-AGI network switch, we are going to encrypt traffic that we approve for the AGI. If we don’t approve the traffic, it’ll got dropped. We’ll do the reverse for traffic generated by the AGI. It is always encrypted prior to leaving the system, and the AGI itself has no control over this function. If the custom network hardware isn’t there to decrypt the approved traffic, it’ll just be babbling noise indecipherable from random data.

  There’s no WiFi module on this hardware, and no public WiFi endpoints in the facility. All WiFi access points in the facility require passwords and have a white-list of approved MAC addresses.

  The reason why controlling access to information matters is because there are hard limits to how far or fast an intelligent entity can reason in a vacuum. There is enough context-dependent information that you can’t reason everything from first principles, because the world’s current state has tons of structure embedded in it that humans put in place, and anything that wants to use human supply chains, logistics, and manufacturing is going to have to respect those totally arbitrary and unpredictable constraints.

  An AGI won’t be able to magically decode arbitrary data. If we don’t give it access to datasheets, technical descriptions, or allow it to interact with data sources, it’ll be just as stuck as any other intelligence faced with an arbitrary number of hypotheses and no way to rule any of them out.

  Depending on how complex the “general intelligence” algorithm turns out to be, just reasoning through if-then logic and Bayesian world modeling could be non-trivial compute time. The more we control information for the AGI, the less feasible plans that involve looking at thousands of CCTV cameras simultaneously will seem.

  3. Safety-critical systems engineering.

  In a world where we cared about AGI being developed safely, the hardware that I’m describing above containing the hardware security module would actually have at least 3 identical processors running on a synchronized clock. The hardware security module would expect 3 identical packets for any I/​O. If that did not occur, an immediate interrupt would power-cycle the system—again, this is outside of the processors running the AGI code itself. This is because even if we assumed the AGI was “aligned”, we don’t want uncontrollable behavior like radiation-induced bit flips, hardware defects from manufacturing, or other unexpected issues to result in erroneous behavior.

  Power for the entire system would be controlled via external, non-AGI systems. There would be multiple easily accessed locations for throwing a physical switch on the power line. In addition to that, there would be network accessible microprocessors for doing the same remotely, with the caveat that use small pyrotechnic charges are therefore irreversible. The default behavior of the system is that the power is controlled via these external systems and on reset, they do not supply power downstream to the AGI system itself. After an unexpected reset, e.g. loss of facility power, it requires manual control to re-enable downstream power to the AGI. This is done on a logically separate power domain from the rest of the facility—i.e. you can turn on power to everything else in the facility, WITHOUT being forced to power anything related to the AGI system.

  This is a world where AGI systems are designed under similar paranoia that you’d use for developing any other hardware/​software system that could kill people, and the resulting pessimism drives us to isolate and limit functionality using non-AGI systems by default, where those non-AGI systems can be designed, implemented, verified, and QA-tested all using traditional systems safety methods.

  4. An emphasis on at-rest encryption and secure-by-default networking.

  In a world where we cared about limiting the damage that AGI could do, we would ask manufacturers of hard drives, solid state drives, and every operating system to require encryption at rest. Your work laptop probably already does this. If you’re paranoid, you’ve got it enabled on your cellphone too. Do it everywhere. This is just good security practice, even without an AGI, but this is doubling down on (2) -- we are controlling access to information. Limit what can be known. Prevent discovery.

  Ditto for the second point. Emphasize encrypted networking, VPNs, firewalls, and anything else that makes it harder for any attacker to find targets. Importantly, and I want to emphasize this point: maybe don’t network factories to the internet and totally remote-controllable. This isn’t actually a problem in today’s world, as I’ve posted about it before, but it’s something to think about going forward. If a failure mode in many stories where the AGI kills everyone is “well it could hack this cool networked factory”, maybe don’t do that? The weird thing about factories is that there kind of aren’t that many in the world that conceivably fit that description? Like, there are lots of factories, but very few that are designed to build arbitrary products. It’s a handful of places in China and they still require manual reconfiguration for new products. Most things called “factories” build specific products. The AGI that somehow takes over the Ford car factory is not going to immediately begin turning out things that are not Ford cars.

  5. Extensive logging, monitoring, interpretability, and circuit breakers.

  Remember all of those non-AGI systems from 2 and 3? They’re constantly logging and telemetering information. If the AGI suddenly starts babbling like an idiot unexpectedly, cut the power. If the AGI is requesting information and has been denied repeatedly, cut the power. If the AGI unexpectedly starts using significantly more power, or the power begins to fluctuate in “weird” ways, cut the power. If any of these non-AGI systems report errors, or failures, cut the power. None of this needs to be controlled by humans. It’s all driven by simple microprocessors running watchdogs. Even failures like “this switch is supposed to report telemetry, but didn’t” will result in killing the power.

  If you can do this for systems whose failure mode is dozens of people die, you can do this for the system whose failure mode is “everybody dies”.

  For the AGI itself, we want interpretability. As much as we can get. We don’t want this to be a function of the AGI, because we don’t trust the AGI. But we don’t need it to be. We’ve got access to the RAM through an external debug interface. We don’t need the AGI to report that state, and we don’t have to trust it.

  Even though we won’t use AGI systems as watchdogs, it is acceptable to use traditional AI/​ML systems for parts of this loop. Maybe no human can parse what the AGI’s RAM says, but if we can train an AI/​ML system to do a good job of producing a human-readable digest, that is better than nothing.

  In a world where we cared about AGI being developed safely, the AGI is still treated like a task-based service. The AGI is given a problem to solve, and when it reports a solution and we finish negotiating with it on the particulars of the solution, we reset the system. We don’t maintain long-running AGI instances because there’s no need. Point it at a problem, get the solution, power-cycle it.

  6. Systems with AGI are assumed to be adversarial.

  Here we are again, back on the top of the mountain.

  The point of the systems safety process is to make some system safe, despite the inherent unsafety of components within it.
  To do that, we look for the ways it can cause harm, and we find mitigations.
  We practice defense-in-depth, with an emphasis on redundancy and at least one-fault-tolerance.

  In a world where we cared about not dying to AGI, it would be insufficient to expect that an aligned AGI would “just work”. It is not enough for there to be a proof that a hypothetical product as-designed is safe. We also need to look at the likelihood of:

  • design failures (the formal proof was wrong because the verification of it had a bug, there is no formal proof, the “formally verified” proof was actually checked by humans and not by an automated theorem prover)

  • manufacturing failures (hardware behavior out-of-spec, missed errata, power failures, bad ICs, or other failure of components)

  • implementation failures (software bugs, compiler bugs, differences between an idealized system in a proof vs the implementation of that system in some runtime or with some language)

  • verification failures (bugs in tests that resulted in a false claim that the software met the formal spec)

  • environment or runtime failures (e.g. radiation-induced upsets like bit flips; Does the system use voting? Is the RAM using ECC? What about the processor itself?)

  • usage failures (is the product still safe if it’s misused? what type of training or compliance might be required? is maintenance needed? is there some type of warning or lockout on the device itself if it is not actively maintained?)

  • process failures (“normalization of deviance”)

  For each of these failure modes, we then look at the worst-case magnitude of that failure. Does the failure result in non-functional behavior, or does it result in erroneous behavior? Can erroneous behavior be detected? By what? Etc. This type of review is called an FMEA. This review process can rule out designs that “seem good on paper” if there’s sufficient likelihood of failures and inability to mitigate them to our desired risk tolerances outside of just the design itself, especially if there exist other solutions in the same design space that do not have similar flaws.

  If we did this process, and we stuck to it, I think we’d come to an awkward conclusion.

  Many things that people have assumed we would give to the AGI to solve, we could just not.

  If the AGI tells us the way to solve climate change is nanobots, we can say “no thanks, give us a different solution”. It doesn’t matter if the AGI promises us this solution is 99.99% efficient and the next best option using boring, conventional technology is only 70% efficient. We can opt to not do things that we don’t think we can verify.

  Or if the AGI gives us the plans for a remote controlled factory that is going to be so efficient because it’s totally controlled by the AGI over the internet—don’t build that factory. Ask the AGI for the schematics, data sheets, and any other design material needed to build a new product, and build them the old fashioned way. Trade efficiency for stability.

  That’s how the system safety process is supposed to work. If after we do an FMEA, we are uncertain about a proposed solution, the ethical and correct response is to reject that solution!

  If we ask the AGI to explain a plan, and it says “Humans can’t comprehend the complexity of this.”, then we should reply “Toss that one then, give us one we can understand”.

  That’s how design reviews work.
  You don’t tell the review board, “You wouldn’t understand why this design is safe.”
  Why would I hold a super-intelligent AGI to a lower standard than I would a human engineer?

  Conclusion

  Is any of this feasible?

  In my mind, the key assumption comes down to whether AGI can be deployed to general-purpose hardware.

  I think the answer might be “no”, which is why we increasingly see the most successful AI/​ML labs in the world are investing in specialized hardware like the Google TPUs, Cerebras Wafer, Nvidia JETSON, Microsoft /​ Graphcore IPU, Mythic AMP, or literal dozens of other examples. All of these are examples of specialized, dedicated HW for AI/​ML systems that replace general-purpose hardware like a CPU or GPU.

  The alternative is a world where a 10-year-old laptop can run an AGI in someone’s living room.

  I have not seen anything yet that makes me think we’re leaning that way. Nothing about the human brain, or our development of AI/​ML systems so far, makes me think that when we create an actual AGI, it’ll be possible for that algorithm to run efficiently on general-purpose hardware.

  In the world that I’m describing, we do develop AGI, but it never becomes ubiquitous. It’s not like a world where every single company has pet AGI projects. It’s the ones you’d expect. The mega-corporations. The most powerful nations. AGI are like nuclear power plants. They’re expensive and hard to build and the companies that do so have zero incentive to give that away. If you can’t spend the billions of dollars on designing totally custom, novel hardware that looks nothing like any traditional general-purpose computer hardware built in the last 40 years, then you can’t develop the platform needed for AGI. And for the few companies that did pull it off, their discoveries and inventions get the dubious honor of being regulated as state secrets, so you can’t buy it on the open market either. This doesn’t mean AI/​ML development stops or ceases. The development of AGI advances that field immensely too. It’s just in the world I’m describing, even after AGI is discovered, we still have development focused on creating increasingly powerful, efficient, and task-focused AI/​ML systems that have no generality or agent-like behavior—a lack of capabilities isn’t a dead-end, it’s yet another reason why this world survives. If you don’t need agents for a problem, then you shouldn’t apply an agent to that problem.

  What links here?

  • AI Train­ing Should Allow Opt-Out by alyssavance (23 Jun 2022 1:33 UTC; 76 points)
  • Which pos­si­ble AI sys­tems are rel­a­tively safe? by Zach Stein-Perlman (21 Aug 2023 17:00 UTC; 42 points)
  • Which pos­si­ble AI sys­tems are rel­a­tively safe? by Zach Stein-Perlman (21 Aug 2023 17:00 UTC; 41 points)
  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (22 Jun 2022 21:04 UTC; 6 points)

• anonymousaisafety 14 Apr 2023 0:37 UTC

  21 points

  14

  on: On “aiming for con­ver­gence on truth”

  Sometimes when you work at a large tech-focused company, you’ll be pulled into a required-but-boring all-day HR meeting to discuss some asinine topic like “communication styles”.

  If you’ve had the misfortune fun of attending one of those meetings, you might remember that the topic wasn’t about teaching a hypothetically “best” or “optimal” communication style. The goal was to teach employees how to recognize when you’re speaking to someone with a different communication style, and then how to tailor your understanding of what they’re saying with respect to them. For example, some people are more straightforward than others, so a piece of seemingly harsh criticism like “This won’t work for XYZ reason.” doesn’t mean that they disrespect you—they’re just not the type of person who would phrase that feedback as “I think that maybe we’ve neglected to consider the impact of XYZ on the design.”

  I have read the many pages of debate on this current disagreement over the past few days. I have followed the many examples of linked posts that were intended to show bad behavior by one side or the other.

  I think Zack and gjm have a good job at communicating with each other despite differences in their preferred communication styles, and in particular, I agree strongly with gjm’s analysis:

  I think this is the purpose of Duncan’s proposed guideline 5. Don’t engage in that sort of adversarial behaviour where you want to win while the other party loses; aim at truth in a way that, if you are both aiming at truth, will get you both there. And don’t assume that the other party is being adversarial, unless you have to, because if you assume that then you’ll almost certainly start doing the same yourself; starting out with a presumption of good faith will make actual good faith more likely.

  And then with Zack’s opinion:

  That said, I don’t think there’s a unique solution for what the “right” norms are. Different rules might work better for different personality types, and run different risks of different failure modes (like nonsense aggressive status-fighting vs. nonsense passive-aggressive rules-lawyering). Compared to some people, I suppose I tend to be relatively comfortable with spaces where the rules err more on the side of “Punch, but be prepared to take a punch” rather than “Don’t punch anyone”—but I realize that that’s a fact about me, not a fact about the hidden Bayesian structure of reality. That’s why, in “‘Rationalist Discourse’ Is Like ‘Physicist Motors’”, I made an analogy between discourse norms and motors or martial arts—there are principles governing what can work, but there’s not going to be a unique motor, a one “correct” martial art.

  I also agree with Zack when they said:

  I’m unhappy with the absence of an audience-focused analogue of TEACH. In the following, I’ll use TEACH to refer to making someone believe X if X is right; whether the learner is the audience or the interlocutor B isn’t relevant to what I’m saying.

  I seldom write comments with the intent of teaching a single person. My target audience is whoever is reading the posts, which is overwhelmingly going to be more than one person.

  From Duncan, I agree with the following:

  It is in fact usually the case that, when two people disagree, each one possesses some scrap of map that the other lacks; it’s relatively rare that one person is just right about everything and thoroughly understands and can conclusively dismiss all of the other person’s confusions or hesitations. If you are trying to see and understand what’s actually true, you should generally be hungry for those scraps of map that other people possess, and interested in seeing, understanding, and copying over those bits which you were missing.

  Almost all of my comments tend to focus on a specific disagreement that I have with the broader community. That disagreement is due to some prior that I hold, that is not commonly held here.

  And from Said, I agree with this:

  Examples?

  This community is especially prone to large, overly-wordy armchair philosophy about this-or-that with almost no substantial evidence that can tie the philosophy back down to Earth. Sometimes that philosophy gets camouflaged in a layer of pseudo-math; equations, lemmas, writing as if the post is demonstrating a concrete mathematical proof. To that end, focusing the community on providing examples is a valuable, useful piece of constructive feedback. I strongly disagree that this is an unfair burden on authors.

  EDIT: I forgot to write an actual conclusion. Maybe “don’t expect everyone to communicate in the same way, even if we assume that all interested parties care about the truth”?

  What links here?

  • anonymousaisafety's comment on Moder­a­tion notes re: re­cent Said/​Dun­can threads by Raemon (14 Apr 2023 22:13 UTC; 47 points)

• anonymousaisafety 21 Jun 2022 7:22 UTC

  21 points

  0

  in reply to: TekhneMakre’s comment on: Pivotal out­comes and pivotal processes

  Oh, sorry, you’re referring to this:

  includes a distributed network of non-nuclear electromagnetic pulse emitters that will physically shut down any tech infrastructure appearing to be running rogue AI agents.

  This just seems like one of those things people say, in the same vein as “melt all of the GPUs”. I think that non-nuclear EMPs are still based on chemical warheads. I don’t know if a “pulse emitter” is a thing that someone could build. Like I think what this sentence actually says is equivalent to saying

  includes a distributed network of non-nuclear ICBMs that will be physically shot at any target believed to be running a rogue AI agent

  and then we can put an asterisk on the word “ICBM” and say it’ll cause an EMP at the detonation site, and only a small explosion.

  But you can see how this now has a different tone to it, doesn’t it? It makes me wonder how the system defines “appears to be running rogue AI agents”, because now I wonder what the % chance of false positives is—since on a false positive, the system launches a missile.

  What happens if this hypothetical system is physically located in the United States, but the rogue AI is believed to be in China or Russia? Does this hypothetical system fire a missile into another country? That seems like it could be awkward if they’re not already on board with this plan.

  because they’re doing something pretty non-trivial, they probably have to be big complex systems. Because they’re big complex systems, they’re hackable. Does this sound right to you? I’m mostly asking you about the step “detecting rogue AI implies hackable”. Or to expand the question, for what tasks XYZ can you feasibly design a system that does XYZ, but is really seriously not hackable even by a significantly superhuman hacker?

  It’s not really about “tasks”, it’s about how the hardware/​software system is designed. Even a trivial task, if done on a general-purpose computer, with a normal network switch, the OS firewall turned off, etc, is going to be vulnerable to whatever exploits exist for applications or libraries running on that computer. Those applications or libraries expose vulnerabilities on a general-purpose computer because they’re connected to the internet to check for updates, or they send telemetry, or they’re hosting a Minecraft server with log4j.

  It seems like you could not feasibly make an unhackable system that takes a bunch of inputs from another (unsafe) system and processes them in a bunch of complex ways using software that someone is constantly updating, because having the ability to update to the latest Detect-O-Matic-v3.4 without knowing in advance what sort of thing the Detect-O-Matic is, beyond that it’s software, seems to imply being Turing-completely programmable, which seems to imply being hackable.

  When you’re analyzing the security of a system, what you’re looking for is “what can the attacker control?”

  If the attacker can’t control anything, the system isn’t vulnerable.

  We normally distinguish between remote attacks (e.g. over a network) and physical attacks (e.g. due to social engineering or espionage or whatever). It’s generally safe to assume that if an attacker has physical access to a machine, you’re compromised.^[1] So first, we don’t want the attacker to have physical access to these computers. That means they’re in a secure facility, with guards, and badges, and access control on doors, just like you’d see in a tech company’s R&D lab.

  That leaves remote attacks. These generally come in two forms:

  1. The attacker tricks you into downloading and running some compromised software. For example, visiting a website with malicious JavaScript, or running some untrusted executable you downloaded because it was supposed to be a cheat engine for a video game but it was actually just a keylogger, or the attacker has a malicious payload in a seemingly innocent file type like a Word document or PDF file and it’s going to exploit a bug in the Word program or Adobe Acrobat program that tries to read that file.

  2. The attacker sends network traffic to the machine which is able to compromise the machine in some way, generally by exploiting open ports or servers running on the target machine.

  All of the attacks in (1) fall under “when you run untrusted code, you will get pwned” umbrella. There’s a bunch of software mitigations for trying to make this not terrible, like admin users vs non-admin users, file system permissions, VM sandboxing, etc, but ultimately it’s just like rearranging deck chairs on the Titanic. It doesn’t matter what you do, someone else is going to find a side channel attack and ruin your day if you let them run code on your machine. So don’t do that. This is actually easier than you might think: plenty of systems are “secure” because they run an incredibly minimal Linux OS (or some RTOS or even just bare metal) and they’re effectively static—the software image is flashed to some SoC’s read-only memory (ROM) by an external debugger^[2], and there’s no capability from within the software to write or update that memory. The processor is not configured for running code outside of that ROM. There’s no user accounts, or “default applications”, or browsers, or anything else other than the code for performing the actual task required by the system.

  For (2), in cases where the system is not supposed to be receiving arbitrary traffic, we solve this by using networking hardware downstream of the system that drops any unexpected traffic, and rate-limits everything. If the attacker’s traffic gets dropped, they can’t control anything. The rate-limiting is to prevent a denial-of-service based on traffic we expect. For the “expected” traffic, the ideal mechanism to prevent abuse is a binary schema for valid messages, a fixed MTU size on messages, and the use of authenticated encryption so that the attacker is unable to generate the traffic without access to encryption keys. Encryption keys can be stored in hardware security modules that are physically located on the senders & receivers within the system.^[3]

  With that in mind, I hope it is clear that the problem isn’t about complexity of a system, or the number of inputs that the system has, or if the system supports software updates.

  The actual issue with the system we’re discussing is that the attacker can control their network traffic, and in this hypothetical, we need to look at arbitrary network traffic to decide if it’s evidence of a rogue AI.^[4] We can’t just drop everything in hardware and ignore messages that don’t fit an authenticated encryption scheme, because we want to read that traffic.

  So we have to assume that the attacker can generate arbitrary payloads, send them to our system, and our system is going to need to not get pwned.

  I think I’d probably approach it with the following thoughts:

  1. I’d design the system so that the traffic to be inspected is distributed across a cluster of computers. This is both to prevent denial-of-service attacks and to prevent targeted attacks like Nethammer. If the system is receiving traffic in excess of what the cluster is designed to handle, it’ll be dropped—while this represents a degradation in monitoring capability, it can be fixed by adding additional machines to the cluster as needed. It’s often useful to gracefully degrade capabilities when a system is under load.

  2. The computers in the cluster that receive traffic have a simple role: send an alert message to a central supervisor if suspicious traffic is detected.

  3. The alert message uses authenticated encryption.

  4. The central supervisor is only communicated with via authenticated encryption messages on a local network. It is not connected to the wider network and it is not possible for the attacker to send traffic to the supervisor machine, unless they first compromised one of the cluster machines.

  The supervisor machine is pretty safe unless a cluster machine is compromised. If a cluster machine is compromised, maybe we think that the attacker can control the alert message, and in doing so, control traffic sent to the central supervisor? Let’s fix that. What do we need for the alert message? Maybe the IP of the packet, geolocation information, reasons why it was suspicious? Etc. We’ll define a binary schema for that information and we’ll stick a serial device or FPGA onto each cluster machine.

  1. The cluster machines send the alert message by setting the relevant fields of the message via the serial device or FPGA registers, and it’s that device which sends the actual alert. Now if the cluster machine is compromised, all the attacker can do is change fields in the alert message, but they can’t send different messages entirely.^[5]

  2. We test the central supervisor to confirm that any permutation of the alert message is still safe to receive. For testing permutations, we normally use a method based on equivalence classes.

  So now attackers can maybe compromise cluster machines because they control the traffic that we’re trying to monitor, but they can’t use a compromised cluster machine to control traffic to the central supervisor, because compromising a cluster machine doesn’t let you change the traffic sent to a supervisor in a way that would be dangerous.

  At this point we can ask what the attacker can do:

  • They can maybe send garbage alert messages.

  • They can maybe send no message at all—i.e. they crash the cluster machine so it fails silent.

  We can mitigate the 2nd problem by requiring a heartbeat from each cluster machine to the central supervisor. A heartbeat is another message sent via authenticated encryption, but it’s sent at a known frequency, like 100 Hz. If an attacker or any other event crashes the machine, the central supervisor will know.

  If the attacker sends a garbage alert message, what is the downside? It could be bad if this system launches missiles, so maybe we’ll have human review on any alert that the central supervisor receives, prior to any action being taken by the system. The human review can include a log of the network traffic that generated the alert, captured by the cluster machine, and a diagnosis of the cluster machine’s telemetry for how it concluded the traffic was suspicious. This is intended to mitigate the possibility of a false positive where the attacker can control where we think a rogue AI is located.

  Now, where we stand is that an attacker can maybe compromise a cluster machine, and in doing so, they can maybe generate an erroneous alert message for a rogue AI, which a human review maybe won’t be able to detect as erroneous, so whatever action gets taken on belief that a rogue AI exists would be sent to a location controlled by the attacker. The magnitude of how bad that is depends on what action is taken. If it is “fire an ICBM”, it could be pretty bad.

  The other possibility is that an attacker compromises a machine in such a way that no alert is generated, but no heartbeat is missed either. This is an interesting failure mode because it’s equivalent to another failure mode that we have not discussed: the system could simply fail to detect the behavior of a rogue AI. In other words, this system doesn’t need to be hacked in order to fail. If the algorithm we’re using for detecting rogue AI has any margin for error at all, then it should be no surprise to anyone that this system is susceptible to a false negative, i.e.: it did not send an alert when it should have.

  The relevant question becomes, what’s the margin of error on the hypothetical algorithm for detecting a rogue AI? If that algorithm has a higher likelihood of a false positive than our hypothetical attacker, after all of our mitigations, than maybe this system isn’t even worth building, depending on how high that likelihood is. Likewise, if the algorithm has a high likelihood of a false negative, such that a rogue AI can reasonably evade detection due to not being suspicious enough, then this system might still not be worth building, because now the most likely failure mode is that the system just doesn’t work reliably enough, hacked or not.

  1. ^

     This is a safe assumption because if the attacker has physical access to a machine, they can do dumb things like pull out the hard drive and put it into another computer that they control. Enable encryption on your drives.

  2. ^

     When I say “external debugger”, this is implying physical access to the system.

  3. ^

     In order to get the encryption keys, the attacker needs physical access to the system. Attacks that would use a side-channel attack won’t work here, because we don’t accept arbitrary traffic, and in order to generate that traffic, you need the keys. It’s a catch-22, unless you have physical access. But if you have physical access, we assume the system is compromised.

  4. ^

     To be clear, I am not convinced that “evidence of a rogue AI” is a meaningful description of behavior.

  5. ^

     This is assuming that we’ve verified the device that sends the alert message, which we’ll have high confidence in because of how simple the implementation will be.

• anonymousaisafety 23 Jun 2022 2:16 UTC

  19 points

  in reply to: DirectedEvolution’s comment on: Air Con­di­tioner Test Re­sults & Discussion

  We can even consider the proposed plan (add a 2nd hose and increase the price by $20) in the context of an actual company.

  The proposed plan does not actually redesign the AC unit around the fact that we now have 2 hoses. It is “just” adding an additional hose.

  Let’s assume that the distribution of AC unit cooling effectively looks something like this graphic that I made in 3 seconds.

  

  
  

  In this image, we are choosing to assume that yes, in fact, 2-hose units are more efficient on average than a 1-hose unit. We are also recognizing that perhaps there is some overlap. Perhaps there are especially bad 2-hose units, and especially good 1-hose units.

  Based on all of the evidence, I’m going to say that the average 1-hose unit does represent the minimum efficiency needed for cooling in an average consumer’s use-case—i.e. it is sufficient for their needs.

  When I consider what would make a 2-hose unit good or bad, I suspect it has a lot to do with how much of the design is built around the fact that there are 2-hoses.

  In your proposal, we simply add a 2nd hose to a unit that was otherwise designed functionally as a 1-hose unit. Let’s consider where that might be plotted on this graph.

  I’m going to claim based on vague engineering intuition /​ judgment /​ experience that it goes right here.

  

  
  

  If I am right about where this proposal falls against the competition, then here’s what we’ve done:

  1. This is not a 1-hose unit any more. Despite it being more efficient than the average 1-hose units, and only slightly more expensive, consumers looking at 1-hose units (because they are concerned about cost) will not see this model. The argument that it is “only $20 more expensive” is irrelevant. Their search results are filtered, they read online that they wanted a one-hose unit, this product has been removed from their consideration.

  2. This is a bad 2-hose unit. It is at the bottom of the efficiency scale, because other 2-hose units were actually designed to take full advantage of the 2-hoses. They will beat you on efficiency, even if they cost more. Wirecutter will list this in the “also ran” when discussing 2-hose units, “So and so sells a 2-hose model, but it was barely more efficient than a 1-hose, we cannot recommend it”.

  3. A consumer looking at 2-hose units is already selecting for efficiency over cost, so they will not buy the “just add another hose” 2-hose unit, since it is on the wrong end of the 2-hose distribution.

  4. You will acquire a reputation as the company that sells “cheap” products—your unit is cheaper than other 2-hose units, but isn’t better because it wasn’t designed as a 2-hose unit, and it was torn apart by reviewers.

  5. Fixing this inefficiency requires actually designing around 2-hoses, which likely results in something like this

  

  
  

  “Minimum viable”, in the context of a “minimum viable product” or MVP, is a term in engineering that represents the minimal thing that a consumer will pay to acquire. This is a product that can actually be sold. It’s not the literal worst in its category, and it has a clear supremacy over cheaper categories. This is also called table stakes. Reviewers will consider it fairly, consumers will not rage review it, etc.

  However, it’s probably also a lot more expensive than the hypothetical “only $20 more” that has been repeatedly stated.

  Even in the scenario where a reviewer does consider the “just add another hose” model when viewing one-hose units, we’ve already established that the one-hose unit is cheaper (by $20! if it’s a $200 unit, that’s 10%), and that the average 1-hose unit is sufficient for some average use-case. Therefore the rational consumer choice is to buy the cheaper one-hose anyway, because it’s irrational to pay more for efficiency that isn’t needed!^[1][2]

  1. ^

     The exception here is some hypothetical consumer who knows, for a fact, that their unique situation requires a two-hose unit, e.g. they tried a one-hose unit already and it was insufficient.

  2. ^

     There’s also an argument here that a rational option is to buy a 1-hose unit, and then if you need slightly more efficiency, just buy & wrap the 1-hose with insulation, as described here. This allows the consumer to purchase at the lower price point and then add efficiency if needed for the cost of the insulation. It’s unclear to me that the “just add another hose” AC would still perform better than an insulated 1-hose.

  What links here?

  • anonymousaisafety's comment on Air Con­di­tioner Test Re­sults & Discussion by johnswentworth (22 Jun 2022 23:12 UTC; 83 points)

• anonymousaisafety 20 Jun 2022 20:03 UTC

  14 points

  2

  in reply to: Eliezer Yudkowsky’s comment on: Pivotal out­comes and pivotal processes

  I am only replying to the part of this post about hardware vulnerabilities.

  Like, superhuman-at-security AGIs rewrote the systems to be formally unhackable even taking into account hardware vulnerabilities like Rowhammer that violate the logical chip invariants?

  There are dozens of hardware vulnerabilities that exist primarily to pad security researcher’s bibliographies.

  Rowhammer, like all of these vulnerabilities, is viable if and only if the following conditions are met:

  • You know the exact target hardware.

  • You also know the exact target software, like the OS, running on that hardware.

    • You need to know what in RAM you’re trying to flip, and where to target to do so, like a page table or some type of bit for the user’s current access level.

  • The target needs to actually execute your code.

    • In attacks where security researchers pad themselves for pulling off Rowhammer remotely, it’s because they use JavaScript or WebGL in a browser, and then pwn devices that use browsers. This is flaw almost entirely reserved for general-purpose compute hardware, because embedded software or other application-specific hardware/​software systems don’t need or have browsers in them.

    • In all other attacks, it involves downloading & executing a program on the target machine. Normally the example is given with cloud VMs whose entire purpose is to run code from an external source. Again, this is reserved for general-purpose compute hardware, because systems that execute code only out of read-only memory will not be able to execute an attacker’s code.

    • There is time on the target to run the Rowhammer attack uninterrupted. It relies on a continuous and degenerate set of instructions. This can be anywhere from minutes to days of time. This means that systems that don’t give uninterrupted time to external code are also not vulnerable.

  • The target OS, or other software, on the system needs to not perform any type of RAM scrubbing. There are papers claiming that variants of Rowhammer work for systems that use ECC + scrubbing, but those papers also assume that the scrubbing happens over hours. If a system has very little RAM, like an embedded processor, it is feasible for hardware to scrub RAM far faster than that.

    • These exploits also add the requirement that the attacker needs to know the exact target RAM and ECC algorithm.

  • The target hardware/​software system needs to not have any hardware level redundancy. You can’t rowhammer a system that has 3 separate computers that compare & vote their state on a synchronized clock. Hardware vulnerabilities are probabilistic attacks. They don’t work anymore if the attack must occur simultaneously, and identically on separate physical systems. This is another reason why we’re able to build systems that function despite hostile environments where bit flips are routine, i.e. high radiation.

  • The target needs to not crash. Semi-randomly flipping bits in roughly the right location in RAM is not something that most software is designed to handle, and in an overwhelming number of cases, trying to execute this attack will crash the system.

  It’s not that Rowhammer isn’t possible in the sense that it cannot be shown to work, but it’s like this paper showing that you can create WiFi signals in air-gapped computers. Or this fun paper for Nethammer showing novel attacks that don’t require code execution on the target machine, except they also don’t allow for controlling where bit flips occur, so the “attack” is isomorphic to an especially hostile radiation environment with a high likelihood of bit-flips, and it relies on the ability for the attacker to swarm the target system with a high volume (500 Mbps?) of network traffic that they control—a network switch that drops unexpected traffic or even just rate-limits it will defeat Nethammer. Note that rate-limiting network traffic is in fact standard practice for high stability systems, because it’s also a protection against much more mundane denial-of-service attacks.

  Consumer systems are vulnerable to attacks, because consumer systems don’t care about stability. Consumers want to have a fast network connection to the internet. There’s no requirement, or need, for that to be true on a system designed for stability, like something in a satellite, or some other safety-critical role. It is possible to have systems that are effectively “not able to be hacked”—they don’t use general-purpose hardware, they don’t have code that can be modified, they have no capability for executing external code, they include hardware level fault tolerance and redundancy, and they have exceptionally limited I/​O. It doesn’t require us presuming “superhuman-at-security AGIs” exist to design these systems.

  Every few weeks researchers publish papers carefully documenting the latest side-channel attacks that result in EVERYTHING EVERYWHERE BEING VULNERABLE FOREVER, and every few weeks attackers continue to do the boring old thing of leaving USB drives lying around for a target to pwn themselves, or letting the target just download the malware directly to their machine. They’re almost all just remixes too—it’s “here’s how to do exploit A (the original), but on hardware systems that implemented mitigations for C, D, and E”. Except exploit A still has all of the preconditions I listed above, and now you’ve got whatever new preconditions the security researchers have in their latest paper.

• anonymousaisafety 28 Jun 2022 16:26 UTC

  12 points

  15

  in reply to: harfe’s comment on: Con­test: An Alien Message

  I have mixed thoughts on this.

  I was delighted to see someone else put forth an challenge, and impressed with the amount of people who took it up.

  I’m disappointed though that the file used a trivial encoding. When I first saw the comments suggesting it was just all doubles, I was really hoping that it wouldn’t turn out to be that.

  I think maybe where the disconnect is occurring is that in the original That Alien Message post, the story starts with aliens deliberately sending a message to humanity to decode, as this thread did here. It is explicitly described as such:

  From the first 96 bits, then, it becomes clear that this pattern is not an optimal, compressed encoding of anything. The obvious thought is that the sequence is meant to convey instructions for decoding a compressed message to follow...

  But when I argued against the capability of decoding binary files in the I No Longer Believe Intelligence To Be Magical thread, that argument was on a tangent—is it possible to decode an arbitrary binary files? I specifically ruled out trivial encodings in my reasoning. I listed the features that make a file difficult to decode. A huge issue is ambiguity because in almost all binary files, the first problem is just identifying when fields start or end.

  I gave examples like

  1. Camera RAW formats

  2. Compressed image formats like PNG or JPG

  3. Video codecs

  4. Any binary protocol between applications

     1. Network traffic

     2. Serialization to or from disk

     3. Data in RAM

  On the other hand, an array of doubles falls much more into this bucket

  data that is basically designed to be interpreted correctly, i.e. the data, even though it is in a binary format, is self-describing.

  With all of the above said, the reason why I did not bother uploading an example file in the first thread is frankly because it would have taken me some number of hours to create and I didn’t think there would be any interest in actually decoding it by enough people to justify the time spent. That assumption seems wrong now! It seems like people really enjoyed the challenge. I will update accordingly, and I’ll likely post my example of a file later this week after I have an evening or day free to do so.

• anonymousaisafety 14 Jun 2022 22:58 UTC

  12 points

  on: Yes, AI re­search will be sub­stan­tially cur­tailed if a lab causes a ma­jor disaster

  This argument is important because it is related to a critical assumption in AGI x-risk, specifically with regard to the effectiveness of regulation.

  If an AGI can be created by any person, in their living room, with a 10 year old laptop, then regulation is going to struggle to make a difference. Case in point: strong encryption was made illegal (and still is) in various places, and yet, teenagers use Signal and the internet runs on HTTPS.

  If, on the other hand, true agent-like AGI turns out to be computationally expensive and requires very specialized hardware to run efficiently, such that only very large corporations can foot the bill to do so (e.g. designing and building increasingly custom hardware like Google TPUs, Nvidia JETSON, Cerebras Wafer, Microsoft /​ Graphcore IPU, Mythic AMP), then regulation is going to be surprisingly effective, in the same way that it has become stupidly difficult to build new nuclear reactors in the United States, despite advances in safety /​ efficiency /​ etc.

  What links here?

  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)

• anonymousaisafety 3 Dec 2022 19:33 UTC

  LW: 11 AF: 8

  2

  AF

  in reply to: Charlie Steiner’s comment on: In­ner and outer al­ign­ment de­com­pose one hard prob­lem into two ex­tremely hard problems

  I wasn’t intending for a metaphor of “biomimicry” vs “modernist”.

  (Claim 1) Wings can’t work in space because there’s no air. The lack of air is a fundamental reason for why no wing design, no matter how clever it is, will ever solve space travel.

  If TurnTrout is right, then the equivalent statement is something like (Claim 2) “reward functions can’t solve alignment because alignment isn’t maximizing a mathematical function.”

  The difference between Claim 1 and Claim 2 is that we have a proof of Claim 1, and therefore don’t bother debating it anymore, while with Claim 2 we only have an arbitrarily long list of examples for why reward functions can be gamed, exploited, or otherwise fail in spectacular ways, but no general proof yet for why reward functions will never work, so we keep arguing about a Sufficiently Smart Reward Function That Definitely Won’t Blow up as if that is a thing that can be found if we try hard enough.

  As of right now, I view “shard theory” sort of like a high-level discussion of chemical propulsion without the designs for a rocket or a gun. I see the novelty of it, but I don’t understand how you would build a device that can use it. Until someone can propose actual designs for hardware or software that would implement “shard theory” concepts without just becoming an obfuscated reward function prone to the same failure modes as everything else, it’s not incredibly useful to me. However, I think it’s worth engaging with the idea because if correct then other research directions might be a dead-end.

  Does that help explain what I was trying to do with the metaphor?

• anonymousaisafety 18 Jun 2022 0:28 UTC

  11 points

  in reply to: faul_sname’s comment on: I No Longer Believe In­tel­li­gence to be “Mag­i­cal”

  Here is an example of a large file that:

  1. contains actual, real information (it is not just random noise)

  2. it’ll compress easily (there’s structure to it)

  3. it’s totally useless to you (you can derive structure, but not meaning)

  It’s 1 megabyte of telemetry captured from a real-time hardware/​software system using a binary encoding for the message frames. The telemetry is, in other words, not self-describing.

  Some facts:

  • A trivial compressor, e.g. zip, can compress this telemetry to a much smaller size.

  • It can also decompress the telemetry.

  • The size of the compressed file + the size of the compressor is less than the uncompressed telemetry.

  However:

  You cannot make any conclusions about the meaning of this data, even after you derive some type of structure from it.

  Knowing that a 4 byte sequence starting at data[N * 20] for each integer N can be delta-encoded such that on average the delta-encoding of data[(N-1) * 20] saves more space than including both data[(N-1) * 20] and data[N * 20] does not tell you what those 4 bytes mean.

  • You can argue that maybe it’s a 4 byte value of some type.
    But you don’t know if it’s stored as Big Endian or Little Endian in this format.

  • You also don’t know if it is signed or unsigned, or if it is even a 2′s complement integer.
    It could be an IEEE 754 32-bit floating point value.

  • It could be a contiguous run of 2 separate 2 byte values that for some reason are correlated in such a way that they can be delta-encoded together efficiently, e.g. the sensor readings of redundant sensors measuring the same physical process, like a temperature.

  • Let’s say that for some reason you’re certain it is representing an unsigned 4-byte integer.
    What does it mean?

    • Is it the value of some sensor, as a rounded integer, like a distance measured in meters where we drop the fractional part?

    • Or are we actually using some type of encoding where we record the sensor value as a integer, but we have a calibration table we apply when post-processing this table to calculate a real physical value, like the look-up table used for thermocouples when interpreting the raw count measured on an ADC?

    • If it’s the latter and you don’t know what look-up table to use (what type of thermocouple is it? Type-K? Type-J?), good luck. Ditto for encoding where the actual physical value is calculated using coefficients for some linear or exponential transform, including an offset.

  • The same arguments apply if we try to assume it is a floating point value.

  If you think this is being unfair because the original question was about image frames, the encoding that has been suggested in the comments for what image data looks like is describing a bitmap—a very simple, trivial way to encode image data, and incredibly inefficient in size for that reason. Even in the case of a bitmap, you’re jumping to the conclusion that you can reliably differentiate between an MxN image of 8-bit-per-channel RGB data vs an NxM image of 8-bit-per-channel RGB data, when the alternative might be that is some different size of image using say 16-bit-per-channel RGB data for higher resolution in colors, or maybe it is actually 8-bit-per-channel RGBA data because we’re including the alpha component for each pixel, or maybe the data is actually stored as HSL channels, or maybe it’s stored as BGR instead of RGB, or maybe the data is actually an MxN image of a 8-bit-per-channel grayscale data, or perhaps 24-bit-per-channel grayscale data. In the latter cases, there are not 3 values per pixel, because it’s grayscale! Not to be confused with storing a grayscale image in RGB channels, e.g. by repeating the same value for all 3 channels.

  And all of the above is ignoring the elephant in the room: Video codecs are not a series of bitmaps. They are compressed and almost always use inter-frame compression meaning that individual video frames are compressed using knowledge of the previous video frames. Therefore, the actual contents of a video does not look like “a series of frames where each frame is a grid of cells where each cell is 3 values”. Likewise, most images on the internet are JPGs and PNGs, which are also compressed, and do not look like a grid of cells of three values.

  Ok, but what about cameras?

  “an image taken by a modern camera contains a surprisingly large amount of easy-to-decode information about the world, in the information-theoretic sense of the word information”

  Cameras used by professional photographers often dump data in a RAW format which may or may not be compressed but is specific to the manufacturer of the camera because it’s tied to the actual camera hardware in the same way that my hypothetical telemetry is tied to the hardware/​software system.

  Here’s the Wikipedia list of RAW formats:

  .3fr, .ari, .arw, .bay, .braw, .crw, .cr2, .cr3, .cap, .data, .dcs, .dcr, .dng, .drf, .eip, .erf, .fff, .gpr, .iiq, .k25, .kdc, .mdc, .mef, .mos, .mrw, .nef, .nrw, .obm, .orf, .pef, .ptx, .pxn, .r3d, .raf, .raw, .rwl, .rw2, .rwz, .sr2, .srf, .srw, .tif, .x3f

  ...
  
  Providing a detailed and concise description of the content of raw files is highly problematic. There is no single raw format; formats can be similar or radically different. Different manufacturers use their own proprietary and typically undocumented formats, which are collectively known as raw format. Often they also change the format from one camera model to the next. Several major camera manufacturers, including Nikon, Canon and Sony, encrypt portions of the file in an attempt to prevent third-party tools from accessing them.

  Is that “easy to decode”?

  There is an argument here that it is still easy because all you need to do is “just” run through all of the various permutations I’ve described above until at the end of the process there is an image that “looks like” a reasonable image. I mean if you dump the color data and it looks like blue and red are swapped, maybe it was stored BGR instead of RGB, and there’s no harm there, right? And now we’ve buried the whole argument with a question of what does it mean for a result to look “reasonable”? When you’re given a totally unknown encoding and you want to decode it, and you have to make assumptions about what the parsed data is going to look like just to evaluate the strength of your decoding, is that very solid ground? Are you certain that the algorithm being described here is “easy”, in the sense that is computationally efficient?

  What links here?

  • anonymousaisafety's comment on Let’s See You Write That Cor­rigi­bil­ity Tag by Eliezer Yudkowsky (20 Jun 2022 10:37 UTC; 25 points)
  • anonymousaisafety's comment on Con­test: An Alien Message by DaemonicSigil (28 Jun 2022 16:26 UTC; 12 points)
  • anonymousaisafety's comment on I No Longer Believe In­tel­li­gence to be “Mag­i­cal” by DragonGod (18 Jun 2022 0:42 UTC; 2 points)