daozaich

Karma: 258

daozaich 26 Nov 2017 22:50 UTC
43 points
on: Security Mindset and Ordinary Paranoia
Hey,
fun that you now post about security. So, I used to work as itsec consultant/reasearcher for some time; let me give my obligatory 2 cents.
On the level of platitudes: my personal view of security mindset is to zero in on the failure modes and tradeoffs that are made. If you additionally have a good intuition on what’s impossible, then you quickly discover either failure modes that were not known to the original designer—or, also quite frequently, the system is broken even before you look at it (“and our system archieves this kind of security, and users are supposed to use it that way”—“lol, you’re fucked”). The following is based on aesthetics/ intuition, and all the attack scenarios are post-hoc rationalizations (still true).
So, now let’s talk passwords. The standard procedure for implementing password-login on the internet is horrible. Absolutely horrible. Let me explain first why it is horrible on the object level, second some ways to do better (as an existence proof), and third why this hasn’t gotten fixed yet, in the spirit of inadequacy analysis.
First: So, your standard approach is the following: Server stores salted and hashed password for each user (using a good key-derivation function ala scrypt, not a hash, you are not stupid). User wants to login, you serve him the password-prompt html, he enters password, his browser sends password via some POST form, you compare, either accept or reject user. Obviously you are using https everywhere.
Failure modes: (1) User has chosen extremely bad password. Well, you obviously rate-limit login attempts. Otherwise, there is no defense against stupid (you maybe check against john-the-ripper’s top N passwords on creation (bloom filter lookup, hit SSD once). Users can often get away with 30 bit of entroy, because attackers can only use rate-limited online attacks.
(2) Someone steals your password database. Does not allow to log into your server, does not allow to log into other servers if user stupidly reused password. Nice! However, attacker can now do offline attempts at the passwords—hence, anyone below 45 bits is fucked, and anyone below 80 bits should feel uncomfortable. This is however unpreventable.
(3) Someone MitMs you (user sits at starbucks). You are using SSL, right? And Eve did not get a certificate from some stupid CA in the standard browser CA list? Standard browser X.509 PKI is broken by design; any Certificate Authority on this list, and any governent housing one of the CAs on this list get a free peek at your SSL connection. Yeah, the world sucks, live with it (PS: DNSSEC is sound, but no one uses it for PKI—because, face it, the world sucks)
(4) Your user gets phished. This does not necessarily mean that the user is stupid; there are lots of ways non-stupid users can get phished (say- some other tab changed the tab-location while the user was not watching, and he/she did not check the address bar again).
(5) You got XSSed. Password gone.
(6) Your user catched a keylogger. Game over.
(7) Your SSL termination point leaks memory (e.g. heartbleed, cloudfare cache bug, etc).
Ok, what could we possibly fix? Terrible password is always game over, semi-bad password + database stolen is always game over , keylogger is
always game over—try two-factor for all of those.
The others?
First, there is no reason for the user to transmit the password in the SSL-encrypted plain. For example, you could send (nonce, salt) to the user, the user computes hash(nonce, scrypt(password, salt)) and sends this. Compare, be happy. Now, if eve reads the http plaintext she still does not know the password and also knows no valid login-token (that’s why the nonce was needed!). This has one giant advantage: You cannot trust yourself to store the password on disk; you should not trust yourself to store it in RAM. There are a thousand things that can go wrong. Oh, btw, Ben could implement this now on lesserwrong!
But this system still stinks. Fundamentally: (1) you should not trust SSL to correctly verify your site to the user (mitm/bad CA), (2) you should not rely on the combination of ill-trained lusers and badly designed browser UI, and (3) you should not trust yourself to render a password box to the user, because your fancy secure development lifecycle-process will not prevent Joe WebDev from dropping XSS into your codebase.
But, we have assets: User really needs only to prove posession of some secret; server knows some other derived secret; we can make it so that phishing becomes mostly harmless.
How? First, the browser needs a magic special API for rendering password forms. This must be visually distinct from web forms, and inaccessible to JS. Second, we need a protocol. Let me design one over a glass of whisky.
Server has a really, really good way of authenticating itself: It knows a fucking hash of the users password. So, an easy way would be: server stores scrypt(“SMP”,username, domain, password). When user enters his password, his user agent (browser) regenerates this shared secret, and then both parties play their favorite zero-knowledge mutual authentication (no server-generated salt: username + domain is enough) -- e.g. socialist millionare [1]. Hence, the user cannot get phished. He can enter all his passwords for all his accounts anywhere in the net, as long as he sticks to the magic browser form.
Now, this still stinks: If Suzy Sysadmin decides to take ownership of your server’s password database, then she can impersonate any user. Meh, use public key crypto: scrypt(“curvexyz”,username, domain, password) as the seed to generate a key-pair in your favorite curve. Server stores the public key, user-agent regenerates the private key, proves posession—AFTER the server has authenticated itself. Now Suzy fails against high-entropy passwords (which resist offline cracking). And when user logs in to Suzy’s rogue impersonation server, then he will still not leak valid credentials.
In other words, password authentication is cryptographically solved.
Now, something really depressing: No one is using a proper solution. First I wanted to tell a tale how browser-vendors and websites will fail forever to coordinate on one, especially since both webdevs (enemies of the people) want to render beautiful password forms, and the CA industry wants to live too (middlemen are never happy to be cut, both economical and cyptographic ones). But then I saw that openssh uses brain-dead password auth only. God, this depresses me, I need more whisky.
Small number of links:
[1] https://en.wikipedia.org/wiki/Socialist_millionaires
[2] https://en.wikipedia.org/wiki/Password-authenticated_key_agreement—our goals have a name, we want “augmented PAKE”!
[3] https://tools.ietf.org/html/rfc8236 -- how a more sober protocol looks like.
[4] https://crypto.stackexchange.com/questions/25338/why-arent-zero-knowledge-proofs-used-for-authentication-in-practice

daozaich 28 Nov 2017 0:21 UTC
7 points
in reply to: ChristianKl’s comment on: Security Mindset and Ordinary Paranoia
*shrugs*
Yeah, ordinary paranoia requires that you have unbound listening on localhost for your DNS needs. Because there should be a mode to ask my ISP-run recursive resolver to deliver the entire cert-chain. Thisis a big fail of DNSSEC (my favorite would be -CD +AD +RD, this flag combination should still be free and means “please recurse; please use dnssec; please don’t check key validity”).
Yes, and DNSSEC over UDP breaks in some networks, then you need to run it via TCP (or do big a big debugging-session in order to figure out what broke).
And sure, DNSSEC implementations can have bugs, the protocol can have minor glitches, many servers suck at choosing good keys or doing key-rollover correctly.
But: Compared to X.509 browser CA and compared to DNS with transport-layer security (like DJB’s horrible dnscurve), this is a sound global PKI and is “realexistierend”. And it has so many good features—for example, the keys to your kingdom reside on airgapped smartcard (offline-signing), instead of waiting in RAM for the next heartbleed (openssl is famous for its easy-to-verify beautiful bug-free code, after all) or sidechannel (hah! virtualization for the win! Do you believe that openssl manages to sign a message without leaking bits via L1cache access patterns? If you are on AWS then the enemy shares the same physical machine, after all) or ordinary break-in.
If you are interested I can write a long text why transport-layer security for DNS misses the fucking point (hint: how do you recover from a poisoned cache? What is the guaranteed time-of-exposure after Suzy ran away with your authorative server? Did these people even think about these questions??!!1 How could DJB of all people brain-fart this so hard?).

daozaich 28 Nov 2017 22:36 UTC
3 points
in reply to: daozaich’s comment on: Security Mindset and Ordinary Paranoia
edit timeout over, but the flags for requesting a chain-of-trust from your recursive resolver/ cache should of course by (+CD +AD +RD).

daozaich 28 Nov 2017 23:29 UTC
13 points
in reply to: John_Maxwell’s comment on: Security Mindset and the Logistic Success Curve
Everything exposed to an attacker, and everything those subsystems interact with, and everything those parts interact with! You have to build all of it robustly!
seems false to me, if you have good isolation—which is what a project like Qubes tries to accomplish.
I agree with you here that Qubes is cool; but the fact that it is (performantly) possible was not obvious before it was cooked up. I certainly failed to come up with the idea of Qubes before hearing it (even after bluepill), and I am not ashamed of this: Qubes is brilliant (and IOMMU is cheating).
Also, in some sense Qubes is doing exactly what Carol says. Qubes only has a chance of working because the fundamental design for hardware-assisted security-by-isolation trumps all other considerations in their trade-offs. The UI is fundamentally constrained (to prevent window-redressing), as is performance (3d accelleration) and ease-of-use. All these constraints were known and documented before even a single line of code was written (afaik). Qubes can only work because it has security as one of its main goals, and has brilliant security people as project leads with infinite internal political capital.
That said, going on a tangent about qubes:
I really want to see painless live-migration of Qubes (migrate an application between different hosts, without interupting—say, from a lousy netbook to a fat workstation and back), this would be a killer feature for non-security-nerds. Unfortunately xen cannot do x86 <-> arm (qemu?); live-migration
smartphone<->workstation would be awesome (just bring my smartphone, plug it in as a boot-drive and continue your work on a fat machine—secure as long as there is no hardware implant).
Re Qubes security: You still have the bad problem of timing-sidechannels which cross VM borders; you should view Qubes as an awesome mitigation, not a solution (not to speak of the not-so-rare xen outbreaks), and you still need to secure your software. That is, Qubes attempts to prevent privelege escalation, not code exec; if the vulnerability is in the application which handles your valuable data, then Qubes cannot help you.

daozaich 29 Nov 2017 10:34 UTC
1 point
in reply to: John_Maxwell’s comment on: Security Mindset and the Logistic Success Curve
Yep. The counter-example would be Apple iOS.
I never expected it to become as secure as it did. And Apple security are clowns (institutionally, no offense inteded for the good people working there), and UI tends to beat security in tradeoffs.

daozaich 2 Dec 2017 20:34 UTC
15 points
in reply to: John_Maxwell’s comment on: Security Mindset and the Logistic Success Curve
Real-world anectdata how one big company (medical equipment) got OK at security:
At some time they decided that security was more important now. Their in-house guy (dev->dev management → “congrats, you are now our chief security guy”) got to hire more consultants for their projects, went to trainings and, crucially, went to cons (e.g. defcon). He was a pretty nice guy, and after some years he became fluent at hacker-culture. In short, he became capable of judging consultant’s work and hiring real security people. And he made some friends on the way. I think this is the best path to aquire institutional knowledge: Take a smart person loyal to the company, immerse them into the knowledgable subculture (spending money on failures on the way), use the aquired knowledge to hire real professionals (really hire, or hire as consultants for projects, or hire to give trainings).
Different big company (not software related), same thing. After some years their security guys became fed up with their lack of internal political capital, quit and switched career to “real” security.

daozaich 14 Dec 2017 20:10 UTC
3 points
on: Against the Linear Utility Hypothesis and the Leverage Penalty
The assumption I’m talking about is that the state of the rest of the universe (or multiverse) does not affect the marginal utility of there also being someone having certain experiences at some location in the uni-/multi-verse.
Now, I am not a friend of probabilities / utilities separately; instead, consider your decision function.
Linearity means that your decisions are independent of observations of far parts of the universe. In other words, you have one system over which your agent optimizes expected utility; and now compare it to the situation where you have two systems. Your utility function is linear iff you can make decisions locally, that is, without considering the state of the other system.
Clearly, almost nothing has a linear decision / utility function.
I think people mistake the following (amazing) heuristic for linear utility: If there are very many local systems, and you have a sufficiently smooth utility and probability distribution for all of them, then you can do mean-field: You don’t need to look, the law of large numbers guarantees strong bounds. In this sense, you don’t need to couple all the systems, they just all couple to the mean-field.
To be more practical: Someone might claim to have almost linear (altruistic) utility for QALYs over the 5 years (so time-discounting is irrelevant). Equivalently, whether some war in the middle east is terrible or not does not influence his/her malaria-focused charity work (say, he/she only decides on this specific topic).
Awesome, he/she does not need to read the news! And this is true to some extent, but becomes bullshit at the tails of the distribution. (the news become relevant if e.g. the nukes fly, because they bring you into a more nonlinear regime for utility; on the other hand, given an almost fixed background population, log-utility and linear-utility are indistinguishable by Taylor’s rule)
Re pascal’s muggle: Obviously your chance of getting a stroke and hallucinating weird stuff outweighs your chance of witnessing magic. I think that it is quite clear that you can forget about the marginal cost of giving 5 bucks to an imaginary mugger before the ambulance arrives to maybe save you; decision-theoretically, you win by precommitting to pay the mugger and call an ambulance once you observe something sufficiently weird.

daozaich 17 Dec 2017 1:26 UTC
1 point
in reply to: Ben Pace’s comment on: Against the Linear Utility Hypothesis and the Leverage Penalty
Thanks, and sorry for presumably messing up the formatting.

daozaich 2 Mar 2018 23:12 UTC
12 points
on: Arguments about fast takeoff
I imagine the “secret sauce” line of thinking as “we are solving certain problems in the wrong complexity class”. Changing complexity class of an algorithm introduces a discontinuity; when near a take-off, then this discontinuity can get amplified into a fast take-off. The take-off can be especially fast if the compute hardware is already sufficient at the time of the break-through.
In other words: In order to expect a fast take-off, you only need to assume that the last crucial sub-problem for recursive self-improvement / explosion is done in the wrong complexity class prior to the discovery of a good algorithm.
For strong historical precedents, I would look for algorithmic advances that improved empirical average complexity class, and at the same time got a speed-up of e.g. 100 x on problem instances that were typical prior to the algorithmic discovery (so Strassen matrix-multiply is out).
For weaker historical precedent, I would look for advances that single-handedly made the entire field viable—that is, prior to the advance one had a tiny community caring about the problem; post the advance, the field (e.g. type of data analysis) became viable at all (hence, very limited commercial / academic interest in the subfield prior to its breakthrough). I think that this is meaningful precedent because people optimize for expected pay-off, and it is sometimes surprising that some small-but-crucial-if-possible subproblem can be solved at all (reasonable quickly)!
And I do believe that there are many parts of modern ML that are in the wrong complexity class (this does not mean that I could do better, nor that I necessarily expect an improvement or even discontinuous jump in usefulness).

daozaich 3 Mar 2018 19:16 UTC
10 points
in reply to: paulfchristiano’s comment on: Arguments about fast takeoff
Not sure. I encountered this once in my research, but the preprint is not out yet (alas, I’m pretty sure that this will still be not enough to reach commercial viability, so pretty niche and academic and not a very strong example).
Regarding “this is not common”: Of course not for problems many people care about. Once you are in the almost-optimal class, there are no more giant-sized fruit to pick, so most problems will experience that large jumps never, once or twice over all of expected human history (sorting is $N log N$ even if you are a super-intelligence) (pulling the numbers 0,1,2 out of my ass; feel free to do better). On the other hand, there is a long tail of problems very few people care about (e.g. because we have no fast solution and hence cannot incorporate a solver into a bigger construction). There, complexity-class jumps do not appear to be so uncommon.
Cheap prediction: Many visual machine-learning algos will gain in complexity class once they can handle compressed data (under the usual assumption that, after ordering by magnitude, coefficients will decay like a power-law for suitable wavelet-transforms of real-world input data; the “compression” introduces a cut-off / discretization, and a cool ML algo would only look at the coefficients it cares about and hence be able to e.g. output a classification in finite time for infinite input data). Also cheap prediction: This will turn out to be not as hot as it sounds, since GPUs (and to a lesser extent CPUs) are just too damn good at dealing with dense matrices and FFT is just too damn fast.
-----
Since I’m bad at computer history, some maybe-examples that spring to mind:
Probable example: Asymmetric crypto.
Very-maybe examples:
Fast fourier transform (except if you want to attribute it to Gauss). Linear programming (simplex algorithm). Multi-grid methods for PDE / the general shift from matrix-based direct solvers to iterative operator-based solvers. Probabilistic polynomial identity testing. Bitcoin (not the currency, rather the solution to trust-less sybil-safe global consensus that made the currency viable). Maybe Zk-snark (to my eternal shame I must admit that I don’t understand the details of how they work).
For large economic impact, possibly discontinuous: MP3 / the moment where good audio compression suddenly became viable and made many applications viable. I think this was more of an engineering advance (fast software decoding, with large hardware overhang on general-purpose CPUs), but others here probably know more about the history.
Everyone, feel free to list other historic examples of kinda-discontinuous algorithmic advances and use your superior historical knowledge to tear down my probably very bad examples.
-----
Separate point: Many processes require “critical mass” (which I called “viability”), and such processes amplify discontinuities / should be modeled as discontinuities.
Physics / maths intuition pump would be phase transitions or bifurcations in dynamical systems; e.g. falling off a saddle-node looks quite discontinuous if you have a separation of time-scales, even if it is not discontinuous once you zoom in far enough. Recursive self-improvement / general learning does have some superficial resemblance to such processes, and does have a separation of time-scales. Tell me if you want me to talk about the analogies more, but I was under the impression that they have been discussed ad-nauseam.
-----
I’ll continue to think about historic examples of complexity-class-like algorithmic advancements with economic impact and post if anything substantial comes to mind.

daozaich 3 Mar 2018 20:06 UTC
4 points
on: The abruptness of nuclear weapons
Just commenting that the progress to thermonuclear weapons represented another discontinuous jump (1-3 orders of magnitude).
Also, whether von Neumann was right depends on the probability for the cold war ending peacefully. If we retrospectively conclude that we had a 90% chance of total thermonuclear war (and just got very lucky in real life) then he was definitely right. If we instead argue from the observed outcome (or historical studies conclude that the eventual outcome was not due to luck but rather due to the inescapable logic of MAD), then he was totally nuts.
Near-misses are not necessarily a very strong guide to estimating retrospective risk. Both sides were incentivized to hide their fail-safes for escalation; to credibly commit to having a twitchy retaliation finger, and at the same time to not actually retaliate (the game is first chicken, then ultimatum, and never prisoner’s dilemma). So I would be very wary of trusting the historical record on “what if Petrov had not kept a cool mind”.

daozaich 7 Mar 2018 1:54 UTC
4 points
on: Takeoff Speed: Simple Asymptotics in a Toy Model.
(1) As Paul noted, the question of the exponent alpha is just the question of diminishing returns vs returns-to-scale.
Especially if you believe that the rate $f = f (R)$ is a product of multiple terms (like e.g. Paul’s suggestion $f = R^{α_{t}} \cdot R^{α_{a}}$ with one exponent for computer tech advances and another for algorithmic advances) then you get returns-to-scale type dynamics (over certain regimes, i.e. until all fruit are picked) with finite-time blow-up.
(2) Also, an imho crucial aspect is the separation of time-scales between human-driven research and computation done by machines (transistors are faster than neurons and buying more hardware scales better than training a new person up to the bleeding edge of research, especially considering Scott’s amusing parable of the alchemists).
Let’s add a little flourish to your model: You had the rate of research $I$ and the cumulative research $R$ ; let’s give a name $C$ to the capability of the AI system. Then, we can model $\partial_{t} R = I = f (R) = g (C) = g (h (R))$ . This is your model, just splitting terms into $h$ , which tells us how hard AI progress is, and $g$ which tells us how good we are at producing research.
Now denote by $q = q (C)$ the fraction of work that absolutely has to be done by humans, and by $ε$ the speed-up factor for silicon over biology. Amdahl’s law gives you $g (C) = \frac{1}{q (C) + ε (1 - q (C)) C}$ , or somewhat simplified $g (C) \geq \frac{1}{q + ε C}$ . This predicts a rate of progress that first looks like $1 / q$ , as long as human researcher input is the limiting factor, then becomes $1 / (ε C)$ when we have AIs designing AIs (recursive self-improvement, aka explosion), and then probably saturates at something (when the AI approaches optimality).
The crucial argument for fast take-off (as far as I understood it) is that we can expect $q (C)$ to hit $q = 0$ at some cross-over $C^{*}$ , and we can expect this to happen with a nonzero derivative $\partial_{C} q (C^{*}) \neq 0$ . This is just the claim that human-level AI is possible, and that the intelligence of the human parts of the AI research project is not sitting at a magical point (aka: this is generic, you would need to fine-tune your model to get something else).
The change of the rate of research output from the $1 / q (C)$ regime to the $1 / (ε C)$ regime sure looks like a hard-take-off singularity to me! And I would like to note that the function $h$ , i.e. the hardness AI research and the diminishing-returns vs returns-to-scale debate does not enter this discussion at any point.
In other words: If you model AI research as done by a team of humans and proto-AIs assisting the humans; and if you assert non-fungibility of humans vs proto-AI-assistents (even if you buy a thousand times more hardware, you still need the generally intelligent human researchers for some parts); and if you assert that better proto-AI-assistents can do a larger proportion of the work (at all); and if you assert that computers are faster than humans; then you get a possibly quite wild change at $q = 0$ .
I’d like to note that the cross-over is not “human-level AI”, but rather “ $q \approx 0$ ” , i.e. an AI that needs (almost) no human assistence to progress the field of AI research.
On the opposing side (that’s what Robin Hanson would probably say) you have the empirical argument that $q$ should decay like a power-law long before we $q = 0$ (“the last 10% take 90% of the work” is a folk formulation for “percentile 90-99 take nine time as much work as percentile 0-89″ aka power law, and is borne out quite well, empirically).
This does not have any impact on whether we cross $q = 0$ with non-vanishing derivative, but would support Paul’s view that the world will be unrecognizably crazy long before $q = 0$ .
PS. I am currently agnostic about the hard vs soft take-off debate. Yeah, I know, cowardly cop-out.
edit: In the above, C kinda encodes how fast / good our AI is and q encodes how general it is compared to humans. All AI singularity stuff tacitly assumes that human intelligence (assisted by stupid proto-AI) is sufficiently general to design an AI that exceeds or matches the generality of human intelligence. I consider this likely. The counterfactual world would have our AI capabilities saturate at some subhuman level for a long time, using terribly bad randomized/evolutionary algorithms, until it either stumbles unto an AI design that has better generality or we suffer unrelated extinction/heat-death. I consider it likely that human intelligence (assisted by proto-AI) is sufficiently general for a take-off. Heat-death is not an exaggeration: Algorithms with exponentially bad run-time are effectively useless.
Conversely, I consider it very well possible that human intelligence is insufficiently general to understand how human intelligence works! (we are really, really bad at understanding evolution/gradient-descent optimized anything, an that’s what we are)

daozaich 8 Mar 2018 23:49 UTC
5 points
on: Prize for probable problems
[Meta: Even low-effort engagement, like “known + keyword” or “you misunderstood everything; read <link>” or “go on talking / thinking” is highly appreciated. Stacks grow from the bottom to the top today, unlike x86 or threads on the internet]
------------
Iterative amplification schemes work by having each version $i + 1$ trained by previous iteration $i$ ; and, whenever version $i$ fails at finding a good answer (low confidence in the prediction), punting the question to $i - 1$ , until it reaches the human overseer at $i = 0$ , which is the ground truth for our purposes. At the same time, the distribution $D_{i}$ of relevant question widens at each iteration, as capabilities improve: For example, a very stupid clippy will need to deal with simple linear optimization problems of its supply chain; a very smart clippy will need to figure out a proper plan for world-domination.
Now, in order to correctly punt questions down the call-stack, each level $i$ when faced with a problem $q$ must decide whether it can solve it (providing training data for level $i + 1$ ), or whether it must punt downwards. More precisely, level $i$ must recognize whether $q$ is inside the distribution $D_{i - 1}$ it was trained with (resp. $D_{i}$ , the distribution of problems it can generate / handle post-training).
False positives (underconfidence) lead to increased load on the human overseers / capability loss. False negatives (overconfidence) lead to safety failures: If they happen during training, then the machine learns wrong values / ethics / whatever-it-should-learn, if they happen in production then they lead to bad actions (I am assuming that no one would run the AI unsupervised, but supervision will, by necessity, be pretty light).
This is a problem, even with a very shallow call-stack (few iterations), but it gets (possibly exponentially!) worse with deeper stacks: After all, if even a single false-negative mistake occurs along the chain, we have a safety incident; naively, the chance of mistakes grows exponentially in the depth, for fixed false-negative rate at each level. In other words, we would need to ensure that the infinite series $\sum_{i} log (1 - p_{i})$ converges, where $p_{i}$ is the false-negative rate at level $i$ .
Something that may makes this problem either simpler or harder is the fact that, once a question has been bumped down a couple of times, it can be very far out-of-distribution. This makes it much easier to get mediocre reliability (the questions bumped down from very high levels are very weird for the low levels), but, at the same time, the low levels are really, really not equipped for dealing with these weird questions. In the clippy example, Paul’s scheme would totally allow the high capability clippy to convince low levels into accepting its proposal by exploiting its weaknesses (adversarial examples).
In different words: The entire scheme needs really good anomaly detection. Anomaly detection is a machine-learning problem, right? So learn this along the way?
The problem is that, while training the anomaly detection at level $i$ , we cannot sample from the distribution $D_{i + N}$ , because we simply don’t know it yet. As we run amplification, we extend both the capability of answering questions and the range of questions that come up.
Note that this would not be a problem if the extensions of supports of the distributions $D_{i}$ were very mild, compared to the larger domain $Ω_{i}$ where the anomaly detection of level $i$ is reliable-by-training. This appears like a very dangerous assumption on the “mind-space” accessible to our training procedure, for the “problem-space” at hand and the specific training procedure. In other words: No universality of safety amplification, if it relies on this assumption. Please link me to relevant discussions if this assumption has been justified anywhere.
A different approach would be to not run the scheme in a hierarchical way; instead try to feed it into itself. This reintroduces the problem of “drift”, i.e. instability: The system will acquire a specific culture, and we can only pray (i.e. must assume / ensure) that desireable states are an attractor, for the specific training procedure. I fear that this is extra-hard to check, because meta-stable systems look pretty stable, until you increase capability: Bad scaling.
A third approach to solving this problem would be to create, separately, a reliable anomaly detector. This may very well be an AI complete problem, but it may be simpler than alignment! Further, robust anomaly detection (recognizing that data points are outside of the training distribution) might be an avenue for differential progress that matters more for safety/alignment than capabilities.
Do you know other approaches for this problem?
------------
When is this harmless?
Well, this is entirely harmless if our problem domain contains only simple questions, where verification of ground truth is dirt cheap. For example, NP questions (like, e.g., all of mathematics! Proof verification is trivial after all). This observation should give us a bad feeling, because an AI that is really good at mathematics / programming and bad at everything else is one of the most dangerous situation we could imagine (magical function optimizers, aka mindless AIXI genies, being the only more hopeless case I can think of). On the other hand, typical NP questions don’t scale down: It is currently entirely infeasible to use machine learning for theorem proving, simply because useful transformations are exponentially rare in the space of possible ones (I am aware of some papers using the mizar library; while the neural net + theorem prover beat the unaided prover, I was less than impressed by the results).
For problem domains that feel more like exptime, this is more likely to be a problem: Say, training to play games like Go. Then, we can play against our ancestors in order to judge performance, and gain access to some kind of ground truth. Unfortunately, (1) strength is not linearly ordered: You clearly can have situations where A beats B beats C beats A, and (2) if we wanted to optimize “strength against perfect play”, aka min-max, then we don’t have access to a perfect opponent during training. Afaik it is usual for training-through-amplification of Game AI to develop “fads”, i.e. cheesy tactics, on the way; sometimes, these recur cyclically. This is also observed for the metagame in many multiplayer videogames. I have a feeling that the Go successes tell us a lot about how MCTS is amazingly stable against cheesy tactics; and who knows how much tweaking deepmind had to do until they got the amplification stable.
Now, safety amplification / value learning has a much, much harder problem: The ground truth is only accessible through examples / very expensive oracle queries (which might be fundamentally unsafe, at very high levels of capability: Don’t let human operators talk to unaligned too-clever AI).
------------
Post-script: Writing this down in clear words made me slightly update against Paul’s amplification schemes eventually growing into a solution. I still think that Paul’s line of research is damn cool and promising, so I’m more playing devil’s advocate here. The possible differential gain for capability in NP problems versus harder-than-NP alignment for this kind of amplification procedure made me slightly more pessimistic about our prospects in general. Moreover, it makes me rather skeptic whether amplification is a net win for safety / alignment in the differential progress view. I want to look more into anomaly detection now, for fun, my own short-term profit and long-term safety.

daozaich 9 Mar 2018 16:53 UTC
5 points
on: Why mathematics works
I strongly disagree that anthropics explains the unreasonable effectiveness of mathematics.
You can argue that a world, where people develop a mind and mathematical culture like ours (with its notion of “modular simplicity”) should be a world where mathematics is effective in everyday phenomena like throwing a spear.
This tells us nothing about what happens if we extrapolate to scales that are not relevant to everyday phenomena.
For example, physics appears to have very simple (to our mind) equations and principles, even at scales that were irrelevant during our evolution. The same kind of thought-processes are useful both for throwing spears / shooting cannons and for describing atoms. This is the unreasonable effectiveness of mathematics.
On the other hand, there are many phenomena where mathematics is not unreasonably effectice; take biological systems. There, our brains / culture have evolved heuristics that are useful on a human scale, but are entirely bogus on a micro-scale or macro-scale. Our mathematics is also really bad at describing the small scales; reductionism is just not that useful for understanding, say, how a genome defines an organism, and our brains / culture are not adapted to understanding this.
I think a counterfactual world, where physics outside the scales of human experience were as incomprehensibly complex (to our minds) as biology outside human scales does sound realistic. It does seem like a remarkable and non-trivial observation that the dynamics of a galaxy, or the properties of a semiconductor, are easy to understand for a culture that learned how a cannonball flies; whereas learning how to cultivate wheat or sheep is not that helpful for understanding cancer.

daozaich 11 Mar 2018 22:04 UTC
1 point
in reply to: norswap’s comment on: Brains and backprop: a key timeline crux
I think part of the assumption is that reflection can be bolted on trivially if the pattern matching is good enough. For example, consider guiding an SMT / automatic theorem prover by deep-learned heuristics, e.g. (https://arxiv.org/abs/1701.06972)[https://arxiv.org/abs/1701.06972] . We know how to express reflection in formal languages; we know how to train intuition for fuzzy stuff; me might learn how to train intuition for formal languages.
This is still borderline useless; but there is no reason, a priori, that such approached are doomed to fail. Especially since labels for training data are trivial (check the proof for correctness) and machine-discovered theorems / proofs can be added to the corpus.

daozaich 6 Apr 2018 12:57 UTC
2 points
on: Against Occam’s Razor
I have a feeling that you mix probability and decision theory. Given some observations, there are two separate questions when considering possible explanations / models:
1. What probability to assign to each model?
2. Which model to use?
Now, our toy-model of perfect rationality would use some prior, e.g. the bit-counting universal/kolmogorov/occam one, and bayesian update to answer (1), i.e. compute the posterior distribution. Then, it would weight these models by “convenience of working with them”, which goes into our expected utility maximization for answering (2), since we only have finite computational resources after all. In many cases we will be willing to work with known wrong-but-pretty-good models like Newtonian gravity, just because they are so much more convenient and good enough.
I have a feeling that you correctly intuit that convenience should enter the question which model to adopt, but misattribute this into the probability—but which model to adopt should formally be bayesian update + utility maximization (taking convenience and bounded computational resources into account), and definitely not “Bayesian update only”, which leads you to the (imho questionable) conclusion that the universal / kolmogorov / occam prior is flawed for computing probability.
On the other hand, you are right that the above toy model of perfect rationality is computationally bad: Computing the posterior distribution after some prior and then weighting by utility/convenience is of stupid if directly computing prior * convenience is cheaper than computing prior and convenience separately and then multiplying. More generally, probability is a nice concept for human minds to reason about reasoning, but we ultimately care about decision theory only.
Always combining probability and utility might be a more correct model, but it is often conceptually more complex to my mind, which is why I don’t try to always adopt it ;)

daozaich 6 Apr 2018 15:03 UTC
3 points
in reply to: zulupineapple’s comment on: Against Occam’s Razor
>But the greatest merit of Occamian prior is that it vaguely resembles the Lazy prior.
...
>With that in mind, I asked what prior would serve this purpose even better and arrived at Lazy prior. The idea of encoding these considerations in a prior may seem like an error of some kind, but the choice of a prior is subjective by definition, so it should be fine.
Encoding convenience * probability into some kind of pseudo-prior such that the expected-utility maximizer is the maximum likelihood model with respect to the pseudo-prior does seem like a really useful computational trick, and you are right that terminology should reflect this. And you are right that the Occam prior has the nice property that weight-by-bit-count is often close to convenience, and hence makes the wrong naive approach somewhat acceptable in practice: That is, just taking the max likelihood model with respect to bit-count should often be a good approx for weight-by-bitcount * convenience (which is the same as weight-by-bitcount for probability and maximize expected utility).
In cases where we know the utility we can regenerate probabilities afterwards. So I would now be really interested in some informal study of how well Occam actually performs in practice, after controlling for utility: You are right that the empirical success of Occam might be only due to the implicit inclusion of convenience (succinct-by-bit-count models are often convenient) when doing the (wrong!) max-likelihood inference. I had not considered this, so thanks also for your post; we both learned something today.
I’d also remark/reiterate the point in favor of the Lazy prior: The really terrible parts of working with Occam (short descriptions that are hard to reason about, aka halting problem) get cancelled out in the utility maximization anyway. Lazy avoids invoking the halting-problem oracle in your basement for computing these terms (where we have the main differences between Occam vs Lazy). So you are right after all: Outside of theoretical discussion we should all stop using probabilities and Occam and switch to some kind of Lazy pseudo-prior. Thanks!
That being said, we all appear to agree that Occam is quite nice as an abstract tool, even if somewhat naive in practice.
A different point in favor of Occam is “political objectivity”: It is hard to fudge in motivated reasoning. Just like the “naive frequentist” viewpoint sometimes wins over Bayes with respect to avoiding politically charged discussions of priors, Occam defends against “witchcraft appears natural to my mind, and the historical record suggests that humans have evolved hardware acceleration for reasoning about witchcraft; so, considering Lazy-prior, we conclude that witches did it” (Occam + utility maximization rather suggests the more palatable formulation “hence it is useful to frame these natural processes in terms of Moloch, Azatoth and Cthulhu battling it out”, which ends up with the same intuitions and models but imho better mental hygiene)

daozaich 6 Apr 2018 16:28 UTC
1 point
in reply to: Wei Dai’s comment on: Kaj’s shortform feed
Regarding measurement of pain:suffering ratio
A possible approach would be to use self-reports (the thing that doctor’s always ask about, pain scale 1-10) vs revealed preferences (how much painkillers were requested? What trade-offs for pain relief do patients choose?).
Obviously this kind of relation is flawed on several levels: Reported pain scale depends a lot on personal experience (very painful events permanently change the scale, ala “I am in so much pain that I cannot walk or concentrate, but compared to my worst experience… let’s say 3?”). Revealed preferences depend a lot on how much people care about the alternatives (e.g. if people have bad health insurance or really important stuff to do they might accept a lot of subjective suffering in order to get out of hospital one day early). Likewise, time preference might enter a lot into revealed preference.
Despite these shortcomings, that’s where I would start thinking about what such a ratio would mean. If one actually did a study with new questionaires, one should definitely ask patients for some examples in order to gauge their personal pain-scale, and combine actual revealed preferences with answers to hypothetical questions “how much money would pain relief be worth to you? How much risk of death? How many days of early hospital release? etc”, even if the offer is not actually on the table.

daozaich 16 Apr 2018 11:32 UTC
2 points
on: A voting theory primer for rationalists
Re SODA: The setup appears to actively encourage candidates to commit to a preference order. Naively, I would prefer a modification along the following lines; could you comment?
(1) Candidates may make promises about their preference order among other candidates; but this is not enforced (just like ordinary pre-election promises). (2) The elimination phase runs over several weeks. In this time, candidates may choose to drop out and redistribute their delegated votes. But mainly, the expected drop-outs will negotiate with expected survivors, in order to get at least some of their policies implemented (with the same kind of enforcement as regular pre-election promises). Hence, this phase is “coalition building”.
An interesting final phase (3), in order to encourage compromise / beneficial trade would be something like: Once we are down to candidates with > 25% approval, we randomize the result. The probability of a candidate to win could be something like the square, or third power, of his approval. The threshold of 25% is in order to prevent complete crackpots from winning the election, and might need to be even more brutal. The randomization serves to allow the two remaining highest approval platforms to negotiate a compromise, weighted by their chance of winning the final lottery. In practice, one would hope that the randomization is never applied: That is, the highest candidate makes enough concessions in order to make the second candidate agree to drop out.
This way, we preserve minority rights, and get a pretty good outcome if the candidates are actually reasonable and trustworthy parties (instead of persons) that are capable of committing to modify the values they will espouse once in power.
Obvious disadvantages are (1) negotiation skill and integrity become essential (integrity in the christiano sense: You need to be predictable in which promises you will break under which contingencies), because the winner is decided though coalition building, rather than voting; and (2) some voters might object to shady backroom deals being the explicit procedure, and (3) randomization may make losers very pissed if the dice are ever rolled (which hopefully is never, because of positive-sum trades).
You can obviously combine this with a 3-2-1 system (where only candidates reaching at least 50% “OK” votes are eligible, and otherwise “Good” votes are counted; if no candidate receives 50% OK, then the election is repeated, under the assumption that Belgium is better than Trump, i.e. better a constitutional crisis and government by interim civil servants than government by crazy elected officials).
edit: I forgot to mention the other big advantage of the probabilistic phase (3): Enforcing continuity, i.e. preventing small factions from gaining oversized influence by playing kingmaker.

daozaich 21 Apr 2018 23:14 UTC
2 points
in reply to: Stuart_Armstrong’s comment on: Weird question: could we see distant aliens?
I think communicating without essentially conquering the Hubble volume is still an interesting question. I would not rule out a future human ethical system that restricts expansion to some limited volume, but does not restrict this kind of omnidirectional communication. Aliens being alien, we should not rule out them having such a value system either.
That being said, your article was really nice. Send multiplying probes everywhere, watch the solar system form and wait for humans to evolve in order to say “hi” is likely to be amazingly cheap.