roystgnr

Karma: 2,757

roystgnr 4 Mar 2024 19:35 UTC
2 points
0
on: Good HPMoR scenes / passages?
The “understandable”+”exploit” category would include my personal favorite introduction, the experiment in Chapter 17. From “Thursday” to “that had been the scariest experimental result in the entire history of science” is about 900 words. This section is especially great because it does the whole “deconstruction of canon”/”reconstruction of canon” bit in one self-contained section; that pattern is one the best aspects of HPMOR but usually the setup and the payoff are dozens of chapters apart, with many so interleaved with the plot that the payoff counts as a major spoiler.

On the other hand, that section works best if you already know what P and NP and RSA cryptography are (and if you’re somehow still not too nerdy to wince at Harry’s lazy way of talking about the former); know your audience.

My kids’ favorite section is the introduction to Draco in Madam Malkin’s in Chapter 5, which IMO qualifies as “understandable”+”gobsmacking”, but it’s a bit long for you, 1500 words from “You are to get fitted for your robes, nothing else” to “poor Madam Malkin and her two poor assistants.”, and I don’t see any way to reasonably cut out a third of that without wrecking the humor.

roystgnr 28 Mar 2023 14:56 UTC
4 points
2
in reply to: Viliam’s comment on: Manifold: If okay AGI, why?
In economics, “we can model utility as logarithmic in wealth”, even after adding human capital to wealth, feels like a silly asymptotic approximation that obviously breaks down in the other direction as wealth goes to zero and modeled utility to negative infinity.

In cosmology, though, the difference between “humanity only gets a millionth of its light cone” and “humanity goes extinct” actually does feel bigger than the difference between “humanity only gets a millionth of its light cone” and “humanity gets a fifth of its light cone”; not infinitely bigger, but a lot more than you’d expect by modeling marginal utility as a constant as wealth goes to zero.
This is all subjective; others’ feelings may differ.

(I’m also open in theory to valuing an appropriately-complete successor to humanity equally to humanity 1.0, whether the successor is carbon or silicon or whatever, but I don’t see how “appropriately-complete” is likely so I’m ignoring the possibility above.)

roystgnr 27 Mar 2023 19:30 UTC
12 points
5
on: GPT-4 is bad at strategic thinking
It’s hard to apply general strategic reasoning to anything in a single forward pass, isn’t it? If your LLM has to come up with an answer that begins with the next token, you’d better hope the next token is right. IIRC this is the popular explanation for why LLM output seems to be so much better when you just add something like “Let’s think step by step” to the prompt.

Is anyone trying to incorporate this effect into LLM training yet? Add an “I’m thinking” and an “I’m done thinking” to the output token set, and only have the main “predict the next token in a way that matches the training data” loss function grade on tokens that aren’t in between those brackets. Then when you hit “What is 45235 + 259719? 304954” in the training set, optimization doesn’t have to discourage multi-step reasoning to reproduce that, because “<thinking>5+9=14, so we carry the 1“ … “</thinking>304954” is still worth just as much as an ex nihilo “304954”. Chess algorithms could do a brief tree search before outputting their final decision.

Add whatever regularization is needed to keep the “train of thought” in English rather than in whatever-cypher-the-optimizer-hits-on, and this would be an increase in safety, not just in capabilities. The more “internal reasoning” is human-readable text rather than maybe-a-posteriori-interpretable activation patterns, the better. You could even expose it to the end user: ask ChatGPT a question and you get a succinct answer, click on the “expose thoughts” button and you get the chain of support for the answer.

roystgnr 9 Sep 2022 19:17 UTC
4 points
1
on: Should you refrain from having children because of the existential risk posed by artificial intelligence?

I have never seen any convincing argument why “if we die from technological singularity it will” have to “be pretty quick”.

The arguments for instrumental convergence apply not just to Resource Acquisition as a universal subgoal but also to Quick Resource Acquistion as a universal subgoal. Even if “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else”, the sooner it repurposes those atoms the larger a light-cone it gets to use them in. Even if an Unfriendly AI sees humans as a threat and “soon” might be off the table, “sudden” is still obviously good tactics. Nuclear war plus protracted conventional war, Skynet-style, makes a great movie, but would be foolish vs even biowarfare. Depending on what is physically possible for a germ to do (and I know of no reason why “long asymptomatic latent phase” and “highly contagious” and “short lethal active phase” isn’t a consistent combination, except that you could only reach it by deliberate engineering rather than gradual evolution), we could all be dead before anyone was sure we were at war.

roystgnr 23 Aug 2022 3:45 UTC
3 points
0
on: Stable Diffusion has been released
That was astonishingly easy to get working, and now on my laptop 3060 I can write a new prompt and generate another 10-odd samples every few minutes. Of course, I do mean 10 odd samples: most of the human images it’s giving me have six fingers on one hand and/or a vaguely fetal-alcohol-syndrome vibe about the face, and none of them could be mistaken for a photo or even art by a competent artist yet. But they’re already better than any art I could make, and I’ve barely begun to experiment with “prompt engineering”; maybe I should have done that on easier subjects before jumping into the uncanny valley of realistic human images headfirst.
Only optimizedSD/optimized_txt2img.py works for me so far, though. scripts/txt2img.py, as well as any version of img2img.py, dies on my 6GB card with RuntimeError: CUDA out of memory.
Update: in the optimization fork at https://github.com/basujindal/stable-diffusion , optimized_txt2img.py works on my GPU as well.

roystgnr 9 Aug 2019 19:42 UTC
7 points
on: Can we really prevent all warming for less than 10B$ with the mostly side-effect free geoengineering technique of Marine Cloud Brightening?
we still need to address ocean acidification
And changes in precipitation patterns (I’ve seen evidence that reducing solar incidence is going to reduce ocean evaporation, independent of temperature).
There’s also the “double catastrophe” problem to worry about. Even if the median expected outcome of a geoengineering process is decent, the downside variance becomes much worse.
I still suspect MCB is our least bad near- to medium-term option, and even in the long term the possibility of targeted geoengineering to improve local climates is awfully tempting, but it’s not a panacea.
As an unrelated aside, that CCC link rates “Methane Reduction Portfolio” as “Poor”; I’d have marked it “Counterproductive” for the moment. The biggest long-term global warming problem is CO2 (thanks to the short half-life of methane), and the biggest obstacle to CO2 emissions reduction is voters who think global warming is oversold. Let the problem get bigger until it can’t be ignored, and then pick the single-use-only low hanging fruit.

roystgnr 17 Dec 2018 16:25 UTC
4 points
in reply to: moridinamael’s comment on: Preschool: Much Less Than You Wanted To Know
Alex has not skipped a grade or put in an some secret fast-track program for kids who went to preschool, because this does not exist.
Even more confounding: my kids have been skipping kindergarten in part because they didn’t go to preschool. My wife works from home, and has spent a lot of time teaching them things and double-checking things they teach themselves.
Preschools don’t do tracking any more than grade schools, so even if in theory they might provide better instruction than the average overworked parent(s), the output will be 100% totally-ready-for-kindergarten (who will be stuck as you describe), which in the long term won’t look as good as a mix of 95% not-quite-as-ready-for-kindergarten (who will catch up as you describe) and 5% ready-for-first-grade (who will permanently be a year ahead).

roystgnr 20 Aug 2018 20:13 UTC
6 points
in reply to: Sniffnoy’s comment on: Dissolving the Fermi Paradox, and what reflection it provides
Gah, of course you’re correct. I can’t imagine how I got so confused but thank you for the correction.

roystgnr 20 Jul 2018 18:52 UTC
1 point
in reply to: Sniffnoy’s comment on: Dissolving the Fermi Paradox, and what reflection it provides
You don’t need any correlation between $X$ and $Y$ to have $E [X Y] \neq E [X] E [Y]$ . Suppose both variables are 1 with probability .5 and 2 with probability .5; then their mean is 1.5, but the mean of their products is 2.25.

roystgnr 5 Jul 2018 22:00 UTC
3 points
in reply to: MrFailSauce’s comment on: Dissolving the Fermi Paradox, and what reflection it provides
Not quite. Expected value is linear but doesn’t commute with multiplication. Since the Drake equation is pure multiplication then you could use point estimates of the means in log space and sum those to get the mean in log space of the result, but even then you’d *only* have the mean of the result, whereas what would really be a “paradox” is if $P (N = 1 | N \neq 0)$ turned out to be tiny.

roystgnr 31 May 2018 21:58 UTC
14 points
on: Expressive Vocabulary
the thing you know perfectly well people mean when they say “chemicals”
I honestly don’t understand what that thing is, actually.
To use an example from a Facebook post I saw this week:
Is P-Menthane-3,8-diol (PMD) a chemical? What about oil from the lemon eucalyptus tree? Oil of Lemon Eucalyptus is typically refined until it’s 70% PMD instead of 2%; does that turn it into a chemical? What if we were to refine it all the way to 100%? What if, now that we’ve got 100% PMD, we just start using PMD synthesized at a chemical plant instead?
I do like the idea from another comment here that
the motte is “technically, everything is a chemical,” and the bailey is “No need to worry about the content of the food you buy.”
But even that can be inverted. People buy “no nitrites added” bacon to try to avoid a dangerous chemical, and they end up getting the result of (all natural! organic!) celery-juice and celery-powder processes. At best, the nitrates in the celery still end up getting converted to nitrites during curing, except that now there’s another loose variable in the production so a balance between “cardiac/cancer risk from high nitrite levels” versus “botulism risk from low nitrite levels” is even harder to achieve. At worst, the consumer falsely believes they’ve no further need to worry about the content of the food they buy, and so they don’t moderate consumption of it they way they would have moderated consumption of “chemicals”.

roystgnr 29 Jan 2018 18:36 UTC
4 points
on: The different types (not sizes!) of infinity
So, does 1+ω make sense? It does, for the ordinals and hyperreals only.
It make sense for cardinals (the size of “a set of some infinite cardinality” unioned with “a set of cardinality 1″ is “a set with the same infinite cardinality as the first set”) and in real analysis (if lim f(x) = infinity, then lim f(x)+1 = infinity) too.
What about −1+ω? That only makes sense for the hyperreals.
And for cardinals (the size of the set difference between “a set of some infinite cardinality” and “a subset of one element” is the same infinite cardinality) and in real analysis (if lim f(x) = infinity, then lim −1+f(x) = infinity) too.

roystgnr 6 Dec 2017 20:42 UTC
4 points
in reply to: yagudin’s comment on: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
I believe the answer to your second question is probably technically “yes”; if there’s any way in which AZ mispredicts relative to a human, then there’s some Ensemble Learning classifier that weights AZ move choices with human move choices and performs better than AZ alone. And because Go has so many possible states and moves at each state, humans would have to be much, much, much worse at play overall for us to conclude that humans were worse along every dimension.
However, I’d bet the answer is practically “no”. If AlphaZero vs the top humans is now an Elo difference of 1200, that gives a predicted human victory rate of about 1/1000. We’d never be able to play enough games to get enough systematic data to identify a dimension along which humans chose better moves. And even if we did, it’s entirely possible that the best response to that identification would be “give AlphaZero more training time on those cases”, not “give AlphaZero a human partner in those cases”. And even if we did decide to give AlphaZero a human partner, how often would the partner’s input end up overriding the move AlphaZero alone would have chosen? Would the human even be able to uselessly pay attention for game after game, just so they could be ready to contribute to a winning move on game 100?

roystgnr 8 Nov 2017 16:04 UTC
2 points
in reply to: magfrump’s comment on: LDL 2: Nonconvex Optimization
Oh, well in that case the point isn’t subtlely lacking, it’s just easily disproven. Given any function from I^N to R, I can take the tensor product with cos(k pi x) and get a new function from I^{N+1} to R which has k times as many non-globally-optimal local optima. Pick a decent k and iterate, and you can see the number growing exponentially with higher dimension, not approaching 0.
Perhaps there’s something special about the functions we try to optimize in deep learning, a property that rules out such cases? That could be. But you’ve said nothing special about (or even defined) a particular class of deep learning problems, rather you’ve made a claim about all higher dimensional optimization problems, a claim which has an infinite number of counterexamples.

roystgnr 7 Nov 2017 17:48 UTC
8 points
in reply to: Allen Rice’s comment on: Moloch’s Toolbox (1/2)
a Leviathan could try to transcend/straddle these fruits/niches and force them upward into a more Pareto optimal condition, maybe even into the non-Nash E. states if we’re extra lucky.
Remember that old Yudkowsky post about Occam’s Razor, wherein he points out how “a witch did it” sounds super-simple, but the word “witch” hides a ton of hidden complexity? I’m pretty sure you’re doing the same thing here with the word “could”. Instead of trying to picture what an imaginary all-powerful leader could do, imagine what a typical leader would do. You’ve just given a speech to Hillary Clinton about why it’s imperative to decimate the university system for economic reasons, or to Donald Trump about how protectionism is always Pareto-suboptimal. Do they react the way you wish them to act? Does their reaction make you want to give them more forceful power, or less?
For that matter, you don’t have to imagine any speeches—every politician and their staff have access to the same internet you do, and can learn everything about coordination problems and their solutions that you can, and even without totalitarian official power they still have enough of a “bully pulpit” and enough credibility, seriousness, and so on to push the world out of at least the shallowest non-globally-optimal local optima. How many of them are doing so? By contrast, how many of them are deliberately setting up more barriers to such changes?

roystgnr 7 Nov 2017 17:32 UTC
4 points
in reply to: ChristianKl’s comment on: Moloch’s Toolbox (1/2)
In the BFR announcment Musk promized that tickets from intercontinental rocket travel for the price of a current economy ticket.
“Full-fare” economy, which is much more expensive than even the “typical” international economy seat tickets you’re thinking of, but yes, and even outsiders don’t think it’s impossible. It is very sensitive to a lot of assumptions—third party spreadsheets I’ve seen say low thousands of dollars per ticket is possible, but it wouldn’t take many assumptions to fall short before prices jumped into the low tens of thousands.
If that’s the price to travel by rocket it would also be the price to go to a space station.
No; a space station trip could even be an order of magnitude more expensive. (Which I admit surprised me too).
An intercontinental ballistic arc can require nearly as much fuel as Low Earth Orbit, true, but some hops are shorter than others, and in every case the difference then goes through an exponential function in the rocket equation. BFR might end up needing half as much propellant on a mere transatlantic flight, and the whole “super cheap flights” plan is predicated on them reducing non-propellant costs to a small multiple of propellant costs, so that’s not a trivial change.
Also, an intercontinental flight can pack people in like sardines, with enough life support to go up and down again. A flight to a space station implicitly includes the cost of lifting and operating (and if it’s going to be big enough to be useful for a full BFR passenger flight, assembling) a space station which is roomy enough and well equipped enough for every passenger to stay a while.
Maybe there are some illnesses where you can benefit from being in zero g
This is a mainstay of classic Sci-Fi, but so far IIRC every effect we’ve found has been in the opposite direction. Zero G is like anti-exercise for the human body, like being bedridden.
and you can have your hospital outside of any jurisdiction.
This makes more sense, and (unlike the rest of my huge digression) gets straight at the heart of the post here, but I don’t think orbit adds any extra sense to the idea. There are a hundred little jurisdictions around the world who would happily issue the medical equivalent of a “Flag of Convenience” and waive whatever regulations you think are stupid in return for getting heavy investment and medical tourism. Even if there weren’t, and you had to go outside every jurisdiction, international waters a hundred miles offshore would have the (questionable!) legal status you’re looking for and would still be cheaper and more convenient than international space a hundred miles up.

roystgnr 23 Oct 2017 16:02 UTC
8 points
on: Four Scopes Of Advice
Failing to follow good strategic advice isn’t even the worst failure mode here; unless you’re lucky you may not be given any strategic advice at all in response to a tactical question. If nobody notices that you’re committing the XY Problem, then you may be given good advice for the tactical problem you asked about, follow it, and end up worse off than you were before with respect to the strategic problem you should have been asking about instead.

roystgnr 23 Oct 2017 13:46 UTC
4 points
on: LDL 2: Nonconvex Optimization
This argument doesn’t seem to take into account selection bias.
We don’t get into a local optimum becuase we picked a random point and wow, it’s a local optimum, what are the odds!?
We get into a local optimum because we used an algorithm that specifically *finds* local optima. If they’re still there in higher dimensions then we’re still liable to fall into them rather than into the global optimum.

roystgnr 16 Oct 2017 15:52 UTC
2 points
on: Why no total winner?
Is there some more general limit to power begetting power that would also affect AGI?
The only one which immediately comes to mind is inflexibility. Often companies shrink or fail entirely because they’re suddenly subject to competition armed with a new idea. Why do the new ideas end up implemented by smaller competitors? The positive feedback of “larger companies have more people who can think of new ideas” is dominated by the negative feedbacks of “even the largest company is tiny compared to its complement” and “companies develop monocultures where everyone thinks the same way” and “companies tend to internally suppress new ideas which would devalue the company’s existing assets”.
Any AGI that doesn’t already have half the world’s brainpower would be subject to the first limitation (which may mean any AGI who hasn’t taken over half the world or just any AGI less than an hour old, depending on how much “foom” the real world turns out to allow, I admit), and an AGI that propagates by self-copying might even be more affected than humans by the second limitation. Whether an AGI was subject to the third limitation or not would depend on its aspirations, I suppose; anything with a “conquer the world” endgame would hardly let itself get trapped in the “just stretch out current revenues as long as possible” mindset of Blockbuster-vs-Netflix...

roystgnr 13 Oct 2017 20:06 UTC
2 points
in reply to: Stuart_Armstrong’s comment on: Toy model of the AI control problem: animated version
You should provide some more explicit license, if you don’t want to risk headaches for others later. “yes [It’s generally intended to be Open Source]” may be enough reassurance to copy the code once, but “yes, you can have it under the new BSD (or LGPL2.1+, or whatever) license” would be useful to have in writing in the repository in case others want to create derived works down the road.
Thanks very much for creating this!