calef

Karma: 660

calef 19 Dec 2013 19:31 UTC
61 points
on: Open thread for December 17-23, 2013
A full half (20/40) of the posts currently under discussion are meetup threads.

Can we please segregate these threads to another forum tab (in the vein of the Main/Discussion split)?

Edit: And only 5 or so of them actually have any comments in them.

calef 19 Nov 2021 2:03 UTC
29 points
on: Ngo and Yudkowsky on AI capability gains
This was frustrating to read.

There’s some crux hidden in this conversation regarding how much humanity’s odds depend on the level of technology (read: GDP) increase we’ll be able to achieve with pre-scary-AGI. It seems like Richard thinks we could be essentially post-scarcity, thus radically changing the geopolitical climate (and possibly making collaboration on an X-risk more likely? (this wasn’t spelled out clearly)). I actually couldn’t suss out what Eliezer thinks from this conversation—possibly that humanity’s odds are basically independent of the achieved level of technology, or that the world ends significantly sooner than we’ll be able to deploy transformative tech, so the point is moot. I wish y’all had nailed this down further.

Despite the frustration, this was fantastic content, and I’m excited for future installments.

calef 13 Aug 2013 4:13 UTC
29 points
on: Does Checkers have simpler rules than Go?
It’s also worth mentioning that checkers is solved.

calef 23 Feb 2015 3:34 UTC
26 points
on: How to debate when authority is questioned, but really not needed?
Honestly, the only “winning” strategy here is to not argue with people on the comments sections of political articles.

If you must, try and cast the argument in a way that avoids the standard red tribe / blue tribe framing. Doing this can be hard because people generally aren’t in the business of having politics debate with an end goal of dissolving an issue—they just want to signal their tribe—hence why arguing on the internet is often a waste of time.

As to the question of authority: how would you expect the conversation to go if you were an economist?

Me: I think money printing by the Fed will cause inflation if they continue like this.

Random commenter: Are you an economist?

Me: Yes actually, I have a PhD in The Economy from Ivy League University.

Random commenter (possible response 1): I don’t believe you, and continue to believe what I believe.

Random commenter (possible response 2): Oh well that’s one of the (Conservative / Liberal) (pick one) schools, they’re obviously wrong and don’t know what they’re talking about.

Random commenter (possible response 3): Economists obviously don’t know what they’re talking about.

Again, it’s a mix of Dunning-Kruger and tribal signalling. There’s not actually any direction an appeal-to-authority debate can go that’s productive because the challenger has already made up their mind about the facts being discussed.

For a handful of relevant lesswrong posts: http://lesswrong.com/lw/axn/6_tips_for_productive_arguments/ http://lesswrong.com/lw/gz/policy_debates_should_not_appear_onesided/ http://lesswrong.com/lw/3k/how_to_not_lose_an_argument/

calef 8 Dec 2014 22:59 UTC
22 points
in reply to: NancyLebovitz’s comment on: Stupid Questions December 2014
According to http://arxiv.org/abs/astro-ph/0503520 we would need to be able to boost our current orbital radius to about 7 AU.

This would correspond to a change in specific orbital energy of 132712440018/(2(1 AU)) to 132712440018 / (2(7 AU)). (where the 12-digit constant is the standard gravitational parameter of the sun. This is like 5.6 10^10 in Joules / Kilogram, or about 3.4 10^34 Joules when we restore the reduced mass of the earth/sun (which I’m approximating as just the mass of the earth).

Wolframalpha helpfully supplies that this is 28 times the total energy released by the sun in 1 year.

Or, if you like, it’s equivalent to the total mass energy of ~3.7 * 10^18 Kilograms of matter (about 1.5% the mass of the asteroid Vespa).

So until we’re able to harness and control energy on the order of magnitude of the total energetic output of the sun for multiple years, we won’t be able to do this any time soon.

There might be an exceedingly clever way to do this by playing with orbits of nearby asteroids to perturb the orbit of the earth over long timescales, but the change in energy we’re talking about here is pretty huge.

calef 30 Jun 2013 7:14 UTC
21 points
on: Harry Potter and the Methods of Rationality discussion thread, part 19, chapter 88-89
Two interesting observations: The most recent utterances of the words “temporal pressure” (similar to the titles of these last two chapters) was in chapter 86 when discussing the Halls of Prophecy:

“”The Hall of Prophecy,” Minerva whispered. She’d read about that place, said to be a great room of shelves filled with glowing orbs, one after another appearing over the years. Merlin himself had wrought it, it was said; the greatest wizard’s final slap to the face of Fate. Not all prophecies conduced to the good; and Merlin had wished for at least those spoken of in prophecy, to know what had been spoken of them. That was the respect Merlin had given to their free will, that Destiny might not control them from the outside, unwitting. Those mentioned within a prophecy would have an glowing orb float to their hand, and then hear the prophet’s true voice speaking. Others who tried to touch an orb, it was said, would be driven mad—or possibly just have their heads explode, the legends were unclear on this point. Whatever Merlin’s original intention, the Unspeakables hadn’t let anyone enter in centuries, so far as she’d heard. Works of the Ancient Wizards had stated that later Unspeakables had discovered that tipping off the subjects of prophecies could interfere with seers releasing whatever temporal pressures they released; and so the heirs of Merlin had sealed his Hall. It did occur to Minerva to wonder (now that she’d spent a few months around Mr. Potter) how anyone could possibly know that; but she also knew better than to ask Albus, in case Albus tried to tell her. Minerva firmly believed that you only ought to worry about Time if you were a clock.”

Next, near the end of chapter 89, Harry “turns away from Dumbledore” twice, almost as if there’s a kink in time after explicitly mentioning “pressure”:

Harry opened his mouth to scream out all his fury, and then closed it again. There wasn’t any point in screaming, it wouldn’t accomplish anything. The unbearable pressure rising inside him couldn’t be let out that way.

’Harry turned away from Dumbledore and looked down at where the remains of Hermione Granger were lying in a pool of blood … another part of him already knew that this event was real, part of the same flawed world that included Azkaban and the Wizengamot chamber and

No

With a fracturing feeling, as though time was still torn to pieces around him, Harry turned away from Dumbledore and looked down at the remains of Hermione Granger lying in a pool of blood with two tourniquets tied around her thigh-stumps, and decided

No.

I do not accept this.”

Hypothesis: Something very strange just happened to Time around Harry, possibly involving a time turner, and things are definitely not as they seem.

It’s almost as if someone is trying to manipulate Harry’s reaction to these events, like when H&C tricks Hermione repeatedly. Fred and George are unconscious at this point, too. It goes without saying that Harry is probably an unreliable narrator.
What links here?
- Qiaochu_Yuan's comment on Harry Potter and the Methods of Rationality discussion thread, part 19, chapter 88-89 by Vaniver (2 Jul 2013 1:20 UTC; 0 points)

calef 23 Mar 2012 18:34 UTC
21 points
on: Nonmindkilling open questions
“What is the probability there is microbial-like life (other than from earth) in our solar system?”

I’m having difficulty giving this a good estimate, myself, actually.

calef 30 Oct 2014 1:19 UTC
20 points
on: Wikipedia articles from the future
This article is marked as controversial and has been locked, see talk page for details.

Quantum computing winter

The Quantum computing winter was the period from 1995 to approximately October 2031 when experimental progress on the creation of fault tolerant quantum computers stalled despite significant effort at constructing the machines. The era ended with the publication of the Kitaev-Kalai-Alicki-Preskill (KKAP) theorem in early 2030 which purported to show that the construction of fault-tolerant quantum computers was in fact impossible due to fundamental constraints. The theorem was not widely accepted until experiments performed by Mikhail Lukin’s group in early 2031 verified the bounds provided in the KKAP theorem.

Early history

Quantum computing technology looked promising in the late 20th and early 21st century due to the celebrated Fault Tolerance theorems, as well as the rapid experimental progress towards satisfying the fault tolerance threshold. The Fault Tolerance theorem, which at the time was thought to be based on reasonable assumptions, guaranteed scalable, fault tolerant quantum computation could be performed—provided an architecture could be built that had an error rate smaller than a known bound.

In the early 2010s, superconducting qubit architectures designed by John Martinis’ group at Google, and then HYPER Inc., looked poised to satisfy the threshold theorems, and considerable work was done to build scaled architectures with many millions of physical qubits by the mid 2020s.

However, despite what seemed to be guarantees via threshold theorems for their architectures, the Martinis group was never able to report large concurrences for more than 12 (disputed) logical qubits.

The scalability wall

Parallel to the development of the scalable, silicon architectures, many groups continued work on other traditional schemes like neutral atoms, trapped ions, and Nuclear Magnetic Resonance (NMR) based devices. These devices, in turn, ran into the now named Scalability Wall of 12 (disputed) entangled encoded qubits. For a discussion on the difference between encoded and physical qubits, see the discussion in Quantum error correction.

The Martinis group hoped that polishing their hardware, and scaling the size of their error correction schemes would allow them to surpass the limit, but progress stalled for more than a decade.

Correlated noise catastrophe

Alexei Kitaev, building on earlier work by Gil Kalai, Robert Alicki, and John Preskill published a series of papers in the late 2020s, culminating in the 2030 theorem now known as the KKAP Theorem, or the Noise Catastrophe Theorem. This proof traced how fundamental limits on the noise experienced by quantum mechanical objects irretrievably destroys the controllability of quantum systems beyond only a few qubits. Uncontrollable correlations were shown to arise in any realistic noise model, essentially disproving the possibility of large scale quantum computation.

Aftermath (This section has been marked as controversial, see the talk page for details)

The immediate aftermath of the publication of the proof was disbelief. Almost all indications pointed towards scalable quantum computation being possible, and that only engineering problems stood in the way of truly scalable quantum computation. The Nobel Prize (2061) winning work of Mikhail Lukin’s team at Harvard only reinforced the shock felt by the Quantum Information community when the bounds provided in the KKAP Theorem’s proof were explicitly saturated via cold atom experiments. Funding in quantum information science rapidly dwindled in the following years, and the field of Quantum Information was nearly abandoned. The field has since been reinvigorated by Kitaev’s recent proof of the possibility of Quantum Gravitational computers in 2061.

calef 2 Aug 2013 18:44 UTC
20 points
on: [Link] AI advances: computers can be almost as funny as people
The link to the actual paper is http://homepages.inf.ed.ac.uk/s0894589/petrovic13unsupervised.pdf

calef 16 Nov 2021 19:54 UTC
17 points
in reply to: Razied’s comment on: Ngo and Yudkowsky on alignment difficulty
I don’t think the issue is the existence of safe prompts, the issue is proving the non-existence of unsafe prompts. And it’s not at all clear that a GPT-6 that can produce chapters from 2067EliezerSafetyTextbook is not already past the danger threshold.

Typical Sneer Fallacy

calef1 Sep 2015 3:13 UTC

16 points

44 comments4 min readLW link

calef 31 Mar 2022 18:20 UTC
16 points
on: [Link] Training Compute-Optimal Large Language Models
Something worth reemphasizing for folks not in the field is that these benchmarks are not like usual benchmarks where you train the model on the task, and then see how good it does on a held-out set. Chinchilla was not explicitly trained on any of these problems. It’s typically given some context like: “Q: What is the southernmost continent? A: Antarctica Q: What is the continent north of Africa? A:” and then simply completes the prompt until a stop token is emitted, like a newline character.

And it’s performing above-average-human on these benchmarks.

calef 27 Sep 2013 4:48 UTC
16 points
on: I played the AI Box Experiment again! (and lost both games)
The secrecy aspect of these games continues to rub me the wrong way.

I understand the argument—that an enumeration of strategies an oracle A.I. might take would only serve as a list of things a critic could point to and claim, “None of these would ever convince me!”.

But the alternative is that critics continue to claim “an oracle A.I. could never convince me!”, and the only ‘critics’ whose minds have been changed are actually just skeptical readers of lesswrong.com already familiar with the arguments of friendly A.I. who happen to invest multiple hours of time actually partaking in a simulation of the whole procedure.

So I suppose my point is two-fold:
1. Anonymous testimony without chatlogs don’t actually convince skeptics of anything.
2. Discussions of actual strategies at worst inform readers of avenues of attack the readers might not have thought about, and at double worst supply people that probably won’t ever be convinced that oracle AIs might be dangerous with a list of things to pretend they’re immune to.

calef 21 Feb 2022 23:44 UTC
15 points
on: Ngo and Yudkowsky on scientific reasoning and pivotal acts
That got people to, I dunno, 6 layers instead of 3 layers or something? But it focused attention on the problem of exploding gradients as the reason why deeply layered neural nets never worked, and that kicked off the entire modern field of deep learning, more or less.
This might be a chicken or egg thing. We couldn’t train big neural networks until we could initialize them correctly, but we also couldn’t train them until we had hardware that wasn’t embarrassing / benchmark datasets that were nontrivial.
While we figured out empirical init strategies fairly early, like Glorot init in 2010, it took until much later that we developed initialization schemes that really Just Worked (He init in 2015 , Dynamical Isometry from Xiao et al 2018)
If I had to blame something, I’d blame GPUs and custom kernel writing getting to the point that small research labs could begin to tinker with ~few million parameter models on essentially single machines + a few GPUs. (The AlexNet model from 2012 was only 60 million parameters!)

Waterfall Ethics

calef30 Jan 2012 21:14 UTC

14 points

30 comments2 min readLW link

calef 4 Dec 2021 1:28 UTC
14 points
on: Shulman and Yudkowsky on AI progress
For what it’s worth, the most relevant difficult-to-fall-prey-to-Goodheartian-tricks measure is probably cross entropy validation loss, as shown in this figure from the GPT-3 paper:
Serious scaling efforts are much more likely to emphasize progress here over Parameter Count Number Bigger clickbait.
Further, while this number will keep going down, we’re going to crash into the entropy of human generated text at some point. Whether that’s within 3 OOM or ten is anybody’s guess, though.

calef 21 May 2015 20:51 UTC
14 points
on: Leaving LessWrong for a more rational life

This argument is, however, nonsense. The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available. The wave-particle duality of quantum physics, or the 11-dimensional space of string theory may defy human intuition, i.e. our built-in intelligence. But we have proven ourselves perfectly capable of understanding the logical implications of models which employ them. We may not be able to build intuition for how a super-intelligence thinks. Maybe—that’s not proven either. But even if that is so, we will be able to reason about its intelligent behaviour in advance, just like string theorists are able to reason about 11-dimensional space-time without using their evolutionarily derived intuitions at all.

This may be retreating to the motte’s bailey, so to speak, but I don’t think anyone seriously thinks that a superintelligence would be literally impossible to understand. The worry is that there will be such a huge gulf between how superintelligences reason versus how we reason that it would take prohibitively long to understand them.

I think a laptop is a good example. There probably isn’t any single human on earth that knows how to build a modern laptop from scratch. There’s are computer scientists that know how the operating system is put together—how the operating system is programmed, how memory is written and retrieved from the various buses; there are other computer scientists and electrical engineers who designed the chips themselves, who arrayed circuits efficiently to dissipate heat and optimize signal latency. Even further, there are material scientists and physicists who designed the transistors and chip fabrication processes, and so on.

So, as an individual human, I don’t know what it’s like to know everything about a laptop all at once in my head, at a glance. I can zoom in on an individual piece and learn about it, but I don’t know all the nuances for each piece—just a sort of executive summary. The fundamental objects with which I can reason have a sort of characteristic size in mindspace—I can imagine 5, maybe 6 balls moving around with distinct trajectories (even then, I tend to group them into smaller subgroups). But I can’t individually imagine a hundred (I could sit down and trace out the paths of a hundred balls individually, of course, but not all at once).

This is the sense in which a superintelligence could be “dangerously” unpredictable. If the fundamental structures it uses for reasoning greatly exceed a human’s characteristic size of mindspace, it would be difficult to tease out its chain of logic. And this only gets worse the more intelligent it gets.

Now, I’ll grant you that the lesswrong community likes to sweep under the rug the great competition of timescales and “size”scales that are going on here. It might be prohibitively difficult, for fundamental reasons, to move from working-mind-RAM of size 5 to size 10. It may be that artificial intelligence research progresses so slowly that we never even see an intelligence explosion—just a gently sloped intelligence rise over the next few millennia. But I do think it’s a maybe not a mistake but certainly naiive to just proclaim, “Of course we’ll be able to understand them, we are generalized reasoners!”.

Edit: I should add that this is already a problem for, ironically, computer-assisted theorem proving. If a computer produces a 10,000,000 page “proof” of a mathematical theorem (i.e., something far longer than any human could check by hand), you’re putting a huge amount of trust in the correctness of the theorem-proving-software itself.

calef 25 May 2022 7:03 UTC
13 points
on: autonomy: the missing AGI ingredient?
The relevant sub-field of RL interested in this calls this “lifelong learning”, though I actually prefer your framing because it makes pretty crisp what we actually want.

I also think that solving this problem is probably closer to “something like a transformer and not very far away”, considering, e.g. memorizing transformers work (https://arxiv.org/abs/2203.08913)

calef 27 Nov 2021 2:59 UTC
13 points
on: larger language models may disappoint you [or, an eternally unfinished draft]
In defense of shot-ness as a paradigm:

Shot-ness is a nice task-ambiguous interface for revealing capability that doesn’t require any cleverness from the prompt designer. Said another way, If you needed task-specific knowledge to construct the prompt that makes GPT-3 reveal it can do the task, it’s hard to compare “ability to do that task” in a task-agnostic way to other potential capabilities.

For a completely unrealistic example that hyperbolically gestures at what I mean: you could spend a tremendous amount of compute to come up with the magic password prompt that gets GPT-3 to reveal that it can prove P!=NP, but this is worthless if that prompt itself contains a proof that P!=NP, or worse, is harder to generate than the original proof.

This is not what if “feels like” when GPT-3 suddenly demonstrates it is able to do something, of course—it’s more like it just suddenly knows what you meant, and does it, without your hinting really seeming like it provided anything particularly clever-hans-y. So it’s not a great analogy. But I can’t help but feel that a “sufficiently intelligent” language model shouldn’t need to be cajoled into performing a task you can demonstrate to it, thus I personally don’t want to have to rely on cajoling.

Regardless, it’s important to keep track of both “can GPT-n be cajoled into this capability?” as well as “how hard is it to cajole GPT-n into demonstrating this capability?”. But I maintain that shot-prompting is one nice way of probing this while holding “cajoling-ness” relatively fixed.

This is of course moot if all you care about is demonstrating that GPT-n can do the thing. Of course you should prompt tune. Go bananas. But it makes a particular kind of principled comparison hard.

Edit: wanted to add, thank you tremendously for posting this—always appreciate your LLM takes, independent of how fully fleshed they might be

calef 26 Nov 2021 7:36 UTC
13 points
in reply to: Eliezer Yudkowsky’s comment on: Christiano, Cotra, and Yudkowsky on AI progress
I believe Sam Altman implied they’re simply training a GPT-3-variant for significantly longer for “GPT-4”. The GPT-3 model in prod is nowhere near converged on its training data.

Edit: changed to be less certain, pretty sure this follows from public comments by Sam, but he has not said this exactly

calef

Typ­i­cal Sneer Fallacy

Water­fall Ethics

Typical Sneer Fallacy

Waterfall Ethics