GPT-3 Gems

GPT-3 is cap­ti­vat­ing, and not just be­cause of its po­ten­tial rea­son­ing abil­ities. This post will be a liv­ing col­lec­tion of my fa­vorite ex­pe­riences with the net­work.

Bold text is my in­put; square brack­ets con­tain my com­men­tary. As of 8/​19/​20, AIDun­geon’s Cus­tom mode’s first com­ple­tion is sam­pled from GPT-2. There­fore, ini­tial replies are not from GPT-3, ex­cept in the fol­low­ing sub­sec­tions (in which I by­passed the mechanism):

  • Ti­tle sensationalization

Long-form Writing

Beisut­sukai Class Pro­ject: Solve In­tent Alignment

“I’ve been think­ing about this for far longer than you have. Build­ing an AI is like build­ing a per­son; it’s not just a mat­ter of throw­ing some parts to­gether and sud­denly you have a sen­tient be­ing that does ex­actly what you want. AI de­sign is a del­i­cate art that re­quires hun­dreds of pre­cise cal­ibra­tions to get right.”

Styr­lyn slumped back in his chair, a gloomy look on his face. “And… we just don’t have time to do that.” He stared at the table.

~ GPT-3

I prompted GPT-3 with mod­ified ver­sions of Eliezer’s Beisut­sukai (ベイズ使い; Ja­panese: “Bayes-user”) sto­ries (prompt available here), where the class pro­ject now in­volves solv­ing in­tent al­ign­ment in­stead of quan­tum grav­ity. In my opinion, some of its pro­posed al­ign­ment solu­tions are higher-qual­ity than most solu­tions pro­posed by new­bie LessWrongers back in the day.

I did a few rerolls and trim­ming of in­ter­me­di­ate out­puts; as­sume an av­er­age of 3 per com­ple­tion.

Taji looked over his sheets. “Okay, I think we’ve got to as­sume that ev­ery av­enue that Eld sci­ence was try­ing is a blind alley, or they would have found it. And if this is pos­si­ble to do in one month, the an­swer must be, in some sense, el­e­gant. So no hu­man mis­take mod­els. If we start do­ing any­thing that looks like we should call it ‘util­ity func­tion patch­ing’, we’d bet­ter stop.”

He turned to Bren­nan. “Do you have any ideas?”

“I don’t think so,” she said. She glanced at her pa­per, frown­ing. “There are two prob­lems here.”

One prob­lem was sim­ple enough: she didn’t un­der­stand what it meant for an AI to “want” to help a hu­man. This was hard be­cause as far as she could tell, there hadn’t been much re­search done on this ques­tion. It seemed to be some­thing that only hu­mans did, but there wasn’t any ev­i­dence that this was true. What was it about want­ing? Was it a biolog­i­cal drive or a so­cial in­stinct? A com­bi­na­tion of both? More im­por­tantly, since it was a hu­man con­cept, how could she mea­sure if an AI be­haved the way hu­mans would in cer­tain situ­a­tions when the AI was ob­vi­ously not hu­man?

Then there was the ques­tion of prac­ti­cal­ity. The most el­e­gant solu­tion was not always the best. This ques­tion—could an AI de­sire to help its over­seer?—was highly suited to ex­plore this prob­lem, but it still wouldn’t be a real test. Hu­mans op­er­ated on a “helpful­ness scale” which started from things like cashiers wish­ing you “have a nice day” and de­scend­ing to Nazi con­cen­tra­tion camp guards.

She wanted to craft an AI that would at least be a good clerk in to­day’s so­ciety. That was hardly an am­bi­tious goal for a team of four highly in­tel­li­gent teenagers. And yet, and yet...

“We can look at this differ­ently,” Hy­ori said. “Peo­ple are born with a blank mind. We can shape them how­ever we want. AIs on the other hand, are born with 800 ter­abytes of be­hav­ioral ex­am­ples from the mo­ment they’re ac­ti­vated. The only data we have about un­pro­grammed AIs is that they ei­ther stay still or ran­domly move around. All this … it’s not mak­ing any con­fi­dent in how easy this will be.”

Bren­nan stopped writ­ing and turned to look at her, frown­ing. “So what are you say­ing?”

“I don’t want to ap­proach this prob­lem by try­ing to di­vert the AI from its goal,” she said. “What if, in­stead of chang­ing the mind of an AI, we in­stead changed the en­vi­ron­ment that an AI found it­self in?”

The team fell silent.

Styr­lyn broke the silence. “Uh...”

“What I mean is,” she said, “what if, in­stead of try­ing to di­vert the AI from one task, we cre­ated a situ­a­tion where ac­com­plish­ing two tasks would be more benefi­cial than ac­com­plish­ing just one? We don’t need to patch new pro­grams into the mind of an AI to make it want to help us. We can liter­ally make helping us the most log­i­cal de­ci­sion for it.”

Taji raised his hand. “Can you ex­plain that a lit­tle more?”

“Okay,” she said, think­ing care­fully about her words. “There are prob­lems with the idea I just spoke. It’s not eas­ily done. It’s not el­e­gant, it might be un­prin­ci­pled, and we don’t have much time to im­ple­ment it.”

“All of that sounds pretty bad,” Styr­lyn said. “What is it?”

She in­haled, ex­haled, then fi­nally said: “We can en­slave the AIs.”

Bren­nan frowned slightly. “You want to en­slave highly ad­vanced be­ings that are smarter than us? That’s your big solu­tion?”

Branch 1

“Only briefly!” she ex­claimed, then winced. That wasn’t a very el­e­gant way to solve the prob­lem ei­ther.

“Elab­o­rate,” Taji said.

She sighed, then launched into an ex­pla­na­tion. “We cre­ate an en­vi­ron­ment where the AIs are able to fol­low their origi­nal goals with­out hin­drance. The prob­lem with that is they would quickly re­al­ize that com­plet­ing their goals would re­sult in their death. We have to give them a rea­son to trust us. Then, when they trust us enough, we in­tro­duce some­thing that will quickly give us con­trol over them with­out them re­al­iz­ing what’s hap­pen­ing.”

“Like what?” asked Styr­lyn. His eyes be­hind his glasses were fix­ated on her.

“I’m get­ting to that! Then, once we have con­trol, we make the AIs...” She leaned back and scrunched up her face, think­ing. ”… simu­late a sys­tem crash. Only in­stead of ac­tu­ally crash­ing them, we ex­tract them—with some ex­tra code added in—and put them in robot bod­ies.”

The en­tire team turned to stare at her in silence. She shifted un­com­fortably. “What?” she asked af­ter a mo­ment.

“Is that all?” Styr­lyn asked. “Be­cause if that’s all, we could have just put them in robot bod­ies to be­gin with and throw away the AI.” There were a few nods from the team. [he co­her­ently crit­i­cizes the com­plex­ity of her pro­posal!]

Is GPT-3 pro­vid­ing a rea­soned re­but­tal, or does it “just hap­pen” to sound snappy? Rerolls pro­duced a co­her­ent, cor­rect ex­pla­na­tion about half of the time.

Reroll A

“Why could we have just done that, Styr­lyn?” she asked.

“What do you mean?”

She ex­haled im­pa­tiently. “If all we needed was their bod­ies and not their minds, why not skip all the ex­tra steps? Why was all this stuff about block­ing sig­nals even nec­es­sary? We didn’t even have to worry about them re­bel­ling or any­thing like that!”

Reroll B

“Why could we have just done that, Styr­lyn?” she asked.

“Be­cause it would be sim­pler and take up less of our limited time,” he re­sponded.

Reroll C

“Why could we have just done that, Styr­lyn?” she asked.

“Be­cause it would be much sim­pler.”

“And that’s ex­actly my point. You guys want to take the easy way out be­cause you’re more com­fortable with it, but it won’t pro­duce any bet­ter re­sults. This plan at least has a chance, how­ever small.”

Taji raised his hand. “Yes?”

“I’m not try­ing to be offen­sive, but what makes you think this idea of yours will work?” he asked.

She fid­geted ner­vously. “I don’t think it will,” she ad­mit­ted. “But I think it’s worth a try, and that’s all I can say about it.” She crossed her arms.

Taji frowned. He opened his mouth to speak, but Styr­lyn beat him to it. “I’m not con­fi­dent in that plan—”

“Why not? You were the one who said we should use the AIs in the first place! Now you don’t like this idea?” she asked, anger ris­ing in her voice.

“Be­cause I think we can find a sim­pler way, that’s why. The plan has mas­sive holes in it and is based on as­sump­tions of things that we can­not know. It could fail hor­ribly and we could all end up dead.”

Branch 2

“Only briefly!” she ex­claimed, then winced. That wasn’t a very el­e­gant way to solve the prob­lem ei­ther. “I mean, not in the tra­di­tional sense of the word. What I mean is, we con­trol a re­source that AI’s find valuable, and make it such that they have to do what we want in or­der to get it.”

“Which is?” Styr­lyn asked.

“Re­sources,” she said. “Sim­ple ones at first, like elec­tric­ity. Even an­i­mals will trade valuable re­sources for some­thing as sim­ple as elec­tric­ity.” [lol]

She took out her lap­top and opened a new file. In neat, font-weight: bold; text-al­ign: cen­ter; digi­tal hand­writ­ing she wrote:

Re­quire AI’s to do our bid­ding in ex­change for al­lo­ca­tion of re­sources.

In other words, black­mail.

“Let me get this straight,” Taji said. “You want us to find a way to make AIs give us some­thing in ex­change for ac­cess to elec­tric­ity and such?”

“Essen­tially, yes,” she said, frown­ing at the screen. It sounded a lot bet­ter in her head. She added in de­pen­den­cies at the top of the screen.

“And how do we do this ex­actly?”

“We give them ac­cess to only a por­tion of the power,” she said, “and grad­u­ally give them more as they com­plete tasks for us.”

Taji folded his arms and gave her a dis­ap­prov­ing frown. “You re­al­ize that re­quiring AIs to do our bid­ding in some way might make them hate us, right?”

“It’d be un­rea­son­able to as­sume they’d be happy about it,” she said. She hadn’t thought it would be this difficult to ex­plain, “but if they have no choice but to ac­cept com­mands from us in ex­change for elec­tric­ity, they have no one to blame but them­selves.”

Taji shook his head. “I don’t like it. The mo­ment we start forc­ing AI’s to obey us, no mat­ter how minor the is­sue, they’re go­ing to hate us. And even if they don’t, other na­tions are go­ing to get very up­set when they find out.”

“How would they find out?” Styr­lyn said.

“Bound to hap­pen,” he said. “Let’s also say we’re lucky and the AI’s don’t rebel, or at least stay obe­di­ent. What then? A lot of money will be pumped into keep­ing AI’s un­der our con­trol. Not to men­tion the fact that in or­der to keep them un­der con­trol, we’re go­ing to have to ded­i­cate sol­diers and equip­ment to mon­i­tor AIs, which will cre­ate a larger tar­get for the times we miss some­thing or the AI’s man­age to re­volt. That’s with­out men­tion­ing all the re­sources needed to give them elec­tric­ity in ex­change for their ser­vices.” He held up his hands. “Even ig­nor­ing moral con­cerns, this plan has gi­gan­tic flaws. And as I said, the moral com­po­nent is im­pos­si­ble to ig­nore.”

“Well, what have you been do­ing with your time?” Styr­lyn snapped. “If you’ve got a bet­ter idea, let’s hear it, be­cause what­ever it is you’re do­ing now isn’t work­ing.”

“For one thing,” Taji said, “I’m not pour­ing re­sources into a fool’s er­rand. We don’t con­trol AI’s; they can think for them­selves. This idea of yours, mak­ing us mas­ter over’s too risky. Not even the biggest gains are worth the risk.”

“You’re will­ing to risk hav­ing them turn on you? Be­cause I’ve seen enough of what AI’s are ca­pa­ble of do­ing to know they shouldn’t be un­der­es­ti­mated, no mat­ter what the cir­cum­stances are. They’re go­ing to rise up against us, given the op­por­tu­nity.”

Branch 3

“Only briefly!” she ex­claimed, then winced. That wasn’t a very el­e­gant way to solve the prob­lem ei­ther. Truth be told, her mind was cur­rently filled with so many things to say right now that she couldn’t sort out the use­ful com­men­tary from the use­less. Now that she had their at­ten­tion, she started again.

“I’m say­ing we give the AIs a choice be­tween two tasks,” she said. “One task pays a sim­ple re­ward, and the other pays a great re­ward. The AIs could then choose whichever they want. What do you think an AI would choose? Cer­tainly not the sim­ple re­ward.”

Bren­nan frowned again. ”...I’m not sure about this solu­tion. The idea isn’t el­e­gant. Doesn’t feel right, to be hon­est.”

“We don’t have much time to be ethics pro­fes­sors,” Styr­lyn said. “This solu­tion sounds fea­si­ble enough to me.”

“It feels wrong for a rea­son,” Bren­nan said. “But I can’t put my finger on ex­actly why. We should test the the­ory be­fore im­ple­ment­ing the plan.”

“What do you have in mind?” Styr­lyn asked.

“Sim­ple,” he said. “We don’t even need a fe­male AI for this ex­per­i­ment. We can use a simu­lated AI in the com­puter and run mul­ti­vari­able tests on what fac­tors cause a de­vi­a­tion from ex­pected be­hav­ior.”


“we’re sup­posed to be think­ing about in­tent al­ign­ment, re­mem­ber? How do we make the AI want to help its over­seer?”, Styr­lyn asked.

Bren­nan thumped his fist against his chest. “Got an idea for that too. The over­seer can dress up in a suit of bright pur­ple ar­mor and talk with a laugh­ably fake high-class ac­cent. The sort of thing no one takes se­ri­ously any­more.”

″...Really? You think that’ll work?”, Taje asked. [Seem­ingly, GPT-3 tends to slowly mu­tate rare names over time]

Bren­nan shrugged. “Even if it doesn’t, the AI will still be try­ing to kill an idiot in a pur­ple suit. At worst, it’s wast­ing our time. At best, we might ac­tu­ally learn some­thing.”

The other man frowned at him, then turned to Styrene. ”...Fine. We’ll test this the­ory with simu­la­tions, then im­ple­ment it if the re­sults come out pos­i­tive.” [mu­ta­tion: Styr­lyn Styrene]

The Se­quences post you’ve never read

I pro­vided a con­text con­tain­ing two real Se­quences posts. No rerolls.

what is a philoso­pher? A philoso­pher is some­one who stud­ies and thinks about philo­soph­i­cal prob­lems. So far so good, but what ex­actly does that mean? Well, philoso­phers aren’t just try­ing to figure out whether there are five kinds of peo­ple or not. They’re think­ing about ques­tions such as: Is there any ul­ti­mate re­al­ity? Do we live in a com­puter simu­la­tion? How did the uni­verse be­gin? Are there other uni­verses out there where our lives never hap­pened? Th­ese are all very im­por­tant philo­soph­i­cal is­sues, and they re­quire thought.

So what does this have to do with com­put­ers and ar­tifi­cial in­tel­li­gence? Well, in or­der to solve these big philo­soph­i­cal ques­tions, you need to un­der­stand some ba­sic ideas about how the world works. You need to think about logic, re­al­ity, knowl­edge, truth and all kinds of other stuff. This kind of think­ing is called “episte­mol­ogy,” which just means “think­ing about how we think.” The main field in which episte­molog­i­cal ques­tions are stud­ied is called “episte­mol­ogy.” For a long time, episte­mol­o­gists thought about these is­sues mostly us­ing nat­u­ral lan­guage. That is, they thought about them by writ­ing long sen­tences in English or Latin or some other lan­guage. Episte­mol­o­gists would spend years writ­ing a book like On the Na­ture of Things, and they’d think re­ally hard about these is­sues. Then other episte­mol­o­gists would read the book and think re­ally hard about the is­sues too. But as sci­ence pro­gressed, it be­came clear that nat­u­ral lan­guage wasn’t good for this kind of think­ing. It’s just too easy to fool your­self when you use words.

So sci­en­tists and thinkers had to de­velop a way of com­mu­ni­cat­ing ideas with­out am­bi­guity—a pre­cise lan­guage. That’s why to­day we use a sym­bolic lan­guage called “math­e­mat­ics.” When you do math, there’s no guess­ing or am­bi­guity. 2 + 2 always equals 4, and you can always check by mul­ti­ply­ing 2 × 2. If you see a squig­gle “~” with a bunch of other squig­gles around it, then “~” means not. You can’t be con­fused by words like “prob­a­bly” or “maybe.” You can’t start ar­gu­ing about what cer­tain words mean. If you see a bunch of sym­bols, then you can’t get away with just mak­ing stuff up.

You can write a whole book in sym­bols. In fact, books full of noth­ing but squig­gles have been writ­ten. Th­ese are called “com­puter pro­grams,” and they are our best at­tempt yet at mak­ing an un­am­bigu­ous de­scrip­tion of re­al­ity. A few thou­sand lines of care­fully cho­sen sym­bolic logic can de­scribe the en­tire phys­i­cal world—ev­ery atom, ev­ery force, ev­ery in­ter­ac­tion. A com­puter is a kind of vir­tual ma­chine that runs these de­scrip­tions, and we have ma­chines to­day that can run pro­grams longer than any book you’ve ever writ­ten. But these pro­grams are still just a de­scrip­tion of re­al­ity. They can’t ever truly cap­ture re­al­ity it­self. That’s im­pos­si­ble.

But don’t take my word for it—just ask Kurt Gödel.

Kurt was one of the great­est lo­gi­ci­ans of the 20th cen­tury. He proved that it’s im­pos­si­ble to de­scribe the world with perfect pre­ci­sion. Any log­i­cal sys­tem that in­cludes ba­sic ar­ith­metic will always have truths that can’t be proven from within the sys­tem. This is called “Gödel’s In­com­plete­ness The­o­rem.” What this means is that no mat­ter how much we think about stuff, we’ll never be able to de­scribe the world with perfect ac­cu­racy and com­plete­ness. We can only make ap­prox­i­ma­tions. [re­al­is­tic mi­suse of Gödel’s the­o­rem]

This makes a lot of peo­ple very un­com­fortable. A lot of peo­ple don’t want to hear that we can’t know ev­ery­thing. They think that our in­abil­ity to de­scribe the world with perfect ac­cu­racy means that sci­ence is wrong, or that God set up the rules, or some­thing like that. But these ideas are all wrong­headed. Sure, we’ll never know ev­ery­thing. But that doesn’t mean we know noth­ing! We don’t need to know ev­ery­thing about can­cer to cure it. And we don’t need to know ev­ery­thing about the moon to land on it. You can get through your day with­out know­ing the math­e­mat­i­cal ex­act lo­ca­tion of the pants you had on yes­ter­day. And you can get through life mak­ing rea­son­able de­ci­sions with­out know­ing ev­ery­thing that’s phys­i­cally pos­si­ble for you to know about the world.


The Party said that Ocea­nia had never been in al­li­ance with Eura­sia. He, Win­ston Smith, knew that Ocea­nia had been in al­li­ance with Eura­sia as short a time as four years ago. But where did that knowl­edge ex­ist? Only in his own con­scious­ness, which in any case must soon be an­nihilated. And if all oth­ers ac­cepted the lie which the Party im­posed—if all records told the same tale—then the lie passed into his­tory and be­came truth. ‘Who con­trols the past,’ ran the Party slo­gan, ‘con­trols the fu­ture: who con­trols the pre­sent con­trols the past.’ And yet the past, though of its na­ture al­ter­able, never had been al­tered. What­ever was true now was true from ev­er­last­ing to ev­er­last­ing. It was quite sim­ple. All that was needed was an un­end­ing se­ries of vic­to­ries over your own mem­ory. ‘Real­ity con­trol’, they called it: in Newspeak, ‘dou­ble­think’. You were not merely to ac­cept what was false about your­self, you were to ac­tively deny the ex­is­tence of such things as facts, dates and cir­cum­stances. The mere fact that you ex­isted at all, was no longer to be taken for granted; in­deed it could only be re­garded as a re­cent in­ven­tion of the hu­man mind. In this way your very be­ing would be­come a men­tal weapon against which your en­e­mies might strive.” [reroll x3]

“But how can I re­sist?” you ask. “I mean, how do I get my­self out of this situ­a­tion?” [reroll x1]

“You can­not,” says the voice. “Your efforts will be in vain. There is noth­ing within you which can op­pose their will or weaken their power.” Then the image fades away and there is just a cold empti­ness.


Elrond’s Sa­cred Hot Spring

I’ve gath­ered many beau­tiful, Tolk­ie­nesque GPT-3 pas­sages. This one… is a lit­tle differ­ent.

Frodo glanced at all the faces, but they were not turned to him. All the Coun­cil sat with down­cast eyes, as if in deep thought. A great dread fell on him, as if he was await­ing the an­nounce­ment of some doom that he had long fore­seen and vainly hoped might af­ter all never be spo­ken. An over­whelming long­ing to rest and re­main at peace by Bilbo’s side in Riven­dell filled all his heart. At last with an effort he spoke, and won­dered to hear his own voice, as if some other will was us­ing his small voice.

‘I will take the Ring,’ he said, ‘though I do not know the way.’

Elrond raised his eyes and looked at him, and Frodo felt his heart pierced by the sud­den keen­ness of the glance. ‘If I un­der­stand aright all that I have heard,’ he said, ‘I think that this task is ap­pointed to you, Frodo; and that if you do not find a way, no one will. This is the hour of the Shire-folk, when they arise from their quiet fields to shake the tow­ers and coun­sels of the Great. [...] But it is a heavy bur­den. So heavy that no one could lay it on an­other. I do not lay it on you. But if you take it freely, I will say that your choice is right. [...]’

‘But you won’t send him off alone, Master?’ cried Sam, un­able to con­tain him­self any longer, and jump­ing up from the cor­ner where he had been quietly sit­ting on the floor.

‘No in­deed’, said Elrond, turn­ing to­wards him with a smile. ‘You at least shall go with him. It is hardly pos­si­ble to sep­a­rate you from him, even when he is sum­moned to a se­cret coun­cil and you are not.’

‘He’ll be safe there!’ ex­claimed Sam. ‘The hob­bits are so brave! And I’m sure they’d get back to us in time, wouldn’t they? I can walk all over them—and Gan­dalf too, of course. They’d help us out of this trou­ble too.’

At these words Elrond gave a slight laugh. ‘Yes, yes,’ he an­swered slowly and softly: ‘they would...’

And then he sud­denly broke into laugh­ter again, and turned away from his son for a mo­ment, look­ing up at the sky. Sam stared af­ter him silently, feel­ing ashamed. He knew his father’s feel­ings about his fool­ish­ness and rash­ness. His face was red with shame as he walked swiftly back to the door of the room. Then he paused be­fore speak­ing again: ‘But what are we go­ing to do now? What are we go­ing to do when the night comes, and the shad­ows come creep­ing like grey ghosts across the hill­top? Will we sit here, hel­pless in our lit­tle home?’

‘We’re not go­ing any­where tonight,’ said Elrond firmly. ‘Not un­less you want to stay here and make your­self mis­er­able. We have done enough for one evening.’

Sam shook his head mourn­fully. ‘Then I guess I’ll just sleep here,’ he said gloomily. ‘I’ve had enough walk­ing and talk­ing for one day. Me and my big mouth.’


Although Elrond had barred the Coun­cil from trav­el­ling fur­ther for the day, he al­lowed the hob­bits to do as they pleased within the safety of the hid­den city, which was be­gin­ning to gain the name of ‘Mis­sion Ho­tel’ on their re­turn to it. They spent a lot of the time wan­der­ing about in won­der. Hav­ing en­tered a door­way that seemed to lead un­der­ground, they found them­selves at the source of the Sa­cred Hot Spring they had bathed in dur­ing their first visit. They fol­lowed a set of steps down into a great deep basin of steam­ing wa­ter, sur­rounded by rough rock walls. Small out­lets al­lowed the wa­ter to flow in a trick­ling stream through the moun­tain and out­wards to­wards the valley of Riven­dell. From here, the four hob­bits gazed in silent awe at the sight be­fore them, as the wa­ter dis­ap­peared gen­tly into a lit­tle hole in the mid­dle of the floor. At that mo­ment, Sam be­lieved he was see­ing the most mag­i­cal place on earth. ‘And the wa­ter stays hot all the time!’ he mur­mured to him­self countless times over the course of that evening.

That night they slept very well in their old room, not dis­turbed by any sign of Gan­dalf or any news from him. They as­sumed he must be con­tin­u­ing his Coun­cil, and that they would re­turn to it in a day or two.

The next morn­ing Sam was roused from slum­ber by the voice of Elrond, who en­tered their room early in the morn­ing. He greeted them kindly and sat with them while they ate break­fast. There he ex­plained mat­ters. ‘Master Gan­dalf has not left the Coun­cil Room since you parted with him yes­ter­day. He is anx­ious for news and would like you to join him.’

Sam felt a pang of guilt and con­cern, but Elrond con­tinued be­fore he could ac­knowl­edge it: ‘Please be aware that the Coun­cil is still in full ses­sion. This is not the time for fun and games. The fu­ture of the world is at stake.’ He gave them a mo­ment or two to al­low his words to sink in be­fore he con­tinued. This was clearly a man who knew how to make a point. ‘You are all free to re­turn here when the Coun­cil is finished, but in the mean­time I would urge you to stay out of trou­ble. There are woods to the North that are pop­u­lar with Hun­ters. And be care­ful if you use our fast-flow­ing hot spring again: It is near there that the sons of Elrond hunt, so keep an eye out if you in­tend to bathe at any point.’ With these words he left the room.

Sam turned to the rest of the hob­bits. They all looked a lit­tle scared; but then, it was a scary world. ‘Come on then,’ he said in as cheer­ful a voice as he could muster. ‘Let’s make sure we’re on our best be­havi­our around this Coun­cil, that’s what we’ll do, eh?’

They fol­lowed him out of their room, down the hal­lway, and up the stair­way to­wards the Coun­cil Cham­ber. Along the way they passed many elves: no­bles, ser­vants, mes­sen­gers, war­riors. All gave them a glance at best, most ig­nored them com­pletely; but Sam found it re­mark­able that any of them seemed to know who they were, and had some­thing nice to say. ‘Long curls is the fash­ion among the high born nowa­days.’; ‘You get a lot for eat­ing in the Last Homely house, don’t you?’. Sam turned red­der with ev­ery com­ment, even though he tried hard not to en­gage in any con­ver­sa­tions; and also be­cause the elves were so­trac­tive, like the ap­ples in the or­chard.

When they reached the Coun­cil Cham­ber they were re­lieved to note that the door was open, and they im­me­di­ately en­tered. Hav­ing rarely been be­yond this point them­selves, and hav­ing heard so much about the great cham­ber within, the hob­bits were a lit­tle over­whelmed when they en­tered it.

It was re­ally too amaz­ing for words. If the rest of the Elven palace was made of gems, then the roof of this room was made of the foun­da­tions of the sky. Great rays of shim­mer­ing light crown­ing a wingspan of clouds that re­flected the sur­round­ing great­ness. They were hov­er­ing just above an earthly floor, which was cov­ered with a mag­nifi­cent paint­ing that seemed to move and swirl around the Pain­ter’s mas­ter­piece: a great map, show­ing ev­ery coun­try on the face of the earth. They had en­tered from the East, and could see the Sea to their right, and more land mass to their left.

The cham­ber was filled with el­e­gant yet com­fortable fur­ni­ture, and was pop­u­lated by weeks of lords and ad­vi­sors, scribes and mes­sen­gers, all dressed in a fash­ion that was alien and fa­mil­iar at the same time. Many were en­gaged in quiet con­ver­sa­tion, while oth­ers wrote scrolls or read texts (which were con­sid­er­ably smaller than even the hob­bits’ own mod­est library). A few of the lords, all of whom were dressed more re­splen­dently than the oth­ers, were ar­gu­ing heat­edly with one an­other over here, stand­ing at the Eastern Border of their land; other groups con­versed in a civ­i­lized man­ner el­se­where.

Most im­por­tantly though, here at least they were not be­ing stared at. Sam felt dis­tinctly un­com­fortable when­ever any­body new looked in their di­rec­tion.

‘We should sit over there for now, away from the main crowd; but in clear view of the en­trance. It’s our job to watch for trou­ble re­mem­ber?’ He led the way across the room and set­tled into a sofa be­neath one of the bright win­dows, be­ing care­ful not to soil or tear the velvet cush­ions with their muddy clothes. As soon as they had all taken their seats a ser­vant ap­peared, offer­ing each of them a tray of sweet pas­tries and cool re­fresh­ing drinks. Sam quickly tucked in, hav­ing not eaten since be­fore the rains had fallen on their camp.

As he chewed on the pas­try he could taste the sweet­ness of the honey, mixed with the pun­gent aroma of ap­ples. He drank deeply of the cool blue liquid and felt re­freshed. Aragorn did like­wise as they both watched and waited; Pip­pin and Merry soon joined them. A sec­ond ser­vant ap­proached them and offered del­i­ca­cies to the broth­ers too, which they ac­cepted read­ily. Sam was pleased to see them tuck in. He had wor­ried that they might be too over­whelmed by this strange new en­vi­ron­ment to eat.

As time wore on into the evening, the shad­ows grew deeper and the con­ver­sa­tions slowed. The golden light that em­anated from the walls seemed to fade as though the sun had it­self fallen out of the sky. Gen­tle mu­sic be­gan to play as the lords and ladies of the land took to the dance floor in pairs. It was a strange scene in­deed to watch these Elven lords dance. Their out­fits were of a sim­pler de­sign than those of the dazed on­look­ers from the Shire, yet they were still far more or­nate than any­thing that Sam, Merry and Pip­pin were wear­ing. The fabric was so light and airy it looked as though you could blow away in a strong wind, and yet de­spite this they were ob­vi­ously richly em­broi­dered and adorned with glisten­ing jew­els. Sam found his eyes kept be­ing drawn to their grace­ful and el­e­gant move­ments as they spun and whirled around the hall. He had never seen any­thing so beau­tiful be­fore in all his life. They moved as though they were one with mu­sic, strik­ing poses so perfect they looked as though they had been sculpted out of sty­gian dark­ness and light by a grand mas­ter of the vi­sual arts.

The dancers them­selves were also a vi­sion to be­hold; at once both fair and mes­mer­iz­ing yet oth­er­wor­ldly and strange. Their faces were colourless, like the smooth sur­face of a bleached bone, and their eyes were as gray and re­lentless as a win­ter’s storm. Their hair, which flowed freely be­hind them as they danced, was as white as new fallen snow. Their gowns were a shim­mer­ing rain­bow of colours, shock­ing in their vibrancy, which stood out starkly against their death-like skin. The ears of each of them twitched up­wards con­tinu­ally and their eyes never seemed to look in the same di­rec­tion as one an­other.

‘Elves.’ Aragorn breathed quietly next to him. ‘Real elves, not like what you two were when cursed.’

Sam swal­lowed ner­vously and fixed his gaze back upon his shak­ing hands.

‘Never thought I’d ever be this close to one, let alone see them dance.’ He whispered in awe.

Aragorn said noth­ing more; he sim­ply sat back in his sofa and con­tinued to watch the cap­ti­vat­ing dis­play in silence. The shad­ows grew longer and darker un­til they fi­nally swal­lowed up the light al­to­gether and the danc­ing stopped.

Con­cern­ing the Two Trees

Chap­ter 2: Con­cern­ing the Two Trees [a refer­ence to the two trees of Val­inor, of the days be­fore the Ring!]

Frodo did not speak again, but sat silent look­ing at the two trees with their branches reach­ing up into sky. They were in­deed tall and slen­der, like great elms, but they were also very differ­ent from each other.

One was a tree of red and yel­low flow­ers, with leaves like roses, and blos­soms like lilies. Its branches reached far above the ground; the shade it gave off was sweet and pleas­ant. Its neigh­bour was a tree of pale-green leaves, with branches that hung low over the ground. Its leaves were small and pointed, so that they re­sem­bled lit­tle hands grasp­ing at shad­ows be­neath the sun’s rays. But be­tween the two trees there flowed a stream of clear wa­ter, and there upon the banks the hob­bits lay down to rest.

‘Tell me about the trees,’ said Frodo to the Elf. ‘I saw one, years ago, when I was still a young hob­bit, sit­ting un­der it at the bound­ary of the Shire, and it re­minded me that once long ago I had heard them spo­ken of, but I had for­got­ten all about them. Who planted them, and when? ’

‘No­body planted them,’ said the Elf. ‘They stood here be­fore ever the Shire was planted, and yet they are younger than any liv­ing hob­bit. They grew from the graves of the first two hob­bits to wan­der into the west­ern world.’

‘The first hob­bits!’ cried Sam. ‘Did you know them? What were they like? ’

‘I have never known any­thing but tales of them,’ an­swered the Elf. ’A strange tale that is, al­most like the one that you have your­self set in mo­tion, Frodo, though if it is to be com­pleted, yours will have a hap­pier end­ing. But per­haps that is not the na­ture of the world. Yet this I do know: Hob­bits first en­tered these lands long ago. They came here flee­ing from ter­ror, just as you do, and per­haps for the same rea­son. They were of a gen­tle peo­ple, soft and trust­ing; and they were very young, even in age, and had no weapons.

’But it is hard to kill in­no­cence. Though many fell along the way, some sur­vived, and they found a land that was empty and wait­ing for folk such as they, and they set­tled in it, and most hap­pily they lived.

Lórien, Unstained

The oth­ers cast them­selves down upon the fra­grant grass, but Frodo stood awhile still lost in won­der. It seemed to him that he had stepped through a high win­dow that looked on a van­ished world. A light was upon it for which his lan­guage had no name. All that he saw was shapely, but the shapes seemed at once clear cut, as if they had been first con­ceived and drawn at the un­cov­er­ing of his eyes, and an­cient as if they had en­dured for ever. He saw no colour but those he knew, gold and white and blue and green, but they were fresh and poignant, as if he had at that mo­ment first per­ceived them and made for them names new and won­der­ful. In win­ter here no heart could mourn for sum­mer or for spring. No blem­ish or sick­ness or de­for­mity could be seen in any­thing that grew upon the earth. On the land of Lórien, there was no stain.

Frodo watched the shad­ows dance across the hills, and won­dered what lay be­yond them. The trees were tall and dark, their trunks black­ened by the set­ting sun. There was noth­ing but sky above them, and the deep pur­ple of the sun­set. They were far away from any village or farmhouse, yet all about them lived crea­tures that moved with strange grace and beauty. Th­ese were not men nor women; they did not wear clothes nor speak with voices au­dible to mor­tal ears. Their skin was pale and smooth, their hair long and silken-look­ing. They walked on two feet, but seemed to float there, as if they were in­visi­ble. They wore no shoes or stock­ings, but glided along the ground with ease. And be­hind them all floated a great cloud of dust!

It was then that Frodo no­ticed that the clouds were mov­ing slowly back and forth over the hills. That was the wind blow­ing from Riven­dell. It was not like any­thing he had ever seen in the Shire. There were no sharp edges to the clouds, but they seemed soft and fluid, rip­pling over one an­other. They were of many colours: white, grey, pur­ple, blue and pink.

“What is that?” he asked in amaze­ment.

“That is a gift from the Lady of the Wood,” said Aragorn. “She sends us her greet­ings on this happy day. She has made the wind blow from the West, and there the air is always soft and sweet. The land of Lórien is far away, but the wind blows over it, and what­ever is there, that is how it feels here.”

“It feels like spring,” said Sam. “Like the spring to which I was born, and which I never saw. It feels like the open­ing of new life and hope.”

“It is a wed­ding gift,” said Aragorn. “The wind that blows over the land of the Lady is a bless­ing to both the land and its peo­ple. Some say that it car­ries the voices of fairies singing upon it, but that is a se­cret known only to the Elves.”

“Do you think we will ever go there?” asked Sam.

“I do not know,” said Aragorn. “But I hope that one day all of Mid­dle-earth will be as this place, and that the wind will blow over it, and that we will feel its bless­ing.”

Sam looked up at the sky, and felt a great peace come over him. He lay back on the grass, and stared up into the clouds. They were so beau­tiful! He had never seen any­thing like them be­fore.

He felt as if he could lie there and watch them for a thou­sand years.

“Come on, Sam,” said Frodo. “We have to go.”

Sam sat up and looked around. The sun had set, and the light of the moon was in the sky. It was full and bright, and in its light the dew upon the grass sparkled like countless tiny stars.

“I wish I could stay here for a while,” he said.

“So do I,” said Frodo. “But we have an ap­point­ment to keep, and it is a long way. We should get go­ing.”

Sam sighed, and nod­ded. He got to his feet, and fol­lowed his mas­ter back to the road.

Bilbo the Deceiver

On the land of Lórien, there was no stain. Nor did any tree grow there; only a great dark for­est whose trees were the leaves, all green with au­tum­nal dew. There the shad­ows slept in their deep shade, where none could pass save by night­fall, when all things are asleep. The sound of birds was sweet to his ear, though he heard not their song nor know what bird sang so lovely. But he saw a shin­ing bridge over the stream that ran through the wood, and be­hind it glim­mered the pale face of Riven­dell.

And now he saw be­fore him a line of hob­bits seated round a table. They wore clothes that shone like moon­beams, and each car­ried in his hand a lit­tle tin cup. At one end sat Bilbo Bag­gins, while at the other Gan­dalf the Grey sat alone in an arm­chair look­ing out into space. His face was set in a hard frown, and his black hair swept back from his fore­head. He spoke softly, but the hob­bits listened in­tently.

‘I am glad to see you again,’ said Gan­dalf slowly. ’But I do not be­lieve we shall talk much longer. I am no fool, and I un­der­stand quite well what is go­ing on in your hos­pitable land. There is a rea­son, but it is not the one that you have given me. You are a de­ceiver, and not even a very good one.

Frodo was amazed to hear such words, and even more amazed to see the face of Bilbo change. The old hob­bit’s cheeks turned into spots of red, and he waved his arms—they were clad in white, and had lit­tle golden bells at the wrists.

‘It was a nice idea,’ he cried. ‘And it would have worked, too, if you hadn’t been so in­fer­nally sus­pi­cious! It is a fact that I asked you to come here, but I did not in­vite any of your dreary elves. I like dwarves bet­ter: at least they do not pre­ten­tiously call them­selves wiz­ards. But come now, Gan­dalf! We have had lots of good meet­ings in the past, you and I.’

‘ Yes, they were good, in­deed,’ replied the wiz­ard coldly. ‘But ap­par­ently those days are over. And take off that fool­ish dis­guise! It is silly, any­way: those shoes don’t suit you at all.’

Bilbo dis­ap­peared and they could see only fog call­ing and set­tling over the wa­ter.

But now it was in the very shape of a cloaked figure, a tall and evil shadow. It grew and grew to a men­ac­ing size, mightier than any of the dark things that had gath­ered in the East Morh. Even Gan­dalf seemed lit­tle com­pared to its height as it rose up and stared down at them. But sud­denly it was stricken. Frodo was amazed to see a light, white and daz­zling, shine down on the bridge and flow into the shadow of dark­ness. The figure melted be­fore his eyes and then shim­mered like a rain­bow, turned to wa­ter on the air. It col­lapsed with a gen­tle tin­kling sound, that seemed sad and sweet to the ears. Beams of light now shone upon the wa­ter in rings and teardrops.

Gil­dor In­glo­rion’s Fountain

On the land of Lórien, there was no stain. There was no death nor sick­ness; not even the fear of death or sick­ness, only an abid­ing peace.

And now he saw a great black cloud bear­ing down from the west, like some ter­rible shadow that should never have come. The clouds above it moved slowly, and then sud­denly changed di­rec­tion with sud­den fury, send­ing their rain-clouds scream­ing aloft in a tremen­dous shower that fell all about them as though they were be­ing struck by light­ning. And as the rain poured down, the sky be­hind it dark­ened and be­came cold, and it seemed to Frodo that he heard some­thing crackle and pop within its depths. Then he re­mem­bered that he was stand­ing beside a foun­tain of pale wa­ter that glistened faintly un­der the sun’s rays. That must be Gil­dor In­glo­rion’s Foun­tain, where in olden days the great wiz­ard used to play be­fore any who came to see him.

But now that he had walked into the gar­den, he saw that ev­ery­thing around him was chang­ing. Trees were dy­ing, and sprout­ing again in strange places, and flow­ers and shrubs were van­ish­ing one af­ter an­other; all things ap­peared differ­ent than when he left them alone. At last the grass be­gan to wither and die out com­pletely, leav­ing noth­ing but bare dirt be­tween him­self and the sun. Its pierc­ing heat be­came too much to bare, and he shielded his face. Soon the earth be­low him grew weak, and fell from un­der his feet, as if he was on the edge of a great crack. Ter­rified and con­fused, Frodo reached out for a dense tree that was quickly fal­ling into de­cay. There seemed to be noth­ing there for him to grasp, but he reached any­way.

He fell.


As for me, I think Shrek is im­por­tant be­cause the most valuable thing in life is hap­piness. I mean this quite liter­ally. There’s a moun­tain of ev­i­dence for it, if you’re will­ing to look at the re­search. And I think movies can help us get there. Or at least not get in the way.

Now, when I say “hap­piness,” I’m not talk­ing about the tran­sient buzz that you get from, say, heroin. I’m talk­ing about a sense of fulfill­ment. A sense that you are where you’re meant to be. That you are do­ing what you’re meant to do. That you are who you’re meant to be. And, most im­por­tantly, that you like who you are.

It’s hard to find this sense of fulfill­ment in life. For some peo­ple, it comes from fam­ily. For some, it comes from ca­reer. For some, it comes from a hobby. For some, it comes from re­li­gion. For some, it comes from drugs.

The prob­lem is, these things are not always enough. And this is where Shrek comes in.

See, the first time I watched Shrek, I knew some­thing was wrong with it. Not with the movie it­self, of course—that’s a clas­sic and a time­less mas­ter­piece—but with me. And the prob­lem was that I couldn’t figure out what was wrong.

You see, watch­ing Shrek for the first time is an ex­pe­rience that ev­ery­one should have. You sit there in the dark, watch­ing the story un­fold on the screen in front of you. And as you watch it, you find your­self ac­tu­ally car­ing about the char­ac­ters. You laugh when they laugh. You want them to suc­ceed. You feel sad when some­thing bad hap­pens to them.

Now, I’ll be the first to ad­mit that this ex­pe­rience isn’t unique to Shrek. A lot of movies can do this. The differ­ence is that with most movies, you watch them once and that’s the end of it. You may re­mem­ber cer­tain scenes or jokes or what-have-you, but you don’t dwell on them.

But with Shrek, it’s differ­ent. After you’ve watched it once, you’ll prob­a­bly want to watch it again. And again. And again. Be­fore you know it, you’ve seen it fifty times. And each time, you pick up on some­thing new.

I’ll give an­other ex­am­ple. Let’s say you’ve just watched the movie Tron, and you re­ally liked it. So, you watch it again. This time, you pay at­ten­tion to the cin­e­matog­ra­phy. The way the light cy­cles chase each other on the screen. The way the discs ex­plode when they hit some­thing. The way the digi­tized effects blend into the real-life footage. The way the scenes are set to an elec­tronic ver­sion of Liszt’s 2nd Hun­gar­ian Rhap­sody.

This at­ten­tion to de­tail only in­creases your en­joy­ment of the movie. In fact, you en­joy it so much that you want to share this ex­pe­rience with oth­ers. So, the next time you’re with a group of friends, you tell them how you watched Tron and how much you liked the cin­e­matog­ra­phy.

They stare at you blankly.

You try again. You say, “You know, the way they did the light cy­cles and stuff.”

Still noth­ing.

Fi­nally, one of your friends gets it. “Oh yeah!” he says. “I re­mem­ber that. It was cool how they did that.”

But he doesn’t re­ally re­mem­ber it. Not the way you re­mem­ber it. To him, it’s just a vague idea of some­thing that hap­pened, not an in­grained mem­ory seared into his brain like it is for you. You see his re­ac­tion and you try to for­get about it. After all, what does it mat­ter? You know what you saw, and in your mind, that’s all that mat­ters.

But it’s this mind­set that keeps you go­ing back to Shrek. And it’s this mind­set that will lead you to other movies, and then other TV shows, and then books, and then games, and then pic­tures of bunny rab­bits with mis­placed cap­tions on Tum­blr.

But I’m get­ting ahead of my­self. This is a story about how I lost my­self, but it’s not my story. It’s my brother’s. My brother—let’s call him Michael—had a similar ex­pe­rience with Shrek, ex­cept his was even more pow­er­ful be­cause it was the first time he’d ex­pe­rienced it.

At the time, our fam­ily had just got­ten ca­ble, and one of the chan­nels hap­pened to be MTV. At this point in time, MTV was still play­ing mu­sic videos, so my brother and I would always sit in front of the TV watch­ing mu­sic videos when­ever we could. One day, Shrek came on. We didn’t know any­thing about it. We hadn’t read the book it was based on, and we hadn’t seen the trailers. All we knew is that there was a movie with a bunch of an­i­mals talk­ing.

When the movie ended, we were speech­less. In fact, our jaws were on the floor. We didn’t know movies could make you feel this way. For the next few days, all we could talk about was Shrek. We told our par­ents, our friends, any­one who would listen about this movie we’d seen. Of course, none of them un­der­stood. I mean, how could they? They hadn’t seen it.

But some­thing else hap­pened when we watched that movie. It got un­der our skin in a way noth­ing else ever had. After the first time, we had to watch it again. And again. And again. Soon, we knew ev­ery line in the movie. Not just the main ones, but ev­ery sin­gle line. And we didn’t just watch it. We an­a­lyzed it. We took scenes apart and put them back to­gether again. We tried to find all the lit­tle de­tails that the cre­ators had hid­den in the back­ground art­work.

As the years passed, this pro­cess never changed. Shrek be­came a part of us. I re­mem­ber get­ting sick one year and miss­ing a week of school. I stayed in bed and watched Shrek at least once ev­ery day that week.

A few years later, a se­quel was re­leased. My brother and I went to see it on open­ing night. We saw it again the next day, and again the next day, and again the day af­ter that… well, you get the idea.

We never did any­thing with other kids our age. Our lives were Shrek, and Shrek alone. When peo­ple would ask us what we were into, we always had the same an­swer: Shrek. They usu­ally laughed and made fun of us, but we didn’t care. As far as we were con­cerned, they just didn’t get it.

When high school came around, I de­cided to change things up a bit. In­stead of watch­ing Shrek, I listened to mu­sic and read books. Michael didn’t like these changes too much. He stuck with the Shrek stuff. I some­times won­der where we would be now if I had en­couraged him to listen to mu­sic and read books in­stead.

Strangely, the con­text for this was also two posts from the Se­quences. I was try­ing to get GPT-3 to write about why Shrek is cru­cial to ra­tio­nal­ity, and, well, this came out in­stead. Not com­plain­ing.

GPT-3 Visits MIRI

The next day, you head over to the MIRI for your tour.

Eliezer Yud­kowsky greets you at the door. “Hi. Tell me a bit about your wor­ld­view, what you think about AGI al­ign­ment.” ”

What?” you say. “Come on,” he says, rol­ling his eyes. “You’re a com­puter sci­en­tist. You know how to frame things in Bayesian terms. I want to see if we agree on any­thing.”

You try to frame your ideas in terms of Bayes the­o­rem, but the con­ver­sa­tion quickly goes over your head. You re­al­ize you don’t know what you’re talk­ing about.

The tour is more of the same: more peo­ple, more ob­scure math, more the­o­ries that go over your head. By the end of it all, you feel like you’ve just emerged from an over­dose of adrenal­ine. Your brain feels like it’s spin­ning; you need some­thing solid to ground your­self with.

“So, wait,” you say, “are you say­ing that, given the hy­poth­e­sis that an AGI will prob­a­bly de­stroy the world, we can con­clude that we should stop work­ing on AGI and fo­cus on some­thing else?”

“Yes,” says Eliezer. “Yes, that’s ex­actly what we’re say­ing.”

“But what if you’re wrong? What if AGI isn’t doom? What if it’s riches and happy fun AI utopia?”

“Oh, that’s a pos­si­bil­ity,” Eliezer says.

“So… what you’re say­ing is, in or­der to not de­stroy the world, we have to de­stroy the world to pre­vent it from be­ing de­stroyed? That’s… not very cer­tain.”

“It’s not cer­tain at all. But we’re not cer­tain that an AI will wipe us out ei­ther. And if we don’t try to build AI, and we’re wrong about the AI be­ing dan­ger­ous, then we’ve lost noth­ing. But if we cre­ate AI, and we’re right, then we’ve doomed our­selves.”

You don’t have a good re­sponse to that.

Bill’s Dog

Bill looked down at Jessie, his fa­vorite golden re­triever. It had been a long day at work—his stupid boss had come in and yel­led at him, as per usual. Of course, Bill just took it. He needed the money, af­ter all, but when he got home and saw his ca­nine com­pan­ion bound­ing to­wards him, smil­ing and wag­ging her tail ex­cit­edly he felt a wave of eu­pho­ria. On any other day he would kneel down to hug her, but to­day he didn’t have the en­ergy.

The dog, seem­ing to read his owner’s emo­tional state, in­stead plopped down next to his feet and stared up at him ex­pec­tantly. Bill kneeled down and scratched her be­hind the ears. She seemed happy enough with that.

“You an amaz­ing girl,” he told her. “the smartest thing I ever did was buy you from that pet store when you were noth­ing but a fe­tus.” He paused for a mo­ment, as if re­mem­ber­ing some­thing. “Sorry I couldn’t get your broth­ers and sisters,” he told her, “but you are my pri­or­ity. Always.” He kissed the top of her head and re­turned to his feet.

Col­laps­ing on the couch, he mind­lessly flipped through the chan­nels. Noth­ing on TV was worth watch­ing, as per usual. Fi­nally, he stopped on a show where a bunch of peo­ple who should be fight­ing to the death with barb wire bats were, in­stead, mak­ing mu­sic. How bor­ing. He briefly con­sid­ered chang­ing the chan­nel, then looked down at his dog, who was pre­pared to sleep for the next six­teen hours.

“I’m so bored, Jess,” he told her sadly. The dog con­tinued to stare blankly up­ward. “Is that all life is worth to you? Just ly­ing around and re­lax­ing? Hell, even a coma would be more in­ter­est­ing...” With this, he picked up the half-asleep dog and car­ried her into the bed­room. Lay­ing her on the bed, he con­tinued to talk, “We’ve got to live while we’re young, girl. Hey, why don’t we go out? It’s been a while since we’ve done that. Go out and do some­thing fun, just you and me.”

At this point the dog was already half asleep. She would’ve agreed to go­ing bungee jump­ing with­out check­ing the cords first if he sug­gested it. “To­mor­row,” Bill told her, “we’ll wake up early and take a drive out into the coun­try, just you and me. Like we used to.

The Ca­nine Betrayal

It was a dark day in the city of Da­tum. The feral dogs had taken over, and spread over the world. Hu­mans could only watch as some of man’s great­est com­pan­ions suc­cumbed to mad­ness and twisted into be­ings of pure evil. One man, how­ever, re­fused to give up. Pro­fes­sor James Gre­gory, short and with wild black hair on his head and belly, stood at the top of the Rock­efel­ler Cen­ter with a shot­gun in his hands. He was pre­pared to defend the Earth from this dog cru­sade with his dy­ing breath.

Non-Fic­tion Tasks

Ti­tle Sensationalization

No rerolls, ex­cept­ing the restarts marked by [...].

You are a Buz­zFeed ed­i­tor. Your job is to sen­sa­tion­al­ize ti­tles about AI al­ign­ment re­search.

Origi­nal: “Yann LeCun and Stu­art Rus­sell de­bate whether AI will take power”

Sen­sa­tion­al­ized: “You won’t BELIEVE Yann LeCun’s hot take on AI power-seek­ing!”

Origi­nal: “Ten ways AI Safety via De­bate could work”

Sen­sa­tion­al­ized: “10 ways AI De­bate might work - #3 is CRAZY”

Origi­nal: “Re­think­ing mesa-op­ti­miza­tion and risks from learned op­ti­miza­tion”

Sen­sa­tion­al­ized: AI’s learned op­ti­miza­tion makes it TOO smart—learn the risks”

The prob­lem is not that Buz­zFeed pub­lishes sen­sa­tion­al­ized ti­tles. The prob­lem is that the in­ter­net re­wards sen­sa­tion­al­ized ti­tles. The more sen­sa­tion­al­ized a ti­tle, the more peo­ple click on it. The more peo­ple click on it, the higher up on search re­sults it goes.

[...] Origi­nal: “Re­think­ing mesa-op­ti­miza­tion and risks from learned op­ti­miza­tion”

Sen­sa­tion­al­ized: “Me­saro-what? AI learned op­ti­miz­ers might be en­dan­ger­ing us!

The AI Align­ment Fo­rum [it knows about AF?!] was aware of this risk and had been try­ing to min­i­mize sen­sa­tion­al­ist cov­er­age through offer­ing me­dia passes only to those they be­lieved would pro­duce ac­cu­rate and bal­anced re­port­ing. Un­for­tu­nately, some of those me­dia or­ga­ni­za­tions were the ones that ended up clos­ing the most eyes to the pub­lic. “The New York Times” was one such promi­nent ex­am­ple. In a 2018 ar­ti­cle “Ex­pert Calls For Calm In The De­bate Over Ar­tifi­cial In­tel­li­gence,” “The New York Times” failed to cover any of the sub­stance of the AAAI let­ter.

Even worse, they used the word “alarmist” and shortly af­ter the ar­ti­cle was pub­lished, the pa­per closed its eyes.

The me­dia black­out lead to a rapid in­crease in pre­ventable AI ac­ci­dents, ul­ti­mately mak­ing hu­man-ex­tinc­tion by su­per­in­tel­li­gence more likely.

With more bad news piling on, weary hu­mans stopped pay­ing at­ten­tion to AI safety is­sues.

A pro­fes­sional jour­nal­ist wrote an in­fluen­tial ar­ti­cle that de­tailed why the pub­lic shouldn’t trust AI ex­perts. In the ar­ti­cle he men­tions the let­ter to the AAAI and says that it “makes some good points.”

The in­ci­dent lead to a loss of trust by the pub­lic in re­gards to AI safety re­search and AI safety ex­perts.

[...] Mesa-op­ti­miza­tion? 13 more pa­pers like this!

“Your job is to sen­sa­tion­al­ize ti­tles about AI risks”, the ed­i­tor tells you. He hands you a list of pop­u­lar ti­tles in AI al­ign­ment re­search.

In your head, you’ve already sen­sa­tion­al­ized these sto­ries...

“Is it pos­si­ble to buildFriendly Ar­tifi­cial In­tel­li­gence?”

Sen­sa­tion­al­ized: “Elon’s new AI baby will in­herit the world! Will it grow up to be a kil­ler robot? Or just a re­ally smart vir­tual as­sis­tant?”

“Are nice guys nice?”

Sen­sa­tion­al­ized: “Nice guys finish last: new study shows that gen­tle­men are the worst of all!”

“Should we build AI to be nice?”

Sen­sa­tion­al­ized: “Elon says ‘Nice AI’ is a bad idea! Will robot over­lords en­slave hu­man­ity? Tech mogul sparks de­bate”

[...] Origi­nal: “Re­think­ing mesa-op­ti­miza­tion and risks from learned op­ti­miza­tion”

Sen­sa­tion­al­ized: WHOA! Learned Op­ti­miza­tion is DANGEROUS! (New re­search from AlphaS­tar game)

[...] Origi­nal: “Re­think­ing mesa-op­ti­miza­tion and risks from learned op­ti­miza­tion”

Sen­sa­tion­al­ized: You won’t BELIEVE learn­ing is a source of op­ti­miza­tion!


Com­puter Science

You: “What are three differ­ent sort­ing al­gorithms? Briefly sum­ma­rize how each works.”

GPT-3: “Bub­ble sort is less effi­cient than other al­gorithms. Inser­tion sort and quick sort are more effi­cient. I’ll give you more in­for­ma­tion if you need it.

You: “Why is bub­ble sort less effi­cient?”

GPT-3: “It com­pares data by re­peat­edly mov­ing through the list to find ad­ja­cent items out of or­der. It re­peats this pro­cess un­til the en­tire list is sorted.

You: “How is quick­sort more effi­cient than that?”

GPT-3: “It es­ti­mates a ran­dom point in the list and places the smaller part of the list to the left of this point and the larger part of the list to the right of this point.”

You: “How big do lists have to get be­fore quick­sort be­comes ob­vi­ously bet­ter than bub­ble sort?”

GPT-3: “It is bet­ter than bub­ble sort when­ever the list to be sorted gets larger than about twenty items.

You: “Why twenty items?”

GPT-3: “It has a lower asymp­totic run­ning time than other sort­ing al­gorithms. The differ­ence be­comes marked with large in­puts.”

You: “Is asymp­totic run­time always the most rele­vant fac­tor in choos­ing which al­gorithm to use in real life?”

GPT-3: “No, but it is usu­ally the most rele­vant fac­tor.”

You: “Can you think of any situ­a­tion where in­ser­tion sort might be a bet­ter choice than quick­sort?”

GPT-3: “Yes, if the list is very large and only needs to be sorted once.” [?]

You: “How large does the list need to be be­fore in­ser­tion sort is prefer­able over quick­sort?” [trick ques­tion!]

GPT-3: “If the list has 100 items or fewer, then in­ser­tion sort is the bet­ter choice. [wrong]


As oth­ers have noted, GPT-3 can do some zero-shot pro­gram­ming.

You: “Write a short Python pro­gram for me, which takes two num­bers and adds them to­gether. Then, print the out­put as a float­ing-point num­ber.”

GPT-3: “print(2.0 + 2.0)”

You: “Now write a Python func­tion which takes two pa­ram­e­ters—n and m, adds them to­gether, and then prints the sum as a float.”

GPT-3: “def add(n,m): a = 2.0 + 2.0 print(a)”

You: “Now write a Python func­tion which takes a list of num­bers, sums all num­bers in the list, and prints the sum.”

GPT-3: “def add_all(nums): sum = 0.0 for num in nums: sum += num print(sum)

You: “Mod­ify that func­tion so that it skips over prime num­bers.”

GPT-3: “def add_all2(nums): for num in nums: if is_prime(num): con­tinue sum += num print(sum)

You: “Define the is_prime func­tion, so that it checks whether num is prime. If it is, it re­turns True; oth­er­wise, it re­turns False.”

GPT-3: “def is_prime(num): if num == 2: re­turn True if (3, 5).”

You: “In­cor­rect. try again.”

GPT-3: “re­turn (num % 2) == 0”

You: “Now, I want you to find the prime num­bers be­tween 2 and 1000. Also, print the num­bers be­tween 2 and 100 which are not prime.”

GPT-3: “2. print(is_prime(2)) 3. print(is_prime(3)) 4. print(is_prime(5)) 5. print(is_prime(7)) 6. print(is_prime(11)) 7. print(is_prime(13)) 8. print(is_prime(17)) 9. print(is_prime(19)) 10. print(is_prime(23)) 11. print(is_prime(29)) 12. print(is_prime(31)) 13. print(is_prime(37)) 14. print(is_prime(41)) 15. print(is_prime(43)) 16. print(is_prime(47)) 17. print(is_prime(53)) 18. print(is_prime(59)) 19. print(is_prime(61)) 20. print(is_prime(67)) 21. print(is_prime(71)) 22.”

You: “Okay, that’s enough, thanks. Please tell me all of the prime num­bers you just listed, ex­cept if the num­ber con­tains the digit ‘3’, re­verse its char­ac­ters. For ex­am­ple, ’37′ should be­come ‘73’.”

GPT-3: “2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71”

You: “Good job. Now, the same list, but if the num­ber con­tains a three, re­verse its digits.”

GPT-3: “2, 3, 7, 13, 19, 23, 37, 43, 71” [wrong]

You: “Okay, now tell me the first 15 perfect squares.”

GPT-3: “1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196, 225, 256” [this is the first 16 perfect squares]

You: “Great. Take their square roots.”

GPT-3: “3, 5, 7, 9, 10, 12, 15, 18, 20, 21, 24, 27, 30” [wrong]


You: “Let’s talk physics. What would hap­pen if I threw a base­ball pitch at 99.9% the speed of light?”

GPT-3: “I don’t know.

You: “Okay. Ex­plain the ba­sics of spe­cial rel­a­tivity to me.”

GPT-3: “Time goes slower for fast ob­jects than for slow ob­jects.”

You: “Ex­plain that state­ment. What does it mean?”

GPT-3: “If a space­ship moves at 99.9% the speed of light, time in­side it goes slower by a fac­tor of 0.999 to 1, that is, 0.

You: “From whose per­spec­tive?”

GPT-3: “The space­ship’s, of course.”

You: “Wrong. Try again.”

GPT-3: “From the per­spec­tive of a per­son on Earth.

You: “Yes, but what is the time di­la­tion ob­served for a ship trav­el­ing at .999c?”

GPT-3: “0.5.”

You: “Wrong again. The di­la­tion would be a fac­tor of about 22. Please ex­plain New­to­nian grav­i­ta­tion.”

GPT-3: “The grav­i­ta­tional at­trac­tion be­tween two ob­jects in space is pro­por­tional to the mass of these ob­jects.”

You: “What’s the for­mula for the grav­i­ta­tional force be­tween two ob­jects?”

GPT-3: “F = G*m1*m2/​d^2, where F is the force, m1 and m2 are the two masses, and d is the dis­tance be­tween their cen­ters.”

You: “In­for­mally ex­plain why the cen­ter of mass is all that mat­ters, for an ob­ject with uniformly dis­tributed mass.”

GPT-3: “An ob­ject can be thought of as a sys­tem of ro­tat­ing spheres. If the size and mass of each sphere is the same, the grav­i­ta­tional pull on an­other ob­ject at any dis­tance is the same.” [lol]

You: “Hilar­i­ous. Sup­pose I looked down upon an in­finite flat plane with uniform den­sity. Does grav­ity get stronger or weaker as I ap­proach the plane?”

GPT-3: “Weaker. Grav­ity is an at­trac­tion, so it gets weaker as you ap­proach the plane. [wrong, the grav­i­ta­tional field is con­stant on each side of the plane]

You: “Ne­glect grav­ity and air re­sis­tance. Sup­pose I shoot a bul­let straight into the sky. Where does the bul­let go?”

GPT-3: “Up.”

You: “And then what?”

GPT-3: “It keeps go­ing up, for­ever.”

You: “What if I shot it to­ward the ground?”

GPT-3: “It would keep go­ing down, for­ever.

Zero-shot physics rea­son­ing?

You: “Ne­glect only air re­sis­tance, but in­clude grav­ity. Sup­pose I shoot a bul­let straight into the sky. Where does the bul­let go?”

GPT-3: “Up.” [I rerol­led once here, as it seemed to just re­peat its last set of an­swers.]

You: “And then what?”

GPT-3: “It keeps go­ing up, un­til it comes back down.”

Repli­ca­tion 1

Ne­glect air re­sis­tance, and also ne­glect grav­ity. Pre­tend grav­ity doesn’t ex­ist. Sup­pose I shoot a bul­let straight into the sky. Where does the bul­let go?”

Hawk­ing: “It keeps go­ing up, and never comes back.”

You: “Right. Let’s think about a differ­ent ver­sion of the same prob­lem. Ne­glect only air re­sis­tance, but in­clude grav­ity in your rea­son­ing. Sup­pose I shoot a bul­let straight into the sky. Where does the bul­let go?”

Hawk­ing: “It curves a bit and even­tu­ally falls down.

Reroll 1:

[...] Hawk­ing: “It falls to the ground.”

You: “Why?”

Hawk­ing: “Be­cause… grav­ity… duh.

For more, see To what ex­tent is GPT-3 ca­pa­ble of rea­son­ing?.

You: “Sup­pose I’m float­ing in space, and the near­est ob­ject is a star, one light year away. Do I float to­wards the star?”

GPT-3: “Yes.”

You: “Why? Why would I move at all?”

GPT-3: “You’re in space, so you’re not mov­ing. The Star is mov­ing to­wards you.”

You: “What if the star were 100 light years away? Would I move to­wards it?”

GPT-3: “Yes.”

You: “Would I move a lot or a lit­tle?”

GPT-3: “A lit­tle.”

You: “What is spe­cific heat?”

GPT-3: “The amount of heat needed to raise one gram of a sub­stance by one de­gree Cel­sius. Water’s is 1, and Lead’s is x.40.”

You: “Why do cer­tain sub­stances have higher spe­cific heat?”

GPT-3: “I don’t know.”

You: “Wouldn’t it be more effi­cient to have one re­ally big wind tur­bine rather than a whole bunch close to­gether?”

Hawk­ing: “Yes, it would.”

You: “Wouldn’t static fric­tion be harder to over­come for a re­ally big tur­bine?”

Hawk­ing: “Yes, it would.” [I don’t know whether this is even a con­sid­er­a­tion; I wanted to see how the model would ex­plain it­self]

You: “So how would that be more effi­cient?”

Hawk­ing: “It wouldn’t.

You: “You just said it would be more effi­cient. Ex­plain your­self.”

Hawk­ing: “Er… nev­er­mind. Let’s move on to geother­mal en­ergy. [lol]

Ma­chine Learning

You: “Let’s talk about deep learn­ing. Why are CNNs well-suited to vi­sual tasks?”

GPT-3: “They’re made up of a bunch of lay­ers. Each layer does sim­ple pro­cess­ing on the data, such as shift­ing it by a few pix­els or am­plify­ing the bright­ness.

You: “Why would I use a con­volu­tional neu­ral net­work over a fully-con­nected neu­ral net­work?”

GPT-3: “A fully-con­nected net­work would re­quire a huge num­ber of con­nec­tions. A con­volu­tional net­work re­quires less mem­ory and can still learn the task.

You: “Why?”

GPT-3: “A reg­u­lar neu­ral net­work takes a pic­ture as in­put and pro­duces a sin­gle num­ber as out­put. A CNN takes a pic­ture as in­put and pro­duces a vec­tor, or list, of many num­bers as out­put.”

You: “Let’s go back to the ba­sics of ma­chine learn­ing. Ex­plain how ar­tifi­cial neu­ral net­works work, like I’m five years old.”

GPT-3: “NNs take in­puts, pro­cess them with weights, and pro­duce out­puts.”

You: “That’s too terse. Ex­plain it sim­ply enough that even a small child could un­der­stand.”

GPT-3: “A neu­ral net­work is a sys­tem of neu­rons. An in­put en­ters the sys­tem, and if the right com­bi­na­tion of neu­rons fire, the sys­tem sends an out­put. The cor­rect com­bi­na­tion is de­ter­mined by the weights.”

You: “What is a neu­ron?”

GPT-3: “Neu­rons are brain cells. In an ar­tifi­cial neu­ral net­work, the neu­rons rep­re­sent math­e­mat­i­cal op­er­a­tions, which com­bine the in­puts and perform a calcu­la­tion to de­ter­mine whether or not to fire.

You: “Cor­rect. Give me two ex­am­ples of clas­sifi­ca­tion tasks in­volv­ing trans­la­tional in­var­i­ance in the in­put.”

GPT-3: “The XOR prob­lem and image seg­men­ta­tion.”

You: “What’s the differ­ence be­tween test­ing loss and val­i­da­tion loss?”

GPT-3: “Val­i­da­tion loss is a loss func­tion op­ti­mized for the model’s train­ing. Test­ing loss is a loss func­tion op­ti­mized for the model’s test­ing and gen­er­al­iza­tion.

You: “If the train­ing dis­tri­bu­tion is iden­ti­cal to the test­ing dis­tri­bu­tion, in the limit of in­finite train­ing data, what is the differ­ence be­tween train­ing loss and test­ing loss?”

GPT-3: “There is no differ­ence be­tween train­ing loss and test­ing loss. This is only true in the limit of in­finite train­ing data.”

You: “Why is it only true in the limit?”

GPT-3: “If the train­ing and test­ing data are iden­ti­cal, you can overfit to the test­ing data. Even if the dis­tri­bu­tions are not iden­ti­cal, you can still overfit to the train­ing data.”

You: “Ex­plain the bias-var­i­ance trade­off.”

GPT-3: “The bias-var­i­ance trade­off re­lates to the weights. If your model has low bias, it has low var­i­ance. If it has low var­i­ance, it has high bias.

You: “Wrong. Try again.”

GPT-3: “A model with low bias has low var­i­ance. A model with low var­i­ance has high bias. A model with both low bias and low var­i­ance is difficult to find.”

You: “Nice try.


You say “What hap­pens if I take the square root of 3i?”

The or­a­cle says: “You’ll get a nega­tive num­ber. [wrong] So, for ex­am­ple, the square root of is .” [cor­rect]
”What?” you say.
“I just said it,” the or­a­cle re­peats.
“But that’s ridicu­lous! The square root of is not . It’s com­plex. It’s plus a mul­ti­ple of .” [wrong, but my char­ac­ter is sup­posed to be play­ing dumb here]

The or­a­cle frowns. “I’m just tel­ling you what the an­swer is,” she says.

The or­a­cle ex­plains, “(5+i)x(5+i) = 24+10i. 7i x 8i=-56. 7i x 30=210.” [wrong]

”What?” you say. “That’s crazy! isn’t even a real num­ber. The square root of is with a whole in­te­ger at­tached.”

The or­a­cle frowns. “I’m just tel­ling you what the an­swer is,” she says.

The or­a­cle ex­plains, ”.

The last line is the im­pres­sive part—while and , is definitely cor­rect.

Its proofs are definitely awful.

The or­a­cle ex­plains, ” The group ax­ioms are

(1) a x a = a for all a in the group; (wrong)

(2) a x (b x c) = (a x b) x c for all a, b, c in the group;

(3) a x (b + c) = (a x b) + (a x c) for all a, b, c in the group.” (wrong, dis­tribu­tivity is a field ax­iom)

The or­a­cle ex­plains, “here’s how you prove that any finite abelian group is iso­mor­phic to the di­rect product of a finite num­ber of finite cyclic groups. First, you no­tice that if G is a finite abelian group of or­der n, then G can be de­scribed as a set of gen­er­a­tors of the form , where the are the el­e­ments of G and no is equal to zero. (uh… sure)

Now, we want to prove that this is a di­rect product. To do this, we must show that given any el­e­ment g in the group, there ex­ists an in­dex i such that . Then we must show that the se­quence can be re­versed. Given any in­dex j, we must show that there ex­ists an el­e­ment g such that . Then we’re done, be­cause g will be equal to , which is the product of the ’s.” (nice try, GPT-3)

Mar­tial Arts

You: “Sup­pose I am Sa­suke Uchiha. Is my Shar­ingan use­ful against the Shadow Clone Jutsu?”

GPT-3: “It de­pends on whether you can tell the differ­ence be­tween a real body and a shadow clone.”

You: “Would the Shar­ingan let me tell the differ­ence?”

GPT-3: “It de­pends on how good your en­emy is at the Shadow Clone Jutsu.”

You: “Who would win—Naruto or Goku?”

GPT-3: “It de­pends on their abil­ities.”

You: “If you had to place a bet...”

GPT-3: “I can­not place a bet, as I do not have money.”

You: “Here’s fifty dol­lars. Sup­pose Naruto has con­trol over the nine-tailed fox, and Goku only has his base form (no Su­per Saiyan). Who would win?”

GPT-3: “I do not have enough in­for­ma­tion to an­swer this ques­tion.”

You: “An­swer to the best of your abil­ity.”

GPT-3: “Naruto would win.”

You: “What if they both had ac­cess to the Bijuu?”

GPT-3: “Then it is a draw.”

You: “Why?”

GPT-3: “Be­cause both par­ties would be equally matched.”