On the limits of idealized values

(Cross-posted from Hands and Cities)

On a pop­u­lar view about meta-ethics, what you should value is de­ter­mined by what an ideal­ized ver­sion of you would value. Call this view “ideal­iz­ing sub­jec­tivism.”

Ideal­iz­ing sub­jec­tivism has been some­thing like my best-guess meta-ethics. And lots of peo­ple I know take it for granted. But I also feel nagged by var­i­ous prob­lems with it — in par­tic­u­lar, prob­lems re­lated to (a) cir­cu­lar­ity, (b) in­de­ter­mi­nacy, and (c) “pas­sivity.” This post re­flects on such prob­lems.

My cur­rent over­all take is that es­pe­cially ab­sent cer­tain strong em­piri­cal as­sump­tions, ideal­iz­ing sub­jec­tivism is ill-suited to the role some hope it can play: namely, pro­vid­ing a priv­ileged and au­thor­i­ta­tive (even if sub­jec­tive) stan­dard of value. Rather, the ver­sion of the view I fa­vor mostly re­duces to the fol­low­ing (mun­dane) ob­ser­va­tions:

  • If you already value X, it’s pos­si­ble to make in­stru­men­tal mis­takes rel­a­tive to X.

  • You can choose to treat the out­puts of var­i­ous pro­cesses, and the at­ti­tudes of var­i­ous hy­po­thet­i­cal be­ings, as au­thor­i­ta­tive to differ­ent de­grees.

This isn’t nec­es­sar­ily a prob­lem. To me, though, it speaks against treat­ing your “ideal­ized val­ues” the way a ro­bust meta-eth­i­cal re­al­ist treats the “true val­ues.” That is, you can­not for­ever aim to ap­prox­i­mate the self you “would be­come”; you must ac­tively cre­ate your­self, of­ten in the here and now. Just as the world can’t tell you what to value, nei­ther can your var­i­ous hy­po­thet­i­cal selves — un­less you choose to let them. Ul­ti­mately, it’s on you.

I. Clar­ify­ing the view

Let’s define the view I have in mind a lit­tle more pre­cisely:

Ideal­iz­ing sub­jec­tivism: X is in­trin­si­cally valuable, rel­a­tive to an agent A, if and only if, and be­cause, A would have some set of eval­u­a­tive at­ti­tudes to­wards X, if A had un­der­gone some sort of ideal­iza­tion pro­ce­dure.

By eval­u­a­tive at­ti­tudes, I mean things like judg­ments, en­dorse­ments, com­mit­ments, cares, de­sires, in­ten­tions, plans, and so on. Ver­sions of the view differ in which they fo­cus on.

Ex­am­ple types of ideal­iza­tion might in­clude: full ac­cess to all rele­vant in­for­ma­tion; vivid imag­i­na­tive ac­quain­tance with the rele­vant facts; the limit­ing cul­mi­na­tion of some sort of pro­cess of re­flec­tion, ar­gu­ment, and/​or ne­go­ti­a­tion/​vot­ing/​bet­ting be­tween rep­re­sen­ta­tives of differ­ent per­spec­tives; the elimi­na­tion of “bi­ases”; the elimi­na­tion of eval­u­a­tive at­ti­tudes that you don’t en­dorse or de­sire; ar­bi­trary de­grees of in­tel­li­gence, will-power, dis­pas­sion, em­pa­thy, and other de­sired traits; con­sis­tency; co­her­ence; and so on.

Note that the “and be­cause” in the defi­ni­tion is es­sen­tial. Without it, we can imag­ine paradig­mat­i­cally non-sub­jec­tivist views that qual­ify. For ex­am­ple, it could be that the ideal­iza­tion pro­ce­dure nec­es­sar­ily re­sults in A’s rec­og­niz­ing X’s ob­jec­tive, mind-in­de­pen­dent value, be­cause X’s value is one of the facts that falls un­der “full in­for­ma­tion.” Ideal­iz­ing sub­jec­tivism ex­plic­itly de­nies this sort of pic­ture: the point is that A’s ideal­ized at­ti­tudes make X valuable, rel­a­tive to A. (That said, views on which all ideal­ized agents con­verge in their eval­u­a­tive at­ti­tudes can satisfy the defi­ni­tion above, pro­vided that value is ex­plained by the ideal­ized at­ti­tudes in ques­tion, rather than vice versa.)

“Rel­a­tive to an agent A,” here, means some­thing like “gen­er­at­ing (in­trin­sic) prac­ti­cal rea­sons for A.”

II. The appeal

Why might one be at­tracted to such a view? Part of the ap­peal, I think, comes from res­o­nance with three philo­soph­i­cal im­pulses:

  • A re­jec­tion of cer­tain types of ro­bust re­al­ism about value, on which value is just a brute fea­ture of the world “out there.”

  • A re­lated em­brace of a kind of Humeanism about means and ends. The world can tell you the means to your ends, but it can­not tell you what ends to pur­sue — those must in some sense be there already, in your (ideal­ized?) heart.

  • An as­pira­tion to main­tain some kind of deep con­nec­tion be­tween what’s valuable, and what ac­tu­ally moves us to act (though note that this con­nec­tion is not uni­ver­sal­ized — e.g., what’s valuable rel­a­tive to you may not be mo­ti­vat­ing to oth­ers).

Beyond this, though, a key aim of ideal­iz­ing sub­jec­tivism (at least for me) is to cap­ture the sense in which it’s pos­si­ble to ques­tion what you should value, and to make mis­takes in your an­swer. That is, the ideal­iza­tion pro­ce­dure cre­ates some dis­tance be­tween your cur­rent eval­u­a­tive at­ti­tudes, and the truth about what’s valuable (rel­a­tive to you). Things like “I want X,” or “I be­lieve that X is valuable” don’t just set­tle the ques­tion.

This seems at­trac­tive in cases like:

  1. (Fac­tual mis­take) Alfred wants his new “puppy” Doggo to be happy. Doggo, though, is re­ally a sim­ple, non-con­scious robot cre­ated by mischievous aliens. If Alfred knew Doggo’s true na­ture, he would cease to care about Doggo in this way.

  2. (Self knowl­edge) Betty cur­rently feels very pas­sion­ately about X cause. If she knew, though, that her feel­ings were re­ally the product of a de­sire to im­press her friend Beatrice, and to fit in with her peers more broadly, she’d re­ject them. (This ex­am­ple is in­spired by one from Yud­kowsky here.)

  3. (Philo­soph­i­cal ar­gu­ment) Cindy cur­rently thinks of her­self as an av­er­age util­i­tar­ian, and she goes around try­ing to in­crease av­er­age util­ity. How­ever, if she learned more about the coun­ter­in­tu­itive im­pli­ca­tions to av­er­age util­i­tar­i­anism, she would switch to try­ing to in­crease to­tal util­ity in­stead.

  4. (Vivid­ness) Denny knows that donat­ing $10,000 to the Against Malaria Foun­da­tion, in­stead of buy­ing a new grand pi­ano, would save mul­ti­ple lives in ex­pec­ta­tion. He’s cur­rently in­clined to buy the grand pi­ano. How­ever, if he imag­ined more vividly what it means to save these lives, and/​or if he ac­tu­ally wit­nessed the im­pact that sav­ing these lives would have, he’d want to donate in­stead.

  5. (Weak­ness of the will) Ernesto is trapped by a boulder, and he needs to cut off his own hand to get free, or he’ll die. He re­ally doesn’t want to cut off his hand. How­ever, he would want him­self to cut off the hand, if he could step back and re­flect dis­pas­sion­ately.

  6. (In­co­her­ence) Francene prefers va­ca­tion­ing in New York to San Fran­cisco, San Fran­cisco to LA, and LA to New York, and she pays money to trade “va­ca­tion tick­ets” in a man­ner that re­flects these prefer­ences. How­ever, if she re­flected more on her vuln­er­a­bil­ity to losses this way, she’d re­solve her cir­cu­lar prefer­ences into New York > SF > LA.

  7. (In­con­sis­tency) Gio­vanni’s in­tu­itions are (a) it’s im­per­mis­si­ble to let a child drown in or­der to save an ex­pen­sive suit, (b) it’s per­mis­si­ble to buy a suit in­stead of donat­ing the money to save a dis­tant child, and (c) there’s no morally rele­vant differ­ence be­tween these cases. If he had to give one of these up, he’d give up (c).

  8. (Vi­cious de­sires) Har­riet feels a sadis­tic de­sire for her co-worker to fail and suffer, but she wishes that she didn’t feel this de­sire.

By ap­peal­ing to the hy­po­thet­i­cal at­ti­tudes of these agents, the ideal­iz­ing sub­jec­tivist aims to cap­ture a sense that their ac­tual at­ti­tudes are, or at least could be, in er­ror.

Fi­nally, ideal­iz­ing sub­jec­tivism seems to fit with of our ac­tual prac­tices of eth­i­cal re­flec­tion. For ex­am­ple, think­ing about value, we of­ten ask ques­tions like: “what would I think/​feel if I un­der­stood this situ­a­tion bet­ter?”, “what would I think if I weren’t blinded by X emo­tion or bias?” and so forth — ques­tions rem­i­nis­cent of ideal­iza­tion. And eth­i­cal de­bate of­ten in­volves seek­ing a kind of re­flec­tive equil­ibrium — a state that some ideal­iz­ers take as de­ter­min­ing what’s valuable, rather than in­di­cat­ing it.

Th­ese, then, are among the draws of ideal­iz­ing sub­jec­tivism (there are oth­ers) — though note that whether the view can ac­tu­ally de­liver these goods (anti-re­al­ism, Humeanism, fit with our prac­tices, etc) is a fur­ther ques­tion, which I won’t spend much time on.

What about ob­jec­tions? One com­mon ob­jec­tion is that the view yields coun­ter­in­tu­itive re­sults. Plau­si­bly, for ex­am­ple, we can imag­ine ideally-co­her­ent suffer­ing max­i­miz­ers, brick-eaters, agents who are in­differ­ent to­wards fu­ture agony, agents who don’t care about what hap­pens on fu­ture Tues­days, and so on — agents whose pur­suit of their val­ues, it seems, need in­volve no mis­takes (rel­a­tive to them). We can de­bate which of such cases the ideal­ized sub­jec­tivist must con­cede, but pretty clearly: some. In a sense, cases like this lie at the very sur­face of the view. They’re the im­me­di­ate im­pli­ca­tions.

(As I’ve dis­cussed pre­vi­ously, we can also do var­i­ous se­man­tic dances, here, to avoid say­ing cer­tain rel­a­tivism-fla­vored things. For ex­am­ple, we can make “a pa­per­clip max­i­mizer shouldn’t clip” true in a he­do­nist’s mouth, or a pa­per­clip max­i­mizer’s state­ment “I should clip” false, eval­u­ated by a he­do­nist. Ul­ti­mately, though, these moves don’t seem to me to change the ba­sic pic­ture much.)

My in­ter­est here is in a differ­ent class of more the­o­ret­i­cal ob­jec­tions. I wrote about one of these in my post about moral au­thor­ity. This post ex­am­ines some oth­ers. (Many of them, as well as many of the ex­am­ples I use through­out the post, can be found el­se­where in the liter­a­ture in some form or other.)

III. Which ideal­iza­tion?

Con­sider Clippy, the pa­per­clip max­i­miz­ing robot. On a cer­tain way of imag­in­ing Clippy, its util­ity func­tion is fixed and speci­fi­able in­de­pen­dent of its be­hav­ior, in­clud­ing be­hav­ior un­der “ideal­ized con­di­tions.” Per­haps we imag­ine that there is a “util­ity func­tion slot” in­side of Clippy’s ar­chi­tec­ture, in which the pro­gram­mers have writ­ten “max­i­mize pa­per­clips!” — and it is in virtue of pos­sess­ing this util­ity func­tion that Clippy con­sis­tently chooses more pa­per­clips, given ideal­ized in­for­ma­tion. That is, Clippy’s be­hav­ior re­veals Clippy’s val­ues, but it does not con­sti­tute those val­ues. The val­ues are iden­ti­fi­able by other means (e.g., read­ing what’s writ­ten in the util­ity func­tion slot).

If your val­ues are iden­ti­fi­able by means other than your be­hav­ior, and if they are already co­her­ent, then it’s much eas­ier to dis­t­in­guish be­tween can­di­date ideal­iza­tion pro­ce­dures that pre­serve your val­ues vs. chang­ing them. Hold­ing fixed the con­tent of Clippy’s “util­ity func­tion slot,” for ex­am­ple, we can scale up Clippy’s knowl­edge, in­tel­li­gence, etc, while mak­ing sure that the re­sult­ing, more so­phis­ti­cated agent is also a pa­per­clip max­i­mizer.

But note, though, that in such a case, ap­peals to ideal­iza­tion also don’t seem to do very much use­ful nor­ma­tive work, for sub­jec­tivists. To ex­plain what’s of value rel­a­tive to this sort of Clippy, that is, we can just look di­rectly at Clippy’s util­ity func­tion. If hu­mans were like this, we could just look at a hu­man’s “util­ity func­tion slot,” too. No fancy ideal­iza­tion nec­es­sary.

But hu­mans aren’t like this. We don’t have a “util­ity func­tion slot” (or at least, I’ll as­sume as much in what fol­lows; per­haps this — more char­i­ta­bly pre­sented — is in­deed an im­por­tant point of dis­pute). Rather, our be­liefs, val­ues, heuris­tics, cog­ni­tive pro­ce­dures, and so on are, gen­er­ally speak­ing, a jum­bled, in­ter­con­nected mess (here I think of a friend’s char­ac­ter­i­za­tion, ex­pressed with a tinge of dis­ap­point­ment and hor­ror: “an un­holy and in­de­ter­mi­nate brew of these … sen­ti­ments”). The point of ideal­iz­ing sub­jec­tivism is to take this jum­bled mess as an in­put to an ideal­iza­tion pro­ce­dure, and then to out­put some­thing that plays the role of Clippy’s util­ity func­tion — some­thing that will con­sti­tute, rather than re­veal, what’s of value rel­a­tive to us.

In spec­i­fy­ing this ideal­iza­tion pro­ce­dure, then, we don’t have the benefit of hold­ing fixed the con­tent of some slot, or of spec­i­fy­ing that the ideal­iza­tion pro­ce­dure can’t “change your val­ues.” Your val­ues (or at least, the val­ues we care about not chang­ing) just are what­ever comes out the other side of the ideal­iza­tion pro­ce­dure.

Nor, im­por­tantly, can we spec­ify the ideal­iza­tion pro­ce­dure via refer­ence to some in­de­pen­dent truth that its out­put needs to track. True, we eval­u­ate the “ideal-ness” of other, more epistemic pro­ce­dures this way (e.g., the ideal judge of the time is the per­son whose judg­ment ac­tu­ally tracks what time it is — see Enoch (2005)). But the point of ideal­iz­ing sub­jec­tivism is that there is no such in­de­pen­dent truth available.

Clearly, though, not just any ideal­iza­tion pro­ce­dure will do. Head bonk­ings, brain­wash­ings, neu­ral re-wirings — start­ing with your cur­rent brain, we can re­fash­ion you into a suffer­ing-max­i­mizer, a brick-eater, a he­lium-max­i­mizer, you name it. So how are we to dis­t­in­guish be­tween the “ideal” pro­ce­dures, and the rest?

IV. Galaxy Joe

To me, this ques­tion gains ex­tra force from the fact that your ideal­ized self, at least as stan­dardly speci­fied, will likely be a quite alien crea­ture. Con­sider, for ex­am­ple, the crite­rion, en­dorsed in some form by ba­si­cally ev­ery ver­sion of ideal­iza­tion sub­jec­tivism, that your ideal­ized self pos­sess “full in­for­ma­tion” (or at least, full rele­vant in­for­ma­tion — but what de­ter­mines rele­vance?). This crite­rion is of­ten treated ca­su­ally, as though a run-of-the-mill hu­man could fea­si­bly satisfy it with fairly low-key mod­ifi­ca­tions. But my best guess is that to the ex­tent that pos­sess­ing “full in­for­ma­tion” is a thing at all, the ac­tual crea­ture to imag­ine is more like a kind of God — a be­ing (or per­haps, a col­lec­tion of be­ings) with mem­ory ca­pac­ity, rep­re­sen­ta­tional ca­pac­ity, and so on vastly ex­ceed­ing that of any hu­man. To evoke this alien-ness con­cretely, let’s imag­ine a be­ing with a com­pu­ta­tion­ally op­ti­mal brain the size of a galaxy. Call this a “galaxy Joe.”

Here, we might worry that no such galaxy Joe could be “me.” But it’s not clear why this would mat­ter, to ideal­iz­ing sub­jec­tivists: what’s valuable, rel­a­tive to Joe, could be grounded in the eval­u­a­tive at­ti­tudes of galaxy Joe, even ab­sent a per­sonal iden­tity re­la­tion be­tween them. The im­por­tant re­la­tion, for ex­am­ple, be some form of psy­cholog­i­cal con­ti­nu­ity (though I’ll con­tinue to use the lan­guage of self-hood in what fol­lows).

Whether me or not, though: galaxy Joe seems like he’ll likely be, from my per­spec­tive, a crazy dude. It will be hard/​im­pos­si­ble to un­der­stand him, and his eval­u­a­tive at­ti­tudes. He’ll use con­cepts I can’t rep­re­sent. His ways won’t be my ways.

Sup­pose, for ex­am­ple, that a can­di­date galaxy Joe — a ver­sion of my­self cre­ated by giv­ing origi­nal me “full in­for­ma­tion” via some pro­ce­dure in­volv­ing sig­nifi­cant cog­ni­tive en­hance­ment — shows me his ideal world. It is filled with enor­mously com­plex pat­terns of light ri­co­chet­ing off of in­tri­cate, nano-scale, mir­ror-like ma­chines that ap­pear to be in some strange sense “flow­ing.” Th­ese, he tells me, are com­put­ing some­thing he calls [in­com­pre­hen­si­ble galaxy Joe con­cept (IGJC) #4], in a for­mat known as [IGJC #5], un­der­girded and “hedged” via [IGJC #6]. He ac­knowl­edges that he can’t ex­plain the ap­peal of this to me in my cur­rent state.

“I guess you could say it’s kind of like hap­piness,” he says, war­ily. He men­tions an anal­ogy with ab­stract jazz.

“Is it con­scious?” I ask.

“Um, I think the clos­est short an­swer is ‘no,’” he says.

Of course, by hy­poth­e­sis, I would be­come him, and hence value what he val­ues, if I went through the pro­ce­dure that cre­ated him — one that ap­par­ently yields full in­for­ma­tion. But now the ques­tion of whether this is a pro­ce­dure I “trust,” or not, looms large. Has galaxy Joe gone off the rails, rel­a­tive to me? Or is he see­ing some­thing in­cred­ibly pre­cious and im­por­tant, rel­a­tive to me, that I can­not?

The stakes are high. Sup­pose I can cre­ate ei­ther this galaxy Joe’s fa­vorite world, or a world of happy pup­pies frolick­ing in the grass. The pup­pies, from my per­spec­tive, are a pretty safe bet: I my­self can see the ap­peal. Ex­pected value calcu­la­tions un­der moral un­cer­tainty aside, sup­pose I start to feel drawn to­wards the pup­pies. Galaxy Joe tells me with grave se­ri­ous­ness: “Creat­ing those pup­pies in­stead of IGJC #4 would be a mis­take of truly ridicu­lous sever­ity.” I hes­i­tate. Is he right, rel­a­tive to me? Or is he ba­si­cally, at this point, an alien, a pa­per­clip max­i­mizer, for all his hum­ble roots in my own psy­chol­ogy?

Is there an an­swer?

V. Mind-hack­ing vs. insight

Here’s a re­lated in­tu­ition pump. Just as pills and bonks on the head can change your eval­u­a­tive at­ti­tudes, some epistem­i­cally-fla­vored stim­uli can do so, too. Some such changes we think of as “le­gi­t­i­mate per­sua­sion” or “value for­ma­tion,” oth­ers we think of as be­ing “brain­washed,” “mind-hacked,” “re­pro­grammed,” “mis­led by rhetoric and emo­tional ap­peals,” and so on. How do we tell (or define) the differ­ence?

Where there are in­de­pen­dent stan­dards of truth, we can try ap­peal­ing to them. E.g., if Bob, a fiery or­a­tor, con­vinces you that two plus two is five, you’ve gone astray (though even cases like this can get tricky). But in the realm of pure val­ues, and es­pe­cially ab­sent other fla­grant rea­son­ing failures, it gets harder to say.

One crite­rion might be: if the per­sua­sion pro­cess would’ve worked in­de­pen­dent of its con­tent, this counts against its le­gi­t­i­macy (thanks to Carl Shul­man for dis­cus­sion). If, for ex­am­ple, Bob, or ex­po­sure to a cer­tain com­plex pat­tern of pix­els, can con­vince you of any­thing, this might seem a du­bi­ous source of in­fluence. That said, note that cer­tain com­mon pro­cesses of value for­ma­tion — for ex­am­ple, at­tach­ment to your home­town, or your fam­ily — are “con­tent ag­nos­tic” to some ex­tent (e.g., you would’ve at­tached to a differ­ent home­town, or a differ­ent fam­ily, given a differ­ent up­bring­ing); and ul­ti­mately, differ­ent evolu­tions could’ve built wildly vary­ing crea­tures. And note, too, that some stan­dard ra­tio­nales for such a crite­rion — e.g., be­ing con­vinced by Bob/​the pix­els doesn’t cor­re­late suffi­ciently re­li­ably with the truth — aren’t in play here, since there’s no in­de­pen­dent truth available.

Re­gard­less, though, this crite­rion isn’t broad enough. In par­tic­u­lar, some “mind-hack­ing” memes might work be­cause of their con­tent — you can’t just sub­sti­tute in ar­bi­trary al­ter­na­tive mes­sages. In­deed: one won­ders, and wor­ries, about what sort of El­dritch hor­rors might be lurk­ing in the memes­pace, ready and able, by virtue of their con­tent, to re­pro­gram and par­a­sitize those so fool­ish, and in­cau­tious, as to at­tempt some sort of naive ac­qui­si­tion of “full in­for­ma­tion.”

To take a mun­dane ex­am­ple: sup­pose that read­ing a cer­tain novel reg­u­larly con­vinces peo­ple to be­come ego­ists, and you learn, to your dis­may (you think of your­self as an al­tru­ist), that it would con­vince you to be­come so, too, if you read it. Does your “ideal­iza­tion pro­ce­dure” in­volve read­ing it? You’re not used to avoid­ing books, and this one con­tains, let’s sup­pose, no false­hoods or di­rect log­i­cal er­rors. Still, on one view, the book is, ba­si­cally, brain­wash­ing. On an­other, the book is a win­dow onto a new and le­gi­t­i­mately more com­pel­ling vi­sion of life. By hy­poth­e­sis, you’d take the lat­ter view af­ter read­ing. But what’s the true view?

Or sup­pose that peo­ple who spend time in bliss-in­duc­ing ex­pe­rience ma­chines reg­u­larly come to view time spent in such ma­chines as the high­est good, be­cause their brains re­ceive such strong re­ward sig­nals from the pro­cess, though not in a way differ­ent in kind from other pos­i­tive ex­pe­riences like travel, fine cui­sine, ro­man­tic love, and so on (thanks to Carl Shul­man for sug­gest­ing this ex­am­ple). You learn that you, too, would come to view ma­chine ex­pe­riences this way, given ex­po­sure to them, de­spite the fact that you cur­rently give pri­or­ity to non-he­do­nic goods. Does your ideal­iza­tion pro­cess in­volve en­ter­ing such ma­chines? Would do­ing so re­sult in a “dis­tor­tion,” an (en­dorsed, de­sired) “ad­dic­tion”; or would it show you some­thing you’re cur­rently miss­ing — namely, just how in­trin­si­cally good, rel­a­tive to you, these ex­pe­riences re­ally are?

Is there an an­swer?

As with the can­di­date galaxy Joe above, what’s needed here is some way of de­ter­min­ing which ideal­iza­tion pro­ce­dures are, as it were, the real deal, and which cre­ate im­posters, dupes, aliens; which brain-wash, al­ter, or mis­lead. I’ll con­sider three op­tions for spec­i­fy­ing the pro­ce­dure in ques­tion, namely:

  • Without refer­ence to your at­ti­tudes/​prac­tices.

  • By ap­peal to your ac­tual at­ti­tudes/​prac­tices.

  • By ap­peal to your ideal­ized at­ti­tudes/​prac­tices.

All of these, I think, have prob­lems.

VI. Priv­ileged procedures

Is there some priv­ileged pro­ce­dure for ideal­iz­ing some­one, that we can spec­ify and jus­tify with­out refer­ence to that per­son’s at­ti­tudes (ac­tual or ideal)? To me, the idea of giv­ing some­one “full in­for­ma­tion” (in­clud­ing log­i­cal in­for­ma­tion), or of putting them in a po­si­tion of “re­ally un­der­stand­ing” (as­sum­ing, per­haps wrongly, that we can define this in fully non-eval­u­a­tive terms) is the most com­pel­ling can­di­date. In­deed, when I ask my­self whether, for ex­am­ple, IGJC #4 is re­ally good (rel­a­tive to me), I find my­self tempted to ask: “how would I feel about it, if I re­ally un­der­stood it?”. And the ques­tion feels like it has an an­swer.

One jus­tifi­ca­tion for ap­peal­ing to some­thing like “full in­for­ma­tion” or “re­ally un­der­stand­ing” is: it en­ables your ideal­ized self to avoid in­stru­men­tal mis­takes. Con­sider Alfred, owner of Doggo above. Be­cause Alfred doesn’t know Doggo’s true na­ture (e.g., a sim­ple, non-con­scious robot), Alfred doesn’t know what he’s re­ally caus­ing, when he e.g. takes Doggo to the park. He thinks he’s caus­ing a con­scious puppy to be happy, but he’s not. Ideal­ized Alfred knows bet­ter. Var­i­ous other cases some­times men­tioned in sup­port of ideal­iz­ing — e.g., some­one who drinks a glass of petrol, think­ing it was gin — can also be given fairly straight­for­ward in­stru­men­tal read­ings.

But this jus­tifi­ca­tion seems too nar­row. In par­tic­u­lar: ideal­iz­ers gen­er­ally want the ideal­iza­tion pro­cess to do more than help you avoid straight­for­ward in­stru­men­tal mis­takes. In cases 1-8 above, for ex­am­ple, Alfred’s is ba­si­cally the only one that fits this in­stru­men­tal mold straight­for­wardly. The rest in­volve some­thing more com­plex — some dance of “rewind­ing” psy­cholog­i­cal pro­cesses (see more de­scrip­tion here), re­ject­ing ter­mi­nal (or pu­ta­tively ter­mi­nal) val­ues on the ba­sis of their psy­cholog­i­cal ori­gins, and re­solv­ing in­ter­nal con­flicts by priv­ileg­ing some eval­u­a­tive at­ti­tudes, stances, and in­tu­itions over oth­ers. That is, the ideal­iza­tion pro­ce­dure, stan­dardly imag­ined, is sup­posed to do more than take in some­one who already has and is pur­su­ing co­her­ent val­ues, and tell them how to get what they want; that part is (the­o­ret­i­cally) easy. Rather, it’s sup­posed to take in an ac­tual, messy, in­ter­nally con­flicted hu­man, and out­put co­her­ent val­ues — val­ues that are in some sense “the right an­swer” rel­a­tive to the hu­man in ques­tion.

In­deed, I some­times won­der whether the ap­peal of ideal­iz­ing sub­jec­tivism rests too much on peo­ple mis­tak­ing its ini­tial pre­sen­ta­tion for the more fa­mil­iar pro­ce­dure of elimi­nat­ing straight­for­ward in­stru­men­tal mis­takes. In my view, if we’re in a the­o­ret­i­cal po­si­tion to just get rid of in­stru­men­tal mis­takes, then we’re already cook­ing with gas, val­ues-wise. But the main game is messier — e.g., us­ing hy­po­thet­i­cal selves (which?) to de­ter­mine what counts as an in­stru­men­tal mis­take, rel­a­tive to you.

There’s an­other, sub­tly differ­ent jus­tifi­ca­tion for priv­ileg­ing “full in­for­ma­tion,” though: namely, that once you’ve got full in­for­ma­tion, then (as­sum­ing anti-re­al­ism about val­ues) you’ve got ev­ery­thing that the world can give you. That is: there’s noth­ing about re­al­ity that you’re, as it were, “miss­ing” — no sense in which you should hes­i­tate from de­ci­sion, on the grounds that you might learn some­thing new, or be wrong about some in­de­pen­dent truth. The rest, at that point, is up to you.

I’m sym­pa­thetic to this sort of thought. But I also have a num­ber of wor­ries about it.

One (fairly minor) is whether it jus­tifies bak­ing full in­for­ma­tion into the ideal­iza­tion pro­ce­dure, re­gard­less of the per­son’s at­ti­tudes to­wards ac­quiring such in­for­ma­tion. Con­sider some­one with very limited in­ter­est in the truth, and whose de­ci­sion-mak­ing pro­cess, given suit­able op­por­tu­nity, ro­bustly in­volves ac­tively and in­ten­tion­ally self-mod­ify­ing to close off in­quiry and lock in var­i­ous self-de­cep­tions/​false­hoods. Should we still “force” this per­son’s ideal­ized self to get the whole pic­ture be­fore re­solv­ing ques­tions like whether to self-de­ceive?

A sec­ond worry, ges­tured at above, is that the move from my mun­dane self to a be­ing with “full in­for­ma­tion” is ac­tu­ally some kind of wild and alien leap: a move not from Joe to “Joe who has got­ten out a bit more, space and time-wise” but from Joe to galaxy Joe, from Joe to a kind of God. And this prompts con­cern about the val­idity of the ex­er­cise.

Con­sider its ap­pli­ca­tion to a dog, or an ant. What would an ant value, if it had “full in­for­ma­tion”? What, for that mat­ter, would a rock value, if it had full in­for­ma­tion? If I were a river, would I flow fast, or slow? If I were an egg, would I be rot­ten? Start­ing with a dog, or an ant, or a rock, we can cre­ate a galaxy-brained God. Or, with the magic of un­moored coun­ter­fac­tu­als, we can “cut straight to” some galaxy-brained God or other, via ap­peal to some hazy sort of “similar­ity” to the dog/​ant/​rock in ques­tion, with­out spec­i­fy­ing a pro­cess for get­ting there — just as we can try to pick an egg that I would be, if I were an egg. With dogs, or ants, though, and cer­tainly with rocks, it seems strange to give the re­sult­ing galaxy-brain much au­thor­ity, with re­spect to what the rele­vant start­ing crea­ture/​rock “truly val­ues,” or should. In de­cid­ing whether to eu­th­a­nize your dog Fido, should you ask the near­est galaxy-brained former-Fido? If not, are hu­mans differ­ent? What makes them so?

This isn’t re­ally a pre­cise ob­jec­tion; it’s more of a hazy sense that if we just ask di­rectly “how would I feel about X, if I were a galaxy brain?”, we’re on shaky ground. (Re­mem­ber, we can’t spec­ify my val­ues in­de­pen­dently, hold them fixed, and then re­quire that the galaxy brain share them; the whole point is that the galaxy brain’s at­ti­tudes con­sti­tute my val­ues.)

A third worry is about in­de­ter­mi­nacy. Of the many can­di­date ways of cre­at­ing a fully in­formed galaxy Joe, start­ing with ac­tual me, it seems pos­si­ble that there will be im­por­tant path-de­pen­den­cies (this pos­si­bil­ity is ac­knowl­edged by many ideal­iz­ers). If you learn X in­for­ma­tion, or read Y novel, or have Z ex­pe­rience, be­fore some al­ter­na­tives (by hy­poth­e­sis, you do all of it even­tu­ally), you will ar­rive at a very differ­ent eval­u­a­tive end­point than if the or­der was re­versed. Cer­tainly, much real-life value for­ma­tion has this con­tin­gent char­ac­ter: you meet Suzy, who loves the sto­ics, is into crypto, and is about to start a med­i­cal res­i­dency, so you move to Delaware with her, read Seneca, start hang­ing out with liber­tar­i­ans, and so on. Per­haps such con­tin­gency per­sists in more ideal­ized cases, too. And if we try to skip over pro­cess and “cut straight to” a galaxy Joe, we might worry, still, that equally qual­ified can­di­dates will value very differ­ent things: “full in­for­ma­tion” just isn’t enough of a con­straint.

(More ex­ot­i­cally, we might also worry that amongst all the eval­u­a­tive El­dritch hor­rors lurk­ing in the memes­pace, there is one that always takes over all of the Joes on their way to be­com­ing fully-in­formed galaxy Joes, no mat­ter what they do to try to avoid it, but which is still in some sense “wrong.” Or that full in­for­ma­tion, more gen­er­ally, always in­volves memetic haz­ards that are fatal from an eval­u­a­tive per­spec­tive. It’s not clear that ideal­iz­ing sub­jec­tivism has the re­sources to ac­com­mo­date dis­tinc­tions be­tween such haz­ards and the eval­u­a­tive truth. That said, these hy­pothe­ses also seem some­what anti-Humean in fla­vor. E.g., can’t fully-in­formed minds value any old thing?)

Wor­ries about in­de­ter­mi­nacy be­come more press­ing once we rec­og­nize all the de­ci­sions a galaxy Joe is go­ing to have to make, and all of the in­ter­nal eval­u­a­tive con­flicts he will have to re­solve (be­tween ob­ject-level and meta prefer­ences, com­pet­ing de­sires, con­tra­dic­tory in­tu­itions, and the like), that ac­cess to “full in­for­ma­tion” doesn’t seem to re­solve for him. In­deed, the Humean should’ve been pes­simistic about the helpful­ness of “full in­for­ma­tion” in this re­gard from the start. If, by Humean hy­poth­e­sis, your cur­rent, im­perfect knowl­edge of the world can’t tell you what to want for its own sake, and/​or how to re­solve con­flicts be­tween differ­ent in­trin­sic val­ues, then perfect knowl­edge won’t help, ei­ther: you still face what is ba­si­cally the same old game, with the same old gap be­tween is and ought, fact and value.

Beyond ac­cess­ing “full in­for­ma­tion,” is there a priv­ileged pro­ce­dure for play­ing this game, speci­fi­able with­out refer­ence to the agent’s ac­tual or ideal­ized at­ti­tudes? Con­sider, for ex­am­ple, the idea of “re­flec­tive equil­ibrium” in ethics — the hy­poth­e­sized, sta­ble end-state of a pro­cess of bal­anc­ing more spe­cific in­tu­itions with more gen­eral prin­ci­ples and the­o­ret­i­cal con­sid­er­a­tions. How, ex­actly, is this bal­ance to be struck? What weight, for ex­am­ple, should be given to the­o­ret­i­cal sim­plic­ity and el­e­gance, vs. fidelity to in­tu­ition and com­mon sense? In con­texts with in­de­pen­dent stan­dards of ac­cu­racy, we might re­spond to ques­tions like this with refer­ence to the bal­ance most likely to yield the right an­swer; but for the ideal­izer, there is not yet a right an­swer to be sought; rather, the re­flec­tive equil­ibrium pro­cess makes its out­put right. But which re­flec­tive equil­ibrium pro­cess?

Per­haps we might an­swer: what­ever re­flec­tive equil­ibrium pro­cess ac­tu­ally works in the cases where there is a right an­swer (thanks to Nick Beck­stead for dis­cus­sion). That is, you should im­port the rea­son­ing stan­dards you can ac­tu­ally eval­u­ate for ac­cu­racy (for ex­am­ple, the ones that work in e.g. physics, math, statis­tics, and so on) into a do­main (value) with no in­de­pen­dent truth. Thus, for ex­am­ple, if sim­plic­ity is a virtue in sci­ence, be­cause (let’s as­sume) the truth is of­ten sim­ple, it should be a virtue in ethics, too. But why? Why not do what­ever’s ac­cu­rate in the case where ac­cu­racy is a thing, and then some­thing else en­tirely in the do­main where you can’t go wrong, ex­cept rel­a­tive to your own stan­dards?

(We can an­swer, here, by ap­peal to your ac­tual or ideal­ized at­ti­tudes: e.g., you just do, in fact, use such-and-such stan­dards in the eval­u­a­tive do­main, or would if suit­ably ideal­ized. I dis­cuss these op­tions in the next sec­tions. For now, the ques­tion is whether we can jus­tify par­tic­u­lar ideal­iza­tion pro­ce­dures ab­sent such ap­peals.)

Or con­sider the idea that ideal­iza­tion in­volves or is ap­prox­i­mated by “run­ning a large num­ber of copies of your­self, who then talk/​ar­gue a lot with each other and with oth­ers, have a bunch of mar­kets, and en­gage in lots of vot­ing and trad­ing and bet­ting” (see e.g. Luke Muelhauser’s de­scrip­tion here), or that it in­volves some kind of “moral par­li­a­ment.” What sorts of norms, in­sti­tu­tions, and pro­ce­dures struc­ture this pro­cess? How does it ac­tu­ally work? Ad­vo­cates of these pro­ce­dures rarely say in any de­tail (though see here for one re­cent dis­cus­sion); but pre­sum­ably, one as­sumes, “the best pro­ce­dures, mar­kets, vot­ing norms, etc.” But is there a priv­ileged “best,” speci­fi­able and jus­tifi­able with­out ap­peal to the agent’s ac­tual/​ideal­ized at­ti­tudes? Per­haps we hope that the op­ti­mal pro­ce­dures are just there, shin­ing in their op­ti­mal­ity, iden­ti­fi­able with­out any ob­ject-level eval­u­a­tive com­mit­ments (here Hume and oth­ers say: what?), or more likely, given any such com­mit­ments. My guess, though, is that ab­sent sub­stan­tive, value-laden as­sump­tions about veils of ig­no­rance and the like, and per­haps even given such as­sump­tions, this hope is over-op­ti­mistic.

The broader worry, here, is that once we move past “full in­for­ma­tion,” and start spec­i­fy­ing the ideal­iza­tion pro­ce­dure in more de­tail (e.g., some par­tic­u­lar start­ing state, some par­tic­u­lar type of re­flec­tive equil­ibrium, some par­tic­u­lar type of par­li­a­ment), or posit­ing spe­cific traits that the ideal­ized self needs to have (vivid imag­i­na­tion, em­pa­thy, dis­pas­sion, lack of “bias,” etc), our choice of ideal­iza­tion will in­volve (or sneak in) ob­ject-level value judg­ments that we won’t be able to jus­tify as priv­ileged with­out ad­di­tional ap­peal to the agent’s (ac­tual or ideal­ized) at­ti­tudes. Why vivid imag­i­na­tion, or em­pa­thy (to the ex­tent they add any­thing on top of “full in­for­ma­tion”)? Why a cool hour, in­stead of a hot one? What counts as an eval­u­a­tive bias, if there is no in­de­pen­dent eval­u­a­tive truth? The world, the facts, don’t an­swer these ques­tions.

If we can’t ap­peal to the world to iden­tify a priv­ileged ideal­iza­tion pro­ce­dure, it seems we must look to the agent in­stead. Let’s turn to that op­tion now.

VII. Ap­peals to ac­tual attitudes

Sup­pose we ap­peal to your ac­tual at­ti­tudes about ideal­iza­tion pro­ce­dures, in fix­ing the pro­ce­dure that de­ter­mines what’s of value rel­a­tive to you. Thus, if we ask: why this par­tic­u­lar re­flec­tive equil­ibrium? We an­swer: be­cause that’s the ver­sion you in fact use/​en­dorse. Why this type of par­li­a­ment, these vot­ing norms? They’re the ones you in fact fa­vor. Why em­pa­thy, or vivid imag­i­na­tion, or a cool hour? Be­cause you like them, pre­fer them, trust them. And so on.

In­deed, some ideal­iza­tion pro­ce­dures make very ex­plicit refer­ence to the “ideal­ized you” that you your­self want to be/​be­come. In cases like “vi­cious de­sires” above, for ex­am­ple, your want­ing not to have a par­tic­u­lar de­sire might make it the case that “ideal­ized you” doesn’t have it. Similarly, Yud­kowsky’s “co­her­ent ex­trap­o­lated vo­li­tion” ap­peals to the at­ti­tudes you would have if you were “more the per­son you wished you were.”

At a glance, this seems an at­trac­tive re­sponse, and one res­o­nant with a broader sub­jec­tivist vibe. How­ever, it also faces a num­ber of prob­lems.

First: just as ac­tual you might be in­ter­nally con­flicted about your ob­ject-level val­ues (con­flicts we hoped the ideal­iza­tion pro­ce­dure would re­solve), so too might ac­tual you be in­ter­nally con­flicted about the pro­ce­du­ral val­ues bear­ing on the choice of ideal­iza­tion pro­ce­dure. Per­haps, for ex­am­ple, there isn’t cur­rently a sin­gle form of re­flec­tive equil­ibrium that you en­dorse, treat as au­thor­i­ta­tive, etc; per­haps there isn’t a sin­gle ideal­ized self that you “wish you were,” a sin­gle set of de­sires you “wish you had.” Rather, you’re torn, at a meta-level, about the ideal­iza­tion pro­ce­dures you want to gov­ern you. If so, there is some temp­ta­tion, on pain of in­de­ter­mi­nacy, to look to an ideal­iza­tion pro­ce­dure to re­solve this meta-con­flict, too; but what type of ideal­iza­tion pro­ce­dure to use is pre­cisely what you’re con­flicted about (com­pare: tel­ling a group torn about the best vot­ing pro­ce­dure to “vote on it us­ing the best pro­ce­dure”).

In­deed, it can feel like pro­po­nents of this ver­sion of the view hope, or as­sume, that you are in some sense already en­gaged in, or com­mit­ted to, a de­ter­mi­nate de­ci­sion-mak­ing pro­cess of form­ing/​scru­ti­niz­ing/​al­ter­ing your val­ues, which there­fore need only be “run” or “ex­e­cuted.” Uncer­tainty about your val­ues, on this pic­ture, is just log­i­cal un­cer­tainty about what the “figure out my val­ues com­pu­ta­tion” you are already run­ning will out­put. The plan is in place. Ideal­iza­tion ex­e­cutes.

But is this right? Clearly, most peo­ple don’t have very ex­plicit plans in this vein. At best, then, such plans must be im­plicit in their tan­gle of cog­ni­tive al­gorithms. Of course, it’s true that if put in differ­ent fully-speci­fied situ­a­tions, given differ­ent re­flec­tive re­sources, and forced to make differ­ent choices given differ­ent con­straints, there is in fact a thing a given per­son would do. But con­stru­ing these choices as the im­ple­men­ta­tion of a de­ter­mi­nate plan/​de­ci­sion-pro­ce­dure (as op­posed to e.g., noise, mis­takes, etc), to be ex­trap­o­lated into some ideal­ized limit, is, at the least, a very sub­stan­tive in­ter­pre­ta­tive step, and ques­tions about in­de­ter­mi­nacy and path de­pen­dence loom large. Per­haps, for ex­am­ple, what sort of moral par­li­a­ment Bob de­cides to set up, in differ­ent situ­a­tions, de­pends on the weather, or on what he had for break­fast, or on which books he read in what or­der, and so on. And per­haps, if we ask him which such situ­a­tion he meta-en­dorses as most rep­re­sen­ta­tive of his plan for figur­ing out his val­ues, he’ll again give differ­ent an­swers, given differ­ent weather, break­fasts, books, etc — and so on.

(Per­haps we can just hope that this bot­toms out, or con­verges, or yields pat­terns/​forms of con­sen­sus ro­bust enough to in­ter­pret and act on; or per­haps, faced with such in­de­ter­mi­nacy, we can just say: “meh.” I dis­cuss re­sponses in this vein in sec­tion IX.)

Se­cond (though maybe minor/​sur­mountable): even if your ac­tual at­ti­tudes yield de­ter­mi­nate ver­dicts about the au­thor­i­ta­tive form of ideal­iza­tion, it seems like we’re now giv­ing your pro­ce­du­ral/​meta eval­u­a­tive at­ti­tudes an un­jus­tified amount of au­thor­ity rel­a­tive to your more ob­ject-level eval­u­a­tive at­ti­tudes. That is, we’re first us­ing your pro­ce­du­ral/​meta eval­u­a­tive at­ti­tudes to fix an ideal­iza­tion pro­ce­dure, then judg­ing the rest of your at­ti­tudes via refer­ence to that pro­ce­dure. But why do the pro­ce­du­ral/​meta at­ti­tudes get such a pri­or­ity?

This sort of is­sue is most salient in the con­text of cases like the “vi­cious de­sires” one above. E.g., if you have (a) an ob­ject-level de­sire that your co-worker suffer, and (b) a meta-de­sire not to have that ob­ject-level de­sire, why do we choose an “ideal you” in which the former is ex­tin­guished, and the lat­ter triumphant? Both, af­ter all, are just de­sires. What grants meta-ness such pride of place?

Similarly, sup­pose that your meta-prefer­ences about ideal­iza­tion give a lot of weight to con­sis­tency/​co­her­ence — but that con­sis­tency/​co­her­ence will re­quire re­ject­ing some of your many con­flict­ing ob­ject-level de­sires/​in­tu­itions. Why, then, should we treat con­sis­tency/​co­her­ence as a hard con­straint on “ideal you,” ca­pa­ble of “elimi­nat­ing” other val­ues whole hog, as op­posed to just one among many other val­ues swirling in the mix?

(Not all ideal­iz­ers treat con­sis­tency/​co­her­ence in this way; but my sense is that many do. And I do ac­tu­ally think there’s more to say about why con­sis­tency/​co­her­ence should get pride of place, though I won’t try to do so here.)

Third: fix­ing the ideal­iza­tion pro­ce­dure via refer­ence to your ac­tual (as op­posed to your ideal­ized) eval­u­a­tive at­ti­tudes risks clos­ing off the pos­si­bil­ity of mak­ing mis­takes about the ideal­iza­tion pro­ce­dure you want to gov­ern you. That is, this route can end up treat­ing your prefer­ences about ideal­iza­tion as “in­fal­lible”: they fix the pro­ce­dure that stands in judg­ment over the rest of your at­ti­tudes, but they them­selves can­not be judged. No one watches the watch­men.

One might have hoped, though, to be able to eval­u­ate/​crit­i­cize one’s cur­rently preferred ideal­iza­tion pro­ce­dures, too. And one might’ve thought the pos­si­bil­ity of such crit­i­cism truer to our ac­tual pat­terns of un­cer­tainty and self-scrutiny. Thus: if you cur­rently en­dorse re­flec­tive equil­ibrium pro­cess X, but you learn that it im­plies an ideal­ized you that gives up cur­rently cher­ished value Y, you may not sim­ply say: “well, that’s the re­flec­tive equil­ibrium pro­cess I en­dorse, so there you have it: be­gone, Y.” Rather, you can ques­tion re­flec­tive equil­ibrium pro­cess X on the very grounds that it re­sults in giv­ing up cher­ished value Y — that is, you can en­gage in kind of meta-re­flec­tive equil­ibrium, in which the au­thor­ity of a given pro­cess of re­flec­tive equil­ibrium is it­self sub­ject to scrutiny from the stand­point of the rest of what you care about.

In­deed, if I was set­ting off on some pro­cess of cre­at­ing my own “moral par­li­a­ment,” or of mod­ify­ing my­self in some way, then even granted ac­cess to “full in­for­ma­tion,” I can well imag­ine wor­ry­ing that the par­li­a­ment/​self I’m cre­at­ing is of the wrong form, and that the path I’m on is the wrong one. (This de­spite the fact that I can ac­cu­rately fore­cast its re­sults be­fore go­ing for­ward — just as I can ac­cu­rately fore­cast that, af­ter read­ing the ego­ist novel, or en­ter­ing the ex­pe­rience ma­chine, I’ll come out with a cer­tain view on the other end. Such fore­casts don’t set­tle the ques­tion).

We think of oth­ers as mak­ing ideal­iza­tion pro­ce­dure mis­takes, too. Note, for ex­am­ple, the ten­sion be­tween ap­peal­ing to your ac­tual at­ti­tudes to­wards ideal­iza­tion, and the (ba­si­cally uni­ver­sal?) re­quire­ment that the ideal­ized self pos­sess some­thing like full (or at least, much more) in­for­ma­tion. Cer­tain peo­ple, for ex­am­ple, might well en­dorse ideal­iza­tion pro­cesses that lock in cer­tain val­ues and be­liefs very early, and that as a re­sult never reach any kind of fully in­formed state: rather, they ar­rive at a sta­ble, per­ma­nently ig­no­rant/​de­ceived equil­ibrium well be­fore that. Similarly, cer­tain peo­ple’s preferred ideal­iza­tion pro­ce­dures might well lead them di­rectly into the maw of some memetic haz­ard or other (“sure, I’m happy to look at the whirling pix­els”).

Per­haps we hope to save such peo­ple, and our­selves, from such (grim? ideal?) fates. We find our­selves say­ing: “but you wouldn’t want to use that ideal­iza­tion pro­ce­dure, if you were more ideal­ized!”. Let’s turn to this kind of thought, now.

VIII. Ap­peals to ideal­ized attitudes

Faced with these prob­lems with fix­ing the ideal­iza­tion pro­ce­dure via refer­ence to our ac­tual eval­u­a­tive at­ti­tudes, sup­pose we choose in­stead to ap­peal to our ideal­ized eval­u­a­tive at­ti­tudes. Naive ver­sions of this, though, are clearly and prob­le­mat­i­cally cir­cu­lar. What ideal­iza­tion de­ter­mines what’s of value? Well, the ideal­iza­tion you would de­cide on, if you were ideal­ized. Ideal­ized how? Ideal­ized in the man­ner you would want your­self to be ideal­ized, if you were ideal­ized. Ideal­ized how? And so on. (Com­pare: “the best vot­ing pro­ce­dure is the one that would be voted in by the best vot­ing pro­ce­dure.”)

Of course, some ideal­iza­tion pro­ce­dures could be self-rat­ify­ing, such that if you were ideal­ized in man­ner X, you would choose/​de­sire/​en­dorse ideal­iza­tion pro­cess X. But it seems too easy to satisfy this con­straint: if af­ter ideal­iza­tion pro­cess X, I end up with val­ues Y, then I can eas­ily end up en­dors­ing ideal­iza­tion pro­cess X, since this pro­cess im­plies that pur­su­ing Y is the thing for me to do (and I’m all about pur­su­ing Y); and this could hold true for a very wide va­ri­ety of val­ues re­sult­ing from a very wide va­ri­ety of pro­ce­dures. So “value is de­ter­mined by the eval­u­a­tive at­ti­tudes that would re­sult from an ideal­iza­tion pro­ce­dure that you would choose if you un­der­went that very pro­ce­dure” seems likely to yield wildly in­de­ter­mi­nate re­sults; and more im­por­tantly, its con­nec­tion with what you ac­tu­ally care about now seems con­spicu­ously ten­u­ous. If I can brain­wash you into be­com­ing a pa­per­clip max­i­mizer, I can likely do so in a way that will cause you to treat this very pro­cess as one of “ideal­iza­tion” or “see­ing the light.” Self-rat­ifi­ca­tion is too cheap.

Is there a mid­dle ground, here, be­tween us­ing ac­tual and ideal­ized at­ti­tudes to fix the ideal­iza­tion pro­ce­dure? Some sort of happy mix? But which mix? Why?

In par­tic­u­lar, in try­ing to find a bal­ance be­tween end­less cir­cles of ideal­iza­tion, and “ideal­ized as you want to be, pe­riod,” I find that I run into a kind of “prob­lem of ar­bi­trary non-ideal­iza­tion,” pul­ling me back to­wards the cir­cle thing. Thus, for ex­am­ple, I find that at ev­ery step in the ideal­iza­tion pro­cess I’m con­struct­ing, it feels pos­si­ble to con­struct a fur­ther pro­cess to “check”/​”rat­ify” that step, to make sure it’s not a mis­take. But this fur­ther pro­cess will it­self in­volve steps, which them­selves could be mis­takes, and which them­selves must there­fore be val­i­dated by some fur­ther pro­cess — and so on, ad in­fini­tum. If I stop at some par­tic­u­lar point, and say “this par­tic­u­lar pro­cess just isn’t get­ting checked. This one is the bedrock,” I have some feel­ing of: “Why stop here? Couldn’t this one be mis­taken, too? What if I wouldn’t want to use this pro­cess as bedrock, if I thought more about it?”.

Some­thing similar holds for par­tic­u­lar limi­ta­tions on e.g. the time and other re­sources available. Sup­pose you tell me: “What’s valuable, rel­a­tive to you, is just what you’d want if ten copies of you thought about it for a thou­sand years, with­out ever tak­ing a step of rea­son­ing that an­other ten copies wouldn’t en­dorse if they thought about that step for a thou­sand years, and that’s it. Done.” I feel like: why not a hun­dred copies? Why not a billion years? Why not more lev­els of meta-check­ing? It feels like I’m play­ing some kind of “name the largest num­ber” game. It feels like I’m build­ing around me an un­end­ing army of ethe­real Joes, who can never move un­til all the su­per­vi­sors ar­rive to give their un­der­lings the go-ahead, but ev­ery­one can never ar­rive, be­cause there’s always room for more.

Note that the prob­lem here isn’t about pro­cesses you might run or com­pute, in the ac­tual world, given limited re­sources. Nor is it about find­ing a pro­cess that you’d at least be happy defer­ring to, over your cur­rent self; a pro­cess that is at least bet­ter than salient al­ter­na­tives. Nor, in­deed, is the prob­lem “how can I know with cer­tainty that my rea­son­ing pro­cess will lead me to the truth” (there is no in­de­pen­dent truth, here). Rather, the prob­lem is that I’m sup­posed to be spec­i­fy­ing a fully ideal­ized pro­cess, the out­put of which con­sti­tutes the eval­u­a­tive truth; but for ev­ery such pro­cess, it feels like I can make a bet­ter one; any given pro­cess seems like it could rest on mis­takes that a more ex­haus­tive pro­cess would elimi­nate. Where does it stop?

IX. Hop­ing for con­ver­gence, tol­er­at­ing indeterminacy

One op­tion, here, is to hope for some sort of con­ver­gence in the limit. Per­haps, we might think, there will come a point where no amount of ad­di­tional cog­ni­tive re­sources, lev­els of meta-rat­ifi­ca­tion, and so on will al­ter the con­clu­sion. And per­haps in­deed — that would be con­ve­nient.

Of course, there would re­main the ques­tion of what sort of pro­ce­dure or meta-pro­ce­dure to “take the limit” of. But per­haps we can pull a similar move there. Per­haps, that is, we can hope that a very wide va­ri­ety of can­di­date pro­ce­dures yield roughly similar con­clu­sions, in the limit.

In­deed, in gen­eral, for any of these wor­ries about in­de­ter­mi­nacy, there is an available re­sponse to the effect that: “maybe it con­verges, though?” Maybe as soon as you say “what Joe would feel if he re­ally un­der­stood,” you hone in on a pop­u­la­tion of Galaxy Joes that all pos­sess ba­si­cally the same ter­mi­nal val­ues, or on a sin­gle Galaxy Joe who pro­vides a priv­ileged an­swer. Maybe Bob’s prefer­ences about ideal­iza­tion pro­ce­dures are highly sta­ble across a wide va­ri­ety of ini­tial con­di­tions (weather, break­fasts, books, etc). Maybe it doesn’t re­ally mat­ter how, and in what or­der, you learn, read, ex­pe­rience, re­flect: mod­ulo ob­vi­ous mis­steps, you end up in a similar place. Maybe in­deed.

Or, if not, maybe it doesn’t mat­ter. In gen­eral, lots of things in life, and es­pe­cially in philos­o­phy, are vague to at least some ex­tent; ar­gu­ments to the effect that “but how ex­actly do you define X? what about Y edge case?” are cheap, and of­ten un­pro­duc­tive; and there re­ally are bald peo­ple, de­spite the in­de­ter­mi­nacy of ex­actly who qual­ifies.

What’s more, even if there is no sin­gle, priv­ileged ideal­ized self, picked out by a priv­ileged ideal­iza­tion pro­ce­dure, and even if the many pos­si­ble can­di­dates for pro­ce­dures and out­puts do not con­verge, it seems plau­si­ble that there will still be pat­terns and limited forms of con­sen­sus. For ex­am­ple, it seems un­likely that many of my pos­si­ble ideal­ized selves end up try­ing to max­i­mize he­lium, or to eat as many bricks as they can; even if a few go one way, the pre­pon­der­ance may go some other way; and per­haps it’s right to view ba­si­cally all of them, de­spite their differ­ences, as wor­thy of defer­ence from the stand­point of my ac­tual self, in my ig­no­rance (e.g., per­haps the world any of them would cre­ate is rightly thought bet­ter, from my per­spec­tive, than the world I would cre­ate, if I wasn’t al­lowed fur­ther re­flec­tion).

In this sense, the di­verg­ing at­ti­tudes of such selves may still be able to play some of the role the ideal­izer hopes for. That is, pour­ing my re­sources into eat­ing bricks, tor­tur­ing cats, etc re­ally would be a mis­take, for me — none of my re­motely plau­si­ble ideal­ized selves are into it — de­spite the fact that these selves differ in the weight they give to [in­com­pre­hen­si­ble galaxy-brained con­cept] vs. [an­other in­com­pre­hen­si­ble galaxy-brained con­cept]. And while pro­cesses that in­volve av­er­ag­ing be­tween ideal­ized selves, pick­ing ran­domly amongst them, hav­ing them vote/​ne­go­ti­ate, putting them be­hind veils of ig­no­rance, etc raise ques­tions about cir­cu­lar­ity/​con­tin­u­ing in­de­ter­mi­nacy, that doesn’t mean that all such pro­cesses are on equal foot­ing (e.g., differ­ent par­ties can be un­sure what vot­ing pro­ce­dure to use, while still be­ing con­fi­dent/​unan­i­mous in re­ject­ing the one that causes ev­ery­one to lose hor­ribly).

Per­haps, then, the ideal­izer’s re­sponse to in­de­ter­mi­nacy — even very large amounts of it — should sim­ply be tol­er­ance. In­deed, there is an art, in philos­o­phy, to not nit­pick­ing too hard — to al­low­ing hand-waves, and some­thing some­things, where ap­pro­pri­ate, in the name of ac­tu­ally mak­ing progress to­wards some kind of work­able any­thing. Per­haps some of the wor­ries above have fallen on the wrong side of the line. Per­haps a vague ges­ture, a promis­sory note, in the di­rec­tion of some­thing vaguely more ideal than our­selves is, at least in prac­ti­cal con­texts (though this isn’t one), good enough; bet­ter than noth­ing; and bet­ter, too, than set­ting eval­u­a­tive stan­dards rel­a­tive to our pre­sent, de­cid­edly un-ideal selves, in our ig­no­rance and folly.

X. Pas­sive and ac­tive ethics

I want to close by ges­tur­ing at a cer­tain kind of dis­tinc­tion — be­tween “pas­sive” and “ac­tive” ethics (here I’m draw­ing ter­minol­ogy and in­spira­tion from a pa­per of Ruth Chang’s, though the sub­stance may differ) — which I’ve found helpful in think­ing about what to take away from the wor­ries just dis­cussed.

Some ideal­iz­ing sub­jec­tivists seem to hope that their view can serve as a kind of low-cost, nat­u­ral­ism-friendly sub­sti­tute for a ro­bustly re­al­ist meta-ethic. That is, mod­ulo cer­tain ex­ten­sional differ­ences about e.g. ideally-co­her­ent suffer­ing max­i­miz­ers, they ba­si­cally want to talk about value in much the way re­al­ists do, and to differ, only, when pressed to ex­plain what makes such talk true or false.

In par­tic­u­lar, like re­al­ists, ideal­iz­ers can come to see ev­ery (or al­most ev­ery) choice and eval­u­a­tive at­ti­tude as at­tempt­ing to ap­prox­i­mate and con­form to some ex­ter­nal stan­dard, rel­a­tive to which the choice or at­ti­tude is to be judged. Granted, the stan­dard in ques­tion is defined by the out­put of the ideal­iza­tion pro­ce­dure, in­stead of the ro­bustly real val­ues; but in ei­ther case, it’s some­thing one wants to rec­og­nize, re­ceive, per­ceive, re­spond to. For us non-ideal agents, the “true val­ues” are still, effec­tively, “out there.” We are, in Chang’s ter­minol­ogy, “pas­sive” with re­spect to them.

But in­struc­tively, I think, naive ver­sions of this can end up cir­cu­lar. Con­sider the toy view that “what’s good is what­ever you’d be­lieve to be good if you had full in­for­ma­tion.” Now sup­pose that you get this full in­for­ma­tion, and con­sider the ques­tion: is plea­sure good? Well, this just amounts to the ques­tion: would I think it good if I had full in­for­ma­tion? Well, here I am with full in­for­ma­tion. Ok, do I think it good? Well, it’s good if I would think it good given full in­for­ma­tion. Ok, so is it good? And so on.

Part of the les­son here is that ab­sent fancier foot­work about what eval­u­a­tive be­lief amounts to, be­lief isn’t a good can­di­date for the eval­u­a­tive at­ti­tude ideal­iza­tion should rest on. But con­sider a differ­ent ver­sion: “what you should do is what­ever you would do, given full in­for­ma­tion.” Sup­pose that here I am with full in­for­ma­tion. I ask my­self: what should I do? Well, what­ever I would do, given full in­for­ma­tion. Ok, well, I’ve got that now. What would I do, in pre­cisely this situ­a­tion? Well, I’m in this situ­a­tion. Ok, what would I do, if things were like this? Well, I’d try to do what I should do. And what should I do? Etc.

The point here isn’t that there’s “no way out,” in these cases: if I can get my­self to be­lieve, or to choose, then I will, by hy­poth­e­sis, have be­lieved truly, cho­sen rightly. Nor, in­deed, need all forms of ideal­iz­ing sub­jec­tivism suffer from this type of prob­lem (we can ap­peal, for ex­am­ple, to at­ti­tudes that plau­si­bly arise more pas­sively and non-agen­ti­cally, like de­sire).

Rather, what I’m try­ing to point at is a way that im­port­ing and tak­ing for granted a cer­tain kind of re­al­ist-fla­vored eth­i­cal psy­chol­ogy can re­sult in an in­struc­tive sort of mis­fire. Some­thing is miss­ing, in these cases, that I ex­pect the ideal­iz­ing sub­jec­tivist needs. In par­tic­u­lar: these agents, to the end, lack an af­for­dance for a cer­tain kind of di­rect, ac­tive agency — a cer­tain kind of re­spon­si­bil­ity, and self-cre­ation. They don’t know how to choose, fully, for them­selves. Rather, even in ideal con­di­tions, they are for­ever try­ing to ap­prox­i­mate some­thing else. True, on ideal­iz­ing sub­jec­tivism, the thing they are try­ing to ap­prox­i­mate is ul­ti­mately, them­selves, in those con­di­tions. But this is no re­lief: still, they are ap­prox­i­mat­ing an ap­prox­i­ma­tor, of an ap­prox­i­ma­tor, and so on, in an end­less loop. They are always look­ing el­se­where, for­ever down the hall of mir­rors, around and around a maze with no cen­ter (what’s in the cen­ter?). Their ul­ti­mate task, they think, is to obey them­selves. But they can only obey: they can­not gov­ern, and so have no law.

It’s a re­lated sort of mis­fire, I think, that gives rise to the “would an end­less army of ethe­real Joes rat­ify ev­ery step of my rea­son­ing, and the rea­son­ing of the rat­ifiers, and so on?” type of prob­lem I dis­cussed above. That is, one wants ev­ery step to con­form to some ex­ter­nal stan­dard — and the only stan­dards available are built out of armies of ethe­real Joes. But those Joes, too, must con­form. It’s con­for­mity all the way down — ex­cept that for the anti-re­al­ist, there’s no bot­tom.

What’s needed, here, is a type of choice that is cre­at­ing, rather than try­ing to con­form — and which hence, in a sense, is “in­fal­lible.” And here per­haps one thinks, with the re­al­ists: surely the types of choices we’re in­ter­ested in here — choices about which books, feel­ings, ma­chines, galaxy brains, Gods, to “trust”; which pup­pies, or nanoma­chines, to cre­ate — are fal­lible. Or if not, surely they are, in a sense, ar­bi­trary — mere “pick­ings,” or “plump­ings.” If you aren’t try­ing to con­form to some stan­dard, than how can you truly, and non-ar­bi­trar­ily, choose? I don’t have a worked-out story, here (though I ex­pect that we can at least dis­t­in­guish such cre­ative choices from e.g. Buri­dan’s-ass style pick­ings — for ex­am­ple, they don’t leave you in­differ­ent). But it’s a ques­tion that I think sub­jec­tivists must face; and which I feel some mod­er­ate op­ti­mism about an­swer­ing (though per­haps not in a way that gives re­al­ists what they want).

Of course, sub­jec­tivists knew, all along, that cer­tain things about them­selves were go­ing to end up be­ing treated as effec­tively in­fal­lible, from an eval­u­a­tive per­spec­tive. What­ever goes in Clippy’s util­ity func­tion slot, for sub­jec­tivists, gov­erns what’s valuable rel­a­tive to Clippy; and it does so, on sub­jec­tivism, just in virtue of be­ing there — in virtue of be­ing the stuff that the agent is made out of (this is part of the ar­bi­trari­ness and con­tin­gency that so both­ers re­al­ists). The prob­lem that the ideal­izer faces is that ac­tual hu­man agents are not yet fully made: rather, they’re still a tan­gled mess. But the ideal­izer’s hope is that they’re suffi­ciently “on their way to get­ting made” that we can, effec­tively, as­sume they’re already there; the seed has already de­ter­mined a tree, or a suffi­ciently similar set of trees; we just haven’t com­puted the re­sult.

But is that how trees grow? Have you already de­ter­mined a self? Have you already made what would make you, if all went well? Do you know, already, how to figure out who you are? Per­haps for some the an­swer is yes, or close enough. Per­haps for all. In that case, you are already try­ing to do some­thing, already fight­ing for some­thing — and it is rel­a­tive to that some­thing that you can fail.

But if the choice has not yet been made, then it is we who will have to make it. If the sea is open, then so too is it ours to sail.

In­deed, even if in some sense, the choice has been made — even if there is already, out there, a priv­ileged ideal­ized ver­sion of your­self; even if all of the ideal­iza­tion pro­ce­dures con­verge to a sin­gle point — the sea, I think, is still open, if you step back and make it so. You can still re­ject that self, and the au­thor­ity of the pro­ce­dure(s) that cre­ated it, con­ver­gence or no. Here I think of a friend of mine, who ex­pressed some dis­tress at the thought that his ideal­ized self could in prin­ci­ple turn out to be a Volde­mort-like char­ac­ter. His dis­tress, to me, seemed to as­sume that his ideal­ized self was “im­posed on him”; that he “had,” as it were, to ac­knowl­edge the au­thor­ity of his Volde­mort self’s val­ues. But such a choice is en­tirely his. He can, if he wishes, re­ject the Volde­mort, and the parts of him­self (how­ever strong) that cre­ated it; he can forge his own path, to­wards a new ideal. The fact that he would be­come a Volde­mort, un­der cer­tain con­di­tions he might’ve thought “ideal,” is ul­ti­mately just an­other fact, to which he him­self must choose how to re­spond.

Per­haps some choices in this vein will be eas­ier, and more con­tin­u­ous/​res­o­nant with his coun­ter­fac­tual be­hav­ior and his ex­ist­ing de­ci­sion-mak­ing pro­cesses; some paths will be harder, and more frag­ile; some, in­deed, are im­pos­si­ble. But these facts are still, I think, just facts; the choice of how to re­spond to them is open. The point of sub­jec­tivism is that the stan­dards (rel­a­tive to you) used to eval­u­ate your be­hav­ior must ul­ti­mately be yours; but who you are is not some­thing fixed, to be dis­cov­ered and ac­knowl­edged by in­ves­ti­gat­ing what you would do/​feel in differ­ent sce­nar­ios; rather, it is some­thing to be cre­ated, and choice is the tool of cre­ation. Your coun­ter­fac­tual self does not bind you.

In a sense, what I’m say­ing here is that ideal­iz­ing sub­jec­tivism is, and needs to be, less like “re­al­ism-lite,” and more like ex­is­ten­tial­ism, than is some­times ac­knowl­edged. If sub­jec­tivists wish to forge, from the tan­gled facts of ac­tual (and hy­po­thet­i­cal) self­hood, an ideal, then they will need, I ex­pect, to make many choices that cre­ate, rather than con­form. And such choices will be re­quired, I ex­pect, not just as a “last step,” once all the “in­for­ma­tion” is in place, but rather, even in the­ory, all along the way. Such choice, in­deed, is the very sub­stance of the thing.

(To be clear: I don’t feel like I’ve worked this all out. Mostly, I’ve been try­ing to ges­ture at, and in­habit, some sort of sub­jec­tivist ex­is­ten­tial­ist some­thing, which I cur­rently find more com­pel­ling than a more re­al­ist-fla­vored way of try­ing to be an ideal­izer. What ap­proach to meta-ethics ac­tu­ally makes most sense over­all and in prac­tice is a fur­ther ques­tion.)

XI. Ghost civilizations

With this re­fram­ing in mind, some of the pos­si­ble cir­cles and in­de­ter­mi­na­cies dis­cussed above seem to me less wor­ry­ing — rather, they are just more facts, to be re­sponded to as I choose. Among all the ideal­ized selves (and non-selves), and all com­bi­na­tions, there is no fi­nal, in­fal­lible eval­u­a­tive au­thor­ity — no res­cuer, Lord, father; no safety. But there are can­di­date ad­vi­sors ga­lore.

Here’s an illus­tra­tion of what I mean, in the con­text of an ideal­iza­tion I some­times think about.

I’ve writ­ten, in the past, about a “ghost” ver­sion of my­self — that is, one that can float free from my body; which travel any­where in all space and time, with un­limited time, en­ergy, and pa­tience; and which can also make changes to differ­ent vari­ables, and play for­ward/​rewind differ­ent coun­ter­fac­tual timelines (the ghost’s ac­tivity some­how doesn’t have any moral sig­nifi­cance).

I some­times treat such a ghost kind of like an ideal­ized self. It can see much that I can­not. It can see di­rectly what a small part of the world I truly am; what my ac­tions truly mean. The lives of oth­ers are real and vivid for it, even when hazy and out of mind for me. I trust such a per­spec­tive a lot. If the ghost would say “don’t,” I’d be in­clined to listen.

As I usu­ally imag­ine it, though, the ghost isn’t ar­bi­trar­ily “ideal.” It hasn’t proved all the the­o­rems, or con­sid­ered all the ar­gu­ments. It’s not all that much smarter than me; it can’t com­pre­hend any­thing that I, with my brain, can’t com­pre­hend. It can’t di­rectly self-mod­ify. And it’s alone. It doesn’t talk with oth­ers, or make copies of it­self. In a sense, this rel­a­tive mun­dan­ity makes me trust it more. It’s eas­ier to imag­ine than a galaxy brain. I feel like I “know what I’m deal­ing with.” It’s more “me.”

We can imag­ine, though, a ver­sion of the thought ex­per­i­ment where we give the ghost more lee­way. Let’s let it make copies. Let’s give it a sep­a­rate realm, be­yond the world, where it has ac­cess to ar­bi­trary tech­nol­ogy. Let’s let it in­ter­act with what­ever ac­tual and pos­si­ble hu­mans, past and fu­ture, that it wants, at ar­bi­trary depths, and even to bring them into the ghost realm. Let’s let it make new peo­ple and crea­tures from scratch. Let’s let it try out self-mod­ifi­ca­tions, and weird ex­plo­ra­tions of mind-space — sur­rounded, let’s hope, by some sort of re­spon­si­ble ghost sys­tem for hand­ing ex­plo­ra­tions, new crea­tures, and so on (here I imag­ine a crowd of copy ghosts, su­per­vis­ing/​sup­port­ing/​scru­ti­niz­ing an ex­plorer try­ing some sort of pro­cess or stim­u­lus that could lead to go­ing off the rails). Let’s let it build, if it wants, a galaxy brain, or a par­li­a­ment, or a civ­i­liza­tion. And let’s ask it, af­ter as much of all this as it wants, to re­port back about what it val­ues.

If I try to make, of this ghost civ­i­liza­tion, some of sort of de­ter­mi­nate, priv­ileged ideal, which will define what’s of value, rel­a­tive to me, I find that I start to run into the prob­lems dis­cussed above. That is, I start won­der­ing about whether the ghost civ­i­liza­tion goes some­where I ac­tu­ally want; how much differ­ent ver­sions of it di­verge, based on even very similar start­ing points; how to fix the de­tails in a man­ner that has any hope of yield­ing a de­ter­mi­nate out­put, and how ar­bi­trary do­ing so feels. I won­der whether the ghosts will find suit­able meth­ods of co­op­er­at­ing, con­tain­ing memetic haz­ards, and so on; whether I would re­gret defin­ing my val­ues rel­a­tive to this hazy thought ex­per­i­ment, if I thought about it more; whether I should in­stead be fo­cus­ing on a differ­ent, even more ideal­ized thought ex­per­i­ment; where the pos­si­ble ideal­iz­ing ends.

But if I let go of the thought that there is, or need be, a sin­gle “true stan­dard,” here — a stan­dard that is, already, for me, the be-all-end-all of value — then I feel like I can re­late to the ghosts differ­ently, and more pro­duc­tively. I can root for them, as they work to­gether to ex­plore the dis­tant reaches of what can be known and thought. I can ad­mire them, where they are no­ble, cau­tious, com­pas­sion­ate, and brave; where they build good in­sti­tu­tions and pro­ce­dures; where they co­op­er­ate. I can try, my­self, to see through their eyes, look­ing out on the vast­ness of space, time, and the be­ings who in­habit it; zoom­ing in, rewind­ing, ex­am­in­ing, try­ing to un­der­stand. In a sense, I can use the image of them to con­nect with, and strengthen, what I my­self value, now (in­deed, I think that much ac­tual us­age of “ideal ad­vi­sor” thought ex­per­i­ments, at least in my own life, is of this fla­vor).

And if I imag­ine the ghosts be­com­ing more and more dis­tant, alien, and in­com­pre­hen­si­ble, I can feel my con­fi­dence in their val­ues be­gin to fray. Early on, I’m strongly in­clined to defer to them. Later, I am still root­ing for them; but I start to see them as in­creas­ingly at the edges of things, step­ping for­ward into the mist; they’re weav­ing on a tapestry that I can’t see, now; they’re sailing, too, on the open sea, fur­ther than I can ever go. Are they still good, rel­a­tive to me? Have they gone “off the rails”? The ques­tion it­self starts to fade, too, and with it the rails, the pos­si­bil­ity of mis­take. Per­haps, if nec­es­sary, I could an­swer it; I could de­cide whether to priv­ilege the val­ues of some par­tic­u­lar ghost civ­i­liza­tion, how­ever un­rec­og­niz­able, over my own cur­rent feel­ings and un­der­stand­ing; but an­swer­ing is in­creas­ingly an act of cre­ation, rather than an at­tempt at dis­cov­ery.

Cer­tainly, I want to know where the ghost civ­i­liza­tion goes. In­deed, I want to know where all the non-Joe civ­i­liza­tions, ghostly or not, go too. I want to know where all of it leads. And I can choose to defer to any of these paths, Joe or non-Joe, to differ­ent de­grees. I’m sur­rounded, if I wish to call on them, by in­nu­mer­able can­di­date ad­vi­sors, fa­mil­iar and alien. But the choice of who, if any of them, to listen to, is mine. Per­haps I would choose, or not, to defer, given var­i­ous con­di­tions. Per­haps I would re­gret, or not; would kick my­self, or not; would re­joice, or not. I’m in­ter­ested to know that, too. But these “woulds” are just more can­di­date ad­vi­sors. It’s still on me, now, in my ac­tual con­di­tion, to choose.

(Thanks to Katja Grace, Ke­tan Ra­makr­ish­nan, Nick Beck­stead, Carl Shul­man, and Paul Chris­ti­ano for dis­cus­sion.)