why would it cease to matter if someone moved it a lightyear to the left?
because then it wouldn’t exist, of course!
here, we are writing down a metaphysics, and appealing to that in order to justify moral intuitions. this works, except the pesky ‘meta-’ prefix keeps everything firmly on the hypothetical side of is/ought.
yes, this seems like the more correct posture (not sure time spent despairing is valuable, though). there is not “others will judge me unworthy”, there is only “this is within / not within my capabilities to effect”.