drift is harmful from the perspective of my current values
True. And that drift would be beneficial from the perspective of your new, drifted-to values.
But neither of those statements have any bearing on whether value drift (in general or any specific instance thereof) is good or bad.
Good question :-)
We could measure them against some non-relativist ethical standard, like “US Dollars lost”, “lives lost”, “person-weeks stuck in traffic” or somesuch.