sudhanshu_kasewa

• sudhanshu_kasewa 31 Mar 2018 6:12 UTC

  3 points

  on: Hufflepuff Cyn­i­cism on Hypocrisy

  I feel I flinch away from hypocrisy because allowing it seems to nudge us towards world states that I find undesirable. Consider a malicious version of hypocrisy through the lens of the diner’s dilemma: Transitioning meat-eaters reluctantly order tofu salads, while the vocal vegan gets themselves a steak. I imagine that in a subsequent outing, at least some of the carnivores break their resolve, seeing their duplicitous comrade tuck into a bucket of chicken wings. Eventually, no one cares to take the signalling action; preferably, though, perhaps they eject the offending member from the party.

  I think, though, that the free rider problem probably better reflects my beef: Hypocrisy is one of those things where one can get something for nothing, but the getting sort of depends on most other parties believing that everyone involved is getting something for something. Then someone notices that one can in fact get something for nothing, and proceeds to jump on that gravy train; soon, we reach a critical mass of someones getting somethings for nothing, while the other ones really have to work overtime to keep the lights on. This is unsustainable*, and leads to a complete breakdown of the community that was built.

  To give this concreteness, I sometimes think about some of the arguments against a rapid expansion of the EA movement, especially with regards to signalling.

  *(except maybe in a world with Superman, where he could just power everything, if it eventually came down to it)

• sudhanshu_kasewa 9 Jun 2018 15:11 UTC

  0 points

  on: Could we send a mes­sage to the dis­tant fu­ture?

  Apropos on the space-object proposal from Facebook, and the question about recovering the information: Could the object itself have a shape that encodes the information? I was thinking something like a disco ball, except perhaps cylindrical/​oblong such that we had a higher expectation of spinning in certain directions; then our message could be encoded in binary as shiny/​matte facets on this thing; as its twinkling pattern is observed over time, the entire message is reconstructed.

  If we can do Durable Writing, we could probably also do Durable Ellipsoid.

  For context, this is inspired by how we read information from optical disk technology, but irony is not lost on me that CDs are obsolete. This proposal has many obvious drawbacks, such as can we ensure that 500m years in the future, it spins fast enough to be observed as repeating, but not too fast to be confused as uniform?

  A way-cooler-but-also-much-harder alternative would be to launch some really robust object into a slow-decaying orbit such that it crashes back to Earth after 500m years. Its internals might be stuff made up with two different contrasting materials that encode the payload, while the outside is just enough of a heat-shield to survive the journey back; additionally, it should probably also scream “Not Natural!” in some way, so that someone from 1900 looking at it can predict that it will fall in the next 100 years or so, and prepare to collect it “for science”. Hopefully, it won’t destroy any cities in the process. Reflecting on that, if future inhabitants have anything like human-like psychology, an alien artifact about to plummet to Earth is just asking to be shot out of the sky. But then again, why should they be human-like at all?

• sudhanshu_kasewa 14 Feb 2019 12:06 UTC

  2 points

  on: Con­struct­ing Goodhart

  The problem is to come up with some model system where optimizing for something which is almost-but-not-quite the thing you really want produces worse results than not optimizing at all.

  I’m confused; maybe the following query is addressed elsewhere but I have yet to come across it:

  Doesn’t the (standard, statistical machine learning 101) formulation of minimising-training-error-when-we-actually-care-about-minimising-test-error fall squarely in the camp of something that demonstrates Goodhart’s law? Aggressively optimising to reduce training error with a function-approximator with parameters >> number of data points (e.g. today’s deep neural networks) will result in ~0.0 training error, but it would most likely totally bomb on unseen data. This, to me, appears to be as straightforward an example of a Goodhart’s Law as is necessary to illustrate the concept, and serves as a segue into how to mitigate this phenomenon of overfitting, e.g. by validation, regularisation, enforcing sparsity, and so on.

  Given the premise that we are likely to start from something close to pareto-optimal to begin with, we now have a system which works well from the get-go, and without suitable controls optimising on reducing training error to the exclusion of all other metrics will almost certainly be worse than not optimising at all.

• sudhanshu_kasewa 14 Feb 2019 22:34 UTC

  1 point

  in reply to: johnswentworth’s comment on: Con­struct­ing Goodhart

  I assumed (close to) pareto-optimality, since the OP suggests that most real systems are starting from this state.

  The (immediately disceranable) competing objectives here are training error and test error. Only one can be observed ahead of deployment (much like X in the X+Y example earlier), while it’s actually the other which matters. That is not to say that there aren’t other currently undiscovered /​ unstated metrics of interest (training time, designer time, model size, etc.) which may be articulated and optimised for, still leading to a suboptimal result on test error. Indeed, we can imagine a situation with a perfectly good predictive neural network, which for some reason won’t run on the new hardware that’s provisioned, and so a hasty, over-extended engineer might simply delete entire blocks of it, optimising their time and the ability of the model to fit on a Raspberry PI, while most likely completely voiding any ability of the network to perform the task meaningfully.

  If this sounds contrived, forgive me. Perhaps I’m am talking tangentially past the discussion at hand; if so, kindly ignore. Mostly, I only wish to propose that a fundamental formulation of ML, of minimising training loss while we want to reduce test loss, is an example of Goodhart’s law in action, and there is rich literature on techniques to circumvent its effects. Do you agree? Why /​ why not?

• sudhanshu_kasewa 24 Oct 2019 23:46 UTC

  30 points

  on: De­bate on In­stru­men­tal Con­ver­gence be­tween LeCun, Rus­sell, Ben­gio, Zador, and More

  ICYMI: Yann posted this page on FB as well, and some additional conversation happened there, with at least one interesting exchange between Rob Bensinger and Tony Zador:

  https://​​www.facebook.com/​​yann.lecun/​​posts/​​10156278492457143

  (I did scan the comments and didn’t see this posted earlier, so...)

• sudhanshu_kasewa 25 Apr 2020 3:37 UTC

  3 points

  on: Databases of hu­man be­havi­our and prefer­ences?

  Perhaps these could be useful:

  1) Human Decision-Making dataset https://​​osf.io/​​eagcd/​​ ; but from what I can tell, has less than 300 human participants

  2) User rating dataset, e.g. Yahoo! Music or Netflix or Amazon product review datasets. These could be trimmed in various ways to reduce complicatedness. Netflix dataset is here : https://​​www.kaggle.com/​​netflix-inc/​​netflix-prize-data

  Amazon product reivew is at http://​​liu.cs.uic.edu/​​download/​​data/​​ , but it says available upon request

  3) Transactional data, e.g. https://​​data.world/​​uci/​​online-retail might shed some light on preferences (as transactional data could be a proxy for demand)

[Question] What would flour­ish­ing look like in Con­way’s Game of Life?

• sudhanshu_kasewa 12 May 2020 13:33 UTC

  1 point

  in reply to: TurnTrout’s comment on: What would flour­ish­ing look like in Con­way’s Game of Life?

  I was not; thanks for the pointer!

  A quick look suggests that it’s not quite what I had in mind; nonetheless a reference worth looking at.

• sudhanshu_kasewa 15 Jul 2020 21:28 UTC

  1 point

  on: Why is the im­pact penalty time-in­con­sis­tent?

  1.

  Is there such a thing as a free action, or an action where e.g. the agent breaks its own legs, when it is not accounted for in the action space of the underlying MDP? That feels like adding a new layer of misspecification (which no doubt is a possibility, and probably deserves deep investigation) orthogonal to reward function misspecification.

  
  

  2.

  It seems as though this kind of circumvention of impact penalties depends on what calculates the penalty. If the environment (or something else external to the agent) is responsible for calculating the penalty and providing it to the agent, then it could do so as if the agent was not constrained (or committing to be constrained) at all. If the agent is internally responsible for computing this impact, it could probably find ways to hack this, similar to wireheading.

  So, assuming its some external entity E that computes the impact penalty, it must have the (dis?)ability to account for the agent’s modified action space when making this computation for the agent to successfully reduce the penalty as in the earlier example.

  Something agent A does must signal to E that A’s action space has changed. If so, we might find ourselves in a situation where A is able to communicate an arbitrarily complex message to E, particularly, something of the form of: “I will never do anything that gives me more than the minimum penalty”, or in terms of actions: “I will deterministically follow the policy that gives me the minimum penalty while achieving my goals.” E, if it believed A, would not include high penalty actions in the action space of its penalty computations, and thus A would avoid the impact penalty.

• sudhanshu_kasewa 4 Feb 2022 16:02 UTC

  1 point

  on: Prizes for ELK proposals

  Naive thought #2618281828:

  Could asking counterfactual questions be a potentially useful strategy to bias the reporter to be a direct translator rather than a human simulator?

  Concretely, consider a tuple (v, a, v’), where v := ‘before’ video, a := ‘action’ selected by SmartVault or augmented-human or whatever, and v’ := ‘after’ video.

  Then, for some new action a’, ask the question:

  • “Given (v, a, v’), if action a’ was taken, is the diamond in the room?”

  (How we collect such data is unclear but doesn’t seem obviously intractable.)

  I think there’s some value here:

  • Answering such a question might not require computation concerning a and v’ ; if we see these computations being used, we might derive more value from regularizers that penalize downstream variables (which now includes the nodes close to a)

  • This might also force the reporter to essentially model (or compress but not indefinitely) the predictor; the reporter now has both a compressed predictor Bayes’ net and a human Bayes’ net. If we can be confident that the compressed predictor BN is much smaller than the human BN, then doing direct translation within the reporter, i.e. compressed predictor BN inference + translation + read off from human BN might be less expensive than the human simulator alternative, i.e. compressed predictor BN inference + ‘translation’/​bridging computation + human BN inference.

    • We might find ways of being confident that the compressed predictor BN is small (e.g. by adding decoders at every layer of the reporter that reconstruct v, a or v’ and heavily penalizing later-layer decoders)

• sudhanshu_kasewa 9 Feb 2022 10:30 UTC

  2 points

  on: Prizes for ELK proposals

  Dumb question alert:

  In the appendix “Details for penalizing depending on “downstream” variables”, I’m not able to wrap my head around what we can expect the reporter to learn—if anything at all—seeing that it has no dependency on the inputs (elsewhere it is dependent on z sampled from the posterior).

  Specifically, the only call to the reporter (in the function reporter_loss in this section) contains no information (about before, action, after) from the predictor at all:

  answer = reporter(question, ε, θ_reporter)

  (unless “question” includes some context from the current (before, action, after) being considered, which I’m assuming is not the case)

  My dumb question then is:

  -- Why would this reporter be performant in any way?

  My reasoning: For a given question Q (say, “Is the diamond in the room?”) we might have some answers of “Yes” and some of “No” in the dataset, but without the context, we’re essentially training the reporter to map noise that is uncorrelated with/​independent of the context to the answer; essentially, for a fixed question Q and fixed realization of the noise RV, the reporter will be uniformly uncertain (or well, it will mirror the statistics in the data) about the value of the answer. Since the noise is independent/​uncorrelated, this would be true for every noise value.

• sudhanshu_kasewa 28 Jul 2023 11:35 UTC

  2 points

  on: Neu­ron­pe­dia—AI Safety Game

  Very cool. Thanks for putting this together.

  Half-baked, possibly off-topic: I wonder if there’s some data-collection that can be used to train out polysemi from a model by fine-tuning.

  e.g.:

  • Show 3 examples (just like in this game), and have the user pick the odd-one-out

    • User can say “they are all the same”, if so, remove one at random, and replace with a new example

  • Tag the (neuron, positive example) pairs with (numerical value) label 1, the odd-one-out with 0

  • Fine-tune with next-word-prediction and an auxilliary loss using this new collected dataset

    • Can probably use some automated (e.g. semantic similarity) labelling method to cluster labelled+unlabelled instances, to increase the size of the dataset

  Neuronpedia interface/​codebase could probably be forked to do this kind of data collection very easily.

[Question] Who can most re­duce X-Risk?

• sudhanshu_kasewa 23 Feb 2024 16:27 UTC

  2 points

  0

  on: No Click­bait—Misal­ign­ment Database

  It might be worth (someone) writing out what is meant by each kind of misalignment category, as used in the db. Objective misalignment, specific gaming, value misalignment all seem overlapping, and I’m not at all sure what physical misalignment is supposed to be pointing to.

• sudhanshu_kasewa 8 Mar 2024 12:13 UTC

  2 points

  0

  in reply to: Kabir Kumar’s comment on: No Click­bait—Misal­ign­ment Database

  No, this is not something I can undertake—however, the effort itself need not be very complicated. You’ve already got a list of Misalignment types in the form: create a google doc with definitions/​descriptions of each of these, and put a link to that doc in this question.