(...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties.
- Saharon Shelah
For my most of my writing see my short-forms (new shortform, old shortform)
Twitter: @FellowHominid
Personal website: https://sites.google.com/view/afdago/home
Currently there seems to be no good way to deal with this conundrum.
One wonders whether the AI alignment community should setup some sort of encrypted partial-sharing partial-transparancy protocol for these kinds of situations.
Shapely values are very cool. Let me mention some cool facts:
They arise in (cooperative) game theory but also in ML when doing credit allocation a combined prediction from mixing predictions from different modules of a system.
One piece of evidence of their fundamentalness is that they arise naturally from the Hodge theory on the hypercube of a coalition game: https://arxiv.org/abs/1709.08318
Another interesting fact I learned from Davidad: Shapley values are not compositional: a group of actors can increase their total Shapley value by forming a single cabal such that individuals within that cabal refuse to cooperate with individuals outside the cabal without the rest of the cabal joining in. This is can be a measure of collusion potential.
Could you give a sneak peak on how Sylvia /Alain use these uniform distributions?
There is a third important aspect of functions-in-the-original-sense that distinguishes them from extensional functions (i.e. collection of input-output pairs): effects.
Describing these ‘intensional’ features is an active area of research in theoretical CS. One important thread here is game semantics; you might like to take a look:
https://link.springer.com/chapter/10.1007/978-3-642-58622-4_1
How can the uniform distribution on the natural numbers be used?
No problem!
Do you mean monoidal categories? I think that’s what the central concept in the Abramsly-Coeke work & the Baez Rosetta stone paper is.
Category theory was developed in the 50s from considerations in algebraic topology. Algebraic topology was an extremely technically sophisticated field already in the 50s (and has by now reached literally incredible abstract heights).
I suppose one could imagine an alternate world where Galois invents category theory but it seems apparent to me that the amount of long-term development significantly divorced from direct applications (as calculus was) was needed for category theory to spring up and mature—indeed it is still in its teenage rebellious phase in the grand scheme of things!
John Baez et al ’s work on Blackboxing is uses several ideas only developed recently and though I am a fan of abstract mathematics in general and category theory in particular I hasten to say this work is only a small part of a big and as of yet mostly unfinished story. I’m optimistic that this might eventually lead to serious insights in understanding agency but at the moment it seems we are still quite far.
A more directed search for agent foundations math is needed, happening (but we need much more!) and likely to bear fruits on the medium-short term but I suspect many of the ingredients are likely things that have been developed with very different motivations in mind.
Edit: see https://www.lesswrong.com/s/LWJsgNYE8wzv49yEc for Critch’s new work on Boundaries
Evidence Manipulation and Legal Admissible Evidence
[This was inspired by Kokotaljo’s shortform on comparing strong with weak evidence]
In the real world the weight of many pieces of weak evidence is not always comparable to a single piece of strong evidence. The important variable here is not strong versus weak per se but the source of the evidence. Some sources of evidence are easier to manipulate in various ways. Evidence manipulation, either consciously or emergently, is common and a large obstactle to truth-finding.
Consider aggregating many (potentially biased) sources of evidence versus direct observation. These are not directly comparable and in many cases we feel direct observation should prevail.
This is especially poignant in the court of law: the very strict laws arounding presenting evidence are a culturally evolved mechanism to defend against evidence manipulation. Evidence manipulation may be easier for weaker pieces of evidence—see the prohibition against hearsay in legal contexts for instance.
It is occasionally suggested that the court of law should do more probabilistic and Bayesian type of reasoning. One reason courts refuse to do so (apart from more Hansonian reasons around elites cultivating conflict suppression) is that naive Bayesian reasoning is extremely susceptible to evidence manipulation.
In the real world the weight of many pieces of weak evidence is not always comparable to a single piece of strong evidence. The important variable here is not strong versus weak per se but the source of the evidence. Some sources of evidence are easier to manipulate in various ways. Evidence manipulation, either consciously or emergently, is common and a large obstactle to truth-finding.
Consider aggregating many (potentially biased) sources of evidence versus direct observation. These are not directly comparable and in many cases we feel direct observation should prevail.
This is especially poignant in the court of law: the very strict laws arounding presenting evidence are a culturally evolved mechanism to defend against evidence manipulation. Evidence manipulation may be easier for weaker pieces of evidence—see the prohibition against hearsay in legal contexts for instance.
It is occasionally suggested that the court of law should do more probabilistic and Bayesian type of reasoning. One reason courts refuse to do so (apart from more Hansonian reasons around elites cultivating conflict suppression) is that naive Bayesian reasoning is extremely susceptible to evidence manipulation.
With all due respect with your brand as LessWrong’s ornamental hermeneutic I’m afraid I’ll need some clarification.
What is the monad of 1 exactly? A monad is a functor—what category are we talking about here?
In particular—what are the unit and multiplication maps?
(my guess: for the unit and but now I’m using nilsquare infinitesimals instead of invertible infinitesimals.)
I’m not sure what tangent space we are talking about—but I assume it’s a Lie group (hyperfinite graph?) and we are looking at the tangent space of the identity element. In this case—what is the clifford algebra of the tangent space? Construction of a clifford algebra needs a choice of inner product (or more generally a quadratic form) - what are we picking here?
Where can I learn more about hyperfinite Brownian motion?
Has this been developed deeply? (I am aware of Nelson’s radically elementary probability book)
Where can I read more about this perspective?
I’m intrigued by the idea of linking the discrete and continuous Fourier transform through nonstandard analysis.
The idea is certainly beautifully elegant—has it been worked out in more detail somewhere?
Beautiful chaotic math energy here Alok, keep it up! =)
Roko’s basilisk is a thought experiment which states that an otherwise benevolent artificial superintelligence (AI) in the future would be incentivized to create a virtual reality simulation to torture anyone who knew of its potential existence but did not directly contribute to its advancement or development.
Why Roko’s basilisk probably doesn’t work for simulation fidelity reasons:
Roko’s basilisk threatens to simulate and torture you in the future if you don’t comply. Simulation cycles cost resources. Instead of following through on torturing our would-be cthulhu worshipper they could spend those resources on something else.
But wait can’t it use acausal magic to precommit to follow through? No.
Acausal arguments only work in situations where agents can simulate each others with high fidelity. Roko’s basilisk can simulate the human but not the other way around! The human’s simulation of Roko’s basilisk is very low fidelity—in particular Roko’s Basilisk is never confused whether or not it is being simulated by a human—it knows for a fact that the human is not able to simulate it.
I thank Jan P. for coming up with this argument.
And actually Barbara, Celarent, Darii, Ferio contain hints of phenomena not completely captured in first-order logic!
I like that this post can be read as in both jest and earnestness and in both readings it contains much Truth and Wisdom. =)
Imagine a data stream
assumed infinite in both directions for simplicity. Here represents the current state ( the “present”) and while and represents the future
Predictible Information versus Predictive Information
Predictible information is the maximal information (in bits) that you can derive about the future given the access to the past. Predictive information is the amount of bits that you need from the past to make that optimal prediction.
Suppose you are faced with the question of whether to buy, hold or sell Apple. There are three options so maximally bits of information—not all of that information might be in contained in the past, there a certain part of irreductible uncertainty (entropy) about the future no matter how well you can infer the past. Think about a freak event & blacks swans like pandemics, wars, unforeseen technological breakthroughs, just cumulative aggregated noise in consumer preference etc. Suppose that irreducible uncertainty is half of leaving us with of (theoretically) predictible information.
To a certain degree, it might be predictible in theory to what degree buying Apple stock is a good idea. To do so, you may need to know many things about the past: Apple’s earning records, position of competitiors, general trends of the economy, understanding of the underlying technology & supply chains etc. The total sum of this information is far larger than
To actually do well on the stock market you additionally need to do this better than the competititon—a difficult task! The predictible information is quite small compared to the predictive information
Note that predictive information is always greater than predictible information: you need to at least bits from the past to predict bits of the future. Often it is much larger.
Mathematical details
Predictible information is also called ‘apparent stored information’ or commonly ‘excess entropy’.
It is defined as the mutual information between the future and the past.
The predictive information is more difficult to define. It is also called the ‘statistical complexity’ or ‘forecasting complexity’ and is defined as the entropy of the steady equilibrium state of the ‘epsilon machine’ of the process.
What is the Epsilon Machine of the process ? Define the causal states as the process as the partition on the sets of possible pasts where two pasts are in the same part / equivalence class when the future conditioned on respectively is the same.
That is . Without going into too much more detail the forecasting complexity measures the size of this creature.
So happy to see this post appear! 🔥
The story about operons and the high interconnectedness of prokaryote genomes makes me wonder: bacteria kick out the antibiotic-coding gene after a few hours… but how does it know which gene to kick out?
Does it have a way to tell which genes are more ‘alien’ than others? (Or are we only talking about plasmids here?) I’ve heard it’s hard to genomic manipulate some genomes because the cells keep kicking out new genes
One could speculate there is some sort of mechanism, perhaps epi-genetic, that is able to tell which genes are more alien / new than others somehow?
I’d love to hear your thoughts