Misplaced Pages

Interaction information

Article snapshot taken from Wikipedia with creative commons attribution-sharealike license. Give it a read and then ask your questions in the chat. We can research this topic together.
(Redirected from Multivariate mutual information) Generalization of mutual information for more than two variables
Venn diagram of information theoretic measures for three variables x, y, and z, represented by the lower left, lower right, and upper circles, respectively. The interaction information is represented by gray region, and it is the only one that can be negative.

In probability theory and information theory, the interaction information is a generalization of the mutual information for more than two variables.

There are many names for interaction information, including amount of information, information correlation, co-information, and simply mutual information. Interaction information expresses the amount of information (redundancy or synergy) bound up in a set of variables, beyond that which is present in any subset of those variables. Unlike the mutual information, the interaction information can be either positive or negative. These functions, their negativity and minima have a direct interpretation in algebraic topology.

Definition

The conditional mutual information can be used to inductively define the interaction information for any finite number of variables as follows:

I ( X 1 ; … ; X n + 1 ) = I ( X 1 ; … ; X n ) − I ( X 1 ; … ; X n ∣ X n + 1 ) , {\displaystyle I(X_{1};\ldots ;X_{n+1})=I(X_{1};\ldots ;X_{n})-I(X_{1};\ldots ;X_{n}\mid X_{n+1}),}

where

I ( X 1 ; … ; X n ∣ X n + 1 ) = E X n + 1 ( I ( X 1 ; … ; X n ) ∣ X n + 1 ) . {\displaystyle I(X_{1};\ldots ;X_{n}\mid X_{n+1})=\mathbb {E} _{X_{n+1}}{\big (}I(X_{1};\ldots ;X_{n})\mid X_{n+1}{\big )}.}

Some authors define the interaction information differently, by swapping the two terms being subtracted in the preceding equation. This has the effect of reversing the sign for an odd number of variables.

For three variables { X , Y , Z } {\displaystyle \{X,Y,Z\}} , the interaction information I ( X ; Y ; Z ) {\displaystyle I(X;Y;Z)} is given by

I ( X ; Y ; Z ) = I ( X ; Y ) − I ( X ; Y ∣ Z ) {\displaystyle I(X;Y;Z)=I(X;Y)-I(X;Y\mid Z)}

where I ( X ; Y ) {\displaystyle I(X;Y)} is the mutual information between variables X {\displaystyle X} and Y {\displaystyle Y} , and I ( X ; Y ∣ Z ) {\displaystyle I(X;Y\mid Z)} is the conditional mutual information between variables X {\displaystyle X} and Y {\displaystyle Y} given Z {\displaystyle Z} . The interaction information is symmetric, so it does not matter which variable is conditioned on. This is easy to see when the interaction information is written in terms of entropy and joint entropy, as follows:

I ( X ; Y ; Z ) = ( H ( X ) + H ( Y ) + H ( Z ) ) − ( H ( X , Y ) + H ( X , Z ) + H ( Y , Z ) ) + H ( X , Y , Z ) {\displaystyle {\begin{alignedat}{3}I(X;Y;Z)&=&&\;{\bigl (}H(X)+H(Y)+H(Z){\bigr )}\\&&&-{\bigl (}H(X,Y)+H(X,Z)+H(Y,Z){\bigr )}\\&&&+H(X,Y,Z)\end{alignedat}}}

In general, for the set of variables V = { X 1 , X 2 , … , X n } {\displaystyle {\mathcal {V}}=\{X_{1},X_{2},\ldots ,X_{n}\}} , the interaction information can be written in the following form (compare with Kirkwood approximation):

I ( V ) = ∑ T ⊆ V ( − 1 ) | T | − 1 H ( T ) {\displaystyle I({\mathcal {V}})=\sum _{{\mathcal {T}}\subseteq {\mathcal {V}}}(-1)^{\left\vert {\mathcal {T}}\right\vert -1}H({\mathcal {T}})}

For three variables, the interaction information measures the influence of a variable Z {\displaystyle Z} on the amount of information shared between X {\displaystyle X} and Y {\displaystyle Y} . Because the term I ( X ; Y ∣ Z ) {\displaystyle I(X;Y\mid Z)} can be larger than I ( X ; Y ) {\displaystyle I(X;Y)} , the interaction information can be negative as well as positive. This will happen, for example, when X {\displaystyle X} and Y {\displaystyle Y} are independent but not conditionally independent given Z {\displaystyle Z} . Positive interaction information indicates that variable Z {\displaystyle Z} inhibits (i.e., accounts for or explains some of) the correlation between X {\displaystyle X} and Y {\displaystyle Y} , whereas negative interaction information indicates that variable Z {\displaystyle Z} facilitates or enhances the correlation.

Properties

Interaction information is bounded. In the three variable case, it is bounded by

− min { I ( X ; Y ∣ Z ) , I ( Y ; Z ∣ X ) , I ( X ; Z ∣ Y ) } ≤ I ( X ; Y ; Z ) ≤ min { I ( X ; Y ) , I ( Y ; Z ) , I ( X ; Z ) } {\displaystyle -\min\{I(X;Y\mid Z),I(Y;Z\mid X),I(X;Z\mid Y)\}\leq I(X;Y;Z)\leq \min\{I(X;Y),I(Y;Z),I(X;Z)\}}

If three variables form a Markov chain X → Y → Z {\displaystyle X\to Y\to Z} , then I ( X ; Z ∣ Y ) = 0 {\displaystyle I(X;Z\mid Y)=0} , but I ( X ; Z ) ≥ 0 {\displaystyle I(X;Z)\geq 0} . Therefore

I ( X ; Y ; Z ) = I ( X ; Z ) − I ( X ; Z ∣ Y ) = I ( X ; Z ) ≥ 0. {\displaystyle I(X;Y;Z)=I(X;Z)-I(X;Z\mid Y)=I(X;Z)\geq 0.}

Examples

Positive interaction information

Positive interaction information seems much more natural than negative interaction information in the sense that such explanatory effects are typical of common-cause structures. For example, clouds cause rain and also block the sun; therefore, the correlation between rain and darkness is partly accounted for by the presence of clouds, I ( rain ; dark ∣ cloud ) < I ( rain ; dark ) {\displaystyle I({\text{rain}};{\text{dark}}\mid {\text{cloud}})<I({\text{rain}};{\text{dark}})} . The result is positive interaction information I ( rain ; dark ; cloud ) {\displaystyle I({\text{rain}};{\text{dark}};{\text{cloud}})} .

Negative interaction information

A car's engine can fail to start due to either a dead battery or a blocked fuel pump. Ordinarily, we assume that battery death and fuel pump blockage are independent events, I ( blocked fuel ; dead battery ) = 0 {\displaystyle I({\text{blocked fuel}};{\text{dead battery}})=0} . But knowing that the car fails to start, if an inspection shows the battery to be in good health, we can conclude that the fuel pump must be blocked. Therefore I ( blocked fuel ; dead battery ∣ engine fails ) > 0 {\displaystyle I({\text{blocked fuel}};{\text{dead battery}}\mid {\text{engine fails}})>0} , and the result is negative interaction information.

Difficulty of interpretation

This section may be confusing or unclear to readers. Please help clarify the section. There might be a discussion about this on the talk page. (June 2023) (Learn how and when to remove this message)

The possible negativity of interaction information can be the source of some confusion. Many authors have taken zero interaction information as a sign that three or more random variables do not interact, but this interpretation is wrong.

To see how difficult interpretation can be, consider a set of eight independent binary variables { X 1 , X 2 , X 3 , X 4 , X 5 , X 6 , X 7 , X 8 } {\displaystyle \{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7},X_{8}\}} . Agglomerate these variables as follows:

Y 1 = { X 1 , X 2 , X 3 , X 4 , X 5 , X 6 , X 7 } Y 2 = { X 4 , X 5 , X 6 , X 7 } Y 3 = { X 5 , X 6 , X 7 , X 8 } {\displaystyle {\begin{aligned}Y_{1}&=\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7}\}\\Y_{2}&=\{X_{4},X_{5},X_{6},X_{7}\}\\Y_{3}&=\{X_{5},X_{6},X_{7},X_{8}\}\end{aligned}}}

Because the Y i {\displaystyle Y_{i}} 's overlap each other (are redundant) on the three binary variables { X 5 , X 6 , X 7 } {\displaystyle \{X_{5},X_{6},X_{7}\}} , we would expect the interaction information I ( Y 1 ; Y 2 ; Y 3 ) {\displaystyle I(Y_{1};Y_{2};Y_{3})} to equal 3 {\displaystyle 3} bits, which it does. However, consider now the agglomerated variables

Y 1 = { X 1 , X 2 , X 3 , X 4 , X 5 , X 6 , X 7 } Y 2 = { X 4 , X 5 , X 6 , X 7 } Y 3 = { X 5 , X 6 , X 7 , X 8 } Y 4 = { X 7 , X 8 } {\displaystyle {\begin{aligned}Y_{1}&=\{X_{1},X_{2},X_{3},X_{4},X_{5},X_{6},X_{7}\}\\Y_{2}&=\{X_{4},X_{5},X_{6},X_{7}\}\\Y_{3}&=\{X_{5},X_{6},X_{7},X_{8}\}\\Y_{4}&=\{X_{7},X_{8}\}\end{aligned}}}

These are the same variables as before with the addition of Y 4 = { X 7 , X 8 } {\displaystyle Y_{4}=\{X_{7},X_{8}\}} . However, I ( Y 1 ; Y 2 ; Y 3 ; Y 4 ) {\displaystyle I(Y_{1};Y_{2};Y_{3};Y_{4})} in this case is actually equal to + 1 {\displaystyle +1} bit, indicating less redundancy. This is correct in the sense that

I ( Y 1 ; Y 2 ; Y 3 ; Y 4 ) = I ( Y 1 ; Y 2 ; Y 3 ) − I ( Y 1 ; Y 2 ; Y 3 | Y 4 ) = 3 − 2 = 1 {\displaystyle {\begin{aligned}I(Y_{1};Y_{2};Y_{3};Y_{4})&=I(Y_{1};Y_{2};Y_{3})-I(Y_{1};Y_{2};Y_{3}|Y_{4})\\&=3-2\\&=1\end{aligned}}}

but it remains difficult to interpret.

Uses

  • Jakulin and Bratko (2003b) provide a machine learning algorithm which uses interaction information.
  • Killian, Kravitz and Gilson (2007) use mutual information expansion to extract entropy estimates from molecular simulations.
  • LeVine and Weinstein (2014) use interaction information and other N-body information measures to quantify allosteric couplings in molecular simulations.
  • Moore et al. (2006), Chanda P, Zhang A, Brazeau D, Sucheston L, Freudenheim JL, Ambrosone C, Ramanathan M. (2007) and Chanda P, Sucheston L, Zhang A, Brazeau D, Freudenheim JL, Ambrosone C, Ramanathan M. (2008) demonstrate the use of interaction information for analyzing gene-gene and gene-environmental interactions associated with complex diseases.
  • Pandey and Sarkar (2017) use interaction information in Cosmology to study the influence of large-scale environments on galaxy properties.
  • A python package for computing all multivariate interaction or mutual informations, conditional mutual information, joint entropies, total correlations, information distance in a dataset of n variables is available .


See also

References

  1. Ting, Hu Kuo (January 1962). "On the Amount of Information". Theory of Probability & Its Applications. 7 (4): 439–447. doi:10.1137/1107041. ISSN 0040-585X.
  2. Wolf, David (May 1, 1996). The Generalization of Mutual Information as the Information between a Set of Variables: The Information Correlation Function Hierarchy and the Information Structure of Multi-Agent Systems (Technical report). NASA Ames Research Center.
  3. ^ Bell, Anthony (2003). The co-information lattice. 4th Int. Symp. Independent Component Analysis and Blind Source Separation.
  4. ^ Yeung, R.W. (May 1991). "A new outlook on Shannon's information measures". IEEE Transactions on Information Theory. 37 (3): 466–474. doi:10.1109/18.79902. ISSN 0018-9448.
  5. Baudot, Pierre; Bennequin, Daniel (2015-05-13). "The Homological Nature of Entropy". Entropy. 17 (5): 3253–3318. Bibcode:2015Entrp..17.3253B. doi:10.3390/e17053253. ISSN 1099-4300.
  6. McGill, William J. (June 1954). "Multivariate information transmission". Psychometrika. 19 (2): 97–116. doi:10.1007/bf02289159. ISSN 0033-3123. S2CID 126431489.
  7. Krippendorff, Klaus (August 2009). "Information of interactions in complex systems". International Journal of General Systems. 38 (6): 669–680. doi:10.1080/03081070902993160. ISSN 0308-1079. S2CID 13923485.
  8. Killian, Benjamin J.; Yundenfreund Kravitz, Joslyn; Gilson, Michael K. (2007-07-14). "Extraction of configurational entropy from molecular simulations via an expansion approximation". The Journal of Chemical Physics. 127 (2): 024107. Bibcode:2007JChPh.127b4107K. doi:10.1063/1.2746329. ISSN 0021-9606. PMC 2707031. PMID 17640119.
  9. LeVine, Michael V.; Perez-Aguilar, Jose Manuel; Weinstein, Harel (2014-06-18). "N-body Information Theory (NbIT) Analysis of Rigid-Body Dynamics in Intracellular Loop 2 of the 5-HT2A Receptor". arXiv:1406.4730 .
  10. "InfoTopo: Topological Information Data Analysis. Deep statistical unsupervised and supervised learning - File Exchange - Github". github.com/pierrebaudot/infotopopy/. Retrieved 26 September 2020.
Category: