A Measure of Synergy Based on Union Information

Abstract

The partial information decomposition (PID) framework is concerned with decomposing the information that a set of (two or more) random variables (the sources) has about another variable (the target) into three types of information: unique, redundant, and synergistic. Classical information theory alone does not provide a unique way to decompose information in this manner and additional assumptions have to be made. One often overlooked way to achieve this decomposition is using a so-called measure of union information—which quantifies the information that is present in at least one of the sources—from which a synergy measure stems. In this paper, we introduce a new measure of union information based on adopting a communication channel perspective, compare it with existing measures, and study some of its properties. We also include a comprehensive critical review of characterizations of union information and synergy measures that have been proposed in the literature.

1. Introduction

Williams and Beer [1] introduced the partial information decomposition (PID) framework as a way to characterize, or analyze, the information that a set of random variables (often called sources) has about another variable (referred to as the target). PID is a useful tool for gathering insights and analyzing the way information is stored, modified, and transmitted within complex systems [2,3]. It has been applied in several areas such as cryptography [4] and neuroscience [5,6], with many other potential use cases, such as in studying information flows in gene regulatory networks [7], neural coding [8], financial markets [9], and network design [10,11].

Consider the simplest case, a three-variable joint distribution $p(y_1,y_2,t)$ describing three random variables: two so-called sources, $Y_1$ and $Y_2$, and a target T. Notice that, despite what the names sources and target might suggest, there is no directionality (causal or otherwise) assumption. The goal of PID is to decompose the information that the sources $Y=(Y_1,Y_2)$ have about T into the sum of four non-negative quantities: the information that is present in both $Y_1$ and $Y_2$, known as redundant information, R; the information that only $Y_1$ (respectively $Y_2$) has about T, known as unique information, $U_1$ (respectively $U_2$); and the synergistic information, S, that is present in the pair $(Y_1,Y_2)$ but not in $Y_1$ or $Y_2$ alone. In this case with two variables, the goal is, thus, to write

$$I(T;Y)=R+U_1+U_2+S,$$

(1)

where $I(T;Y)$ is the mutual information between T and Y [12]. The redundant information R, because it is present in both $Y_1$ and $Y_2$, is also referred to as intersection information and denoted as $I_{\cap }$. Finally, $I_{\cup }$ refers to union information, i.e., the amount of information provided by at least one of the sources; in the case of two sources, $I_{\cup }=U_1+U_2+R$, thus $S=I(T;Y)-I_{\cup }$.

Because unique information and redundancy satisfy the relationship $U_i=I(T;Y_i)-R$ (for $i\in \{1,2\}$), it turns out that defining how to compute one of these quantities (R, $U_i$, or S) is enough to fully determine the others [1]. Williams and Beer [1] suggested a set of axioms that a measure of redundancy should satisfy, and proposed a measure of their own. Those axioms became well known as the Williams–Beer axioms, although the measure they proposed has subsequently been criticized for not capturing informational content, but only information size [13]. It is worth noting that as the number of variables grows the number of terms appearing in the PID of $I(T;Y)$ grows super-exponentially [14].

Stimulated by that initial work, other measures of information and other sets of axioms for information decomposition have been introduced; see, for example, the work by Bertschinger et al. [15], Griffith and Koch [16], and James et al. [17], for different measures of redundant, unique, and synergistic information. There is no consensus about what axioms any measure should satisfy or whether a given measure captures the information that it should capture, except for the Williams–Beer axioms. Today, there is still debate about what axioms different measures of information should satisfy, and there is no general agreement on what is an appropriate PID [17,18,19,20,21].

Most PID measures that have been suggested thus far are either measures of redundant information, e.g., [1,13,21,22,23,24,25,26], or measures of unique information, e.g., [15,17]. Alternatively, it is possible to define the union information of a set of sources as the amount of information provided by at least one of those sources. Synergy is then defined as the difference between the total information and union information [22].

In this paper, we introduce a new measure of union information based on the information channel perspective that we already pursued in earlier work [26] and study some of its properties. The resulting measure leads to a novel information decomposition that is particularly suited for analyzing how information is distributed in channels.

The rest of the paper is organized as follows. A final subsection of this section introduces the notation used throughout the paper. In Section 2, we recall some properties of PID and take a look at how the degradation measure for redundant information introduced by Kolchinsky [22] decomposes information in bivariate systems, while also pointing out some drawbacks of that measure. Section 3 presents the motivation for our proposed measure, its operational interpretation, its multivariate definition, as well as some of its drawbacks. In Section 4, we propose an extension of the Williams–Beer axioms for measures of union information and show that our proposed measure satisfies those axioms. We review all properties that have been proposed both for measures of union information and synergy, and either accept or reject them. We also compare different measures of synergy and relate them, whenever possible. Finally, Section 5 presents concluding remarks and suggestions for future work.

Notation

For two discrete random variables $X\in \mathcal{X}$ and $Z\in \mathcal{Z}$, their Shannon mutual information $I(X;Z)$ is given by $I(X;Z)=I(Z;X)=H\left(X\right)-H\left(X\right|Z)=H(Z)-H(Z\left|X\right)$, where $H\left(X\right)=-{\sum }_{x\in \mathcal{X}}p\left(x\right)logp\left(x\right)$ and $H\left(X\right|Z)=-{\sum }_{x\in \mathcal{X}}{\sum }_{z\in \mathcal{X}}p(x,z)logp\left(x\right|z)$ are the entropy and conditional entropy, respectively [12]. The conditional distribution $p\left(z\right|x)$ corresponds, in an information-theoretical perspective, to a discrete memoryless channel with a channel matrix K, i.e., such that $K[x,z]=p\left(z\right|x)$ [12]. This matrix is row-stochastic: $K[x,z]\ge 0$, for any $x\in \mathcal{X}$ and $z\in \mathcal{Z}$, and ${\sum }_{z\in \mathcal{Z}}K[x,z]=1$, for any x.

Given a set of n discrete random variables (sources), $Y_1\in {\mathcal{Y}}_1,\dots ,Y_n\in {\mathcal{Y}}_n$, and a discrete random variable $T\in \mathcal{T}$ (target) with joint distribution (probability mass function) $p(y_1,\dots ,y_n,t)$, we consider the channels $K^{\left(i\right)}$ between T and each $Y_i$, that is, each $K^{\left(i\right)}$ is a $\left|\mathcal{T}\right|\times |{\mathcal{Y}}_i|$ row-stochastic matrix with the conditional distribution $p\left(y_i\right|t)$. For a vector y, $y_{-i}$ refers to the same vector without the $i^{th}$ component.

We say that three random variables, say $X,Y,Z$, form a Markov chain (which we denote by $X-Y-Z$ or by $X\perp Z\mid Y$) if X and Z are conditionally independent, given Y.

2. Background

2.1. PID Based of Channel Orders

In its current versions, PID is agnostic to causality in the sense that, like mutual information, it is an undirected measure, i.e., $I(T;Y)=I(Y;T)$. Some measures indirectly presuppose some kind of directionality to perform PID. Take, for instance, the redundancy measure introduced by Kolchinsky [22], based on the so-called degradation order ${⪯}_d$ between communication channels (see recent work by Kolchinsky [22] and Gomes and Figueiredo [26] for definitions):

$$I_{\cap }^{\mathrm{d}}(Y_1,Y_2,\dots ,Y_m\to T):=\underset{K^Q:K^Q{⪯}_d{K}^{\left(i\right)},i\in \{1,\dots ,m\}}{sup}I(Q;T).$$

(2)

In the above equation, Q is the output of the channel $T\to Q$, which we also denote as $K^Q$. When computing the information shared by the m sources, $I_{\cap }^{\mathrm{d}}(Y_1,Y_2,\dots ,Y_m\to T)$, the perspective is that there is a channel with a single input T and m outputs $Y_1,\dots ,Y_m$. This definition of $I_{\cap }^{\mathrm{d}}$ corresponds to the mutual information of the most informative channel, under the constraint that this channel is dominated (in the degradation order ${⪯}_d$ sense) by all channels $K^{\left(1\right)},\dots ,K^{\left(m\right)}$. Since mutual information was originally introduced to formalize the capacity of communication channels, it is not surprising that measures that presuppose channel directionality are found useful in this context. For instance, the work of James et al. [27] concluded that only if one assumes the directionality $T\to Y_i$ does one obtain a valid PID from a secret key agreement rate, which supports the approach of assuming this directionality perspective.

Although it is not guaranteed that the structure of the joint distribution $p(y_1,\dots ,y_n,t)$ is compatible with the causal model of a single input and multiple output channels (which implies that the sources are conditionally independent, given T), one may always compute such measures, which have interesting and relevant operational interpretations. In the context of PID, where the goal is to study how information is decomposed, such measures provide an excellent starting point. Although it is not guaranteed that there is actually a channel (or a direction) from T to $Y_i$, we can characterize how information about T is spread through the sources. In the case of the degradation order, $I_{\cap }^{\mathrm{d}}$ provides insight about the maximum information obtained about T if any $Y_i$ is observed.

Arguably, the most common scenario in PID is finding out something about the structure of the information the variables $Y_1,\dots ,Y_n$ have about T. In a particular system of variables characterized by its joint distribution, we do not make causal assumptions, so we can adopt the perspective that the variables $Y_i$ are functions of T, hence obtaining the channel structure. Although this channel structure may not be faithful [28] to the conditional independence properties implied by $p(y_1,\dots ,y_n,t)$, this channel perspective allows for decomposition of $I(Y;T)$ and for drawing conclusions about the inner structure of the information that Y has about T. Some distributions, however, cannot have this causal structure. Take, for instance, the distribution generated by $T=Y_1\mathrm{xor}{Y}_2$, where $Y_1$ and $Y_2$ are two equiprobable and independent binary random variables. We will call this distribution XOR. For this well-known distribution, we have $Y_1\perp Y_2$ and $Y_1\cancel{\perp }{Y}_2|T$, whereas the implied channel distribution that $I_{\cap }^{\mathrm{d}}$ assumes yields the exact opposite dependencies, that is, $Y_1\cancel{\perp }{Y}_2$ and $Y_1\perp Y_2|T$; see Figure 1 for more insight.

Consider the computation of $I_{\cap }^{\mathrm{d}}(Y_1,Y_2\to T)$ for the XOR distribution. This measure argues that, since

$$K^{\left(1\right)}=K^{\left(2\right)}=\left[\begin{array}{cc}0.5& 0.5\\ 0.5& 0.5\end{array}\right],$$

then a solution to $I_{\cap }^{\mathrm{d}}(Y_1,Y_2\to T)$ is given by the channel $K^Q=K^{\left(1\right)}$ and redundancy is computed as $I(Q;T)$, yielding 0 bits of redundancy, and consequently 1 bit of synergy (as computed from (1)). Under this channel perspective (as in Figure 1b), $I_{\cap }^{\mathrm{d}}$ is not concerned with, for example, $p\left(Y_1\right|T,Y_2)$ or $p(Y_1,Y_2)$. If all that is needed to compute redundancy is $p\left(T\right),K^{\left(1\right)}$, and $K^{\left(2\right)}$, this would lead to the wrong conclusion that the outcome $(Y_1,Y_2,T)=(0,0,1)$ has non-null probability, which it does not. With this, we do not mean that $I_{\cap }^{\mathrm{d}}$ is an incomplete or incorrect measure to perform PID, we are using its insights to point us in a different direction.

2.2. Motivation for a New PID

At this point, the most often used approaches to PID are based on redundancy measures. Usually, these are in one of the two following classes:

• Measures that are not concerned with information content, only information size, which makes them easy to compute even for distributions with many variables, but at the cost that the resulting decompositions may not give realistic insights into the system, precisely because they are not sensitive to informational content. Examples are $I_{\cap }^{\mathrm{WB}}$ [1] or $I_{\cap }^{\mathrm{MMI}}$ [23], and applications of these can be found in [29,30,31,32].
• Measures that satisfy the Blackwell property—which arguably do measure information content—but are insensitive to changes in the sources’ distributions $p(y_1,\dots ,y_n)={\sum }_{t}p(y_1,\dots ,y_n,t)$ (as long as $p\left(T\right),K^{\left(1\right)},\dots ,K^{\left(n\right)}$ remain the same). Examples are $I_{\cap }^{\mathrm{d}}$ [22] (see Equation (2)) or $I_{\cap }^{\mathrm{BROJA}}$ [15]. It should be noted that $I_{\cap }^{\mathrm{BROJA}}$ is only defined for the bivariate case, that is, for distributions with at most two sources, described by $p(y_1,y_2,t)$. Applications of these can be found in [33,34,35].

Particularly, $I_{\cap }^{\mathrm{d}}$ and $I_{\cap }^{\mathrm{BROJA}}$ satisfy the so-called (*) assumption [15], which argues that redundant and unique information should only depend on the marginal distribution of the target $p\left(T\right)$ and on the conditional distributions of the sources given the target, that is, on the stochastic matrices $K^{\left(i\right)}$. James et al. [17] (Section 4) and Ince [21] (Section 5) provide great arguments as to why the (*) assumption should not hold in general, and we agree with them.

Towards motivating a new PID, let us look at how $I_{\cap }^{\mathrm{d}}$ decomposes information in the bivariate case. Any measure that is based on a preorder between channels and which satisfies Kolchinsky’s axioms yields similar decompositions [26], thus there is no loss of generality in focusing on $I_{\cap }^{\mathrm{d}}$. We next analyze three different cases.

• Case 1: There is an ordering between the channels, that is, w.l.o.g., $K^{\left(2\right)}{⪯}_d{K}^{\left(1\right)}$. This means that $I(Y_2;T)\le I(Y_1;T)$ and the decomposition (as in (1)) is given by $R=I(Y_2;T)$, $U_2=0$, $U_1=I(Y_1;T)-I(Y_2;T)$, and $S=I(Y;T)-I(Y_1;T)$. Moreover, if $K^{\left(1\right)}{\cancel{⪯}}_d{K}^{\left(2\right)}$, then $S=0$.
  As an example, consider the leftmost distribution in Table 1, which satisfies $T=Y_1$. In this case,

  $$K^{\left(1\right)}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]{⪰}_d{K}^{\left(2\right)}=\left[\begin{array}{cc}0.5& 0.5\\ 0.5& 0.5\end{array}\right],$$

  yielding $R=0$, $U_1=1$, $U_2=0$, and $S=0$, as expected, because $T=Y_1$.

Table 1. Three joint distributions $p(yt,y_2,y_2)$ used to exemplify the three cases. Left: joint distribution satisfying $T=Y_1$. Middle: distribution satisfying $T=(Y_1,Y_2)$, known as the COPY distribution. Right: the so-called BOOM distribution (see text).

t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$	t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$	t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	0	1/4	(0,0)	0	0	1/4	0	0	2	1/6
0	0	1	1/4	(0,1)	0	1	1/4	1	0	0	1/6
1	1	0	1/4	(1,0)	1	0	1/4	1	1	2	1/6
1	1	1	1/4	(1,1)	1	1	1/4	2	0	0	1/6
								2	2	0	1/6
								2	2	1	1/6

Table 1. Three joint distributions $p(yt,y_2,y_2)$ used to exemplify the three cases. Left: joint distribution satisfying $T=Y_1$. Middle: distribution satisfying $T=(Y_1,Y_2)$, known as the COPY distribution. Right: the so-called BOOM distribution (see text).

t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$	t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$	t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	0	1/4	(0,0)	0	0	1/4	0	0	2	1/6
0	0	1	1/4	(0,1)	0	1	1/4	1	0	0	1/6
1	1	0	1/4	(1,0)	1	0	1/4	1	1	2	1/6
1	1	1	1/4	(1,1)	1	1	1/4	2	0	0	1/6
								2	2	0	1/6
								2	2	1	1/6

• Case 2: There is no ordering between the channels and the solution of $I_{\cap }^{\mathrm{d}}(Y_1,Y_2\to T)$ is a trivial channel, in the sense that it has no information about T. The decomposition is given by $R=0$, $U_2=I(Y_2;T)$, $U_1=I(Y_1;T)$, and $S=I(Y;T)-I(Y_1;T)-I(Y_2;T)$, which may lead to a negative value of synergy. An example of this is provided later.
  As an example, consider the COPY distribution with $Y_1$ and $Y_2$ i.i.d. Bernoulli variables with parameter 0.5, shown in the center of Table 1. In this case, channels $K^{\left(1\right)}$ and $K^{\left(2\right)}$ have the forms

  $$K^{\left(1\right)}=\left[\begin{array}{cc}1& 0\\ 1& 0\\ 0& 1\\ 0& 1\end{array}\right],K^{\left(2\right)}=\left[\begin{array}{cc}1& 0\\ 0& 1\\ 1& 0\\ 0& 1\end{array}\right],$$

  with no degradation order between them. This yields $R=0$, $U_1=U_2=1$, and $S=0$.
• Case 3: There is no ordering between the channels and $I_{\cap }^{\mathrm{d}}(Y_1,Y_2\to T)$ is achieved by a nontrivial channel $K^Q$. The decomposition is given by $R=I(Q;T)$, $U_2=I(Y_2;T)-I(Q;T)$, $U_1=I(Y_1;T)-I(Q;T)$, and $S=I(Y;T)+I(Q;T)-I(Y_1;T)-I(Y_2;T)$.
  As an example, consider the BOOM distribution [17], shown on the right-hand side of Table 1. In this case, channels $K^{\left(1\right)}$ and $K^{\left(2\right)}$ are

  $$K^{\left(1\right)}=\left[\begin{array}{ccc}1& 0& 0\\ 1/2& 1/2& 0\\ 1/3& 0& 2/3\end{array}\right],K^{\left(2\right)}=\left[\begin{array}{ccc}0& 0& 1\\ 1/2& 0& 1/2\\ 2/3& 1/3& 0\end{array}\right],$$

  and there is no degradation order between them. However, there is a nontrivial channel $K^Q$ that is dominated by both $K^{\left(1\right)}$ and $K^{\left(2\right)}$ that maximizes $I(Q;T)$. One of its versions is

  $$K^Q=\left[\begin{array}{ccc}0& 1& 0\\ 0& 3/4& 1/4\\ 1/3& 1/3& 1/3\end{array}\right],$$

  yielding $R\approx 0.322$, $U_1=U_2\approx 0.345$, and $S\approx 0.114$.

This class of approaches has some limitations, as is the case for all PID measures. In the bivariate case, the definition of synergy S from a measure of redundant information is the completing term such that $I(Y;T)=S+R+U_1+U_2$ holds. The definition of $I_{\cap }^{\mathrm{d}}$ supports the argument that if $K^{\left(2\right)}{⪯}_d{K}^{\left(1\right)}$ and $K^{\left(1\right)}{\cancel{⪯}}_d{K}^{\left(2\right)}$, then there is no synergy. This makes intuitive sense because, in this case, $T-Y_1-Y_2$ is a Markov chain (see Section 1 for the definition), consequently, $I(Y;T)=I(Y_1;T)$, that is, $Y_1$ has the same information about T as the pair $Y=(Y_1,Y_2)$.

If there is no ${⪯}_d$ ordering between the channels, as in the COPY distribution (Table 1, middle), the situation is more complicated. We saw that the decomposition for this distribution yields $R=0$, $U_1=U_2=1$, and $S=0$. However, suppose we change the distribution such that $T=(1,1)$ has probability 0 and the other outcomes have probability $1/3$. For example, consider the distribution in

Table 2. Tweaked COPY distribution, now without the outcome $(T,Y_1,Y_2)=((1,1),1,1)$.

T	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
(0,0)	0	0	1/3
(0,1)	0	1	1/3
(1,0)	1	0	1/3

. For this distribution, we have $I(Y;T)\approx 1.585$. Intuitively, we would expect that $I(Y;T)$ would be decomposed as $R=0$, $U_1=U_2=I(Y;T)/2$, and $S=0$, just as before, so that the proportions $U_i/I(Y;T)$, for $i\in \{1,2\}$, in both distributions remain the same, whereas redundancy and synergy would remain zero. That is, we do not expect that removing one of the outcomes while maintaining the remaining outcomes equiprobable would change the types of information in the system. However, if we perform this and compute the decomposition yielded by $I_{\cap }^{\mathrm{d}}$, we obtain $R=0$, $U_1=U_2=0.918\ne I(Y;T)/2$, and $S=-0.251$, i.e., a negative synergy, arguably meaningless.

There are still many open questions in PID. One of those questions is: Should measures of redundant information be used to measure synergy, given that they compute it as the completing term in Equation (1). We agree that using a measure of redundant information to compute the synergy in this way may not be appropriate, especially because the inclusion–exclusion principle (IEP) should not necessarily hold in the context of PID; see [22] for comments on the IEP.

With these motivations, we propose a measure of union information for PID that shares with $I_{\cap }^{\mathrm{d}}$ the implicit view of channels. However, unlike $I_{\cap }^{\mathrm{d}}$ and $I_{\cap }^{\mathrm{BROJA}}$—which satisfy the (*) assumption, and thus, are not concerned with the conditional dependencies in $p\left(y_i\right|t,y_{-i})$—our measure defines synergy as the information that cannot be computed from $p\left(y_i\right|t)$, but can be computed from $p\left(y_i\right|t,y_{-i})$. That is, we propose that synergy be computed as the information that is not captured by assuming conditional independence of the sources, given the target.

3. A New Measure of Union Information

3.1. Motivation and Bivariate Definition

Consider a distribution $p(y_1,y_2,t)$ and suppose there are two agents, agent 1 and agent 2, whose goal is to reduce their uncertainty about T by observing $Y_1$ and $Y_2$, respectively. Suppose also that the agents know $p\left(t\right)$, and that agent i has access to its channel distribution $p\left(y_i\right|t)$. Many PID measures make this same assumption, including $I_{\cap }^{\mathrm{d}}$. When agent i works alone to reduce the uncertainty about T, since it has access to $p\left(t\right)$ and $p\left(y_i\right|t)$, it also knows $p\left(y_i\right)$ and $p(y_i,t)$, which allows it to compute $I(Y_i;T)$: the amount of uncertainty reduction about T achieved by observing $Y_i$.

Now, if the agents can work together, that is, if they have access to $Y=(Y_1,Y_2)$, then they can compute $I(Y;T)$, because they have access to $p(y_1,y_2|t)$ and $p\left(t\right)$. On the other hand, if the agents are not able to work together (in the sense that they are not able to observe Y together, but only $Y_1$ and $Y_2$, separately) yet can communicate, then they can construct a different distribution q given by $q(y_1,y_2,t):=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t)$, i.e., a distribution under which $Y_1$ and $Y_2$ are conditionally independent given T, but have the same marginal $p\left(t\right)$ and the same individual conditionals $p\left(y_1\right|t)$ and $p\left(y_2\right|t)$.

The form of q in the previous paragraph should be contrasted with the following factorization of p, which entails no conditional independence assumption: $p(y_1,y_2,t)=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t,y_1)$. In this sense, we would propose to define union information, for the bivariate case, as follows:

$$\begin{array}{cc}\hfill & I_{\cup }(Y_1\to T)=I_q(Y_1;T)=I_p(Y_1;T),\hfill \\ \hfill & I_{\cup }(Y_2\to T)=I_q(Y_2;T)=I_p(Y_2;T),\hfill \\ \hfill & I_{\cup }(Y_1,Y_2\to T)=I_q(Y;T),\hfill \\ \hfill & I_{\cup }((Y_1,Y_2)\to T)=I_p(Y;T),\hfill \end{array}$$

(3)

where the subscript refers to the distribution under which the mutual information is computed. From this point forward, the absence of a subscript means that the computation is performed under the true distribution p. As we will see, this is not yet the final definition, for reasons to be addressed below.

Using the definition of synergy derived from a measure of union information [22], for the bivariate case we have

$$\begin{array}{c}\hfill S(Y_1,Y_2\to T):=I(Y;T)-I_{\cup }(Y_1,Y_2\to T).\end{array}$$

(4)

Synergy is often posited as the difference between the whole and the union of the parts. For our measure of union information, the ‘union of the parts’ corresponds to the reduction in uncertainty about T—under q—that agents 1 and 2 can obtain by sharing their conditional distributions. Interestingly, there are cases where the union of the parts is better than the whole, in the sense that $I_{\cup }(Y_1,Y_2\to T)>I(Y;T)$. An example of this is given by the Adapted ReducedOR distribution, originally introduced by Ince [21] and adapted by James et al. [17], which is shown in the left-hand side of

Table 3. Left: The Adapted Reduced OR distribution, where $r\in [0,1]$. Right: The corresponding distribution $q(t,y_1,y_2)=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t)$.

t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$	t	${\mathit{y}}_1$	${\mathit{y}}_2$	$\mathit{q}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	0	1/2	0	0	0	1/2
1	0	0	r/4	1	0	0	1/8
1	1	0	(1 − r)/4	1	1	0	1/8
1	0	1	(1 − r)/4	1	0	1	1/8
1	1	1	r/4	1	1	1	1/8

, where $r\in [0,1]$. This distribution is such that $I_q(Y;T)$ does not depend on r ($I_q(Y;T)\approx 0.549$), since neither $p\left(t\right)$ nor $p\left(y_1\right|t)$ and $p\left(y_2\right|t)$ depend on r; consequently, $q(t,y_1,y_2)$ also does not depend on r, as shown in the right-hand side of Table 3.

It can be easily shown that if $r>0.5$, then $I_q(Y;T)>I(Y;T)$, which implies that synergy, if defined as in (4), could be negative. How do we interpret the fact that there exist distributions such that $I_q(Y;T)>I(Y;T)$? This means that under distribution q, which assumes $Y_1$ and $Y_2$ are conditionally independent given T, $Y_1$ and $Y_2$ reduce the uncertainty about T more than in the original distribution. Arguably, the parts working independently and achieving better results than the whole should mean there is no synergy, as opposed to negative synergy.

The observations in the previous paragraphs motivate our definition of a new measure of union information as

$$\begin{array}{c}\hfill I_{\cup }^{\mathrm{CI}}(Y_1,Y_2\to T):=min\{I(Y;T),I_q(Y;T)\},\end{array}$$

(5)

with the superscript CI standing for conditional independence, yielding a non-negative synergy:

$$\begin{array}{c}\hfill S^{\mathrm{CI}}(Y_1,Y_2\to T)=I(Y;T)-I_{\cup }^{\mathrm{CI}}(Y_1,Y_2\to T)=max\{0,I(Y;T)-I_q(Y;T)\}.\end{array}$$

(6)

Note that for the bivariate case we have zero synergy if $p(t,y_2,y_2)$ is such that $Y_1{\perp }_p{Y}_2|T$, that is, if the outputs are indeed conditionally independent given T. Moreover, $I_{\cup }^{\mathrm{CI}}$ satisfies the monotonicity axiom from the extension of the Williams–Beer axioms to measures of union information (to be mentioned in Section 4.1), which further supports this definition. For the bivariate source case, the decomposition of $I(Y;T)$ derived from a measure of union information is given by

$$\left\{\begin{array}{c}{I}_{\cup }\left(Y_1\right)=I(Y_1;T)=U_1+R\hfill \\ I_{\cup }\left(Y_2\right)=I(Y_2;T)=U_2+R\hfill \\ I_{\cup }(Y_1,Y_2)=U_1+U_2+R\hfill \\ I_{\cup }\left(Y_{12}\right)=I(Y_{12};T)=U_1+U_2+R+S\hfill \end{array}\right.\iff \left\{\begin{array}{c}S=I(Y;T)-I_{\cup }(Y_1,Y_2)\hfill \\ U_1=I_{\cup }(Y_1,Y_2)-I(Y_2;T)\hfill \\ U_2=I_{\cup }(Y_1,Y_2)-I(Y_1;T)\hfill \\ R=I(Y_1;T)-U_1=I(Y_2;T)-U_2\hfill \end{array}\right..$$

3.2. Operational Interpretation

For the bivariate case, if $Y_1$ and $Y_2$ are conditionally independent given T (Figure 1b), then $p\left(y_1\right|t)$ and $p\left(y_2\right|t)$ (and $p\left(t\right)$) suffice to reconstruct the original joint distribution $p(y_1,y_2,t)$, which means the union of the parts is enough to reconstruct the whole, i.e., there is no synergy between $Y_1$ and $Y_2$. Conversely, a distribution generated by the DAG in Figure 1c does not satisfy conditional independence (given T), hence we expect positive synergy, as is the case for the XOR distribution, and indeed our measure yields 1 bit of synergy for this distribution. These two cases motivate the operational interpretation of our measure of synergy: it is the amount of information that is not captured by assuming conditional independence of the sources (given the target).

Recall, however, that some distributions are such that $I_q(Y;T)>I_p(Y;T)$, i.e., such that the union of the parts ‘outperforms’ the whole. What does this mean? It means that under q, $Y_1$ and $Y_2$ have more information about T than under p: the constructed distribution q, which drops the conditional dependence of $Y_1$ and $Y_2$ given T, reduces the uncertainty that Y has about T more than the original distribution p. In some cases, this may happen because the support of q is larger than that of p, which may lead to a reduction in uncertainty under q that cannot be achieved under p. In these cases, since we are decomposing $I_p(Y;T)$, we revert to saying that the union information that a set of variables has about T is equal to $I_p(Y;T)$, so that our measure satisfies the monotonicity axiom (later introduced in Definition 2). We will comment on this compromise between satisfying the monotonicity axiom and ignoring dependencies later.

3.3. General (Multivariate) Definition

To extend the proposed measure to an arbitrary number $n\ge 2$ of sources, we briefly recall the synergy lattice [18,36] and the union information semi-lattice [36]. For $n=3$, these two lattices are shown in Figure 2. For the sake of brevity, we will not address the construction of the lattices or the different orders between sources. We refer the reader to the work of Gutknecht et al. [36] for an excellent overview of the different lattices, the orders between sources, and the construction of different PID measures.

In the following, we use the term source to mean a subset of the variables $\{Y_1,\dots ,Y_n\}$, or a set of such subsets, we drop the curly brackets for clarity and refer to the different variables by their indices, as is common in most works on PID. The decomposition resulting from a measure of union information is not as direct to obtain as one obtained from a measure of redundant information, as the solution for the information atoms is not a Möbius inversion [14]. One must first construct the measure of synergy for source $\alpha$ by writing

$$\begin{array}{c}\hfill S^{\mathrm{CI}}(\alpha \to T)=I(Y;T)-I_{\cup }^{\mathrm{CI}}(\alpha \to T),\end{array}$$

(7)

which is the generalization of (6) for an arbitrary source $\alpha$. In the remainder of this paper, we will often omit “$\to T$” from the notation (unless it is explicitly needed), with the understanding that the target variable is always referred to as T. Also for simplicity, in the following, we identify the different agents that have access to different distributions as the distributions they have access to.

It is fairly simple to extend the proposed measure to an arbitrary number of sources, as illustrated in the following two examples.

Example 1. 

To compute $I_{\cup }^{CI}\left(\left(Y_1,Y_2\right),Y_3\right)$, agent $(Y_1,Y_2)$ knows $p(y_1,y_2|t)$, thus it can also compute, by marginalization, $p\left(y_1\right|t)$ and $p\left(y_2\right|t)$. On the other hand, agent $Y_3$ only knows $p\left(y_3\right|t)$. Recall that both agents also have access to $p\left(t\right)$. By sharing their conditionals, the agents can compute $q_1(y_1,y_2,y_3,t):=p\left(t\right)p(y_1,y_2|t)p\left(y_3\right|t)$, and also $q_2(y_1,y_2,y_3,t):=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t)p\left(y_3\right|t)$. After this, they may choose whichever distribution has the highest information about T, while still holding the view that any information gain larger than $I(Y;T)$ must be disregarded. Consequently, we write

$$I_{\cup }^{CI}\left((Y_1,Y_2),Y_3\right)=min\left\{I(Y;T),max\left\{I_{q_1}(Y;T),I_{q_2}(Y;T)\right\}\right\}.$$

Example 2. 

Slightly more complicated is the computation of $I_{\cup }^{CI}\left((Y_1,Y_2),(Y_1,Y_3),(Y_2,Y_3)\right)$. In this case, the three agents may compute four different distributions, two of which are the same $q_1$ and $q_2$ defined in the previous paragraph, and the other two are $q_3(y_1,y_2,y_3,t):=p\left(t\right)p(y_1,y_3|t)p\left(y_2\right|t)$, and $q_4(y_1,y_2,y_3,t):=p\left(t\right)p\left(y_1\right|t)p(y_2,y_3|t)$.

Given these insights, we propose the following measure of union information.

Definition 1. 

Let $A_1,\dots ,A_m$ be an arbitrary collection of $m\ge 1$ sources (recall sources may be subsets of variables). Assume that no source is a subset of another source and no source is a deterministic function of other sources (if there is, remove it). We define

$$I_{\cup }^{CI}(A_1,\dots ,A_m\to T)=min\left\{I(A;T),\underset{q\in \mathcal{Q}}{max}{I}_q(A;T)\right\},$$

where $A={\displaystyle \bigcup _{i=1}^m}{A}_i$ and Q is the set of all different distributions that the m agents can construct by combining their conditional distributions and marginalizations thereof.

For instance, in Example 1 above, $A=\{Y_1,Y_2\}\cup \left\{Y_3\right\}=\{Y_1,Y_2,Y_3\}$; in Example 2, $A=\{Y_1,Y_2\}\cup \{Y_1,Y_3\}\cup \{Y_2,Y_3\}=\{Y_1,Y_2,Y_3\}$. In Example 1, $\mathcal{Q}=\{q_1,q_2\}$, whereas in Example 2, $\mathcal{Q}=\{q_1,q_2,q_3,q_4\}$. We now justify the conditions in Definition 1 and the fact that they do not entail any loss of generality.

• The condition that no source is a subset of another source (which also excludes the case where two sources are the same) implies no loss of generality: if one source is a subset of another, say $A_i\subseteq A_j$, then $A_i$ may be removed without affecting either A or $\mathcal{Q}$, thus yielding the same value for $I_{\cup }^{\mathrm{CI}}$. The removal of source $A_i$ is also performed for measures of intersection information, but under the opposite condition: whenever $A_j\subseteq A_i$.
• The condition that no source is a deterministic function of other sources is slightly more nuanced. In our perspective, an intuitive and desired property of measures of both union and synergistic information is that their value should not change whenever one adds a source that is a deterministic function of sources that are already considered. We provide arguments in favor of this property in Section 4.2.1. This property may not be satisfied by computing $I_{\cup }^{\mathrm{CI}}$ without previously excluding such sources. For instance, consider $p(t,y_1,y_2,y_3)$, where $Y_1$ and $Y_2$ are two i.i.d. random variables following a Bernoulli distribution with parameter 0.5, $Y_3=Y_2$ (that is, $Y_3$ is deterministic function of $Y_2$), and $T=Y_1\mathrm{AND}{Y}_2$. Computing $I_{\cup }^{\mathrm{CI}}(Y_1,Y_2,Y_3)$ without excluding $Y_3$ (or $Y_2$) yields $I_{\cup }^{\mathrm{CI}}(Y_1,Y_2,Y_3)=I_q(Y_1,Y_2,Y_3;T)\approx 0.6810$ and $I_{\cup }^{\mathrm{CI}}(Y_1,Y_2)=I_q(Y_1,Y_2;T)\approx 0.5409$. This issue is resolved by removing deterministic sources before computing $I_{\cup }^{\mathrm{CI}}$.

We conclude this section by commenting on the monotonicity of our measure. Suppose we wish to compute the union information of sources $\{(Y_1,Y_2),Y_3\}$ and $\{Y_1,Y_2,Y_3\}$. PID theory demands that $I_{\cup }^{\mathrm{CI}}((Y_1,Y_2),Y_3)\ge I_{\cup }^{\mathrm{CI}}(Y_1,Y_2,Y_3)$ (monotonicity of union information). Recall our motivation for $I_{\cup }^{\mathrm{CI}}((Y_1,Y_2),Y_3)$: there are two agents, the first has access to $p(y_1,y_2|t)$ and the second to $p\left(y_3\right|t)$. The two agents assume conditional independence of their variables and construct $q^{\prime }(y_1,y_2,y_3,t)=p\left(t\right)p(y_1,y_2|t)p\left(y_3\right|t)$. The story is similar for the computation of $I_{\cup }^{\mathrm{CI}}(Y_1,Y_2,Y_3)$, in which case we have three agents that construct $q^{″}(y_1,y_2,y_3,t)=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t)p\left(y_3\right|t)$. Now, it may be the case that $I_{q^{\prime }}(Y;T)<I_{q^{″}}(Y;T)$; considering only these two distributions would yield $I_{\cup }^{\mathrm{CI}}((Y_1,Y_2),Y_3)<I_{\cup }^{\mathrm{CI}}(Y_1,Y_2,Y_3)$, contradicting monotonicity for measures of union information. To overcome this issue, for the computation of $I_{\cup }^{\mathrm{CI}}((Y_1,Y_2),Y_3)$—and other sources in general—the agent that has access to $p(y_1,y_2|t)$ must be allowed to disregard the conditional dependence of $Y_1$ and $Y_2$ on T, even if it holds in the original distribution p.

4. Properties of Measures of Union Information and Synergy

4.1. Extension of the Williams–Beer Axioms for Measures of Union Information

As Gutknecht et al. [36] rightfully notice, the so-called Williams–Beer axioms [1] can actually be derived from parthood distribution functions and the consistency equation [36]. Consequently, they are not really axioms but consequences of the PID framework. As far as we know, there has been no proposal in the literature for the equivalent of the Williams–Beer axioms (which refer to measures of redundant information) for measures of union information. In the following, we extend the Williams–Beer axioms to measures of union information and show that the proposed $I_{\cup }^{\mathrm{CI}}$ satisfies these axioms. Although we just argued against calling them axioms, we keep the designation Williams–Beer axioms because of its popularity. Although the following definition is not in the formulation of Gutknecht et al. [14], we suggest formally defining union information from the formulation of parthood functions as

$$I_{\cup }(Y_1,\dots ,Y_m;T)=\sum _{\exists i:f\left(Y_i\right)=1}\Pi \left(f\right),$$

(8)

where f refers to a parthood function and $\Pi \left(f\right)$ is the information atom associated with the parthood function f. Given this formulation, the following properties must hold.

Definition 2. 

Let $A_1,\dots ,A_m$ be an arbitrary number $m\ge 2$ of sources. A measure of union information $I_{\cup }$ is said to satisfy the Williams–Beer axioms for union information measures if it satisfies:

1. 

Symmetry: $I_{\cup }$ is symmetric in the $A_i$’s.

2. 

Self-redundancy: $I_{\cup }\left(A_i\right)=I(A_i;T)$.

3. 

Monotonicity: $I_{\cup }(A_1,\dots ,A_{m-1},A_m)\ge I_{\cup }(A_1,\dots ,A_{m-1})$.

4. 

Equality for monotonicity: $A_m\subseteq A_{m-1}\Rightarrow I_{\cup }(A_1,\dots ,A_{m-1},A_m)=I_{\cup }(A_1,\dots ,A_{m-1})$.

Theorem 1. 

$I_{\cup }^{CI}$ satisfies the Williams–Beer axioms for measures of union information given in Definition 2.

Proof. 

We address each of the axioms in turn.

1.

Symmetry follows from the symmetry of mutual information, which in turn is a consequence of the well-known symmetry of joint entropy.

2.

Self-redundancy follows from the fact that agent i has access to $p\left(A_i\right|T)$ and $p\left(T\right)$, which means that $p(A_i,T)$ is one of the distributions in the set $\mathcal{Q}$, which implies that $I_{\cup }\left(A_i\right)=I(A_i;T)$.

3.

To show that monotonicity holds, begin by noting that

$$I\left(\bigcup _{i=1}^m{A}_i;T\right)\ge I\left(\bigcup _{i=1}^{m-1}{A}_i;T\right),$$

due to the monotonicity of mutual information. Let ${\mathcal{Q}}_m$ be the set of distributions that the sources $A_1,\dots ,A_m$ can construct and ${\mathcal{Q}}_{m-1}$ that which the sources $A_1,\dots ,A_{m-1}$ can construct. Since ${\mathcal{Q}}_{m-1}\subseteq {\mathcal{Q}}_m$, it is clear that

$$\underset{q\in {\mathcal{Q}}_m}{max}{I}_q\left(\bigcup _{i=1}^m{A}_i;T\right)\ge \underset{q\in {\mathcal{Q}}_{m-1}}{max}{I}_q\left(\bigcup _{i=1}^{m-1}{A}_i;T\right).$$

Consequently,

$$min\left\{I\left(\bigcup _{i=1}^m{A}_i;T\right),\underset{q\in {\mathcal{Q}}_m}{max}{I}_q\left(\bigcup _{i=1}^m{A}_i;T\right)\right\}\ge min\left\{I\left(\bigcup _{i=1}^{m-1}{A}_i;T\right),\underset{q\in {\mathcal{Q}}_{m-1}}{max}{I}_q\left(\bigcup _{i=1}^{m-1}{A}_i;T\right)\right\},$$

which means monotonicity holds.

4.

Finally, the proof of equality for monotonicity is the same that was used above to show that the assumption that no source is a subset of another source entails no loss of generality. If $A_m\subseteq A_{m-1}$, then the presence of $A_m$ is irrelevant: $A={\displaystyle \bigcup _{i=1}^m}{A}_i={\displaystyle \bigcup _{i=1}^{m-1}}{A}_i$ and ${\mathcal{Q}}_m={\mathcal{Q}}_{m-1}$, which implies that $I_{\cup }(A_1,\dots ,A_{m-1},A_m)=I_{\cup }(A_1,\dots ,A_{m-1})$.

   □

4.2. Review of Suggested Properties: Griffith and Koch [16]

We now review properties of measures of union information and synergy that have been suggested in the literature in chronological order. The first set of properties was suggested by Griffith and Koch [16], with the first two being the following.

• Duplicating a predictor does not change synergistic information; formally,

  $$S(A_1,\dots ,A_m\to T)=S(A_1,\dots ,A_m,A_{m+1}\to T),$$

  where $A_{m+1}=A_i$, for some $i=1,\dots ,m$. Griffith and Koch [16] show that this property holds if the equality for monotonicity property holds for the “corresponding” measure of union information (“corresponding” in the sense of Equation (7)). As shown in the previous subsection, $I_{\cup }^{\mathrm{CI}}$ satisfies this property, and so does the corresponding synergy $S^{\mathrm{CI}}$.
• Adding a new predictor can decrease synergy, which is a weak statement. We suggest a stronger property: Following the monotonicity property, adding a new predictor cannot increase synergy, which is formally written as

  $$S(A_1,\dots ,A_m\to T)\ge S(A_1,\dots ,A_m,A_{m+1}\to T).$$

  This property simply follows from monotonicity for the corresponding measure of union information, which we proved above holds for $I_{\cup }^{\mathrm{CI}}$.

The next properties for any measure of union information were also suggested by Griffith and Koch [16]:

1.

Global positivity: $I_{\cup }(A_1,\dots ,A_m)\ge 0$.

2.

Self-redundancy: $I_{\cup }\left(A_i\right)=I(A_i;T)$.

3.

Symmetry: $I_{\cup }(A_1,\dots ,A_m)$ is invariant under permutations of $A_1,\dots ,A_m$.

4.

Stronger monotonicity: $I_{\cup }(A_1,\dots ,A_m)\le I_{\cup }(A_1,\dots ,A_m,A_{m+1})$, with equality if there is some $A_i$ such that $H\left(A_{m+1}\right|A_i)=0$.

5.

Target monotonicity: for any (discrete) random variables T and Z, $I_{\cup }(A_1,\dots ,A_m\to T)\le I_{\cup }(A_1,\dots ,A_m\to (T,Z))$.

6.

Weak local positivity: for $n=2$ the derived partial informations are non-negative. This is equivalent to

$$max\left\{I(Y_1;T),I(Y_2;T)\right\}\le I_{\cup }(Y_1,Y_2)\le I(Y;T).$$

7.

Strong identity: $I_{\cup }(T\to T)=H\left(T\right)$.

We argued before that self-redundancy and symmetry are properties that follow trivially from a well-defined measure of union information [36]. In the following, we discuss in more detail properties 4 and 5, and return to the global positivity property later.

4.2.1. Stronger Monotonicity

Property 4 in the above list was originally called monotonicity by Griffith and Koch [16]; we changed its name because we had already defined monotonicity in Definition 2, a weaker condition than stronger monotonicity. The proposed inequality clearly follows from the monotonicity of union information (the third Williams–Beer axiom). Now, if there is some $A_i$ such that $H\left(A_{m+1}\right|A_i)=0$ (equivalently, if $A_{m+1}$ is a deterministic function of $A_i$), Griffith and Koch [16] suggest that we must have equality. Recall Axiom 4 (equality for monotonicity) in the extension of the WB axioms (Definition 2). It states that equality must hold if $A_m\subseteq A_{m-1}$. In this context, $A_m$ and $A_{m-1}$ are sets of random variables, for example, $A_m=\{Y_1,Y_2\}$ and $A_{m-1}=\{Y_1,Y_2,Y_3\}$. There is a different point of view we may take. The only way that $A_m$ is a subset of $A_{m-1}$ is if $A_m$, when viewed as a random vector (in this case, write $A_m=(Y_1,Y_2)$ and $A_{m-1}=(Y_1,Y_2,Y_3)$), is a subvector of $A_{m-1}$. A subvector of a random vector is a deterministic function, and no information gain can come from applying a deterministic function to a random vector. As such, there is no information gain when one considers $A_m$, a function of $A_{m-1}$, if one already has access to $A_{m-1}$. Griffith and Koch [16] argue similarly, there is no information gain by considering $A_{m+1}$—a function of $A_i$—in addition to $A_i$. In conclusion, considering the ‘equality for monotonicity’ strictly through a set-inclusion perspective, stronger monotonicity does not follow. On the other hand, extending the idea of set inclusion to the more general context of functions of random variables, then stronger monotonicity follows, because $\left\{A_{m+1}\right\}=\left\{f\left(A_i\right)\right\}$ is a subset of $\left\{A_i\right\}$, hence there is no information gain by considering $A_{m+1}=f\left(A_i\right)$ in addition to $A_i$. As such, we obtain $I_{\cup }(A_1,\dots ,A_m)=I_{\cup }(A_1,\dots ,A_m,A_{m+1})$. Consequently, it is clear that stronger monotonicity must hold for any measure of union information. We, thus, argue in favor of extending the concept of subset inclusion of property 4 in Definition 2 to include the concept of deterministic functions.

4.2.2. Target Monotonicity

Let us move on to target monotonicity, which we argue should not hold in general. This precise same property was suggested, but for a measure of redundant information, by Bertschinger et al. [19]; they argue that a measure of redundant information should satisfy

$$I_{\cap }(A_1,\dots ,A_m\to T)\le I_{\cap }(A_1,\dots ,A_m\to (T,Z)),$$

for any discrete random variable Z, as they argue that this property ‘captures the intuition that if $A_1,\dots ,A_m$ share some information about T, then at least the same amount of information is available to reduce the uncertainty about the joint outcome of $(T,Z)$’. Since most PID approaches have been built upon measures of redundant information, it is simpler to refute this property. Consider $I_{\cap }^{\mathrm{d}}$, which we argue is one of the most well-motivated and accepted measures of redundant information (as defined in (2)): it satisfies the WB axioms, it is based on the famous Blackwell channel preorder—thus inheriting a well-defined and rigorous operation interpretation—it is based on channels, just as $I(X;T)$ was originally motivated based on channels, and is defined for any number of input variables, which is more than most PID measures can accomplish. Consider also the distribution presented in

Table 4. Counter-example distribution for target monotonicity.

T	Z	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}(\mathit{t},\mathit{z},{\mathit{y}}_1,{\mathit{y}}_2)$
0	(0,0)	0	0	1/4
0	(0,1)	0	1	1/4
0	(1,0)	1	0	1/4
1	(1,1)	1	1	1/4

, which satisfies $T=Y_1\mathrm{AND}{Y}_2$ and $Z=(Y_1,Y_2)$.

From a game theory perspective, since neither agent ($Y_1$ or $Y_2$) has an advantage when predicting T (because the channels that each agent has access to have the same conditional distributions), neither agent has any unique information. Moreover, redundancy—as computed by $I_{\cap }^{\mathrm{d}}(Y_1,Y_2\to T)$—evaluates to approximately 0.311. However, when considering the pair $(T,Z)$, the structure that was present in T is now destroyed, in the sense that now there is no degradation order between the channels that each agent has access to. Note that $p((t,z),y_1,y_2)$ is a relabeling of the COPY distribution. As such, $I_{\cap }^{\mathrm{d}}\left(Y_1,Y_2\to (T,Z)\right)=0<I_{\cap }^{\mathrm{d}}(Y_1,Y_2\to T)$, contradicting the property proposed by Bertschinger et al. [19].

For a similar reason, we believe that this property should not hold for a general measure of union information, even if the measure satisfies the extension of the Williams–Beer axioms, as our proposed measure does. For instance, consider the distribution presented in

Table 5. Counter-example distribution for target monotonicity.

T	Z	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}(\mathit{t},\mathit{z},{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	1	0	0.419
1	1	2	1	0.203
2	1	3	0	0.007
0	0	3	1	0.346
2	2	4	4	0.025

.

This distribution yields $I_{\cup }^{\mathrm{CI}}(Y_1,Y_2\to T)\approx 0.91>0.90\approx I_{\cup }^{\mathrm{CI}}(Y_1,Y_2\to (T,Z))$, meaning target monotonicity does not hold. This happens because although $I_p(Y;T)\le I_p(Y;T,Z)$, it is not necessarily true that $I_q(Y;T)\le I_q(Y;T,Z)$. The union information measure derived from the degradation order between channels, defined as the ‘dual’ of (2), also agrees with our conclusion [22]. For the distribution in Table 4 we have $I_{\cup }^{\mathrm{d}}(Y_1,Y_2\to T)\approx 0.331>0=I_{\cup }^{\mathrm{d}}\left(Y_1,Y_2\to (T,Z)\right)$, for the same reason as above: considering $(T,Z)$ as the target variable destroys the structure present in T. We agree with the remaining properties suggested by Griffith and Koch [16] and we will address those later.

4.3. Review of Suggested Properties: Quax et al. [37]

Moving on to additional properties, Quax et al. [37] suggest the following properties for a measure of synergy:

1.

Non-negativity: $S(A_1,\dots ,A_m\to T)\ge 0$.

2.

Upper-bounded by mutual information: $S(Y\to T)\le I(Y;T)$.

3.

Weak symmetry: $S(A_1,\dots ,A_m\to T)$ is invariant under any reordering of $A_1,\dots ,A_m$.

4.

Zero synergy about a single variable: $S(Y_i\to T)=0$ for any $i\in \{1,\dots ,n\}$.

5.

Zero synergy in a single variable: $S(Y\to Y_i)=0$ for any $i\in \{1,\dots ,n\}$.

Let us comment on the proposed ‘zero synergy’ properties (4 and 5) under the context of PID. Property 4 seems to have been proposed with the rationale that synergy can only exist for at least two sources, which intuitively makes sense, as synergy is often defined as ‘the information that is present in the pair, but that is not retrievable from any individual variable’. However, because of the way a synergy-based PID is constructed—or weak-synergy, as Gutknecht et al. [36] call it—synergy must be defined as in (7), so that, for example, in the bivariate case, $S(Y_1\to T):=I(Y;T)-I_{\cup }(Y_1\to T)=I(Y_2;T|Y_1)$, because of self-redundancy of union information and the chain rule of mutual information [12], and since $I(Y_2;T|Y_1)$ is in general larger than 0, we reject the property ‘zero synergy about a single variable’. We note that our rejection of this property is based on the PID perspective. There may be other areas of research where it makes sense to demand that any (single) random variable alone has no synergy about any target, but under the PID framework, this must not happen, particularly so that we obtain a valid information decomposition.

Property 5, ‘zero synergy in a single variable’, on the other hand, must hold because of self-redundancy. That is because, for any $i\in \{1,\dots ,n\}$, $S(Y\to Y_i):=I(Y;Y_i)-I_{\cup }(Y\to Y_i)=I(Y_i;Y_i)-I(Y;Y_i)=H\left(Y_i\right)-H\left(Y_i\right)=0$.

4.4. Review of Suggested Properties: Rosas et al. [38]

Based on the proposals of Griffith et al. [24], Rosas et al. [38] suggested the following properties for a measure of synergy:

• Target data processing inequality: if $Y-T_1-T_2$ is a Markov chain, then $S(Y\to T_1)\ge S(Y\to T_2)$.
• Channel convexity: $S(Y\to T)$ is a convex function of $P\left(T\right|Y)$ for a given $P\left(Y\right)$.

We argue that a principled measure of synergy does not need to satisfy these properties (in general). Consider the distribution presented in

Table 6. $T_1=$ COPY, $T_2=$ XOR.

${\mathit{T}}_2$	${\mathit{T}}_1$	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}({\mathit{t}}_2,{\mathit{t}}_1,{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	0	0	1/4
1	1	0	1	1/4
1	2	1	0	1/4
0	3	1	1	1/4

, in which $T_1$ is a relabeling of the COPY distribution and $T_2=Y_1\mathrm{xor}{Y}_2$.

Start by noting that since $T_2$ is a deterministic function of $T_1$, then $Y-T_1-T_2$ is a Markov chain. Since $Y_1\perp Y_2|T_1$, our measure $S^{\mathrm{CI}}(Y_1,Y_2\to T_1)=I(Y;T_1)-I_{\cup }^{\mathrm{CI}}(Y_1,Y_2\to T_1)=0$ leads to zero synergy. On the other hand, $S^{\mathrm{CI}}(Y_1,Y_2\to T_2)=1$, contradicting the first property suggested by Rosas et al. [38]. This happens because $Y_1{\cancel{\perp }}_p{Y}_2|T_2$, so synergy is positive. The loss of conditional independence of the inputs (given the target) when one goes from considering the target $T_1$ to $T_2$ is the reason why synergy increases. It can be easily seen that $S^d$, the measure of synergy derived from Kolchinsky’s proposed union information measure $I_{\cup }^d$ [22], agrees with this. A simpler way to see this is by noticing that the XOR distribution must yield 1 bit of synergistic information, and many PID measures do not yield 1 bit of synergistic information for the COPY distribution.

The second suggested property argues that synergy should be a convex function of $P\left(T\right|Y)$, for fixed $P\left(Y\right)$. Our measure of synergy does not satisfy this property, even though it is derived from a measure of union information that satisfies the extension of the WB axioms. For instance, consider the XOR distribution with one extra outcome. We introduce it in

Table 7. Adapted XOR distribution.

T	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	0	$r/4$
1	0	0	$(1-r)/4$
1	1	0	1/4
1	0	1	1/4
0	1	1	1/4

and parameterize it using $r=p(T=0|Y=(0,0))\in [0,1]$. Notice that this modification does not affect $P\left(Y\right)$.

Synergy, as measured by $S^{\mathrm{CI}}(Y_1,Y_2\to T)$, is maximized when r equals 1 (the distribution becomes the standard XOR) and minimized when r equals 0. We do not see an immediate reason as to why a general synergy function should be convex in $p\left(t\right|y)$, or why it should have a unique minimizer as a function of r. Recall that a function S is convex if $\forall t\in [0,1],\forall x_1,x_2\in D$, we have

$$S(t{x}_1+(1-t)x_2)\le tS\left(x_1\right)+(1-t)S\left(x_2\right).$$

In the following, we slightly abuse the notation of the input variables of a synergy function. Our synergy measure $S^{\mathrm{CI}}$, when considered as a function of r, does not satisfy this inequality. For the adapted XOR distribution, take $t=0.5$, $x_1=0$, and $x_2=0.5$. We have

$$S^{\mathrm{CI}}(0.5\times 0+0.5\times 0.5)=S^{\mathrm{CI}}\left(0.25\right)\approx 0.552$$

and

$$0.5\times S^{\mathrm{CI}}\left(0\right)+0.5\times S^{\mathrm{CI}}\left(0.5\right)\approx 0.5\times 0.270+0.5\times 0.610\approx 0.440,$$

contradicting the property of channel convexity. $S^{\mathrm{d}}$ agrees with this. We slightly change $p\left(y\right)$ in the above distribution to obtain a new distribution, which we present in

Table 8. Adapted XOR distribution v2.

T	${\mathit{Y}}_1$	${\mathit{Y}}_2$	$\mathit{p}(\mathit{t},{\mathit{y}}_1,{\mathit{y}}_2)$
0	0	0	$r/10$
1	0	0	$(1-r)/10$
1	1	0	4/10
1	0	1	4/10
0	1	1	1/10

.

This distribution does not satisfy the convexity inequality, since

$$S^{\mathrm{d}}(0.5\times 0+0.5\times 0.5)\approx 0.338>0.3095\approx 0.5S^{\mathrm{d}}\left(0\right)+0.5S^{\mathrm{d}}\left(0.5\right).$$

This can be easily seen since $K^{\left(1\right)}=K^{\left(2\right)}$ for any $r\in [0,1]$, hence we may choose $K^Q=K^{\left(1\right)}$ to compute $S^{\mathrm{d}}=I(Y;T)-I(Q;T)$, which is not convex for this particular distribution. To conclude this section, we present a plot of $S^{\mathrm{CI}}(Y_1,Y_2)$ and $S^{\mathrm{d}}(Y_1,Y_2)$ as a function of r in Figure 3, for the distribution presented in Table 7.

With this we do not mean that these properties should not hold for an arbitrary PID. We are simply showing that some properties must be satisfied in the context of PID (such as the WB axioms), whereas other properties are not necessary for PID (such as the previous two properties).

4.5. Relationship with the Extended Williams–Beer Axioms

We now prove which of the introduced properties are implied by the extension of the Williams–Beer axioms for measures of union information. In what follows, assume that the goal is to decompose the information present in the distribution $p(y,t)=p(y_1,\dots ,y_n,t)$.

Theorem 2. 

Let $I_{\cup }$ be a measure of union information that satisfies the extension of the Williams–Beer axioms (symmetry, self-redundancy, monotonicity, and equality for monotonicity) for measures of union information as in Definition (2). Then, $I_{\cup }$ also satisfies the following properties of Griffith and Koch [16]: global positivity, weak local positivity, strong identity, “duplicating a predictor does not change synergistic information”, and “adding a new predictor cannot increase synergy”.

Proof. 

We argued before that the last two properties follow from the definition of a measure of union information. Global positivity is a direct consequence of monotonicity and the non-negativity of mutual information: $I_{\cup }(A_1,\dots ,A_m)\ge I_{\cup }\left(A_1\right)=I(A_1;T)\ge 0$.

Weak local positivity holds because monotonicity and self-redundancy imply that $I_{\cup }(Y_1,Y_2)\ge I_{\cup }\left(Y_1\right)=I(Y_1;T)$, as well as $I_{\cup }(Y_1,Y_2)\ge I(Y_2;T)$, hence $max\{I(Y_1;T),I(Y_2;T)\}$$\le I_{\cup }(Y_1,Y_2)$. Moreover, $I_{\cup }(Y_1,Y_2)\le I_{\cup }(Y_1,Y_2,Y_{12})=I_{\cup }\left(Y_{12}\right)=I(Y;T)$.

Strong identity follows trivially from self-redundancy, since $I_{\cup }(T\to T)=I(T;T)=H\left(T\right)$.    □

Theorem 3. 

Consider a measure of union information that satisfies the conditions of Theorem 2. If synergy is defined as in Equation (7), it satisfies the following properties of [37]: non-negativity, upper-bounded by mutual information, weak symmetry, and zero synergy in a single variable.

Proof. 

Non-negativity of synergy and upper-bounded by mutual information follow from the definition of synergy and from the fact that for whichever source $(A_1,\dots ,A_m)$, with $m\ge 1$, we have that $I_{\cup }\left(A_1,\dots ,A_m\to T\right)\le I(Y;T)$.

Weak symmetry follows trivially from the fact that both $I\left(Y;T\right)$ and $I_{\cup }\left(A_1,\dots ,A_m\to T\right)$ are symmetric in the relevant arguments.

Finally, zero synergy in a single variable follows from self-redundancy together with the definition of synergy, as shown above.    □

5. Previous Measures of Union Information and Synergy

We now review other measures of union information and synergy proposed in the literature. For the sake of brevity, we will not recall all their definitions, only some important conclusions. We suggest the interested reader consult the bibliography for more information.

5.1. Qualitative Comparison

Griffith and Koch [16] review three previous measures of synergy:

• $S^{\mathrm{WB}}$, derived from $I_{\cap }^{\mathrm{WB}}$, the original redundancy measure proposed by Williams and Beer [1], using the IEP;
• the whole-minus-sum (WMS) synergy, $S^{\mathrm{WMS}}$;
• the correlational importance synergy, $S^{\Delta I}$.

These synergies can be interpreted as resulting directly from measures of union information; that is, they are explicitly written as $S(\alpha \to T)=I(Y;T)-I_{\cup }(\alpha \to T)$, where $I_{\cup }$ may not necessarily satisfy our intuitions of a measure of union information, as in Definition 2, except for $S^{\Delta I}$, which has the form of a Kullback–Leibler divergence.

Griffith and Koch [16] argue that $S^{\mathrm{WB}}$overestimates synergy, which is not a surprise, as many authors criticized $I_{\cap }^{\mathrm{WB}}$ for not measuring informational content, only informational values [13]. The WMS synergy, on the other hand, which can be written as a difference of total correlations, can be shown to be equal to the difference between synergy and redundancy for $n=2$, which is not what is desired in a measure of synergy. For $n>2$, the authors show that the problem becomes even more exacerbated: $S^{\mathrm{WMS}}$ equals synergy minus the redundancy counted multiple times, which is why the authors argue that $S^{\mathrm{WMS}}$underestimates synergy. Correlational importance, $S^{\Delta I}$, is known to be larger than $I(Y;T)$ for some distributions, excluding it from being an appropriately interpretable measure of synergy.

Faced with these limitations, Griffith and Koch [16] introduce their measure of union information, which they define as

$$\begin{array}{cc}\hfill I_{\cup }^{\mathrm{VK}}(A_1,\dots ,A_m\to T):=& \underset{p^{*}}{min}{I}_{p^{*}}\left(\bigcup _{i=1}^m{A}_i;T\right)\hfill \\ \hfill & \mathrm{s}.\mathrm{t}.p^{*}(A_i,T)=p(A_i,T),i=1,\dots ,m,\hfill \end{array}$$

(9)

where the minimization is over joint distributions of $A_1,\dots ,A_m,T$, alongside the derived measure of synergy $S^{\mathrm{VK}}(\alpha \to T)=I(Y;T)-I_{\cup }^{\mathrm{VK}}(\alpha \to T)$. This measure quantifies union information as the least amount of information that source $\alpha$ has about T when the source–target marginals (as determined by $\alpha$) are fixed by p. Griffith and Koch [16] also established the following inequalities for the synergistic measures they reviewed:

$$\begin{array}{c}\hfill max\left\{0,S^{\mathrm{WMS}}(\alpha \to T)\right\}\le S^{\mathrm{VK}}(\alpha \to T)\le S^{\mathrm{WB}}(\alpha \to T)\le I\left(\bigcup _{i=1}^m{A}_i;T\right),\end{array}$$

(10)

where $\alpha =(A_1,\dots ,A_m)$. At the time, Griffith and Koch [16] did not provide a way to analytically compute their measure. Later, Kolchinsky [22] showed that the measure of union information derived from the degradation order, $I_{\cup }^{\mathrm{d}}$, is equivalent to $I_{\cup }^{\mathrm{VK}}$, and provided a way to compute it. For this reason, we will only consider $I_{\cup }^{\mathrm{d}}$.

After the work of Griffith and Koch [16] in 2014, we are aware of only three other suggested measures of synergy:

• $S^{\mathrm{MSRV}}$, proposed by Quax et al. [37], where MSRV stands for maximally synergistic random variable;
• synergistic disclosure, $S^{\mathrm{SD}}$, proposed by Rosas et al. [38];
• $S^d$, proposed by Kolchinsky [22].

The first two proposals do not define synergy via a measure of union information. They define synergy through an auxiliary random variable, Z, which has positive information about the whole—that is, $I(Z;Y)>0$—but no information about any of the parts—that is, $I(Z;Y_i)=0,i=1,\dots ,n$. While this property has an appealing operational interpretation, we believe that it is too restrictive; that is, we believe that information can be synergistic, even if it provides some positive information about some part of Y.

The authors of $S^{\mathrm{MSRV}}$ show that their proposed measure is incompatible with PID and that it cannot be computed for all distributions, as it requires the ability to compute orthogonal random variables, which is not always possible [37]. A counter-intuitive example for the value of this measure can be seen for the AND distribution, defined by $T=Y_1\mathrm{AND}{Y}_2$, with $Y_1$ and $Y_2$ i.i.d. taking values in $\{0,1\}$ with equal probability. In this case, $S^{\mathrm{MSRV}}=0.5$, a value that we argue is too large, because whenever $Y_1$ (respectively $Y_2$) is 0, then T does not depend on $Y_2$ (respectively $Y_1$) (which happens with probability 0.75). Consequently, $S^{\mathrm{MSRV}}/I(Y;T)\approx 0.5/0.811\approx 0.617$ may be too large of a synergy ratio for this distribution. As the authors note, the only other measure that agrees with $S^{\mathrm{MSRV}}$ for the AND distribution is $S^{\mathrm{WB}}$, which Griffith and Koch [16] argued also overestimates synergy.

Concerning $S^{\mathrm{SD}}$, we do not have any criticism, except for the one already pointed out by Gutknecht et al. [36]: they note that the resulting decomposition from $S^{\mathrm{SD}}$ is not a standard PID, in the sense that it does not satisfy a consistency equation (see [36] for more details), which implies that ‘… the atoms cannot be interpreted in terms of parthood relations with respect to mutual information terms …. For example, we do not obtain any atoms interpretable as unique or redundant information in the case of two sources’ [36]. Gutknecht et al. [36] suggest a very simple modification to the measure so that it satisfies the consistency equation.

For the AND distribution, $S^{\mathrm{SD}}$ evaluates to approximately $0.311$, as does $S^{\mathrm{d}}$, whereas our measure yields $S^{\mathrm{CI}}\approx 0.270$, as the information that the parts cannot obtain when they combine their marginals, under distribution q. This shows that these four measures are not equivalent.

5.2. Quantitative Comparison

Griffith and Koch [16] applied the synergy measures they reviewed to other distributions. We show their results below in

Table 9. Application of the measures reviewed in Griffith and Koch [16] ($S^{\mathrm{WB}}$, $S^{\mathrm{WMS}}$, and $S^{\Delta I}$), $S^{\mathrm{SD}}$ introduced by Rosas et al. [38], $S^{\mathrm{d}}$ introduced by Kolchinsky [22], and our measure of synergy $S^{\mathrm{CI}}$ to different distributions. The bottom four distributions are trivariate. We write DNF to mean that a specific computation did not finish within 10 min.

Example	${\mathit{S}}^{\mathbf{WB}}$	${\mathit{S}}^{\mathbf{WMS}}$	${\mathit{S}}^{\mathbf{\Delta }\mathit{I}}$	${\mathit{S}}^{\mathbf{d}}$	${\mathit{S}}^{\mathbf{SD}}$	${\mathit{S}}^{\mathbf{CI}}$
XOR	1	1	1	1	1	1
AND	0.5	0.189	0.104	0.5	0.311	0.270
COPY	1	0	0	0	1	0
RDNXOR	1	0	1	1	1	1
RDNUNQXOR	2	0	1	1	DNF	1
XORDUPLICATE	1	1	1	1	1	1
ANDDUPLICATE	0.5	−0.123	0.038	0.5	0.311	0.270
XORLOSES	0	0	0	0	0	0
XORMULTICOAL	1	1	1	1	DNF	1

and compare them with the synergy resulting from our measure of union information, $S^{\mathrm{CI}}$, with the measure of Rosas et al. [38], $S^{\mathrm{SD}}$, and that of Kolchinsky [22], $S^{\mathrm{d}}$. Since the code for the computation of $S^{\mathrm{MSRV}}$ is no longer available online, we do not present it.

We already saw the definition of the AND, COPY, and XOR distributions. The XORDUPLICATE and ANDDUPLICATE are built from the XOR and the AND distributions by inserting a duplicate source variable $Y_3=Y_2$. The goal is to test if the presence of a duplicate predictor impacts the different synergy measures. The definitions of the remaining distributions are presented in Appendix A. Some of these are trivariate, and for those we compute synergy as

$$\begin{array}{c}\hfill S(Y_1,Y_2,Y_3\to T)=I(Y_1,Y_2,Y_3;T)-I_{\cup }(Y_1,Y_2,Y_3\to T),\end{array}$$

(11)

unless the synergy measure is directly defined (as opposed to being defined via a union information measure). We now comment on the results. It should be noted that Kolchinsky [22] suggested that unique information $U_1$ and $U_2$ should be computed from measures of redundant information, and excluded information $E_1$ and $E_2$ should be computed from measures of union information, as in our case. However, since we will only present the decompositions for the bivariate case and in this case $E_1=U_2$ and $E_2=U_1$, we present the results considering unique information, as is mostly performed in the literature.

• XOR yields $I(Y;T)=1$. The XOR distribution is the hallmark of synergy. Indeed, the only solution of (1) is $(S,R,U_1,U_2)=(1,0,0,0)$, and all of the above measures yield 1 bit of synergy.
• AND yields $I(Y;T)\approx 0.811$. Unlike XOR, there are multiple solutions for (1), and none is universally agreed upon, since different information measures capture different concepts of information.
• COPY yields $I(Y;T)=2$. Most PID measures argue one of two different possibilities for this distribution. They suggest that the solution is either $(S,R,U_1,U_2)=(1,1,0,0)$ or $(0,0,1,1)$. Our measure suggests that all information flows uniquely from each source.
• RDNXOR yields $I(Y;T)=2$. In words, this distribution is the concatenation of two XOR ‘blocks’, each of which has its own symbols, and not allowing the two blocks to mix. That is, both $Y_1$ and $Y_2$ can determine in which XOR block the resulting value T will be—which intuitively means that they both have this information, meaning it is redundant—but neither $Y_1$ nor $Y_2$ have information about the outcome of the XOR operation—as is expected in the XOR distribution—which intuitively means that such information must be synergistic. All measures except $S^{\mathrm{WMS}}$ agree with this.
• RDNUNQXOR yields $I(Y;T)=4$. According to Griffith and Koch [16], it was constructed to carry 1 bit of each information type. Although the solution is not unique, it must satisfy $U_1=U_2$. Indeed, our measure yields the solution $(S,R,U_1,U_2)=(1,1,1,1)$, like most measures except $S^{\mathrm{WB}}$ and $S^{\mathrm{WMS}}$. This confirms the intuition by Griffith and Koch [16] that $S^{\mathrm{WB}}$ and $S^{\mathrm{WMS}}$ overestimate and underestimate synergy, respectively. In fact, in the decomposition resulting from $S^{\mathrm{WB}}$, there are 2 bits of synergy and 2 bits of redundancy, which we argue cannot be the case, as this would imply that $U_1=U_2=0$, and given the construction of this distribution, it is clear that there is some unique information since, unlike in RDNXOR, the XOR blocks are allowed to mix, thus $(T,Y_1,Y_2)=(1,0,1)$ is a possible outcome, but so is $(T,Y_1,Y_2)=(2,0,2)$. That is not the case with RDNXOR. On the other hand, $S^{\mathrm{WMS}}$ yields zero synergy and redundancy, with $U_1$ and $U_2$ each evaluating to 2 bits. Since this distribution is a mix of blocks satisfying a relation of the form $T=Y_1\mathrm{xor}{Y}_2$, we argue that there must be some non-null amount of synergy, which is why we claim that $S^{\mathrm{WMS}}$ is not valid.
• XORDUPLICATE yields $I(Y;T)=1$. All measures correctly identify that the duplication of a source should not change synergy, at least for this particular distribution.
• ANDDUPLICATE yields $I(Y;T)\approx 0.811$. Unlike in the previous example, both $S^{\mathrm{WMS}}$ and $S^{\Delta I}$ yield a change in their synergy value. This is a shortcoming, since duplicating a source should not increase either synergy or union information. The other measures are not affected by the duplication of a source.
• XORLOSES yields $I(Y;T)=1$. Its distribution is the same as XOR but with a new source $Y_3$ satisfying $T=Y_3$. As such, since $Y_3$ uniquely determines T, we expect no synergy. All measures agree with this.
• XORMULTICOAL yields $I(Y;T)=1$. Its distribution is such that any pair $(Y_i,Y_j)$, $i,j=1,2,3,i\ne j$ is able to determine T with no uncertainty. All measures agree that the information present in this distribution is purely synergistic.

From these results, we agree with Griffith and Koch [16] that $S^{\mathrm{WB}}$, $S^{\mathrm{WMS}}$, and $S^{\Delta I}$ are not good measures of synergy: they do not satisfy many of our intuitions and overestimate synergy, not being invariant to duplicate sources or taking negative values. For these reasons, and those presented in Section 5, we reject those measures of synergy. In the next section, we comment on the remaining measures $S^{\mathrm{d}}$, $S^{\mathrm{SD}}$, and $S^{\mathrm{CI}}$.

5.3. Relation to Other PID Measures

Kolchinsky [22] introduced $I_{\cup }^{\mathrm{d}}$ and showed that this measure is equivalent to $I_{\cup }^{\mathrm{VK}}$ [16] and to $I_{\cup }^{\mathrm{BROJA}}$ [15], in the sense that the three of them achieve the same optimum value [22]. The multivariate extension of $I_{\cup }^{\mathrm{BROJA}}$ was proposed by Griffith and Koch [16], defined as

$$I_{\cup }^{\mathrm{BROJA}}(A_1,\dots ,A_m\to T):=\underset{\widetilde{A_1},\dots ,\widetilde{A_m}}{min}I(\widetilde{A_1},\dots ,\widetilde{A_m};T)\mathrm{such}\mathrm{that}\forall iP(\widetilde{A_i},T)=P(A_i,T),$$

which we present because it makes it clear what conditions are enforced upon the marginals. There is a relation between $I_{\cup }^{\mathrm{BROJA}}(A_1,\dots ,A_m)=I_{\cup }^{\mathrm{d}}(A_1,\dots ,A_m)$ and $I_{\cup }^{\mathrm{CI}}(A_1,\dots ,A_m)$ whenever the sources $A_1,\dots A_m$ are singletons. In this case, and only in this case, the set $\mathcal{Q}$ involved in the computation of $I_{\cup }^{\mathrm{CI}}(A_1,\dots ,A_m)$ has only one element: $q(t,a_1,\dots ,a_m)=p\left(t\right)p\left(a_1\right|t)\dots p\left(a_m\right|t)$. Since this distribution, as well as the original distribution p, are both admissible points in $I_{\cup }^{\mathrm{d}}$, we have that $I_{\cup }^{\mathrm{d}}\le I_{\cup }^{\mathrm{CI}}$, which implies that $S^{\mathrm{d}}\ge S^{\mathrm{CI}}$. On the other hand, if there is at least one source $A_1,\dots ,A_m$ that is not a singleton, the measures are not trivially comparable. For example, suppose we wish to compute $I_{\cup }((Y_1,Y_2),(Y_2,Y_3))$. We know that the solution of $I_{\cup }^{\mathrm{d}}((Y_1,Y_2),(Y_2,Y_3))$ is a distribution $p^{*}$ whose marginals $p^{*}(y_1,y_2,t)$ and $p^{*}(y_2,y_3,t)$ must coincide with the marginals under the original p. However, in the computation of $I_{\cup }^{\mathrm{CI}}((Y_1,Y_2),(Y_2,Y_3))$, it may be the case that the solution $p^{*}$ of $I_{\cup }^{\mathrm{d}}((Y_1,Y_2),(Y_2,Y_3))$ is not in the set $\mathcal{Q}$, involved in the computation of $I_{\cup }^{\mathrm{CI}}$, and it achieves a lower mutual information with T. That is, it might be the case that $I_{p^{*}}(Y;T)<I_q(Y;T)$, for all $q\in \mathcal{Q}$. In such a case, we would have $I_{\cup }^{\mathrm{d}}>I_{\cup }^{\mathrm{CI}}$.

It is convenient to be able to upper-bound certain measures with other measures. For example, Gomes and Figueiredo [26] (see that paper for the definitions of these measures) showed that for any source $(A_1,\dots ,A_m),m\ge 1$,

$$I_{\cap }^{\mathrm{d}}(A_1,\dots ,A_m)\le I_{\cap }^{\ln }(A_1,\dots ,A_m)\le I_{\cap }^{\mathrm{mc}}(A_1,\dots ,A_m).$$

However, we argue that the inability to draw such strong conclusions (or bounds) is a positive aspect of PID. This is because there are many different ways to define the information (be it redundant, unique, union, etc.) that one wishes to capture. If one could trivially relate all measures, it would mean that it would be possible to know a priori how those measures would behave. Consequently, this would imply the absence of variability/freedom in how to measure different information concepts, as those measures would capture, non-equivalent but similar types of information, as they would all be ordered. It is precisely because one cannot order different measures of information trivially that PID provides a rich and complex framework to distinguish different types of information, although we believe that PID is still in its infancy.

James et al. [17] introduced a measure of unique information, which we recall now. In the bivariate case—i.e., consider $p(y_1,y_2,t)$—let q be the maximum entropy distribution that preserves the marginals $p(y_1,t)$ and $p(y_2,t)$, and let r be the maximum entropy distribution that preserves the marginals $p(y_1,t)$, $p(y_2,t)$, and $p(y_1,y_2)$. Although there is no closed form for r, which has to be computed using an iterative algorithm [39], it may be shown that the solution for q is $q(y_1,y_2,t)=p\left(t\right)p\left(y_1\right|t)p\left(y_2\right|t)$ (see, e.g., [17]). This is the same distribution q that we consider for the bivariate decomposition (3). James et al. [17] suggest defining unique information $U_i$ as the least change (in sources–target mutual information) that involves the addition of the $(Y_i,T)$ marginal constraint, that is,

$$\begin{array}{c}\hfill U_1=min\{I_q(Y_1;T|Y_2),I_r(Y_1;T|Y_2)\},\end{array}$$

(12)

and analogously for $U_2$. They show that their measure yields a non-negative decomposition for the bivariate case. Since $I(Y_1;T|Y_2)=S+U_1$, some algebra leads to

$$\begin{array}{c}\hfill S^{\mathrm{dep}}=I(Y;T)-min\{I_q(Y;T),I_r(Y;T)\},\end{array}$$

(13)

where $S^{\mathrm{dep}}$ is the synergy resulting from the decomposition of James et al. [17] in the bivariate case. Recall that our measure of synergy for the bivariate case is given by

$$\begin{array}{c}\hfill S^{\mathrm{CI}}=I(Y;T)-min\{I_q(Y;T),I_p(Y;T)\}.\end{array}$$

(14)

The similarity is striking. Computing $S^{\mathrm{dep}}$ for the bivariate distributions in Table 9 shows that it coincides with the decomposition given by our measure, although this is not the case in general. We could not obtain $S^{\mathrm{dep}}$ for the RDNUNQXOR distribution because the algorithm that computes r did not finish in the allotted time of 10 min. James et al. [17] showed that for whichever bivariate distribution $I_r(Y;T)\le I_p(Y;T)$; therefore, for the bivariate case we have $S^{\mathrm{CI}}\le S^{\mathrm{dep}}$. Unfortunately, the measure of unique information proposed by James et al. [17], unlike the usual proposals of intersection or union information, does not allow for the computation of the partial information atoms in the complete redundancy lattice if $n>2$. The authors also comment that it is not clear if their measure satisfies monotonicity when $n>2$. Naturally, our measure is not the same as $S^{\mathrm{dep}}$, so it does not retain the operational interpretation of unique information $U_i$ being the least amount that influences $I(Y;T)$ when the marginal constraint $(Y_i,T)$ is added to the resulting maximum entropy distributions. Given the form of $S^{\mathrm{dep}}$, one could define $I_{\cup }^{\mathrm{dep}}:=min\{I_q(Y;T),I_r(Y;T)\}$ and study its properties. Clearly, it does not satisfy the self-redundancy axiom, but we wonder if it could be adjusted so that it satisfies all of the proposed axioms. The $n=2$ decomposition retains the operational interpretation of the original measure, but it is not clear whether this is true for $n>2$. For the latter case, the maximum entropy distributions that we wrote as q and r have different definitions [17]. We leave this for future work.

6. Conclusions and Future Work

In this paper, we introduced a new measure of union information for the partial information decomposition (PID) framework, based on the channel perspective, which quantifies synergy as the information that is beyond conditional independence of the sources, given the target. This measure has a clear interpretation and is very easy to compute, unlike most measures of union information or synergy, which require solving an optimization problem. The main contributions and conclusions of the paper can be summarized as follows.

• We introduced new measures of union information and synergy for the PID framework, which thus far was mainly developed based on measures of redundant or unique information. We provided its operational interpretation and defined it for an arbitrary number of sources.
• We proposed an extension of the Williams–Beer axioms for measures of union information and showed our proposed measure satisfies them.
• We reviewed, commented on, and rejected some of the previously proposed properties for measures of union information and synergy in the literature.
• We showed that measures of union information that satisfy the extension of the Williams–Beer axioms necessarily satisfy a few other appealing properties, as well as the derived measures of synergy.
• We reviewed previous measures of union information and synergy, critiqued them, and compared them with our proposed measure.
• The proposed conditional independence measure is very simple to compute.
• We provide code for the computation of our measure for the bivariate case and for source $\{\left\{Y_1\right\},\left\{Y_2\right\},\left\{Y_3\right\}\}$ in the trivariate case.

Finally, we believe this paper opens several avenues for future research, thus we point out several directions to be pursued in upcoming work:

• We saw that the synergy yielded by the measure of James et al. [17] is given by $S^{\mathrm{dep}}=I(Y;T)-min\{I_q(Y;T),I_r(Y;T)\}$. Given its analytical expression, one could start by defining a measure of union information as $I_{\cup }(Y_1,Y_2\to T)=min\{I_q(Y;T),I_r(Y;T)\}$, possibly tweak it so it satisfies the WB axioms, study its properties, and possibly extend it to the multivariate case.
• Our proposed measure may ignore conditional dependencies that are present in p in favor of maximizing mutual information, as we commented in Section 3.3. This is a compromise so that the measure satisfies monotonicity. We believe this is a potential drawback of our measure, and we suggest the investigation of a measure similar to ours, but that does not ignore conditional dependencies that it has access to.
• Extending this measure for absolutely continuous random variables.
• Implementing our measure in the dit package [40].
• This paper reviewed measures of union information and synergy, as well as properties that were suggested throughout the literature. Sometimes this was by providing examples where the suggested properties fail, and other times simply by commenting. We suggest performing something similar for measures of redundant information.

7. Code Availability

The code is publicly available at https://github.com/andrefcorreiagomes/CIsynergy/ and requires the dit package [40] (accessed on 12 January 2024).