The Logical Consistency of Simultaneous Agnostic Hypothesis Tests

Abstract

Simultaneous hypothesis tests can fail to provide results that meet logical requirements. For example, if A and B are two statements such that A implies B, there exist tests that, based on the same data, reject B but not A. Such outcomes are generally inconvenient to statisticians (who want to communicate the results to practitioners in a simple fashion) and non-statisticians (confused by conflicting pieces of information). Based on this inconvenience, one might want to use tests that satisfy logical requirements. However, Izbicki and Esteves shows that the only tests that are in accordance with three logical requirements (monotonicity, invertibility and consonance) are trivial tests based on point estimation, which generally lack statistical optimality. As a possible solution to this dilemma, this paper adapts the above logical requirements to agnostic tests, in which one can accept, reject or remain agnostic with respect to a given hypothesis. Each of the logical requirements is characterized in terms of a Bayesian decision theoretic perspective. Contrary to the results obtained for regular hypothesis tests, there exist agnostic tests that satisfy all logical requirements and also perform well statistically. In particular, agnostic tests that fulfill all logical requirements are characterized as region estimator-based tests. Examples of such tests are provided.

1. Introduction

One of the practical shortcomings of simultaneous test procedures is that they can lack logical consistency [1,2]. As a result, recent papers have discussed minimum logical requirements and methods that achieve these requirements [3,4,5,6,7]. For example, it has been argued that simultaneous tests ought to be in agreement with the following criterion: if hypothesis A implies hypothesis B, a procedure that rejects B should also reject A.

In particular, Izbicki and Esteves [3] and da Silva et al. [7] examine classical and bayesian simultaneous tests with respect to four consistency properties:

• Monotonicity: if A implies B, then a test that does not reject A should not reject B.
• Invertibility: A test should reject A if and only if it does not reject not-A.
• Union consonance: If a test rejects A and B, then it should reject $A\cup B$.
• Intersection consonance: If a test does not reject A and does not reject B, then it should not reject $A\cap B$.

Izbicki and Esteves [3] prove that the only tests that are fully coherent are trivial tests based on point estimation, which are generally void of statistical optimality. This finding suggests that alternatives to the standard “reject versus accept” tests should be explored.

Such an alternative are agnostic tests [8], which can take the following decisions: (i) accept an hypothesis (decision 0); (ii) reject it (decision 1); or (iii) noncommittally neither accept or reject it; thus abstaining or remaining agnostic about the other two actions (decision $\frac{1}{2}$). Decision (iii) is also called a no-decision classification. The set of samples, $x\in \mathcal{X}$, for which one abstains from making a decision about a given hypothesis is called a no-decision region [8]. An agnostic test enables one to explicitly deal with the difference between “accepting a hypothesis H” and “not rejecting H (remaining agnostic)”. This distinction will be made clearer in Section 5, which derives agnostic tests under a Bayesian decision-theoretic standpoint by means of specific penalties for false rejection, false acceptance and excessive abstinence.

We use the above framework to revisit the logical consistency of simultaneous hypothesis tests. Section 2 defines agnostic testing scheme (ATS), a transformation that assigns to each statistical hypothesis an agnostic test function. This definition is illustrated with bayesian and frequentist examples, using both existing and novel agnostic tests. Section 3 generalizes the logical requirements in [3] to agnostic testing schemes. Section 4 presents tests that satisfy all of these logical requirements. Section 5 obtains, under the Bayesian decision-theoretic paradigm, necessary and sufficient conditions on loss functions to ensure that Bayes tests meet each of the logical requirements. All theorems are proved in the Appendix.

2. Agnostic Testing Schemes

This section describes the mathematical setup for agnostic testing schemes. Let $\mathcal{X}$ denote the sample space, Θ the parameter space and $L_x\left(\theta \right)$ the likelihood function at the point $\theta \in \mathsf{\Theta }$ generated by the data $x\in \mathcal{X}$. We denote by $\mathcal{D}=\{0,\frac{1}{2},1\}$ the set of all decisions that can be taken when testing a hypothesis: accept (0), remain agnostic ($\frac{1}{2}$) and reject (1). By an agnostic hypothesis test (or simply agnostic test) we mean a decision function from $\mathcal{X}$ to $\mathcal{D}$ [8,9]. Similar tests are commonly used in machine learning in the context of classification [1,2]. Moreover, let $\mathsf{\Phi }=\{\varphi :\varphi :\mathcal{X}⟶\mathcal{D}\}$ be the set of all (agnostic) hypothesis tests. The following definition adapts testing schemes [3] to agnostic tests.

Definition 1 (Agnostic Testing Scheme; ATS). 

Let $\sigma \left(\mathsf{\Theta }\right)$, a σ-field of subsets of the parameter space Θ, be the set of hypotheses to be tested. An ATS is a function $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ that, for each hypothesis $A\in \sigma \left(\mathsf{\Theta }\right)$, assigns the test $\mathcal{L}\left(A\right)\in \mathsf{\Phi }$ for testing A.

A way of creating an agnostic testing scheme is to find a collection of statistics and to compare them to thresholds:

Example 1. 

For every $A\in \sigma \left(\mathsf{\Theta }\right)$, let $s_A:\mathcal{X}⟶\mathbb{R}$ be a statistic. Let $c_1,c_2\in \mathbb{R}$, with $c_1\ge c_2$, be fixed thresholds. For each $A\in \sigma \left(\mathsf{\Theta }\right)$, one can define $\mathcal{L}\left(A\right):\mathcal{X}\to \mathcal{D}$ by

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{s}_A\left(x\right)>c_1\hfill \\ \frac{1}{2}\hfill & \mathrm{if}{c}_1\ge s_A\left(x\right)>c_2\hfill \\ 1\hfill & \mathrm{if}{c}_2\ge s_A\left(x\right)\hfill \end{array}\right.$$

The ATS in Example 1 rejects a hypothesis if the value of the statistic $s_A$ is small, accepts it if this value is large, and remains agnostic otherwise. If $s_A\left(x\right)$ is a measure of how much evidence that x brings about A, then this ATS rejects a hypothesis if the evidence brought by the data is small, accepts it if this evidence is large, and remains agnostic otherwise. The next examples present particular cases of this ATS These examples will be explored in the following sections.

Example 2 (ATS based on posterior probabilities). 

Let $\mathsf{\Theta }={\mathbb{R}}^d$ and $\sigma \left(\mathsf{\Theta }\right)=\mathcal{B}\left(\mathsf{\Theta }\right),$ the Borelians of ${\mathbb{R}}^d$. Assume that a prior probability $\mathbb{P}$ in $\sigma \left(\mathsf{\Theta }\right)$ is fixed, and let $c_1,c_2\in (0,1)$, with $c_1\ge c_2$, be fixed thresholds. For each $A\in \sigma \left(\mathsf{\Theta }\right)$, let $\mathcal{L}\left(A\right):\mathcal{X}\to \mathcal{D}$ be defined by

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}\mathbb{P}\left(A\right|x)>c_1\hfill \\ \frac{1}{2}\hfill & \mathrm{if}{c}_1\ge \mathbb{P}\left(A\right|x)>c_2\hfill \\ 1\hfill & \mathrm{if}{c}_2\ge \mathbb{P}\left(A\right|x)\hfill \end{array}\right.$$

where $\mathbb{P}(.|x)$ is the posterior distribution of θ, given x. This is essentially the test that Ripley [10] proposed in the context of classification, which was also investigated by Babb et al. [9]. When $c_1=c_2,$ this ATS is a standard (non-agnostic) Bayesian testing scheme.

Example 3 (Likelihood Ratio Tests with fixed threshold). 

Let $\mathsf{\Theta }={\mathbb{R}}^d$ and $\sigma \left(\mathsf{\Theta }\right)=\mathcal{P}\left(\mathsf{\Theta }\right),$ the set of the parts of ${\mathbb{R}}^d$. Let $c_1,c_2\in (0,1)$, with $c_1\ge c_2$, be fixed thresholds. For each $A\in \sigma \left(\mathsf{\Theta }\right)$, let ${\lambda }_x\left(A\right)=\frac{{sup}_{\theta \in A}{L}_x\left(\theta \right)}{{sup}_{\theta \in \mathsf{\Theta }}{L}_x\left(\theta \right)}$ be the likelihood ratio statistic for sample $x\in \mathcal{X}$. Define $\mathcal{L}$ by

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{\lambda }_x\left(A\right)>c_1\hfill \\ \frac{1}{2}\hfill & \mathrm{if}{c}_1\ge {\lambda }_x\left(A\right)>c_2\hfill \\ 1\hfill & \mathrm{if}{c}_2\ge {\lambda }_x\left(A\right)\hfill \end{array}\right.$$

When $c_1=c_2$, this is the standard likelihood ratio with fixed threshold (non-agnostic) testing scheme [3].

A similar test to that of Example 3 is developed by Berg [8]; however, the values of the cutoffs $c_1$ and $c_2$ are allowed to change with the hypothesis of interest, and they are chosen so as to control the level of significance and the power of each of the tests.

Example 4 (FBST ATS). 

Let $\mathsf{\Theta }={\mathbb{R}}^d,$ $\sigma \left(\mathsf{\Theta }\right)=\mathcal{B}\left({\mathbb{R}}^d\right),$ and $f\left(\theta \right)$ be the prior probability density function (p.d.f.) for θ. Suppose that, for each $x\in \mathcal{X}$, there exists $f\left(\theta \right|x),$ the p.d.f. of the posterior distribution of θ, given x. For each hypothesis $A\in \sigma \left(\mathsf{\Theta }\right),$ let

$$T_x^A=\left\{\mathit{\theta }\in \mathsf{\Theta }:f\left(\mathit{\theta }\right|\mathit{x})>\underset{\theta \in A}{sup}f\left(\mathit{\theta }\right|\mathit{x})\right\}$$

be the set tangent to the null hypothesis and let $e{v}_x\left(A\right)=1-P(\theta \in T_x^A|\mathit{x})$ be the Pereira–Stern evidence value for A [11]. Let $c_1,c_2\in (0,1)$, with $c_1\ge c_2$, be fixed thresholds. One can define an ATS $\mathcal{L}$ by

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}e{v}_x\left(A\right)>c_1\hfill \\ \frac{1}{2}\hfill & \mathrm{if}{c}_1\ge e{v}_x\left(A\right)>c_2\hfill \\ 1\hfill & \mathrm{if}{c}_2\ge e{v}_x\left(A\right)\hfill \end{array}\right.$$

When $c_1=c_2$, this ATS reduces to the standard (non-agnostic) FBST testing scheme [3].

The following example presents a novel ATS based on region estimators.

Example 5 (Region Estimator-based ATS). 

Let $R:\mathcal{X}⟶\mathcal{P}\left(\mathsf{\Theta }\right)$ be a region estimator of θ. For every $A\in \sigma \left(\mathsf{\Theta }\right)$ and $x\in \mathcal{X}$, one can define an ATS $\mathcal{L}$ via

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}R\left(x\right)\subseteq A\hfill \\ 1\hfill & \mathrm{if}R\left(x\right)\subseteq A^c\hfill \\ \frac{1}{2}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Hence, $\mathcal{L}\left(A\right)\left(x\right)=\frac{\mathbb{I}(R\left(x\right)\subseteq A^c)+\mathbb{I}(R\left(x\right)⊈A)}{2}$. See Figure 1 for an illustration of this procedure.

Notice that for continuous Θ, Example 5 does not accept precise (i.e., null Lebesgue measure) hypotheses, yielding either rejection or abstinence (unless region estimates are themselves precise). Therefore, the performance of region estimator-based ATS’s is in agreement with the prevailing position among both bayesian and frequentist statisticians: to accept a precise hypothesis is inappropriate. From a Bayesian perspective, precise null hypothesis usually have zero posterior probabilities, and thus should not be accepted. From a Frequentist perspective, not rejecting a hypothesis is not the same as accepting it. See Berger and Delampady [12] and references therein for a detailed account on the controversial problem of testing precise hypotheses.

In principle, R can be any region estimator. However, some choices of R lead to better statistical performance. For example, from a frequentist, one might choose R to be a confidence region. This choice is explored in the next example.

Example 6. 

From a frequentist perspective, one might choose R in Example 5 to be a confidence region: if the region estimator has confidence at least $1-\alpha$, then type I error probability, ${sup}_{\theta \in A}\mathbb{P}(\mathcal{L}\left(A\right)\left(X\right)=1|\theta )$, is smaller than α for each of the hypothesis tests. Indeed,

$$\underset{\theta \in A}{sup}\mathbb{P}(\mathcal{L}\left(A\right)\left(X\right)=1|\theta )=\underset{\theta \in A}{sup}\mathbb{P}({\theta }^{\prime }\notin R\left(X\right)\mathrm{for}\mathrm{every}{\theta }^{\prime }\in A|\theta )\le \underset{\theta \in A}{sup}\mathbb{P}(\theta \notin R\left(X\right)|\theta )\le \alpha .$$

If R is a confidence region, then this ATS also controls the Family Wise Error Rate (FWER, [13]), as shown in Section 3.1.

Consider $X_1,\dots ,X_{20}|\mu \stackrel{i.i.d.}{\sim }Normal(\mu ,1)$. In Figure 2, we illustrate how the probability of each decision, $\mathbb{P}\left(\mathcal{L}\right(A\left)\right(X)=d|\mu )$ for $d\in \{0,\frac{1}{2},1\}$, varies as a function of μ for three hypotheses: (i) $\mu <0$; (ii) $\mu =0$; and (iii) $0<\mu <1$. We consider the standard region estimator for μ, $R\left(X\right)=[\overline{X}-z_{1-\alpha /2}\frac{1}{\sqrt{n}};\overline{X}+z_{1-\alpha /2}\frac{1}{\sqrt{n}}]$ with $\alpha =5\%$. These curves represent the generalization of the standard power function to agnostic hypothesis tests. Notice that $\mu =0$ is never accepted, and that, under the null hypothesis, all tests have at most 5% of probability of rejecting H.

The next two examples show other cases of the ATS in Example 5 that use region estimators based on the measures of evidence in Examples 3 and 4.

Example 7 (Region Likelihood Ratio ATS). 

For a fixed value $c\in (0,1)$, define the region estimate $R_c\left(x\right)=\{\theta \in \mathsf{\Theta }:{\lambda }_x\left(\left\{\theta \right\}\right)\ge c\}$, where ${\lambda }_x$ is the likelihood ratio statistics from Example 3. For every $A\in \sigma \left(\mathsf{\Theta }\right)$ and $x\in \mathcal{X}$, the ATS based on this region estimator (Example 5) satisfies $\mathcal{L}\left(A\right)\left(x\right)=1\iff A\bigcap R_c\left(x\right)=\varnothing$, and $\mathcal{L}\left(A\right)\left(x\right)=0\iff A^c\bigcap R_c\left(x\right)=\varnothing$. It follows that this ATS can be written as

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}{\lambda }_x\left(A^c\right)<c\hfill \\ 1\hfill & \mathrm{if}{\lambda }_x\left(A\right)<c\hfill \\ \frac{1}{2}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

Example 8 (Region FBST ATS). 

For a fixed value of $c\in (0,1)$, let ${\mathrm{HPD}}_c^x$ be the Highest Posterior Probability Density region with probability $1-c$, based on observation x [3,14]. For every $A\in \sigma \left(\mathsf{\Theta }\right)$ and $x\in \mathcal{X}$, the ATS based on this region estimator (Example 5) satisfies $\mathcal{L}\left(A\right)\left(x\right)=1\iff A\bigcap {\mathrm{HPD}}_c^x=\varnothing$, and $\mathcal{L}\left(A\right)\left(x\right)=0\iff A^c\bigcap {\mathrm{HPD}}_c^x=\varnothing$. It follows that this ATS can be written as

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}e{v}_x\left(A^c\right)<c\hfill \\ 1\hfill & \mathrm{if}e{v}_x\left(A\right)<c\hfill \\ \frac{1}{2}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

In the sequence, we introduce four logical coherence properties for agnostic testing schemes and investigate which tests satisfy them.

3. Coherence Properties

3.1. Monotonicity

Monotonicity restricts the decisions that are available for nested hypotheses. If hypothesis Aimplies hypothesis B (i.e., $A\subseteq B$), then a testing scheme that rejects B should also reject A. Monotonicity has received a lot of attention in the literature (e.g., [5,6,15,16,17,18,19]). It can be extended to ATS’s in the following way.

Definition 2 (Monotonicity). 

$\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is monotonic if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$, $A\subset B$ implies that $\mathcal{L}\left(A\right)\ge \mathcal{L}\left(B\right)$.

$\mathcal{L}$ is monotonic if, for every hypotheses $A\subset B$,

• if $\mathcal{L}$ accepts A, then it also accepts B.
• if $\mathcal{L}$ remains agnostic about A, then it either remains agnostic about B or accepts B.

Next, we illustrate some monotonic agnostic testing schemes.

Example 9 (Tests based on posterior probabilities). 

The ATS from Example 2 is monotonic. Indeed, $A\subset B$ implies that $\mathbb{P}\left(A\right|x)\le \mathbb{P}(B\left|x\right)$ $\forall x\in \mathcal{X}$, and hence $\mathbb{P}\left(A\right|x)>c_i$ implies that $\mathbb{P}\left(B\right|x)>c_i$ for $i=1,2$.

Example 10 (Likelihood Ratio Tests with fixed threshold). 

The ATS from Example 3 is monotonic. This is because if $A,B\in \sigma \left(\mathsf{\Theta }\right)$ are such that $A\subset B$, then ${sup}_{\theta \in A}{L}_x\left(\theta \right)\le {sup}_{\theta \in B}{L}_x\left(\theta \right)$, $\forall x\in \mathcal{X},$ which implies that ${\lambda }_x\left(A\right)\le {\lambda }_x\left(B\right)$. It follows that ${\lambda }_x\left(A\right)>c_i$ implies that ${\lambda }_x\left(B\right)>c_i$ for $i=1,2$.

Example 11 (FBST). 

The ATS from Example 4 is monotonic. In fact, let $A,B\in \sigma \left(\mathsf{\Theta }\right)$ be such that $A\subset B$. We have ${sup}_{B}f\left(\theta \right|x)\ge {sup}_{A}f\left(\theta \right|x)$ $\forall x\in \mathcal{X}.$ Hence, $T_x^B\subseteq T_x^A$, and, therefore, $e{v}_x\left(A\right)\le e{v}_x\left(B\right)$. It follows that $e{v}_x\left(A\right)>c_i$ implies that $e{v}_x\left(B\right)>c_i$ for $i=1,2$.

Notice that p-values and Bayes factors are not (coherent) measures of support for hypotheses [19,20], and therefore using them in a similar fashion as in Examples 2–4 would not lead to monotonic agnostic testing schemes. On the other hand, any monotonic statistic $s_A$ does, however, provide a monotonic ATS, because, if $A\subseteq B$, $s_A\left(x\right)>c_i$ implies that $s_B\left(x\right)>c_i$ for $i=1,2$. Another example of such statistic is the s-value defined by Patriota [6]. As a matter of fact, every ATS is, in a sense, associated to monotonic statistics as shown in the next theorem.

Theorem 1. 

Let $\mathcal{L}$ be an agnostic testing scheme. $\mathcal{L}$ is monotonic if, and only if, there exist a sequence of test statistics ${\left(s_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)},$ $s_A:\mathcal{X}⟶I\subseteq \mathbb{R},$ with $s_A\le s_B$ whenever $A\subset B$, $A,B\in \sigma \left(\mathsf{\Theta }\right)$, and cutoffs $c_1,c_2\in I$, $c_1\ge c_2$, such that for every $A\in \sigma \left(\mathsf{\Theta }\right)$ and $x\in \mathcal{X}$,

$$\begin{array}{c}\hfill \mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & if{s}_A\left(x\right)>c_1\hfill \\ \frac{1}{2}\hfill & if{c}_1\ge s_A\left(x\right)>c_2\hfill \\ 1\hfill & if{c}_2\ge s_A\left(x\right)\hfill \end{array}\right.\end{array}$$

(1)

Example 12 (Region Estimator). 

The ATS from Example 5 is monotonic, because if $A\subseteq B,$ $A,B\in \sigma \left(\mathsf{\Theta }\right)$, then

$$\mathcal{L}\left(A\right)\left(x\right)=\frac{\mathbb{I}(R\left(x\right)\subseteq A^c)+\mathbb{I}(R\left(x\right)⊈A)}{2}\ge \frac{\mathbb{I}(R\left(x\right)\subseteq B^c)+\mathbb{I}(R\left(x\right)⊈B)}{2}=\mathcal{L}\left(B\right)\left(x\right),$$

as $\mathbb{I}(R\left(x\right)\subseteq B^c)\le \mathbb{I}(R\left(x\right)\subseteq A^c)$ and $\mathbb{I}\left(R\right(x)⊈B)\le \mathbb{I}\left(R\right(x)⊈A)$. Because this ATS is monotonic, it also controls Family Wise Error Rate [21].

3.2. Union Consonance

Finner and Strassburger [4] and Izbicki and Esteves [3] investigated the following logical property, named union consonance: if a (non-agnostic) testing scheme rejects each of the hypotheses A and B, it should also reject their union $A\cup B$. In other words, a TS cannot accept the union while rejecting its components. In this section, we adapt the concept of union consonance to the framework of agnostic testing schemes by considering two extensions for such desideratum: the weak and the strong union consonance.

Definition 3 (Weak Union Consonance). 

An ATS $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is weakly consonant with the union if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$, and for every $x\in \mathcal{X}$,

$$\begin{array}{c}\hfill \mathcal{L}\left(A\right)\left(x\right)=1\mathrm{and}\mathcal{L}\left(B\right)\left(x\right)=1\mathrm{implies}\mathcal{L}(A\cup B)\left(x\right)\ne 0\end{array}$$

This is exactly the definition of union consonance for non-agnostic testing schemes. Notice that, according to such definition, it is possible to remain agnostic about $A\cup B$ while rejecting A and B.

Remark 1. 

Izbicki and Esteves [3] show that if a non agnostic testing scheme $\mathcal{L}$ satisfies union consonance, then for every finite set of indices I and for every ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right),min{\left\{\mathcal{L}\left(A_i\right)\right\}}_{i\in I}=1$ implies that $\mathcal{L}\left({\cup }_{i\in I}{A}_i\right)\ne 0$. This is not the case for weak union consonant agnostic testing schemes; we leave further details to Section 4.3.

The second definition of union consonance is more stringent than the first one:

Definition 4 (Strong Union Consonance). 

An $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is strongly consonant with the union if, for every arbitrary set of indices I and for every ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$ such that ${\cup }_{i\in I}{A}_i\in \sigma \left(\mathsf{\Theta }\right)$, and for every $x\in \mathcal{X}$,

$$min{\left\{\mathcal{L}\left(A_i\right)\left(x\right)\right\}}_{i\in I}=1\mathrm{implies}\mathcal{L}\left({\cup }_{i\in I}{A}_i\right)\left(x\right)=1$$

Definition 3 is less stringent than Definition 4 in two senses: (i) the latter imposes the (strict) rejection of a union of hypotheses whenever each of them is rejected while the former imposes just non-acceptance (rejection or abstention) of the union is such circumstances; and (ii) in Definition 4 consonance is required to hold for every set (possibly infinite) of hypotheses as opposed to Definition 3 which applies only to pairs of hypotheses. Notice that if an ATS is strongly consonant with union, it is also weakly consonant with union, and that both definitions are indeed extensions of the concept presented by Izbicki and Esteves [3].

The following examples show ATSs that are consonant with union.

Example 13 (Tests based on posterior probabilities). 

Consider again Example 2 with the restriction $c_1\ge 2c_2$. If A and B are rejected after observing $x\in \mathcal{X}$, then

$$\mathbb{P}(A\cup B|x)\le \mathbb{P}\left(A\right|x)+\mathbb{P}\left(B\right|x)\le 2c_2\le c_1,$$

and therefore $A\cup B$ cannot be accepted. Thus, with this restrictions, that ATS is weakly consonant with union. The restriction $c_1\ge 2c_2$ is not only sufficient to ensure weak union consonance, but it is actually necessary to ensure it holds for every prior distribution (see Theorem 2). Notice, however, that this ATS is not strongly consonant with union in general.

Example 14 (Likelihood Ratio Tests with fixed threshold). 

The ATS of Example 3 is strongly consonant with union. Indeed, let I be an arbitrary set of indices and ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$ be such that ${\cup }_{i\in I}{A}_i\in \sigma \left(\mathsf{\Theta }\right)$. For every $x\in \mathcal{X}$, ${\lambda }_x\left({\cup }_{i\in I}{A}_i\right)={sup}_{i\in I}\left\{{\lambda }_x\left(A_i\right)\right\}$ [3]. It follows that if ${\lambda }_x\left(A_i\right)\le c_2$ for every $i\in I$, then ${\lambda }_x\left({\cup }_{i\in I}{A}_i\right)\le c_2$. Thus, if $\mathcal{L}$ rejects all hypotheses $A_i$ after x is observed, it also rejects ${\cup }_{i\in I}{A}_i$. In addition, $\mathcal{L}$ is also weakly consonant with union.

Example 15 (FBST). 

The ATS from Example 4 is also strongly consonant with union. Indeed, let I be an arbitrary set of indices and ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$ be such that ${\cup }_{i\in I}{A}_i\in \sigma \left(\mathsf{\Theta }\right)$. For every $x\in \mathcal{X}$, $e{v}_x\left({\cup }_{i\in I}{A}_i\right)={sup}_{i\in I}\left\{e{v}_x\left(A_i\right)\right\}$ [22]. Strong union consonance holds due to the same argument from Example 14. It follows that $\mathcal{L}$ is also weakly consonant with union.

Example 16 (Region Estimator). 

The TS from Example 5 satisfies strong union consonance. Indeed, let I be an arbitrary set of indices and ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$ be such that ${\cup }_{i\in I}{A}_i\in \sigma \left(\mathsf{\Theta }\right)$. If $\mathcal{L}\left(A_i\right)\left(x\right)=1$, then $R\left(x\right)\subseteq A_i^c$. Hence, if $\mathcal{L}\left(A_i\right)\left(x\right)=1$ for every $i\in I$, $R\left(x\right)\subseteq {\bigcap }_{i\in I}{A}_i^c={\left({\bigcup }_{i\in I}{A}_i\right)}^c$, and, therefore, ${\bigcup }_{i\in I}{A}_i$ is rejected. It follows that $\mathcal{L}$ is also weakly consonant with union.

3.3. Intersection Consonance

The third property we investigate, named intersection consonance [3], states that if a (non agnostic) testing scheme cannot accept hypotheses A and B while rejecting its intersection. We consider two extensions of such definition to agnostic testing schemes.

Definition 5 (Weak Intersection Consonance). 

An ATS $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is consonant with the intersection if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$ and $x\in \mathcal{X}$,

$$\begin{array}{c}\hfill \mathcal{L}\left(A\right)\left(x\right)=0\mathrm{and}\mathcal{L}\left(B\right)\left(x\right)=0\mathrm{implies}\mathcal{L}(A\cap B)\left(x\right)\ne 1.\end{array}$$

This is exactly the definition of intersection consonance for non-agnostic testing schemes. Notice that it is possible to accept A and B while being agnostic about $A\cap B$.

The second definition of intersection consonance is more stringent:

Definition 6 (Strong Intersection Consonance). 

An ATS $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is strongly consonant with the intersection if, for every arbitrary set of indices I and for every ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$ such that ${\cap }_{i\in I}{A}_i\in \sigma \left(\mathsf{\Theta }\right)$, and for every $x\in \mathcal{X}$,

$$max{\left\{\mathcal{L}\left(A_i\right)\left(x\right)\right\}}_{i\in I}=0\mathrm{implies}\mathcal{L}\left({\cap }_{i\in I}{A}_i\right)\left(x\right)=0.$$

As in the case of union consonance, Definition 5 is less stringent than Definition 6 in two senses: (i) the latter imposes the (strict) acceptance of an intersection of hypotheses whenever each of them is accepted while the former imposes just non-rejection (acceptance or abstention) of the intersection is such circumstances; and (ii) in Definition 6 consonance is required to hold for every set (possibly infinite) of hypotheses as opposed to Definition 5 which applies only to pairs of hypotheses. Notice that if an ATS is strongly consonant with intersection, it is also weakly consonant with intersection, and that both definitions are indeed extensions of the concept presented by Izbicki and Esteves [3].

Example 17 (Tests based on posterior probabilities). 

Consider Example 2 with the restriction $c_2\le 2c_1-1$. If A and B are accepted when $x\in \mathcal{X}$ is sampled, then $\mathbb{P}\left(A\right|x)>c_1$ and $\mathbb{P}\left(B\right|x)>c_1$. By Fréchet inequality, it follows that

$$\mathbb{P}(A\cap B|x)\ge \mathbb{P}\left(A\right|x)+\mathbb{P}\left(B\right|x)-1>2c_1-1\ge c_2$$

and, therefore, $A\cap B$ cannot be rejected. It follows that weak intersection consonance holds. The restriction $c_2\le 2c_1-1$ is not only sufficient to ensure weak intersection consonance, but it is actually necessary to ensure this property holds for every prior distribution; see Theorem 2. Notice, however, that this ATS is not strongly consonant with intersection in general (Take, for example, $\mathsf{\Theta }=[0,1]$, $\mathbb{P}=\lambda$ (the Lebesgue measure), $A=[0,2/3]$, $B=[1/3,1]$, and $c_1=3/5$).

The ATS based on the likelihood ratio statistic from Example 3 does not satisfy intersection consonance, because there are examples in which ${\lambda }_x(A\cap B)=0$, while ${\lambda }_x\left(A\right)>0$ and ${\lambda }_x\left(B\right)>0$ (Consider, for example, that every $\theta \in \mathsf{\Theta }$ has the same likelihood and $A\cap B=\varnothing$). Similarly, the ATS based on FBST from Example 4 is not consonant with intersection, because there are examples such that $e{v}_x(A\cap B)=0$, while $e{v}_x\left(A\right)>0$ and $e{v}_x\left(B\right)>0$. ATSs based on region estimators are consonant with intersection.

Example 18 (Region Estimator). 

The TS from Example 5 satisfies both strong and weak intersection consonance. Indeed, let I be an arbitrary set of indices and ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$ be such that ${\cap }_{i\in I}{A}_i\in \sigma \left(\mathsf{\Theta }\right)$. If $\mathcal{L}\left(A_i\right)\left(x\right)=0$ for every $i\in I$, then $R\left(x\right)\subseteq A_i$ for every $i\in I$. It follows that $R\left(x\right)\subseteq {\cap }_{i\in I}{A}_i$, and hence ${\cap }_{i\in I}{A}_i$ is accepted.

It follows that the ATSs from Examples 7 and 8 are are also consonant with intersection. Hence, it is possible to use e-values and likelihood ratio statistics to define ATS that are consonant with intersection.

Example 19 (ANOVA). 

In [3], the authors present an example which we now revisit. Suppose that $X_1,\dots ,X_{20}$ are i.i.d. $N({\mu }_1,{\sigma }^2);$ $X_{21},$ $\dots ,$ $X_{40}$ are i.i.d. $N({\mu }_2,{\sigma }^2)$ and $X_{41},\dots ,X_{60}$ are i.i.d. $N({\mu }_3,{\sigma }^2).$ Consider the following hypotheses:

$$\begin{array}{c}\hfill H_0^{(1,2,3)}:{\mu }_1={\mu }_2={\mu }_3{H}_0^{(1,2)}:{\mu }_1={\mu }_2{H}_0^{(1,3)}:{\mu }_1={\mu }_3\end{array}$$

and suppose that we observe the following means and standard-deviations on the data: ${\overline{X}}_1=0.15;S_1=1.09;{\overline{X}}_2=-0.13;S_2=0.5{\overline{X}}_3=-0.38;S_3=0.79$. Using the likelihood ratio statistics, we have the following p-values for these hypotheses:

$$\begin{array}{c}\hfill p_{H_0^{(1,2,3)}}=0.0498p_{H_0^{(1,2)}}=0.2564p_{H_0^{(1,3)}}=0.0920\end{array}$$

Therefore, the testing scheme given by the likelihood ratio tests with common level of significance $\alpha =5\%$ rejects $H_0^{(1,2,3)}$ but does not reject either $H_0^{(1,2)}$ or $H_0^{(1,3)}.$ It follows intersection consonance does not hold. Now, consider the region estimator ATS based on the region estimate given by [23] for this setting,

$$\begin{array}{c}\hfill R\left(x\right)=\left\{({\mu }_1,{\mu }_2,{\mu }_3)\in {\mathbb{R}}^3:{\mu }_1-{\mu }_2\in \left[-1.65,2.21\right],{\mu }_2-{\mu }_3\in \left[-1.68,2.18\right],{\mu }_1-{\mu }_3\in \left[-1.40,2.46\right]\right\}\end{array}$$

All hypotheses $H_0^{(1,2,3)},H_0^{(1,2)},$ and $H_0^{(1,3)}$ intercept both $R\left(x\right)$ and its complement, so that one remains agnostic about all of them. As expected, intersection consonance holds using this ATS.

3.4. Invertibility

Invertibility formalizes the notion of simultaneous tests free from the labels “null” and “alternative” for the hypotheses of interest and has been suggested by several authors, specially under a Bayesian perspective [3,24,25].

Definition 7 (Invertibility). 

An ATS $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is invertible if, for every $A\in \sigma \left(\mathsf{\Theta }\right)$,

$$\begin{array}{c}\hfill \mathcal{L}\left(A^c\right)=1-\mathcal{L}\left(A\right)\end{array}$$

Example 20 (Tests based on posterior probabilities). 

The ATS from Example 2 is invertible for every prior distribution if and only if $c_2=1-c_1$.

Example 21 (Region Estimator). 

The ATS from Example 5 is invertible. Indeed,

$$\mathcal{L}\left(A\right)\left(x\right)=\frac{\mathbb{I}(R\left(x\right)\subseteq A^c)+\mathbb{I}(R\left(x\right)⊈A))}{2}=\frac{(1-\mathbb{I}(R\left(x\right)⊈A^c))+(1-\mathbb{I}(R\left(x\right)\subseteq A))}{2}=1-\mathcal{L}\left(A^c\right)\left(x\right)$$

It follows that the ATS from Examples 7 and 8 are also invertible.

4. Satisfying All Properties

Is it possible to construct non-trivial agnostic testing schemes that satisfy all consistency properties simultaneously? Contrary to the case of non agnostic testing schemes [3], the answer is yes. We next examine this question considering three desiderata: the weak desiderata (Section 4.1), the strong desiderata (Section 4.2), and the n-weak desiderata (Section 4.3).

4.1. Weak Desiderata

Definition 8 (Weakly Consistent ATS). 

An ATS, $\mathcal{L}$, is said to be weakly consistent if $\mathcal{L}$ is monotonic (Definition 2), invertible (Definition 7), weakly consonant with the union (Definition 3), and weakly consonant with the intersection (Definition 5).

Example 22 (Region Estimator). 

The ATS from Example 5 was already shown to satisfy all consistency properties from Definition 8 (Examples 12, 16, 18 and 21). Thus, it is a weakly consistent ATS.

It follows that the ATSs Examples 7 and 8, based on measures of support (likelihood ratio statistics and e-values), are weakly consistent ATSs.

Example 23 (Tests based on posterior probabilities). 

Consider Example 2. We have seen that the following restrictions are sufficient to guarantee union weak consonance (Example 13), weak intersection consonance (Example 17) and invertibility (Example 20), respectively: $c_1\ge 2c_2$, $2c_1-1\ge c_2$ and $c_2=1-c_1$. It follows from these relations and the fact that this ATS is monotonic (Example 9) that if $c_1>2/3$ and $c_2=1-c_1$, then it is weakly consistent, whatever the prior distribution for θ is.

The next theorem shows necessary and sufficient conditions for agnostic tests based on posterior distribution (with possibly different thresholds $c_1$ and $c_2$ for each hypothesis of interest) to satisfy each of the coherence properties.

Theorem 2. 

Let $\mathsf{\Theta }={\mathbb{R}}^d$ and $\sigma \left(\mathsf{\Theta }\right)=\mathcal{B}\left(\mathsf{\Theta }\right),$ the Borelians of ${\mathbb{R}}^d$. Let $\mathbb{P}$ be a prior probability measure in $\sigma \left(\mathsf{\Theta }\right)$. For each $A\in \sigma \left(\mathsf{\Theta }\right)$, let $\mathcal{L}\left(A\right):\mathcal{X}\to \mathcal{D}$ be defined by

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & if\mathbb{P}\left(A\right|x)>c_1^A\hfill \\ \frac{1}{2}\hfill & if{c}_1^A\ge \mathbb{P}\left(A\right|x)>c_2^A\hfill \\ 1\hfill & if{c}_2^A\ge \mathbb{P}\left(A\right|x)\hfill \end{array}\right.$$

where $\mathbb{P}(.|x)$ is the posterior distribution of θ, given x, and $0\le c_2^A\le c_1^A\le 1$. This is a generalization of the ATS of Example 2. Assume that the likelihood function is positive for every $x\in \mathcal{X}$ and $\theta \in \mathsf{\Theta }$. Such ATS satisfies:

1. 

Monotonicity for every prior distribution if, and only if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$ with $A\subseteq B,$ $c_2^A\ge c_2^B$ and $c_1^A\ge c_1^B$

2. 

Weak union consonance for every prior distribution if, and only if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$ such that $A\ne B$, $c_2^A+c_2^B\le c_1^{A\cup B}$

3. 

Weak intersection consonance for every prior distribution if, and only if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$ such that $A\ne B$, $c_1^A+c_1^B-1\ge c_2^{A\cap B}$

4. 

Invertibility for every prior distribution if, and only if, for every $A\in \sigma \left(\mathsf{\Theta }\right)$, $c_1^A=1-c_2^{A^c}$

It follows from Theorem 2 that if the cutoffs used in each of the tests ($c_1$ and $c_2$) are required to be the same for all hypothesis of interest, then the conditions in Example 23 are not only sufficient, but they are also necessary to ensure that all (weak) consistency properties hold for every prior distribution for θ.

4.2. Strong Desiderata

Definition 9 (Fully Consistent ATS). 

An ATS, $\mathcal{L}$, is said to be fully consistent if $\mathcal{L}$ is monotonic (Definition 2), invertible (Definition 7), strongly consonant with the union (Definition 4), and strongly consonant with the intersection (Definition 6).

The following theorem shows that, under mild assumptions, the only ATSs that are fully consistent are those based on region estimators.

Theorem 3. 

Assume that for every $\theta \in \mathsf{\Theta }$, $\left\{\theta \right\}\in \sigma \left(\mathsf{\Theta }\right)$. An ATS is fully consistent if, and only if, it is a region estimator-based ATS (Example 5).

Hence, the only way to create a fully consistent ATS is by designing an appropriate region estimator and using Example 5. In particular, ATSs based on posterior probabilities (Example 2) are typically not fully consistent. It should be emphasized that when the region estimator that characterizes a fully consistent ATS $\mathcal{L}$ maps $\mathcal{X}$ to singletons of Θ, no sample point will lead to abstention, as either $R\left(x\right)\subseteq A$ or $R\left(x\right)\subseteq A^c$, for every $A\in \sigma \left(\mathsf{\Theta }\right)$. In such situations, region estimators reduce to point estimator which charaterize full consistent non-agnostic TSs [3].

In the next section, we consider a desiderata for simultaneous tests which is not as strong as that of Definition 9, but which is more stringent that that of Definition 8.

4.3. n-Weak Desiderata

In Section 3.2 and Section 3.3, weak consonance was defined for two hypotheses only. It is however possible to define it for $n<\infty$ hypotheses:

Definition 10 (Weak n-union Consonance) 

An A-TS $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ satisfies weak n-union consonant if, for every finite set of indices I, with $\left|I\right|\le n$, for every ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$, and for every $x\in \mathcal{X}$

$$\begin{array}{c}\hfill min{\left\{\mathcal{L}\left(A_i\right)\left(x\right)\right\}}_{i\in I}=1\mathrm{implies}\mathcal{L}\left({\cup }_{i\in I}{A}_i\right)\left(x\right)\ne 0.\end{array}$$

Definition 11 (Weak n-intersection Consonance) 

An ATS $\mathcal{L}:\sigma \left(\mathsf{\Theta }\right)\to \mathsf{\Phi }$ is weak n-intersection consonant if, for every finite set of indices I, with $\left|I\right|\le n$, for every ${\left\{A_i\right\}}_{i\in I}\subseteq \sigma \left(\mathsf{\Theta }\right)$, and for every $x\in \mathcal{X}$

$$\begin{array}{c}\hfill max{\left\{\mathcal{L}\left(A_i\right)\left(x\right)\right\}}_{i\in I}=0\mathrm{implies}\mathcal{L}\left({\cap }_{i\in I}{A}_i\right)\left(x\right)\ne 1.\end{array}$$

Although in the context of non agnostic testing schemes (union or intersection) consonance holds for $n=2$ if, and only if, it holds for every $n\in \mathbb{N}$ [3], this is not the case in the agnostic setting. We hence define

Definition 12 (n-Weakly Consistent ATS) 

An ATS, $\mathcal{L}$, is said to be n-weakly consistent if $\mathcal{L}$ is monotonic (Definition 2), invertible (Definition 7), n-weakly consonant with the union (Definition 10), and n-weakly consonant with the intersection (Definition 11).

Example 24 (Region Estimator). 

The ATS from Example 5 satisfies weak n-union and weak n-intersection consonance. The argument is the same as that presented in Examples 16 and 18. It follows that this is a n-weakly consistent ATS.

Example 25 (Tests based on posterior probabilities). 

Consider Example 2. In order to guarantee weak n-union consonance for every prior, it is necessary and sufficient to have $c_1\ge n{c}_2$. Moreover, to guarantee weak n-intersection consonance for every prior, it is necessary and sufficient to have $c_2\le n{c}_1-(n-1)$. It follows from these conditions and Example 20 that the following restrictions are necessary and sufficient to guarantee monotonicity, n-union consonance, n-intersection consonance and invertibility: $c_1>n/(n+1)$ and $c_2=1-c_1$. Hence, these conditions are sufficient to guarantee this ATS is n-weakly consistent. Now, because these conditions are also necessary, it follows that this ATS is n-weakly consistent for every $n>1$ if, and only if, it remains agnostic about every hypothesis which has probability in $(0,1)$.

5. Decision-Theoretic Perspective

In this section, we investigate agnostic testing schemes from a Bayesian decision-theoretic perspective. First, we define an ATS generated by a family of loss functions. Note that, in the context of agnostic tests, a loss function is a function $L:\mathcal{D}\times \mathsf{\Theta }\to \mathbb{R}$ that assigns to each $\theta \in \mathsf{\Theta }$ the loss $L(d,\theta )$ for making the decision $d\in \{0,\frac{1}{2},1\}$.

Definition 13 (ATS generated by a family of loss functions). 

Let $(\mathcal{X}\times \mathsf{\Theta },\sigma (\mathcal{X}\times \mathsf{\Theta }),\mathbb{P})$ be a Bayesian statistical model. Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of loss functions, where $L_A:\mathcal{D}\times \mathsf{\Theta }\to \mathbb{R}$ is the loss function to be used to test $A\in \sigma \left(\mathsf{\Theta }\right)$. An ATS generated by the family of loss functions ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is any ATS $\mathcal{L}$ defined over the elements of $\sigma \left(\mathsf{\Theta }\right)$ such that, $\forall A\in \sigma \left(\mathsf{\Theta }\right)$, $\mathcal{L}\left(A\right)$ is a Bayes test for hypothesis A against $\mathbb{P}$.

Example 26 (Bayesian ATS generated by a family of error-wise constant loss functions). 

For $A\in \sigma \left(\mathsf{\Theta }\right)$, consider the loss function $L_A$ of the form of

Table 1. The loss function for the hypothesis $\theta \in A$ used in Example 26.

Decision	State of Nature
	$\theta \in A$	$\theta \notin A$
0 (accept A)	0	$b_A$
$\frac{1}{2}$ (remain agnostic about A)	$a_A$	$c_A$
1 (reject A)	$d_A$	0

, where all entries are assumed to be non negative. This is a generalization of standard $0-1-c$ loss functions to agnostic tests in the sense that it penalizes not only false acceptance and false rejection with constant losses $b_A$ and $d_A$, respectively, but also an eventual abstention from deciding between accepting and rejecting A with the values $a_A$ and $c_A$. If $b_A{d}_A>a_A{b}_A+c_A{d}_A$, then the Bayes test against $L_A$ consists in rejecting A if $\mathbb{P}\left(A\right|x)<\frac{c_A}{d_A+c_A-a_A}$, accept A if $\mathbb{P}\left(A\right|x)>\frac{b_A-c_A}{a_A+b_A-c_A}$, and remain agnostic otherwise. It follows that the following ATS is generated by the family of loss functions ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$:

$$\mathcal{L}\left(A\right)\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}\mathbb{P}\left(A\right|x)>\frac{b_A-c_A}{a_A+b_A-c_A}\hfill \\ \frac{1}{2}\hfill & \mathrm{if}\frac{b_A-c_A}{a_A+b_A-c_A}\ge \mathbb{P}\left(A\right|x)>\frac{c_A}{d_A+c_A-a_A}\hfill \\ 1\hfill & \mathrm{if}\frac{c_A}{d_A+c_A-a_A}\ge \mathbb{P}\left(A\right|x)\hfill \end{array}\right.$$

Notice that if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$, $a_A=a_B$, $b_A=b_B$, $c_A=c_B$, and $d_A=d_B$, this ATS matches that from Example 2 for a particular value of $c_1$ and $c_2$.

We restrict out attention to ATSs generated by proper losses, a concept we adapt from [3] to agnostic tests:

Definition 14 (Proper losses). 

A family of loss functions ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has proper losses if

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{L}_A(0,\theta )<L_A(\frac{1}{2},\theta )<L_A(1,\theta )\hfill & ,\text{if}\theta \in A\hfill \\ L_A(0,\theta )>L_A(\frac{1}{2},\theta )>L_A(1,\theta )\hfill & ,\text{if}\theta \notin A\hfill \\ L_A(\frac{1}{2},\theta )<\frac{L_A(0,\theta )+L_A(1,\theta )}{2}\hfill & ,\text{for}\text{all}\theta \hfill \end{array}\right.\end{array}$$

Definition 14 states that (i) by taking a correct decision we lose less than by taking a wrong decision; (ii) by remaining agnostic we do not lose as much as when taking a wrong decision, but we lose more than by taking a correct decision; and (iii) it is better to remain agnostic about A than to flip a coin to decide if we reject or accept this hypothesis.

Example 27 (Bayesian ATS generated by a family of error-wise constant loss functions). 

In order to ensure that the loss in Example 26 is proper, the following restrictions must be satisfied: $0<a_A<d_A/2\mathrm{and}0<c_A<b_A/2.$ In particular, these conditions imply those stated in Example 26.

5.1. Monotonicity

We now turn our attention towards characterizing Bayesian monotonic ATS using a decision-theoretic framework. In order to do this, we first adapt the concept of relative losses [3] to the context of agnostic testing schemes.

Definition 15 (Relative Loss). 

Let $L_A$ be a loss function for testing hypothesis A. The relative losses $r_A^{(1,\frac{1}{2})}:\mathsf{\Theta }\to \mathbb{R}$ and $r_A^{(\frac{1}{2},0)}:\mathsf{\Theta }\to \mathbb{R}$ are defined by

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{r}_A^{(1,\frac{1}{2})}\left(\theta \right)\hfill & =L_A(1,\theta )-L_A(\frac{1}{2},\theta )\hfill \\ r_A^{(\frac{1}{2},0)}\left(\theta \right)\hfill & =L_A(\frac{1}{2},\theta )-L_A(0,\theta )\hfill \end{array}\right.\end{array}$$

The relative losses thus measure the difference between the losses of rejecting a given hypothesis and remaining agnostic about it, as well as the difference between the losses of remaining agnostic and accepting it. In order to guarantee that a Bayesian ATS is monotonic, certain constraints on the relative losses must be imposed. The next definition presents one of such assumptions, which we interpret in the sequence.

Definition 15 (Relative Loss). 

Let ${\mathcal{D}}_{>}^2=\{(1,\frac{1}{2}),(\frac{1}{2},0)\}$. ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has monotonic relative losses if the family ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is proper and, for all $A,B\in \sigma \left(\mathsf{\Theta }\right)$ such that $A\subset B$ and for all $(i,j)\in {\mathcal{D}}_{>}^2$,

$$r_B^{(i,j)}\left(\theta \right)\ge r_A^{(i,j)}\left(\theta \right)\forall \theta \in \mathsf{\Theta }$$

Let $A,B\in \sigma \left(\mathsf{\Theta }\right)$ with $A\subseteq B$. If $\theta \in A$, both A and B are true, so ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ having monotonic relative losses reflects the situation in which the rougher error of rejecting B compared to rejecting A (with respect to remaining agnostic about these hypotheses) should be assigned a larger relative loss. Similarly, the rougher error of remaining agnostic about B should be assigned a larger relative loss than remaining agnostic about A (with respect to correctly accepting these hypotheses). If $\theta \in B$ but $\theta \notin A$, these conditions are a consequence of the assumption that the family ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is proper. The case $\theta \notin B$ can be interpreted in a similar fashion as the case $\theta \in A$.

The following example presents necessary and sufficient conditions to ensure that the loss functions from Example 26 yield monotonic relative losses.

Example 28. 

Consider the losses presented in Example 26. Assuming the losses are proper (see Example 27), the conditions required to ensure ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has monotonic relative losses are

$$a_A\le a_B,c_B\le c_A,c_B-b_B\ge c_A-b_A\text{ and }{d}_B-a_B\ge d_A-a_A$$

Notice that these restrictions imply that $b_A\ge b_B$.

As a particular example, let $k>2$ and λ be a finite measure in $\sigma \left(\mathsf{\Theta }\right)$ with $\lambda \left(\mathsf{\Theta }\right)>0$. The following assignments yield a proper and monotonic loss: for every $A\in \sigma \left(\mathsf{\Theta }\right)$, $b_A=\lambda \left(A^c\right),$ $a_A=\lambda \left(A\right)/k,$ $c_A=\lambda \left(A^c\right)/k,$ and $d_A=\lambda \left(A\right)$. Another particular case is when $a_A=a_B,b_A=b_B,c_A=c_B,$ and $d_A=d_B$ for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$.

Another concept that helps us characterizing the Bayesian monotonic agnostic testing schemes is that of balanced relative losses, which we adapt from [7].

Definition 17 (Balanced Relative Loss). 

${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has balanced relative losses if, for all $A,B\in \sigma \left(\mathsf{\Theta }\right)$ such that $A\subset B$, for all ${\theta }_1\in A$ and ${\theta }_2\in B^c$, and for all $(i,j)\in D_{>}^2$,

$$\begin{array}{c}\hfill \frac{r_A^{(i,j)}\left({\theta }_1\right)}{r_A^{(i,j)}\left({\theta }_2\right)}\ge \frac{r_B^{(i,j)}\left({\theta }_1\right)}{r_B^{(i,j)}\left({\theta }_2\right)}\end{array}$$

Lemma 1. 

If ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has monotonic relative losses, then ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has balanced relative losses.

The following result shows that balanced relative losses characterize Bayesian monotonic ATS.

Theorem 4. 

Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of proper loss functions. Assume that for every $\theta \in \mathsf{\Theta }$ and $x\in \mathcal{X}$, $L_x\left(\theta \right)>0$. For every prior π for θ, let ${\mathcal{L}}^{\pi }$ denote a Bayesian ATS generated by ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$. There exists a monotonic ${\mathcal{L}}^{\pi }$ for every prior π if, and only if, ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has balanced relative losses.

Example 29. 

In Example 28, we obtained conditions on the loss functions ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ from Example 26 in order to guarantee that family to have monotonic relative losses. From Lemma 1 and Theorem 4, it follows that such family of loss functions yield monotonic Bayesian ATSs whatever the prior for θ is. In other words, there are family of loss functions that induce monotonic tests based on posterior probabilities.

5.2. Union Consonance

We now turn our attention towards characterizing union consonant Bayesian ATS using a decision theoretic framework.

Definition 18. 

${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is compatible with weak union consonance if there exists no $A,B\in \sigma \left(\mathsf{\Theta }\right)$, ${\theta }_1,{\theta }_2,{\theta }_3\in \mathsf{\Theta }$ and $p_1,p_2,p_3\ge 0$ such that $p_1+p_2+p_3=1$ and

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{p}_1·r_A^{(1,\frac{1}{2})}\left({\theta }_1\right)+p_2·r_A^{(1,\frac{1}{2})}\left({\theta }_2\right)+p_3·r_A^{(1,\frac{1}{2})}\left({\theta }_3\right)\hfill & <0\hfill \\ p_1·r_B^{(1,\frac{1}{2})}\left({\theta }_1\right)+p_2·r_B^{(1,\frac{1}{2})}\left({\theta }_2\right)+p_3·r_B^{(1,\frac{1}{2})}\left({\theta }_3\right)\hfill & <0\hfill \\ p_1·r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_1\right)+p_2·r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_2\right)+p_3·r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_3\right)\hfill & >0\hfill \end{array}\right.\end{array}$$

Definition 18 states that the family of loss functions ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ being compatible with weak union consonance cannot induce any Bayesian ATS on the basis of which one may prefer rejecting both hypotheses A and B over remaining agnostic about them while accepting $A\cup B$ rather than abstaining.

As we will see in the next theorem, proper loss functions compatible with weak union consonance characterize Bayesian ATSs that are weakly consonant with the union.

Theorem 5. 

Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of proper loss functions. Assume that for every $\theta \in \mathsf{\Theta }$ and $x\in \mathcal{X}$, $L_x\left(\theta \right)>0$. For every prior π for θ, let ${\mathcal{L}}^{\pi }$ denote a Bayesian ATS generated by ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$. There exists an ATS ${\mathcal{L}}^{\pi }$ that is weakly consonant with the union for every priori π if, and only if, ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is compatible with weak union consonance.

Example 30. 

We saw that the ATS from Example 2 is a Bayes test against a particular proper loss (Examples 26 and 27) and that it is weakly consonant with the union (Example 13). It follows from Theorem 5 that the family of loss functions that lead to this ATS are compatible with weak union consonance.

Definition 19 (Union consonance-balanced relative losses [7]). 

${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has union consonance-balanced relative losses if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$, ${\theta }_1\in A\cup B$ and ${\theta }_2\in {(A\cup B)}^c$,

$$\begin{array}{cc}& \frac{r_A^{(1,\frac{1}{2})}\left({\theta }_1\right)}{r_A^{(1,\frac{1}{2})}\left({\theta }_2\right)}\le \frac{r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_1\right)}{r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_2\right)},\mathbf{or}\hfill \\ & \frac{r_B^{(1,\frac{1}{2})}\left({\theta }_1\right)}{r_B^{(1,\frac{1}{2})}\left({\theta }_2\right)}\le \frac{r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_1\right)}{r_{A\cup B}^{(\frac{1}{2},0)}\left({\theta }_2\right)}\hfill \end{array}$$

Corollary 1. 

Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of proper loss functions. Assume that for every $\theta \in \mathsf{\Theta }$ and $x\in \mathcal{X}$, $L_x\left(\theta \right)>0$. If ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ does not have union consonance-balanced relative losses, then there exists a prior π such that every Bayesian ATS, ${\mathcal{L}}^{\pi }$, is not weakly consonant with the union.

5.3. Intersection Consonance

Next, we characterize intersection consonant Bayesian ATS under a Bayesian perspective.

Definition 20. 

${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is compatible with weak intersection consonance if there exists no $A,B\in \sigma \left(\mathsf{\Theta }\right)$, ${\theta }_1,{\theta }_2,{\theta }_3\in \mathsf{\Theta }$ and $p_1,p_2,p_3\ge 0$ such that $p_1+p_2+p_3=1$ and

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{p}_1·r_A^{(\frac{1}{2},0)}\left({\theta }_1\right)+p_2·r_A^{(\frac{1}{2},0)}\left({\theta }_2\right)+p_3·r_A^{(\frac{1}{2},0)}\left({\theta }_3\right)\hfill & >0\hfill \\ p_1·r_B^{(\frac{1}{2},0)}\left({\theta }_1\right)+p_2·r_B^{(\frac{1}{2},0)}\left({\theta }_2\right)+p_3·r_B^{(\frac{1}{2},0)}\left({\theta }_3\right)\hfill & >0\hfill \\ p_1·r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_1\right)+p_2·r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_2\right)+p_3·r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_3\right)\hfill & <0\hfill \end{array}\right.\end{array}$$

Definition 20 states that the family of loss functions ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ being compatible with weak intersection consonance cannot induce any Bayesian ATS on the basis of which one may prefer accepting both hypotheses A and B to remaining agnostic about them while rejecting $A\cap B$ rather than abstaining.

As we will see in the next theorem, proper loss functions compatible with weak intersection consonance characterize Bayesian ATSs that are weakly consonant with the intersection .

Theorem 6. 

Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of proper loss functions. Assume that for every $\theta \in \mathsf{\Theta }$ and $x\in \mathcal{X}$, $L_x\left(\theta \right)>0$. For every prior π for θ, let ${\mathcal{L}}^{\pi }$ denote a Bayesian ATS generated by ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$. There exists an ATS ${\mathcal{L}}^{\pi }$ that is weakly consonant with the intersection for every prior π if, and only if, ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is compatible with weak intersection consonance.

Example 31. 

We saw that the ATS from Example 2 is a Bayes test against a particular proper loss (Examples 26 and 27) and that it is weakly consonant with the intersection (Example 17). It follows from Theorem 6 that the family of loss functions that lead to this ATS are compatible with weak intersection consonance.

Definition 21 (Intersection consonance-balanced relative losses [7]). 

${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has intersection consonance-balanced relative losses if, for every $A,B\in \sigma \left(\mathsf{\Theta }\right)$, ${\theta }_1\in A\cap B$ and ${\theta }_2\in {(A\cap B)}^c$,

$$\begin{array}{cc}& \frac{r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_1\right)}{r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_2\right)}\le \frac{r_A^{(\frac{1}{2},0)}\left({\theta }_1\right)}{r_A^{(\frac{1}{2},0)}\left({\theta }_2\right)},\mathbf{or}\hfill \\ & \frac{r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_1\right)}{r_{A\cap B}^{(1,\frac{1}{2})}\left({\theta }_2\right)}\le \frac{r_B^{(\frac{1}{2},0)}\left({\theta }_1\right)}{r_B^{(\frac{1}{2},0)}\left({\theta }_2\right)}\hfill \end{array}$$

Corollary 2. 

Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of proper loss functions. Assume that for every $\theta \in \mathsf{\Theta }$ and $x\in \mathcal{X}$, $L_x\left(\theta \right)>0$. If ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ does not have intersection consonance-balanced relative losses, then there exists a prior π such that every Bayesian ATS, ${\mathcal{L}}^{\pi }$, is not weakly consonant with the intersection.

We end this section by noting that although we focused our results on weak consonance, they can be extended to strong consonance using the same techniques presented in the Appendix.

5.4. Invertibility

Finally, we examine invertible Bayesian ATSs from a decision-theoretic standpoint.

Definition 22 (Invertible Relative Losses). 

${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has invertible relative losses if, for every $A\in \sigma \left(\mathsf{\Theta }\right)$, for all ${\theta }_1\in A$, ${\theta }_2\in A^c$ and $(i,j)\in D_{>}^2$,

$$\begin{array}{cc}\hfill \frac{r_A^{(i,j)}\left({\theta }_1\right)}{r_A^{(i,j)}\left({\theta }_2\right)}& =\frac{r_{A^c}^{(i,j)}\left({\theta }_1\right)}{r_{A^c}^{(i,j)}\left({\theta }_2\right)}\hfill \end{array}$$

We end this section by showing that invertible Bayesian ATSs are determined by family of loss functions that fulfill the conditions of Definition 22.

Theorem 7. 

Let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be a family of proper loss functions. Assume that for every $\theta \in \mathsf{\Theta }$ and $x\in \mathcal{X}$, $L_x\left(\theta \right)>0$. For every prior π for θ, let ${\mathcal{L}}^{\pi }$ denote a Bayesian ATS generated by ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$. There exists an ATS ${\mathcal{L}}^{\pi }$ that is invertible for every prior π if, and only if, ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ has invertible relative losses.

Example 32. 

For every $A\in \sigma \left(\mathsf{\Theta }\right)$, let ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ be such that $L_A(1,\theta )=L_{A^c}(0,\theta )$ and $L_A(\frac{1}{2},\theta )=L_{A^c}(\frac{1}{2},\theta )$. It is easily seen that the conditions from Definition 22 hold. Theorem 7 then implies that any Bayesian ATS generated by ${\left(L_A\right)}_{A\in \sigma \left(\mathsf{\Theta }\right)}$ is invertible.

6. Final Remarks

Agnostic tests allow one to explicitly capture the difference between “not rejecting” and “accepting” a null hypothesis. When the agnostic decision is chosen, the null hypothesis is neither rejected or accepted. This possibility aligns with the idea that although precise null hypotheses can be tested, they shouldn’t be accepted. This idea is followed by the region based agnostic tests derived in this paper, which can either remain agnostic or reject precise null hypotheses.

This distinction provides a solution to the problem raised by Izbicki and Esteves [3], in which all (non-agnostic) logically coherent tests were shown to be based on point estimators which lack statistical optimality. We show that agnostic tests based on region estimators satisfy logical consistency and also allow statistical optimality. For example, agnostic tests based on frequentist confidence intervals control family wise error. Similarly, agnostic tests based on posterior density regions are shown to be an extension of the Full Bayesian Significance Test [11].

Future research includes investigating the consequences and generalizations of the logical requirements in this paper. For example, one could study what kinds of trivariate logic derive from the different definition of logical consistency studied in this paper. One could also generalize these logical requirements to generalized agnostic tests, in which one can decide among different degrees of agnosticism. The scale of such degrees can be either discrete or continuous. One could also investigate region estimator-based ATSs with respect to other optimality criteria such as statistical power.

The results of this paper can also be tied to the philosophical literature that studies the consequences and importance of precise hypothesis. Agnostic tests can be used to revisit the role of testing precise hypotheses in science. Agnostic tests also provide a framework to interpret the scientific meaning of measures of possibility or significance of precise hypotheses.

multiple