Deriving Proper Uniform Priors for Regression Coefficients, Parts I, II, and III

Abstract

It is a relatively well-known fact that in problems of Bayesian model selection, improper priors should, in general, be avoided. In this paper we will derive and discuss a collection of four proper uniform priors which lie on an ascending scale of informativeness. It will turn out that these priors lead us to evidences that are closely associated with the implied evidence of the Bayesian Information Criterion (BIC) and the Akaike Information Criterion (AIC). All the discussed evidences are then used in two small Monte Carlo studies, wherein for different sample sizes and noise levels the evidences are used to select between competing C-spline regression models. Also, there is given, for illustrative purposes, an outline on how to construct simple trivariate C-spline regression models. In regards to the length of this paper, only one half of this paper consists of theory and derivations, the other half consists of graphs and outputs of the two Monte Carlo studies.

1. Introduction

Using informational consistency requirements, Jaynes [1] derived the form of maximal non-informative priors for location parameters, that is, regression coefficients, to be uniform. However, this does not tell us what the limits of these uniform probability distributions should be, that is, what particular uniform distribution to use. If we are faced with a parameter estimation problem, then these limits of the uniform prior are irrelevant, as we may scale the product of the improper uniform prior and the likelihood to one, which gives us a properly normalized posterior for our regression coefficients. However, if we are faced with a problem of model selection, then the volume covered by the uniform prior is an integral part of the evidence which is used to rank the various competing regression models.

In this paper we will give the four proper uniform priors originally derived in [2]. These priors lie on an ascending scale of informativeness. It will turn out, as we discuss the Bayesian Information Criterion (BIC), the Akaike Information Criterion (AIC), and the results of a small Monte Carlo study, that these priors lead us to evidences that are closely associated with the implied evidences of the BIC and the AIC, as these evidences fill in the space between and around the BIC and AIC on a continuum of conservativeness, in terms of the number of parameters of the chosen regression analysis models.

This paper is structured as follows. First we give an introduction to the evidence construct, that “too-often-ignored half of Bayesian inference” [3], as we give an outline on how to use these evidences in Bayesian model selection. Then we describe the normal multiple regression models for both known and unknown $\sigma$s, after which we specify the conditions under which improper priors become problematic for model selection. This specification brings us naturally to a continuum of informativeness on which priors of regression coefficients may be located. After these preliminaries, we proceed to give the derivations of the four proper uniform priors, originally derived in [2], by way of the results in [4], which are neither grossly ignorant nor grossly knowledgeable. Having checked the coverage of these priors, we address the question what constitutes data and what constitutes prior information. We then discuss the evidences that are associated with our proper priors, as we connect these evidences to the BIC and AIC reference procedures and give the posterior probability distribution of the unknown regression coefficients and the consequent predictive probability distribution that is associated with these proper priors. In Appendix A we report on two small Monte Carlo studies with the C-spline regression models, in order to give the reader a sense for all the discussed evidences. Also, a collection of three simple trivariate C-spline regression models will be discussed in Appendix B, in order to provide the reader with a low-level, hands-on introduction into C-splines [5].

2. The Evidence and Bayesian Model Selection

Bayesian probability theory has four fundamental constructs, namely, the prior, the likelihood, the posterior, and the evidence. These constructs are related in the following way:

$$\mathrm{posterior}=\frac{\mathrm{prior}\times \mathrm{likelihood}}{\mathrm{evidence}}.$$

(1)

Most of us will be familiar with the prior, likelihood, and posterior. However, the evidence concept is less universally known, as most people come to Bayes by way of the more compact relationship [6]:

$$\mathrm{posterior}\propto \mathrm{prior}\times \mathrm{likelihood},$$

(2)

which does not make any explicit mention of the evidence construct. In what follows, we will employ the correct, though notationally more cumbersome, relation (1), and forgo of the more compact, but incomplete, Bayesian shorthand (2). This is done so the reader may develop some feeling for the evidence construct, and how this construct relates to the other three Bayesian constructs (i.e., the prior, likelihood, and posterior.)

Let $p\left(\left.\theta \right|I\right)$ be the prior of some parameter $\theta$, where I is the prior information model of the unknown $\theta$. Let $p\left(\left.D\right|\theta ,M\right)$ be the probability of the data D conditional on the value of parameter $\theta$ and the likelihood model M which is used; the probability of the data is also known as the likelihood of the parameter $\theta$. Let $p\left(\left.\theta \right|D,M,I\right)$ be the posterior distribution of the parameter $\theta$, conditional on the data D, the likelihood model M, and the prior information model I. Then

$$p\left(\left.\theta \right|D,M,I\right)=\frac{p\left(\left.\theta \right|I\right)p\left(\left.D\right|\theta ,M\right)}{\int p\left(\left.\theta \right|I\right)p\left(\left.D\right|\theta ,M\right)d\theta }=\frac{p\left(\left.\theta \right|I\right)p\left(\left.D\right|\theta ,M\right)}{p\left(\left.D\right|M,I\right)},$$

(3)

where

$$p\left(\left.D\right|M,I\right)=\int p\left(\left.\theta ,D\right|M,I\right)d\theta =\int p\left(\left.\theta \right|I\right)p\left(\left.D\right|\theta ,M\right)d\theta$$

(4)

is the evidence, that is, the marginalized likelihood of both the likelihood model M and the prior information model I.

Now, if we have a set of likelihood models $M_i$ (e.g., a collection of regression models) we wish to choose from, and just the one prior information model I (e.g., an ignorance model), then we may do so by computing the evidence values $p\left(\left.D\right|M_i,I\right)$.

Let $p\left(M_i\right)$ and $p\left(\left.M_i\right|D,I\right)$ be, respectively, the prior and posterior probability of the likelihood model $M_i$. Then the posterior probability distribution of these likelihood models is given as

$$p\left(\left.M_i\right|D,I\right)=\frac{p\left(M_i\right)p\left(\left.D\right|M_i,I\right)}{{\sum }_{i}p\left(M_i\right)p\left(\left.D\right|M_i,I\right)}.$$

(5)

For $p\left(M_i\right)=p\left(M_j\right)$ for $i\ne j$, the posterior probabilities (5) will reduce to the normalized evidence values:

$$p\left(\left.M_i\right|D,I\right)=\frac{p\left(\left.D\right|M_i,I\right)}{{\sum }_{i}p\left(\left.D\right|M_i,I\right)}.$$

(6)

So, if we assign equal prior probabilities to our likelihood models $M_i$, then we may rank these models by way of their respective evidence values, where the model with the highest evidence value is the model which has the highest posterior probability of all the models that were taken into consideration [7,8].

3. The Normal Multiple Regression Model (Known Sigma)

Let the model M for the response vector $\mathbf{y}$ be

$$\mathbf{y}=X\mathbf{\beta }+\mathbf{e},$$

(7)

where X is some $N\times m$ predictor matrix, $\mathbf{\beta }$ is the $m\times 1$ vector with regression coefficients and $\mathbf{e}$ is the $N\times 1$ error vector to which we assign a multivariate normal distribution, that is,

$$p\left(\left.\mathbf{e}\right|\sigma \right)=\frac{1}{{\left(2\pi {\sigma }^2\right)}^{N/2}}exp\left(-\frac{{\mathbf{e}}^T\mathbf{e}}{2{\sigma }^2}\right),$$

(8)

or, equivalently, $\mathbf{e}\sim MN\left(\mathbf{0},{\sigma }^2\mathbf{I}\right)$, where $\mathbf{I}$ is the $N\times N$ identity matrix and $\sigma$ is some known standard deviation. By way of a simple Jacobian transformation from $\mathbf{e}$ to $\mathbf{y}$ in (8), we then may obtain the likelihood function of the $\mathbf{\beta }$s:

$$p\left(\left.\mathbf{y}\right|\sigma ,X,\mathbf{\beta },M\right)=\frac{1}{{\left(2\pi {\sigma }^2\right)}^{N/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)\right].$$

(9)

If we assign a uniform prior to the unknown regression coefficients $\mathbf{\beta }$ [6]

$$p\left(\left.\mathbf{\beta }\right|I\right)=C,\mathbf{\beta }\in D_{\mathbf{\beta }},$$

(10)

where C is a yet unspecified normalizing constant, I is the prior information regarding the unknown $\mathbf{\beta }$s which we have at our disposal, and $D_{\mathbf{\beta }}$ is the prior domain of the $\mathbf{\beta }$s, then the probability distribution of both $\mathbf{\beta }$ and $\mathbf{y}$ is derived as

$$p\left(\left.\mathbf{\beta },\mathbf{y}\right|\sigma ,X,M,I\right)=p\left(\left.\mathbf{\beta }\right|I\right)p\left(\left.\mathbf{y}\right|\sigma ,X,\mathbf{\beta },M\right)=\frac{C}{{\left(2\pi {\sigma }^2\right)}^{N/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)\right].$$

(11)

By integrating the unknown $\mathbf{\beta }$s out of (11) over the prior domain $D_{\mathbf{\beta }}$, we obtain the evidence of model M:

$$p\left(\left.\mathbf{y}\right|\sigma ,X,M,I\right)={\int }_{D_{\mathbf{\beta }}}p\left(\left.\mathbf{\beta },\mathbf{y}\right|\sigma ,X,M,I\right)d\mathbf{\beta }.$$

(12)

The evidence (12) is used both to normalize (11) into a posterior distribution, (1), as well as to choose between competing regression models, (5) and (6). In order to evaluate the evidence (12), we rewrite (11) as [6]

$$p\left(\left.\mathbf{\beta },\mathbf{y}\right|\sigma ,X,M,I\right)=\frac{C}{{\left(2\pi {\sigma }^2\right)}^{N/2}}exp\left\{-\frac{1}{2{\sigma }^2}\left[{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)+{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]\right\},$$

(13)

where

$$\widehat{\mathbf{\beta }}={\left(X^{T}X\right)}^{-1}{X}^T\mathbf{y}\mathrm{and}\widehat{\mathbf{y}}=X\widehat{\mathbf{\beta }}.$$

(14)

We then factor (13) as

$$\begin{array}{cc}\hfill p\left(\left.\mathbf{\beta },\mathbf{y}\right|\sigma ,X,M,I\right)& =\frac{C}{{\left|X^{T}X\right|}^{1/2}{\left(2\pi {\sigma }^2\right)}^{\left(N-m\right)/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)\right]\hfill \\ & \times \frac{{\left|X^{T}X\right|}^{1/2}}{{\left(2\pi {\sigma }^2\right)}^{m/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right].\hfill \end{array}$$

(15)

The last term in (15) is in the multivariate normal form [6], so it should evaluate to 1 when integrated over the $\mathbf{\beta }$s. Stated differently, for a prior domain $D_{\mathbf{\beta }}$ which is centered correctly and ‘wide enough’, we have, by way of the factorization (15), that the evidence (12) tends to the equality

$$p\left(\left.\mathbf{y}\right|\sigma ,X,M,I\right)=\frac{C}{{\left|X^{T}X\right|}^{1/2}{\left(2\pi {\sigma }^2\right)}^{\left(N-m\right)/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)\right].$$

(16)

By way of (13), (16) and the product rule (1), we obtain the posterior of the unknown $\mathbf{\beta }$s, [6]:

$$p\left(\left.\mathbf{\beta }\right|\sigma ,\mathbf{y},X,M,I\right)=\frac{p\left(\left.\mathbf{\beta },\mathbf{y}\right|\sigma ,X,M,I\right)}{p\left(\left.\mathbf{y}\right|\sigma ,X,M,I\right)}=\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(2\pi {\sigma }^2\right)}^{m/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right].$$

(17)

This posterior of the unknown $\mathbf{\beta }$s has a mean of $\widehat{\mathbf{\beta }}={\left(X^{T}X\right)}^{-1}{X}^T\mathbf{y}$, (14), and a covariance matrix of ${\left(X^{T}X/{\sigma }^2\right)}^{-1}$.

In the parameter estimation problem, that is, the derivation of the posterior distribution (17), any reference to the normalizing constant C of the uniform prior (10) has fallen away. In contrast, in the model selection problem, that is, the derivation of the evidence (16), C is still present.

In closing, note that different $N\times m_i$ predictor matrices $X_i$ correspond with different likelihood models $M_i$ in (5) and (6). It is to be understood that in what follows we will construct proper uniform priors for a generic likelihood model M which has a generic $N\times m$ predictor matrix X, as we drop the sub-index j in both X and M in order to remove some of the notational clutter in our equations.

4. The Normal Multiple Regression Model (Unknown Sigma)

In case of unknown $\sigma$, we may assign the Jeffreys prior for scaling parameters [6]:

$$p\left(\left.\sigma \right|I\right)=\frac{A}{\sigma },$$

(18)

where A is some normalizing constant, to the unknown $\sigma$ in (11), in order to lose this unknown nuisance parameter by way of integration:

$$p\left(\left.\mathbf{\beta },\mathbf{y}\right|X,M,I\right)=\int p\left(\left.\sigma ,\mathbf{\beta },\mathbf{y}\right|X,M,I\right)d\sigma =\int p\left(\left.\sigma ,\mathbf{\beta },\mathbf{y}\right|X,M,I\right)d\sigma ,$$

(19)

where (11) and (18),

$$p\left(\left.\sigma \right|I\right)p\left(\left.\mathbf{\beta },\mathbf{y}\right|\sigma ,X,M,I\right)=\frac{AC}{{\left(2\pi \right)}^{N/2}{\sigma }^{N+1}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)\right].$$

(20)

We may conveniently factorize (20) as,

$$\begin{array}{cc}\hfill p\left(\left.\sigma ,\mathbf{\beta },\mathbf{y}\right|X,M,I\right)& =\frac{A\mathsf{\Gamma }\left(N/2\right)}{2{\pi }^{N/2}}\frac{C}{{\left[{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)\right]}^{N/2}}\hfill \\ & \times \frac{2}{\mathsf{\Gamma }\left(N/2\right)}{\left[\frac{{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)}{2}\right]}^{N/2}\frac{1}{{\sigma }^{N+1}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)\right].\hfill \end{array}$$

(21)

The last term in (21) evaluates to 1 when integrated over $\sigma$, as it has the form of an inverted gamma distribution [6], from which it follows that

$$p\left(\left.\mathbf{\beta },\mathbf{y}\right|X,M,I\right)=\frac{A\mathsf{\Gamma }\left(N/2\right)}{2{\pi }^{N/2}}\frac{C}{{\left[{\left(\mathbf{y}-X\mathbf{\beta }\right)}^T\left(\mathbf{y}-X\mathbf{\beta }\right)\right]}^{N/2}},$$

(22)

By integrating the unknown $\mathbf{\beta }$s out of (22) over the prior domain $D_{\mathbf{\beta }}$, we obtain the evidence of model M:

$$p\left(\left.\mathbf{y}\right|X,M,I\right)={\int }_{D_{\mathbf{\beta }}}p\left(\left.\mathbf{\beta },\mathbf{y}\right|X,M,I\right)d\mathbf{\beta }.$$

(23)

In order to evaluate the evidence (23), we rewrite (22) as [6]

$$p\left(\left.\mathbf{\beta },\mathbf{y}\right|X,M,I\right)=\frac{A\mathsf{\Gamma }\left(N/2\right)}{2{\pi }^{N/2}}\frac{C}{{\left[{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)+{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]}^{N/2}}.$$

(24)

We then factor (24) as

$$\begin{array}{cc}\hfill p\left(\left.\mathbf{\beta },\mathbf{y}\right|X,M,I\right)& =\frac{1}{{\left|X^{T}X\right|}^{1/2}}\frac{C}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^{N-m}}\frac{A\mathsf{\Gamma }\left[\left(N-m\right)/2\right]}{2{\pi }^{\left(N-m\right)/2}}\hfill \\ & \times \frac{\mathsf{\Gamma }\left(N/2\right)}{\mathsf{\Gamma }\left[\left(N-m\right)/2\right]}\frac{{\left|X^{T}X\right|}^{1/2}}{{\pi }^{m/2}}\frac{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^{N-m}}{{\left[{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2+{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]}^{N/2}},\hfill \end{array}$$

(25)

where

$${∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2={\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right),$$

(26)

and where the last term in (25) is in the multivariate Student-t form [6]. So, for a prior domain $D_{\mathbf{\beta }}$ which is centered correctly and “wide enough”, we have, by way of the factorization (25), that the evidence (23) tends to the equality

$$p\left(\left.\mathbf{y}\right|X,M,I\right)=\frac{1}{{\left|X^{T}X\right|}^{1/2}}\frac{C}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^{N-m}}\frac{A\mathsf{\Gamma }\left[\left(N-m\right)/2\right]}{2{\pi }^{\left(N-m\right)/2}}.$$

(27)

If we divide (24) by the evidence (27), we obtain, by way of the product rule (1), the posterior of the unknown $\mathbf{\beta }$s, [6]:

$$p\left(\left.\mathbf{\beta }\right|\mathbf{y},X,M,I\right)=\frac{v^{v/2}\mathsf{\Gamma }\left(N/2\right){\left|\frac{1}{s^2}{X}^{T}X\right|}^{1/2}}{\mathsf{\Gamma }\left[\left(N-m\right)/2\right]{\pi }^{m/2}}{\left[v+{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T\left(\frac{1}{s^2}{X}^{T}X\right)\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]}^{-N/2}.$$

(28)

where

$$s^2=\frac{1}{v}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2\mathrm{and}v=N-m.$$

(29)

This posterior of the unknown $\mathbf{\beta }$s has a mean of $\widehat{\mathbf{\beta }}={\left(X^{T}X\right)}^{-1}{X}^T\mathbf{y}$, (14), and a covariance matrix of ${\left(X^{T}X/s^2\right)}^{-1}$, (29).

Again, in the parameter estimation problem, that is, the derivation of the posterior distribution (28), any reference to the normalizing constant C of the uniform prior (10) has, seemingly, fallen away. In contrast, in the model selection problem, that is, the derivation of the evidence (27), C is still present.

5. The Problem with Improper Priors

In problems of model comparison between competing (regression) models one generally must take care not to use improper priors, be they uniform or not. Since improper priors may introduce inverse infinities in the evidence factors which do not cancel out if one proceeds to compute the posterior probabilities of the respective models [9]. We will demonstrate this fact and its consequences with a simple example in which we assign improper uniform priors to the respective regression coefficients.

Suppose that we want to compare two regression models:

$$M_1:\mathbf{y}=X_1{\mathbf{\beta }}_1+{\mathbf{e}}_1\mathrm{and}{M}_2:\mathbf{y}=X_2{\mathbf{\beta }}_2+{\mathbf{e}}_2,$$

(30)

where $X_1$ is an $N\times m_1$ predictor matrix and $X_2$ an $N\times m_2$, with $m_2>m_1$, and where both ${\mathit{e}}_1$ and ${\mathit{e}}_1$ are multivariate normally distributed $MN\left(\mathbf{0},{\sigma }^2\mathbf{I}\right)$, where $\mathbf{I}$ is the $N\times N$ identity matrix and $\sigma$ is some known standard deviation, (8). Let the uniform prior of a regression coefficient be given as

$$p\left(\left.{\beta }_j\right|I\right)=\frac{1}{2B},\mathrm{for}-B\le {\beta }_j\le B,$$

(31)

for $j=1,\dots ,m$. If $B\to \infty$, then (31) will tend to the improper Jeffreys prior for location parameters [6]:

$$p\left(\left.{\beta }_j\right|I\right)d{\beta }_j\propto d{\beta }_j,\mathrm{for}-\infty \le {\beta }_j\le \infty ,$$

(32)

where “∝” is the proportionality sign that absorbs the normalizing constant $1/\left(2\infty \right)$. Let the uniform prior of m regression coefficients be given as, (31),

$$p\left(\left.\mathbf{\beta }\right|I\right)=\prod _{j=1}^{m}p\left(\left.{\beta }_j\right|I\right)={\left(\frac{1}{2B}\right)}^m,\mathrm{for}\mathbf{\beta }\in D_{\mathbf{\beta }},$$

(33)

where $D_{\mathbf{\beta }}$ is an m-dimensional cube which is centered at the origin. Substituting (33) into (10), we find the evidences:

$$p\left(\left.\mathbf{y}\right|\sigma ,X_i,M_i,I\right)=\frac{A}{{\left(2B\right)}^{m_i}}{L}_i,$$

(34)

for $i=1,2$, where (27)

$$L_i=\frac{1}{{\left|X^{T}X\right|}^{1/2}}\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^{N-m}}\frac{\mathsf{\Gamma }\left[\left(N-m\right)/2\right]}{2{\pi }^{\left(N-m\right)/2}},$$

(35)

and $m_i$ is the number of columns in the $N\times m_i$ predictor matrix $X_i$, and ${\widehat{\mathbf{y}}}_i$ is the regression model estimate (14)

$${\widehat{\mathbf{\beta }}}_i={\left(X_i^T{X}_i\right)}^{-1}{X}_i^T\mathbf{y}\mathrm{and}{\widehat{\mathbf{y}}}_i=X_i{\widehat{\mathbf{\beta }}}_i.$$

(36)

If we assign equal prior probabilities to $M_1$ and $M_2$, then we find posterior model probabilities, (6) and (34):

$$p\left(\left.M_1\right|\sigma ,X_1,\mathbf{y},I\right)=\frac{L_1}{L_1+\frac{1}{{\left(2B\right)}^{m_2-m_1}}{L}_2}\mathrm{and}p\left(\left.M_2\right|\sigma ,X_2,\mathbf{y},I\right)=\frac{\frac{1}{{\left(2B\right)}^{m_2-m_1}}{L}_2}{L_1+\frac{1}{{\left(2B\right)}^{m_2-m_1}}{L}_2},$$

(37)

as $m_2>m_1$, (30). So, if in (31) we let $B\to \infty$, then the posterior model probabilities (37) will tend to

$$p\left(\left.M_1\right|\sigma ,X_1,\mathbf{y},I\right)\to \frac{L_1}{L_1}=1\mathrm{and}p\left(\left.M_2\right|\sigma ,X_2,\mathbf{y},I\right)\to \frac{0}{L_1+0}=0.$$

(38)

It can be seen that the assigning of an improper Jeffreys’ prior to location parameters (32) will make that the regression model with the least number of regression coefficients, or, equivalently, number of predictors, is automatically chosen over any model which has more regression coefficients.

Improper priors can introduce inverse infinities in the evidence factors, as ${\left(2B\right)}^{-m_2+m_1}$ in (37), which do not cancel out if one proceeds to compute the posterior probabilities of the respective models. However, if the parameter in question is shared by all the competing models, like, for example, the parameter $\sigma$ in (1), then the inverse infinities will cancel out, like A cancels out in (37). This is why care must be taken to let the prior for the regression coefficients $\mathbf{\beta }$, (10), be proper, while, at the same time, as both a mathematical and a modeling convenience, one may let the prior of $\sigma$, (18), be improper.

6. A Continuum of Informativeness

The Jeffreys prior for location parameters (32),

$$p\left(\left.{\beta }_j\right|I\right)d{\beta }_j\propto d{\beta }_j,\mathrm{for}-\infty \le {\beta }_j\le \infty ,$$

represents a limit of gross ignorance as we are even ignorant about the possible limits of the parameters ${\beta }_j$. This gross ignorance leads to evidences that are extremely conservative in that they will always choose the regression model with least number of regression coefficients, (38).

An opposite limit of gross knowledgeableness is the empirical “sure thing” prior [3]:

$$p\left(\left.\mathbf{\beta }\right|\widehat{\mathbf{\beta }},“\mathrm{sure}\mathrm{thing}”\right)=\delta \left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right),$$

(39)

where $\delta$ is the multivariate Dirac delta function for which we have

$${\int }_{-\infty }^{\infty }\delta \left(\mathbf{x}-\mathbf{c}\right)f\left(\mathbf{x}\right)d\mathbf{x}=f\left(\mathbf{c}\right).$$

(40)

The evidence that corresponds with the “sure thing” prior may be derived as, (9), (14), (18), (26), (39), and (40):

$$\begin{array}{cc}\hfill p\left(\left.\mathbf{y}\right|X,\widehat{\mathbf{\beta }},“\mathrm{sure}\mathrm{thing}”\right)& ={\int }_0^{\infty }{\int }_{-\infty }^{\infty }p\left(\left.\sigma ,\mathbf{\beta },\mathbf{y}\right|X,\widehat{\mathbf{\beta }},“\mathrm{sure}\mathrm{thing}”\right)d\mathbf{\beta }d\sigma \hfill \\ & ={\int }_0^{\infty }{\int }_{-\infty }^{\infty }p\left(\left.\sigma \right|I\right)p\left(\left.\mathbf{\beta }\right|\widehat{\mathbf{\beta }},“\mathrm{sure}\mathrm{thing}”\right)p\left(\left.\mathbf{y}\right|\sigma ,X,\mathbf{\beta },M\right)d\mathbf{\beta }d\sigma \hfill \\ & \propto \frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},\hfill \end{array}$$

(41)

where the “∝” symbol is used to absorb the factor $A\mathsf{\Gamma }\left(N/2\right)/\left(2{\pi }^{N/2}\right)$.

Since an increase in the number of predictors m tends to decrease the length of the error vector $∥\mathbf{y}-\widehat{\mathbf{y}}∥$, with a limit length of zero as the number of predictors m tends to the sample size N:

$$∥\mathbf{y}-\widehat{\mathbf{y}}∥\to 0,\mathrm{as}m\to N,$$

(42)

we have that, (41) and (42),

$$p\left(\left.\mathbf{y}\right|X,\widehat{\mathbf{\beta }},“\mathrm{sure}\mathrm{thing}”\right)\to \infty ,\mathrm{as}m\to N.$$

(43)

So, the gross knowledgeableness of the “sure thing” prior leads to evidences that are extremely liberal in that they will tend to choose regression models which have the largest number of regression coefficients.

In what follows we will derive a suite of priors on the continuum of informativeness that are more informed than the improper Jeffreys prior for location parameters (32) and less knowledgeable than the “sure thing” prior (39). It will be shown that the corresponding evidences, as a consequence, will be less conservative than the evidence (34) in its limit of $B\to \infty$, and less liberal than the maximum likelihood evidence (41).

7. A Proper Ignorance Prior

We now proceed to construct a more informed, proper (i.e., non-zero) inverse normalizing “constant” C for the prior (10). By way of (7) and (14), we have for a $N\times m$ predictor matrix X of rank m that

$$\mathbf{\beta }={\left(X^{T}X\right)}^{-1}{X}^T\left(\mathbf{y}-\mathbf{e}\right)={\left(X^{T}X\right)}^{-1}{X}^T\mathbf{z},$$

(44)

where

$$\mathbf{z}=\mathbf{y}-\mathbf{e},$$

(45)

and $\mathbf{e}\sim MN\left(\mathbf{0},{\sigma }^2\mathbf{I}\right)$, (8). Closer inspection of (44) shows us that the parameter space of $\mathbf{\beta }$ is constrained by the difference vector $\mathbf{z}$.

For the special case where the predictor matrix X is an $N\times 1$ vector $\mathbf{x}$ we have that

$$\mathbf{\beta }=\frac{{\mathbf{x}}^T\mathbf{z}}{{\mathbf{x}}^T\mathbf{x}}=\frac{∥\mathbf{x}∥∥\mathbf{z}∥}{{∥\mathbf{x}∥}^2}cos\varphi ,$$

(46)

where $\varphi$ is the angle between the predictor vector $\mathbf{x}$ and the difference vector $\mathbf{z}$. Given that $-1\le cos\varphi \le 1$, we may by way of (46) put definite bounds on $\beta$:

$$-\frac{max∥\mathbf{z}∥}{∥\mathbf{x}∥}\le \beta \le \frac{max∥\mathbf{z}∥}{∥\mathbf{x}∥}.$$

(47)

So, if we assign a uniform distribution to the regression coefficient $\beta$, then this uniform distribution is defined on a line-piece of length $2max∥\mathbf{z}∥/∥\mathbf{x}∥$. It follows that for the case of just the one regression coefficient, the prior (10) is

$$p\left(\left.\beta \right|\mathbf{x},max∥\mathbf{z}∥,I\right)=\frac{∥\mathbf{x}∥}{2max∥\mathbf{z}∥}$$

(48)

where (48) is understood to be defined on the interval (47) which is centered at the origin.

In order to generalize (48) to the general multivariate case, we first must generalize (47) to its multivariate case. This may be done as follows [4]. Let X be a $N\times m$ predictor matrix consisting of m independent vectors ${\mathbf{x}}_j$. The vectors ${\mathbf{x}}_j$, because of their independence, then will span a m-dimensional subspace $S_m$. It follows, trivially, that we may decompose $\mathbf{z}$ into a part that lies inside of this subspace and a part that lies outside, say,

$$\mathbf{z}=\widehat{\mathbf{z}}+\mathbf{n},$$

(49)

where $\widehat{\mathbf{z}}$ is the part of $\mathbf{z}$ that is projected on $S_m$ and $\mathbf{n}$ is the part of $\mathbf{z}$ that is orthogonal to $S_m$. The orthogonality of $\mathbf{n}$ to $S_m$ implies that

$${\mathbf{x}}_j^T\mathbf{n}=0,$$

(50)

for $j=1,\dots ,m$, whereas the fact that $\widehat{\mathbf{z}}$ is a projection on $S_m$ implies that

$$\widehat{\mathbf{z}}=\sum _{j=1}^m{\mathbf{x}}_j{\beta }_j,$$

(51)

where, by construction, (49), (50), and the assumed independence of the ${\mathbf{x}}_j$,

$${\beta }_j=\frac{{\mathbf{x}}_j^T\mathbf{z}}{{\mathbf{x}}_j^T{\mathbf{x}}_j}=\frac{{\mathbf{x}}_j^T\left(\widehat{\mathbf{z}}+\mathbf{n}\right)}{{\mathbf{x}}_j^T{\mathbf{x}}_j}=\frac{{\mathbf{x}}_j^T\widehat{\mathbf{z}}}{{\mathbf{x}}_j^T{\mathbf{x}}_j}=\frac{∥\widehat{\mathbf{z}}∥}{∥{\mathbf{x}}_j∥}cos{\varphi }_j.$$

(52)

Now, because of the independence of the ${\mathbf{x}}_j$ we have that

$${\mathbf{x}}_i^T{\mathbf{x}}_j=0,$$

(53)

for $i\ne j$. So, if we take the norm of (51) we find

$${∥\widehat{\mathbf{z}}∥}^2={∥\sum _{j=1}^m{\mathbf{x}}_j{\beta }_j∥}^2={∥\widehat{\mathbf{z}}∥}^2\sum _{j=1}^m{cos}^2{\varphi }_j.$$

(54)

It follows from (54) that the angles ${\varphi }_j$ in (52) must obey the constraint

$$\sum _{j=1}^m{cos}^2{\varphi }_j=1.$$

(55)

Combining (52) and (55), we see that the regression coefficients ${\beta }_j$ must lie on the surface of an m-variate ellipsoid centered at the origin and with axes which have respective lengths of

$$r_j=\frac{∥\widehat{\mathbf{z}}∥}{∥{\mathbf{x}}_j∥}.$$

(56)

Since

$$∥\widehat{\mathbf{z}}∥\le ∥\mathbf{z}∥\le max∥\mathbf{z}∥,$$

(57)

the axes (56) may be maximized through our prior knowledge of the maximal length of of the outcome variable $\mathbf{z}$:

$$max{r}_j=\frac{max∥\mathbf{z}∥}{∥{\mathbf{x}}_j∥}.$$

(58)

It follows that the regression coefficients ${\beta }_j$ are constrained to lie in the m-variate ellipsoid that is centered at the origin and has axes of length (58). If we substitute (58) into the identity for the volume of an m-variate ellipsoid

$$V=\frac{{\pi }^{m/2}}{\mathsf{\Gamma }\left[(m+2)/2\right]}\prod _{j=1}^m{r}_j,$$

(59)

we find that the parameter space of $\mathbf{\beta }$ has a maximal prior volume of

$$V=\frac{{\pi }^{m/2}}{\mathsf{\Gamma }\left[(m+2)/2\right]}\frac{{\left(max∥\mathbf{z}∥\right)}^m}{{\prod }_{j=1}^m∥{\mathbf{x}}_j∥}.$$

(60)

Now, let $X\equiv \left[{\mathbf{x}}_1\cdots {\mathbf{x}}_m\right]$. Then for orthogonal predictors ${\mathbf{x}}_j$ the product of the norms is equivalent to the square root of the determinant of $X^{T}X$, that is,

$$\prod _{j=1}^m∥{\mathbf{x}}_j∥={\left|X^{T}X\right|}^{1/2},$$

(61)

which is also the volume of the parallelepiped defined by the vectors ${\mathbf{x}}_j$. If the predictor matrix X is non-orthogonal, then we may use a Gram–Schmidt process to transform X to the orthogonal matrix $\tilde{X}$, say, where, because of invariance of the volume of a parallelepiped under orthogonalization,

$${\left|{\tilde{X}}^T\tilde{X}\right|}^{1/2}={\left|X^{T}X\right|}^{1/2}.$$

(62)

So, by way of (60), (61), and (62), it follows that (47) generalizes to the statement that for general (i.e., non-orthogonal) $N\times m$ predictor matrices X the regression coefficient vectors $\mathbf{\beta }$ are constrained to lie in an m-dimensional ellipsoid which is centered on the origin and has a volume of

$$V=\frac{{\pi }^{m/2}}{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}\frac{{\left(max∥\mathbf{z}∥\right)}^m}{{\left|X^{T}X\right|}^{1/2}}.$$

(63)

And the inverse of this volume gives us the corresponding multivariate generalization of the uniform prior (48):

$$p\left(\left.\mathbf{\beta }\right|X,max∥\mathbf{z}∥,I\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}{{\pi }^{m/2}}\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(max∥\mathbf{z}∥\right)}^m},$$

(64)

where (64) is understood to be defined on some ellipsoid having volume (63) and a centroid located at the origin.

Because of the triangle inequality [10], we have that

$$∥\mathbf{y}-\mathbf{e}∥\le ∥\mathbf{y}∥+∥\mathbf{e}∥.$$

(65)

From (45) and (65), it follows trivially that

$$max∥\mathbf{z}∥\le max∥\mathbf{y}∥+max∥\mathbf{e}∥.$$

(66)

As to the first term in the right-hand of (66), let $max\left|y\right|$ be a prior assessment of the maximum absolute value of the dependent variable y. Then we may assign the following simple bound on the length of the vector $\mathbf{y}$:

$$max∥\mathbf{y}∥=\sqrt{N}max\left|y\right|.$$

(67)

As to the second term in the right-hand of (66), the error vector $\mathbf{e}$ has known multivariate probability distribution (8). If we rewrite the elements in $\mathbf{e}$ as a function of its norm $∥\mathbf{e}∥$ and the angles ${\alpha }_1,\dots ,{\alpha }_{N-1}$ [6]

$$\begin{array}{cc}\hfill e_1& =∥\mathbf{e}∥cos{\alpha }_{1}cos{\alpha }_{2}cos{\alpha }_3\cdots cos{\alpha }_{N-q}cos{\alpha }_{N-q+1}\cdots cos{\alpha }_{N-3}cos{\alpha }_{N-2}cos{\alpha }_{N-1}\hfill \\ \hfill e_2& =∥\mathbf{e}∥cos{\alpha }_{1}cos{\alpha }_{2}cos{\alpha }_3\cdots cos{\alpha }_{N-q}cos{\alpha }_{N-q+1}\cdots cos{\alpha }_{N-3}cos{\alpha }_{N-2}sin{\alpha }_{N-1}\hfill \\ \hfill e_3& =∥\mathbf{e}∥cos{\alpha }_{1}cos{\alpha }_{2}cos{\alpha }_3\cdots cos{\alpha }_{N-q}cos{\alpha }_{N-q+1}\cdots cos{\alpha }_{N-3}sin{\alpha }_{N-2}\hfill \\ \hfill ⋮& \\ \hfill e_q& =∥\mathbf{e}∥cos{\alpha }_{1}cos{\alpha }_{2}cos{\alpha }_3\cdots cos{\alpha }_{N-q}sin{\alpha }_{N-q+1}\hfill \\ \hfill ⋮& \\ \hfill e_{N-2}& =∥\mathbf{e}∥cos{\alpha }_{1}cos{\alpha }_{2}sin{\alpha }_3\hfill \\ \hfill e_{N-1}& =∥\mathbf{e}∥cos{\alpha }_{1}sin{\alpha }_2\hfill \\ \hfill e_N& =∥\mathbf{e}∥sin{\alpha }_1,\hfill \end{array}$$

(68)

where $0<∥\mathbf{e}∥<\infty$, $-\pi /2<{\alpha }_i<\pi /2$, for $i=1,2,\dots ,N-2$, and $0<{\alpha }_{N-1}<2\pi$, and which has as its Jacobian

$$J={∥\mathbf{e}∥}^{N-1}{cos}^{N-2}{\alpha }_1{cos}^{N-3}{\alpha }_2\cdots cos{\alpha }_{N-2},$$

(69)

then it may be checked that the polar transformation (68) gives, as it should,

$${\mathbf{e}}^T\mathbf{e}=e_1^2+e_2^2+\cdots +e_N^2={∥\mathbf{e}∥}^2.$$

(70)

So, by way of (69) and (70), we may map (8) from a Cartesian to a polar coordinate system. This gives the transformed probability distribution

$$p\left(\left.∥\mathbf{e}∥,{\alpha }_1,{\alpha }_2,\dots ,{\alpha }_{N-1}\right|\sigma \right)=\frac{{∥\mathbf{e}∥}^{N-1}}{{\left(2\pi {\sigma }^2\right)}^{N/2}}exp\left(-\frac{{∥\mathbf{e}∥}^2}{2{\sigma }^2}\right){cos}^{N-2}{\alpha }_1{cos}^{N-3}{\alpha }_2\cdots cos{\alpha }_{N-2}.$$

(71)

Using the identities

$${\int }_{\pi /2}^{\pi /2}{cos}^{N-i-1}{\alpha }_{i}d{\alpha }_i=\frac{\mathsf{\Gamma }\left[\left(N-i\right)/2\right]}{\mathsf{\Gamma }\left[\left(N-i-1\right)/2+1\right]},$$

(72)

for $i=1,\dots ,N-2$, and

$${\int }_0^{2\pi }d{\alpha }_{N-1}=2\pi ,$$

(73)

we may integrate (71) over the $N-1$ nuisance variables ${\alpha }_i$ and, so, obtain the univariate probability distribution of the norm $∥\mathbf{e}∥$,

$$p\left(\left.∥\mathbf{e}∥\right|\sigma ,I\right)=\frac{2{∥\mathbf{e}∥}^{N-1}}{{\left(2{\sigma }^2\right)}^{N/2}\mathsf{\Gamma }\left(N/2\right)}exp\left(-\frac{{∥\mathbf{e}∥}^2}{2{\sigma }^2}\right),$$

(74)

which has a mean

$$E\left(\left.∥\mathbf{e}∥\right|\sigma ,I\right)=\frac{\sqrt{2}\mathsf{\Gamma }\left[\left(N+1\right)/2\right]}{\mathsf{\Gamma }\left(N/2\right)}\sigma \approx \sqrt{N-1}\sigma$$

(75)

and a standard deviation

$$\mathrm{std}\left(\left.∥\mathbf{e}∥\right|\sigma ,I\right)=\sqrt{N-{\left\{\frac{\sqrt{2}\mathsf{\Gamma }\left[\left(N+1\right)/2\right]}{\mathsf{\Gamma }\left(N/2\right)}\right\}}^2}\sigma \approx \frac{\sigma }{\sqrt{2}}.$$

(76)

By way of (75) and (76), we may set a probabilistic bound on $max∥\mathbf{e}∥$ in (66), that is, we may let $max∥\mathbf{e}∥$ be the k-sigma upper bound

$$max∥\mathbf{e}∥=UB\left(∥\mathbf{e}∥\right)=E\left(\left.∥\mathbf{e}∥\right|\sigma ,I\right)+k\mathrm{std}\left(\left.∥\mathbf{e}∥\right|\sigma ,I\right)\approx \left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma .$$

(77)

In what follows, we will assume sample sizes $N>1$ and, consequently, treat the right-hand approximation in (77) as an equality.

By way of (64), (66), (67), and (77), we then obtain the proper ignorance prior [2]

$$p\left(\left.\mathbf{\beta }\right|X,max\left|y\right|,k,\sigma ,I\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]{\left|X^{T}X\right|}^{1/2}}{{\pi }^{m/2}{\left[\sqrt{N}max\left|y\right|+\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma \right]}^m},$$

(78)

where, as in (64), it is understood that (78) is defined on some ellipsoid which has the origin as its centroid. The proper ignorance prior simplifies to

$$p\left(\left.\mathbf{\beta }\right|X,max\left|y\right|,k,\sigma ,I\right)\approx {\left(\frac{\sigma }{max\left|y\right|+\sigma }\right)}^m{\left(\frac{1}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(\pi {\sigma }^2\right)}^{m/2}},$$

(79)

for $k<<\sqrt{2N}$, where k is some sigma-level for the upper bound (77).

8. A More Informed Manor’s Prior

If apart from the maximum absolute value $max\left|y\right|$ we also have prior knowledge about the minimum and maximum values of y, then we may rewrite (7) as

$$\frac{miny+maxy}{2}\mathbf{1}+\left(\mathbf{y}-\frac{miny+maxy}{2}\mathbf{1}\right)=X\mathbf{\beta }+\mathbf{e},$$

(80)

where $\mathbf{1}$ is a vector of ones and $\left(miny+maxy\right)/2$ is the center of the interval $\left[miny,maxy\right]$. Let

$$c=\frac{{\mathbf{x}}^T\left[\frac{1}{2}\left(miny+maxy\right)\mathbf{1}\right]}{{\mathbf{x}}^T\mathbf{x}}\mathrm{and}\mathbf{w}=\left(\mathbf{y}-\frac{miny+maxy}{2}\mathbf{1}\right)-\mathbf{e}.$$

(81)

Then (47) becomes

$$c-\frac{max∥\mathbf{w}∥}{∥\mathbf{x}∥}\le \beta \le c+\frac{max∥\mathbf{w}∥}{∥\mathbf{x}∥}.$$

(82)

It follows that for the case of just one regression coefficient, the prior (10) is given as

$$p\left(\left.\beta \right|\mathbf{x},max∥\mathbf{w}∥,I\right)=\frac{∥\mathbf{x}∥}{2max∥\mathbf{w}∥},$$

(83)

where(83) is understood to be defined on the interval (82) which is centered at c, (81). Let

$$\mathbf{c}=\frac{miny+maxy}{2}{\left(X^{T}X\right)}^{-1}{X}^T\mathbf{1}.$$

(84)

Then, for the case where X is a $N\times m$ predictor matrix, (82) generalizes to the statement that $\mathbf{\beta }$ is constrained to lie in an m-dimensional ellipsoid which has a centroid $\mathbf{c}$ and a volume [4]

$$V=\frac{{\pi }^{m/2}}{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}\frac{{\left(max∥\mathbf{w}∥\right)}^m}{{\left|X^{T}X\right|}^{1/2}}.$$

(85)

The inverse of this volume gives us the corresponding multivariate generalization of the uniform prior (83):

$$p\left(\left.\mathbf{\beta }\right|X,max∥\mathbf{w}∥,I\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}{{\pi }^{m/2}}\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(max∥\mathbf{w}∥\right)}^m}.$$

(86)

Since $\left(miny+maxy\right)/2$ is the center of the interval $\left[miny,maxy\right]$ which has a range of $\left(maxy-miny\right)$, we have that

$$max∥\mathbf{y}-\frac{miny+maxy}{2}\mathbf{1}∥=\sqrt{N}\frac{maxy-miny}{2}.$$

(87)

So it follows, (45), (65), (66), (77), (81), and (87), that

$$max∥\mathbf{w}∥=\sqrt{N}\frac{maxy-miny}{2}+\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma .$$

(88)

Substituting (88) into (86), we obtain the more informed Manor’s prior [2]

$$p\left(\left.\mathbf{\beta }\right|X,miny,maxy,k,\sigma ,I\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]{\left|X^{T}X\right|}^{1/2}}{{\pi }^{m/2}{\left[\sqrt{N}\frac{maxy-miny}{2}+\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma \right]}^m},$$

(89)

where it is understood that (89) is defined on some ellipsoid which has as its centroid $\mathbf{c}$, (84). Manor’s prior simplifies to

$$p\left(\left.\mathbf{\beta }\right|X,miny,maxy,k,\sigma ,I\right)\approx {\left(\frac{\sigma }{\frac{maxy-miny}{2}+\sigma }\right)}^m{\left(\frac{1}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(\pi {\sigma }^2\right)}^{m/2}},$$

(90)

for $k<<\sqrt{2N}$, where k is some sigma-level for the upper bound (77).

9. An Even More Informed Neeley’s Prior

Alternatively, if we have prior knowledge about the mean $\nu$ and the variance ${\phi }^2$ of the dependent variable y, then, based on that information alone, by way of a maximum entropy argument [11], which also lets us assign (8) to the error vector $\mathbf{e}$ in (7), we may assign a normal distribution as an informative prior to this dependent variable; that is,

$$\mathbf{y}\sim MN\left(\nu \mathbf{1},{\phi }^2\mathbf{I}\right).$$

(91)

Let

$$\mathbf{u}=\mathbf{y}-\nu \mathbf{1}.$$

(92)

By way of (8), (91), (92), and the fact that the mean and variance of a sum of stochastics are the sum of, respectively, the means and variances of those stochastics [12], we then have

$$\mathbf{u}\sim MN\left[\mathbf{0},\left({\phi }^2+{\sigma }^2\right)\mathbf{I}\right].$$

(93)

Since $\mathbf{e}$ and $\mathbf{u}$ both have a zero mean vector and a diagonal covariance matrix, (8) and (93), it follows from (77) that

$$max∥\mathbf{u}∥=UB\left(∥\mathbf{u}∥\right)\approx \left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sqrt{{\phi }^2+{\sigma }^2}.$$

(94)

In what follows, we will assume sample sizes $N>1$ and, consequently, treat the right-hand approximation in (94) as an equality. Substituting (94) into (86), we obtain the even more informed Neeley’s prior [2]

$$p\left(\left.\mathbf{\beta }\right|X,\phi ,k,\sigma ,I\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]{\left|X^{T}X\right|}^{1/2}}{{\pi }^{m/2}{\left[\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sqrt{{\phi }^2+{\sigma }^2}\right]}^m},$$

(95)

where it is understood, as in (89), that (95) is defined on some ellipsoid, which, however, now has a centroid located at

$$\mathbf{c}={\left(X^{T}X\right)}^{-1}{X}^T\left(\nu \mathbf{1}\right).$$

(96)

Neeley’s prior simplifies to

$$p\left(\left.\mathbf{\beta }\right|X,\phi ,k,\sigma ,I\right)\approx {\left(\frac{\sigma }{\sqrt{{\phi }^2+{\sigma }^2}}\right)}^m{\left(\frac{1}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(\pi {\sigma }^2\right)}^{m/2}},$$

(97)

for $k<<\sqrt{2N}$, where k is some sigma-level for the upper bound (77).

10. The Parsimonious Constantineau’s Prior

By way of (7) and (14), we may, in principle, come to the inequality

$$\mathbf{\beta }={\left(X^{T}X\right)}^{-1}{X}^T\left(\mathbf{y}-\mathbf{e}\right)=\widehat{\mathbf{\beta }}-{\left(X^{T}X\right)}^{-1}{X}^T\mathbf{e},$$

(98)

where $\mathbf{e}\sim MN\left(\mathbf{0},{\sigma }^2\mathbf{I}\right)$, (8). So for the special case of an $N\times 1$ predictor vector $\mathbf{x}$, we have that

$$\beta =\widehat{\beta }+\frac{{\mathbf{x}}^T\mathbf{e}}{{\mathbf{x}}^T\mathbf{x}}=\widehat{\beta }+cos\varphi \frac{∥\mathbf{x}∥∥\mathbf{e}∥}{{∥\mathbf{x}∥}^2},$$

(99)

where $\varphi$ is the angle between the predictor vector $\mathbf{x}$ and the error vector $\mathbf{e}$. Given that $-1\le cos\varphi \le 1$, we may by way of (77) and (99) put the following bounds on $\beta$:

$$\widehat{\beta }-\frac{\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma }{∥\mathbf{x}∥}\le \beta \le \widehat{\beta }+\frac{\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma }{∥\mathbf{x}∥}.$$

(100)

For the case where X is a $N\times m$ predictor matrix, (100) generalizes to the statement that $\mathbf{\beta }$ is constrained to lie in an m-dimensional ellipsoid which is centered on $\widehat{\mathbf{\beta }}$ and has a volume of

$$V=\frac{{\pi }^{m/2}}{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}\frac{{\left[\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma \right]}^m}{{\left|X^{T}X\right|}^{1/2}}.$$

(101)

The inverse of this volume gives us the parsimonious Constantineau’s prior [2]

$$p\left(\left.\mathbf{\beta }\right|X,k,\sigma ,I,S\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]{\left|X^{T}X\right|}^{1/2}}{{\pi }^{m/2}{\left[\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma \right]}^m},$$

(102)

where S is the stipulation

$$S\equiv ‘‘\mathrm{centroid}\mathrm{prior}\mathrm{located}\mathrm{at}\widehat{\mathbf{\beta }}.’’$$

(103)

This prior simplifies to

$$p\left(\left.\mathbf{\beta }\right|X,k,\sigma ,I,S\right)\approx {\left(\frac{1}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(\pi {\sigma }^2\right)}^{m/2}},$$

(104)

for $k<<\sqrt{2N}$, where k is some sigma-level for the upper bound (77).

Constantineau’s prior (102) is the most parsimonious of the proposed priors, as it has the smallest k-sigma parameter space volume V. But it will materialize later on that there is an even more parsimonious “stipulation prior” already out there, be it only by implication.

11. The Coverage of the Proposed Priors

In order to demonstrate that (16) tends to hold as an equality for the proposed proper uniform priors, we only need to show that (16) does so for Constantineau’s prior (102), as this prior is the most parsimonious of the proposed priors. That is, we will need to show that the second right-hand term of (15), for all intents and purposes, evaluates to 1 when integrated over $D_{\mathbf{\beta }}$, the domain implied by (101):

$${\int }_{D_{\mathbf{\beta }}}\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(2\pi {\sigma }^2\right)}^{m/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]d\mathbf{\beta }\to 1.$$

(105)

Let $XK=\tilde{X}$ be a transformation of the predictor matrix X such that the columns in $\tilde{X}$ are orthogonal, or, equivalently, ${\tilde{X}}^T\tilde{X}$ is diagonal. Then (105) may be evaluated by way of the transformation

$$\mathbf{\beta }=K\left(\gamma -\widehat{\gamma }\right)+\widehat{\mathbf{\beta }},$$

(106)

which has a Jacobian of $\left|K\right|$. Because of the fact that [6]

$$\left|K\right|{\left|X^{T}X\right|}^{1/2}={\left|K^T\left(X^{T}X\right)K\right|}^{1/2}={\left|{\tilde{X}}^T\tilde{X}\right|}^{1/2}$$

(107)

and the orthogonality of $\tilde{X}$ together with (61), we may rewrite the integrand in (105) for the transformation (106) as

$$\frac{\left|K\right|{\left|X^{T}X\right|}^{1/2}}{{\left(2\pi {\sigma }^2\right)}^{m/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\gamma -\widehat{\gamma }\right)}^T{\tilde{X}}^T\tilde{X}\left(\gamma -\widehat{\gamma }\right)\right]=\prod _{j=1}^m\frac{∥{\tilde{\mathbf{x}}}_j∥}{\sqrt{2\pi {\sigma }^2}}exp\left[-\frac{{∥{\tilde{\mathbf{x}}}_j∥}^2}{2{\sigma }^2}{\left({\gamma }_j-{\widehat{\gamma }}_j\right)}^2\right].$$

(108)

Also, if we go from X to the orthogonal $\tilde{X}$ in (108), then the prior (102) undergoes (by construction) a corresponding transformation, (61),

$$p\left(\left.\mathbf{\beta }\right|\tilde{X},\sigma ,I,\tilde{S}\right)=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]{\left|{\tilde{X}}^T\tilde{X}\right|}^{1/2}}{{\pi }^{m/2}{\left[\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma \right]}^m}=\frac{\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}{{\pi }^{m/2}}\prod _{j=1}^m\frac{∥{\tilde{\mathbf{x}}}_j∥}{\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma },$$

(109)

where k is the sigma-level of the upper bound of the length of the error vector, (77), and $\tilde{S}$ is the transformed stipulation

$$\tilde{S}\equiv ‘‘\mathrm{centroid}\mathrm{prior}\mathrm{located}\mathrm{at}\widehat{\gamma }.$$

(110)

Because of the orthogonality of the ${\tilde{\mathbf{x}}}_j$, the fact that (109) is the inverse of the volume of the prior accessible parameter space, and the fact that this volume is in the form of an ellipsoid with axes of length (59)

$$r_j=\frac{\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)\sigma }{∥{\tilde{\mathbf{x}}}_j∥},$$

(111)

it follows that the rotated parameter space (106) is defined by the ellipsoid

$$\frac{{\left({\gamma }_1-{\widehat{\gamma }}_1\right)}^2}{{\sigma }^2/{∥{\tilde{\mathbf{x}}}_1∥}^2}+\frac{{\left({\gamma }_2-{\widehat{\gamma }}_2\right)}^2}{{\sigma }^2/{∥{\tilde{\mathbf{x}}}_2∥}^2}+\cdots +\frac{{\left({\gamma }_m-{\widehat{\gamma }}_m\right)}^2}{{\sigma }^2/{∥{\tilde{\mathbf{x}}}_m∥}^2}={\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)}^2.$$

(112)

The transformation

$${\gamma }_j={\eta }_j\frac{\sigma }{∥{\tilde{\mathbf{x}}}_j∥}+{\widehat{\gamma }}_j,$$

(113)

for $j=1,2,\dots ,m$, has a Jacobian of

$$J=\prod _{j=1}^m\frac{\sigma }{∥{\tilde{\mathbf{x}}}_j∥}.$$

(114)

By way of (106), (108), (113), and (114), we find for the integral in (105) that

$${\int }_{D_{\mathbf{\beta }}}\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(2\pi {\sigma }^2\right)}^{m/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T{X}^{T}X\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]d\mathbf{\beta }={\int }_{D_{\mathbf{\eta }}}\frac{1}{{\left(2\pi \right)}^{m/2}}exp\left(-\frac{{\mathbf{\eta }}^T\mathbf{\eta }}{2}\right)d\mathbf{\eta },$$

(115)

where the parameter space $D_{\mathbf{\eta }}$ is defined as a sphere which has a radius $\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)$ and is centered at the origin, (112) and (113):

$${\eta }_1^2+{\eta }_2^2+\cdots +{\eta }_m^2={\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)}^2.$$

(116)

By way of the polar transformation (68) and steps (69) through (73), we find that the right-hand side of (115) evaluates as

$${\int }_0^{\sqrt{N-1}+\frac{k}{\sqrt{2}}}\frac{2{∥\mathbf{\eta }∥}^{m-1}}{2^{m/2}\mathsf{\Gamma }\left(m/2\right)}exp\left(-\frac{{∥\mathbf{\eta }∥}^2}{2}\right)d∥\mathbf{\eta }∥=1-\frac{\mathsf{\Gamma }\left[\frac{m}{2},\frac{{\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)}^2}{2}\right]}{\mathsf{\Gamma }\left(\frac{m}{2}\right)},$$

(117)

where $\mathsf{\Gamma }\left(a,b\right)$ and $\mathsf{\Gamma }\left(a\right)$ are the incomplete and the ordinary (Euler) gamma functions, respectively:

$$\mathsf{\Gamma }\left(a,b\right)={\int }_b^{\infty }{t}^{a-1}exp\left(-t\right)dt\mathrm{and}\mathsf{\Gamma }\left(a\right)=\mathsf{\Gamma }\left(a,0\right).$$

(118)

Substituting (117) into (115), we find that requirement (105) translates to the equivalent requirement

$$1-\frac{\mathsf{\Gamma }\left[\frac{m}{2},\frac{{\left(\sqrt{N-1}+\frac{k}{\sqrt{2}}\right)}^2}{2}\right]}{\mathsf{\Gamma }\left(\frac{m}{2}\right)}\to 1.$$

(119)

And it may be checked (numerically) that this requirement holds for $k=6$, (77), even in the (extreme) limit case where the number of predictors m tends to the sample size N. Moreover, it may be checked, by setting $k=0$, that it is the $k/\sqrt{2}$ term in Constantineau’s prior (102) which ensures that requirement (119) holds for the limit case where m tends to N.

12. What is the Data?

Before we go on, we now will discuss two questions that need addressing. The first question is whether or not the predictor matrix X is part of the data. The second question is whether or not the stipulation (103) makes the proposed parsimonious Constantineau’s prior empirical or not.

In answer to the first question, in Bayesian regression analysis the predictor variables in X are assumed to be [6]: “fixed non-stochastic variables,” or, alternatively, “random variables distributed independently of the $\mathbf{e}$, with a pdf not [italics by Zellner himself] involving the parameters ${\beta }_j$ and $\sigma$.” Stated differently, the likelihood $p\left(\left.\mathbf{y}\right|\sigma ,X,\mathbf{\beta },M\right)$ is a probability of the response vector $\mathbf{y}$, and not of the predictor matrix X. Following this line of reasoning, the predictor matrix X should not be considered to be part of the data. Rather, X is part of the prior problem structure, in that for a given predictor matrix X a corresponding response vector $\mathbf{y}$ is obtained in the data gathering phase. So, where in [4] (i.e., Part I of this research) it was proposed that in order to construct a parsimonious prior for regression coefficients one needed to assign a minimal value to the determinant of $X^{T}X$ based on the prior information at hand, a non-trivial task. It was argued in [2] (i.e., Part II) that the predictor matrix X is not a part of the data and, consequently, may be used for the construction of proper priors.

In answer to the second question, if “we adopt the posture of the scrupulous fair judge who insists that fairness in comparing models requires that each is delivering the best performance of which it is capable, by giving each the best possible prior probability for its parameters” [11], then we may defend the use of the cheap and cheerful prior (102), with its stipulation (103), as being the prior that represents some limit of parsimony, which is not influenced by our state of ignorance regarding the dependent variable y. However, if we “consider it necessary to be cruel realists and judge each model taking into account the prior information we actually have pertaining to it, that is, we penalize a model if we do not have the best possible prior information about the dependent variable y, although that is not really a fault of the model itself” [11], then we will be forced to revert to the more solemn priors (78), (89), and (95).

13. The Corresponding Evidences

By way of (10), we may substitute (78) into (16), and so obtain the evidence value of the likelihood model M and prior information I, conditional on $\sigma$:

$$p\left(\left.\mathbf{y}\right|k,\sigma ,X,max\left|y\right|,M,I\right)\approx \frac{2^{m/2}\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}{{\left[\sqrt{N}\left(\frac{max\left|y\right|}{\sigma }+1+\frac{k}{\sqrt{2N}}\right)\right]}^m}\frac{1}{{\left(2\pi {\sigma }^2\right)}^{N/2}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)\right].$$

(120)

If $\sigma$ is unknown, then, as both a mathematical and a modeling convenience (see discussion of Section 5), we may assign the improper Jeffreys prior for scaling parameters (18):

$$p\left(\left.\sigma \right|I\right)=\frac{A}{\sigma },$$

(121)

where A is some normalizing constant, to the unknown $\sigma$ in the evidence (120), in order to integrate with respect to this unknown parameter:

$$p\left(\left.\mathbf{y}\right|k,X,M,I\right)=\int p\left(\left.\sigma ,\mathbf{y}\right|k,X,M,I\right)d\sigma =\int p\left(\left.\sigma \right|I\right)p\left(\left.\mathbf{y}\right|k,\sigma ,X,M,I\right)d\sigma ,$$

(122)

where, (120) and (121),

$$p\left(\left.\sigma ,\mathbf{y}\right|k,X,max\left|y\right|,M,I\right)\approx \frac{2^{m/2}\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}{{\left[\sqrt{N}\left(\frac{max\left|y\right|}{\sigma }+1+\frac{k}{\sqrt{2N}}\right)\right]}^m}\frac{A}{{\left(2\pi \right)}^{N/2}{\sigma }^{N+1}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)\right].$$

(123)

We may conveniently factorize (123) as,

$$\begin{array}{cc}\hfill p\left(\left.\sigma ,\mathbf{y}\right|k,X,max\left|y\right|,M,I\right)& \approx \frac{2^{m/2}\mathsf{\Gamma }\left[\left(m+2\right)/2\right]}{{\left[\sqrt{N}\left(\frac{max\left|y\right|}{\sigma }+1+\frac{k}{\sqrt{2N}}\right)\right]}^m}\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N}\frac{A\mathsf{\Gamma }\left(N/2\right)}{2{\pi }^{N/2}}\hfill \\ & \times \frac{2}{\mathsf{\Gamma }\left(N/2\right)}{\left(\frac{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2}{2}\right)}^{N/2}\frac{1}{{\sigma }^{N+1}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)\right].\hfill \end{array}$$

(124)

The last term in (124) evaluates to 1 when integrated over $\sigma$, as it has the form of an inverted gamma distribution [6]. Also, the last term in (124) will tend to a Dirac delta distribution as $N\to \infty$, [9]; that is,

$$\frac{2}{\mathsf{\Gamma }\left(N/2\right)}{\left(\frac{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2}{2}\right)}^{N/2}\frac{1}{{\sigma }^{N+1}}exp\left[-\frac{1}{2{\sigma }^2}{\left(\mathbf{y}-\widehat{\mathbf{y}}\right)}^T\left(\mathbf{y}-\widehat{\mathbf{y}}\right)\right]\to \delta \left(\sigma -\frac{∥\mathbf{y}-\widehat{\mathbf{y}}∥}{\sqrt{N}}\right).$$

(125)

So, by way of (125), the property (40), and the factorization (124), we have that the evidence (122) evaluates as

$$p\left(\left.\mathbf{y}\right|k,X,max\left|y\right|,M,I\right)\propto {\left(\frac{1+\frac{k}{\sqrt{2N}}}{\frac{\sqrt{N}max\left|y\right|}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}+1+\frac{k}{\sqrt{2N}}}\right)}^m{\left(\frac{2}{\sqrt{2N}+k}\right)}^m\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(126)

where k is the upper-bound sigma level of the maximum length of the error vector $max∥\mathbf{e}∥$, (77). If we assume that $k<<\sqrt{2N}$, then the evidence (126) simplifies to

$$p\left(\left.\mathbf{y}\right|k,X,max\left|y\right|,M,I\right)\propto {\left(\frac{\sqrt{N}max\left|y\right|}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}+1\right)}^{-m}{\left(\frac{2}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N}.$$

(127)

Likewise, if we substitute (89), (95), and (102) into (16), integrate over $\sigma$, and assume $k<<\sqrt{2N}$, we obtain the respective approximate evidence values:

$$p\left(\left.\mathbf{y}\right|k,X,miny,maxy,M,I\right)\propto {\left(\frac{\sqrt{N}\frac{maxy-miny}{2}}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}+1\right)}^{-m}{\left(\frac{2}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(128)

and

$$p\left(\left.\mathbf{y}\right|k,X,\phi ,M,I\right)\propto {\left(\frac{N{\phi }^2}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2}+1\right)}^{-m/2}{\left(\frac{2}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(129)

and

$$p\left(\left.\mathbf{y}\right|k,X,M,I,S\right)\propto {\left(\frac{2}{N}\right)}^m\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(130)

where the “∝” symbol is used to absorb the common factors $A\mathsf{\Gamma }\left(N/2\right)/\left(2{\pi }^{N/2}\right)$, which are shared by all the competing regression models and which cancel out as the posterior probabilities of these models are computed.

The above evidences can be deconstructed into a goodness of fit factor, which is also the implied evidence (41) of the “sure thing” prior (39):

$$\mathrm{Goodness}\mathrm{of}\mathrm{Fit}=\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(131)

and an Occam factor which penalizes the shrinkage of the posterior accessible parameter space of $\mathbf{\beta }$ relative to the prior accessible space. Now, all Occam factors are a monotonic decreasing function in the number of predictors m. But only the Occam factors of the “cruelly realistic” evidences (127)–(129) have terms which are dependent upon our state of prior knowledge regarding the dependent variable y.

If in the construction of the priors (79), (90), or (97) we make prior value assignments that grossly overestimate the maximum absolute value, range, and standard deviation, respectively, of the dependent variable y, then the Occam factors in the corresponding evidences, (127)–(129), stand ready to punish us for making consequent prior parameter space assignments that are too voluminous. Whereas, if we make prior value assignments that grossly underestimate these aspects of the dependent variable y, then the Occam factors of the cruelly realistic evidences (127)–(129) will tend to the Occam factor of the “scrupulously fair” evidence (130), as the cruelly realistic evidences, as a consequence, tend to the scrupulously fair evidence.

For prior value assignments that approximate the underlying ‘true’ values of the maximum absolute value, range, and standard deviation, respectively, of the dependent variable y, the Occam factors of the evidences (127)–(129) tend to the inequality:

$$\mathrm{Occam}\mathrm{Factor}\le 2^{-m/2}{\left(\frac{2}{N}\right)}^m\mathsf{\Gamma }\left(\frac{m+2}{2}\right),$$

(132)

seeing that for accurate prior value assignments we have that, (125),

$$\frac{\sqrt{N}max\left|y\right|}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}\ge \frac{\sqrt{N}\left(maxy-miny\right)/2}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}>\frac{\sqrt{N}\phi }{∥\mathbf{y}-\widehat{\mathbf{y}}∥}\ge \frac{\sqrt{N}\sigma }{∥\mathbf{y}-\widehat{\mathbf{y}}∥}\approx 1,$$

(133)

where $\phi$ is the prior standard deviation of y which is estimated by the root mean square error of a simple intercept-only regression model and $\sigma$ is the prior model error which is estimated by the root mean square error of the full regression model.

Note that equality will hold in (132) only for the evidence (129) of an intercept-only regression model in combination with an accurate prior value assignment for $\phi$, because only then do we have that $\phi$ is approximated by $∥\mathbf{y}-\widehat{\mathbf{y}}∥/\sqrt{N}$.

14. Connecting the Derived Evidences with the BIC and the AIC

In order to get our bearings for the proposed priors and their consequent evidences, we will connect the Bayesian Information Criterion (BIC) and the Akaike Information Criterion (AIC) to these evidences.

The BIC is given as [13]

$$\mathrm{BIC}=mlogN+2Nlog∥\mathbf{y}-\widehat{\mathbf{y}}∥,$$

(134)

where, given any two estimated models, the model with the lower value of BIC is the one to be preferred. The BIC has an implied evidence of

$$p\left(\left.\mathbf{y}\right|X,\mathrm{BIC},S\right)\propto exp\left(-\frac{1}{2}\mathrm{BIC}\right)={\left(\frac{1}{N}\right)}^{m/2}\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(135)

where S is the stipulation (103)

$$S\equiv ‘‘\mathrm{centroid}\mathrm{prior}\mathrm{located}\mathrm{at}\widehat{\mathbf{\beta }}.’’$$

and where we assume that the factor $A\mathsf{\Gamma }\left(N/2\right)/\left(2{\pi }^{N/2}\right)$ has been absorbed in the proportionality sign. For $k<<\sqrt{2N}$, the BIC evidence (135) differs from the Constantineau’s evidence (130) by an approximate factor

$$\frac{p\left(\left.\mathbf{y}\right|k,X,M,I,S\right)}{p\left(\left.\mathbf{y}\right|X,\mathrm{BIC},S\right)}\approx 2^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right).$$

(136)

Let $c_{\mathrm{BIC}}$ be the factor by which the lengths of the axes of the parameter space of the implied BIC prior differs from the lengths of the axes of the parameter space of Constantineau’s prior (102). Then we have that

$$p\left(\left.\mathbf{y}\right|X,\mathrm{BIC},S\right)=\frac{1}{c_{\mathrm{BIC}}^m}p\left(\left.\mathbf{y}\right|k,X,M,I,S\right),$$

(137)

as the lengths of the prior ellipsoid parameter spaces factor inversely into their corresponding evidences. Combining (136) and (137), and making use of the Stirling approximation

$$log\mathsf{\Gamma }\left(\frac{m+2}{2}\right)=\frac{m}{2}log\frac{m}{2}-\frac{m}{2}+O\left(\sqrt{m}\right),$$

(138)

we find that the axes of the implied BIC prior tend to be longer by a factor

$$c_{\mathrm{BIC}}\approx {\left[2^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\right]}^{1/m}\approx {\left(\frac{m}{e^1}\right)}^{1/2}.$$

(139)

than the axes of Constantineau’s prior. It follows that the implied BIC prior is approximately given as, (104) and (139),

$$p\left(\left.\mathbf{\beta }\right|X,\sigma ,\mathrm{BIC},S\right)\approx {\left(\frac{e^1}{m}\right)}^{m/2}{\left(\frac{1}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(\pi {\sigma }^2\right)}^{m/2}}.$$

(140)

And it may be checked that the requirement (105) holds for this implied prior, as we have that the equivalent requirement (119),

$$1-\frac{\mathsf{\Gamma }\left(\frac{m}{2},\frac{e^{-1}mN}{2}\right)}{\mathsf{\Gamma }\left(\frac{m}{2}\right)}=1,$$

(141)

holds for $N\ge m\ge 3$, where it is understood that in a regression analysis the number of parameters m may never exceed the sample size N.

The AIC is given as [13]

$$\mathrm{AIC}=2m+2Nlog∥\mathbf{y}-\widehat{\mathbf{y}}∥,$$

(142)

where, given any two estimated models, the model with the lower value of AIC is the one to be preferred. The AIC has an implied evidence of

$$p\left(\left.\mathbf{y}\right|X,\mathrm{AIC},S\right)\propto exp\left(-\frac{1}{2}\mathrm{AIC}\right)=e^{-m}\frac{1}{{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^N},$$

(143)

where S is the stipulation (103). For $k<<\sqrt{2N}$, the AIC evidence (143) differs from Constantineau’s evidence (130) by an approximate factor

$$\frac{p\left(\left.\mathbf{y}\right|X,M,I,S\right)}{p\left(\left.\mathbf{y}\right|X,\mathrm{AIC},S\right)}\approx e^m{\left(\frac{2}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right).$$

(144)

Let $c_{\mathrm{AIC}}$ be the factor by which the lengths of the axes of the parameter space of the implied BIC prior differs from the lengths of the axes of the parameter space of Constantineau’s prior (102). Then we have that, (137),

$$p\left(\left.\mathbf{y}\right|X,\mathrm{AIC},S\right)=\frac{1}{c_{\mathrm{AIC}}^m}p\left(\left.\mathbf{y}\right|X,M,I,S\right).$$

(145)

Combining (144) and (145), and making use of the Stirling approximation (138), we find that the axes of the implied AIC prior tend to be shorter by a factor

$$c_{\mathrm{AIC}}\approx {\left[e^m{\left(\frac{2}{N}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\right]}^{1/m}\approx {\left(\frac{e^{1}m}{N}\right)}^{1/2}$$

(146)

than the axes of Constantineau’s prior. It follows that the implied AIC prior is approximately given as, (104) and (146),

$$p\left(\left.\mathbf{\beta }\right|X,\sigma ,\mathrm{AIC},S\right)\approx {\left(\frac{1}{e^{1}m}\right)}^{m/2}\mathsf{\Gamma }\left(\frac{m+2}{2}\right)\frac{{\left|X^{T}X\right|}^{1/2}}{{\left(\pi {\sigma }^2\right)}^{m/2}}.$$

(147)

Now, if we look at the coverage of the AIC prior (147), then we find that, (119),

$$1-\frac{\mathsf{\Gamma }\left[\frac{m}{2},\frac{e^{1}m}{2}\right]}{\mathsf{\Gamma }\left(\frac{m}{2}\right)}=1,$$

(148)

even as $m\to 1$. Moreover, it would seem that the second argument of the incomplete gamma function in (148) is the threshold level below which, for a given first argument of $m/2$, the requirement (147) no longer holds for general m, as we have for $m\to \infty$ that, on the one hand,

$$1-\frac{\mathsf{\Gamma }\left[\frac{m}{2},\frac{m}{2}\right]}{\mathsf{\Gamma }\left(\frac{m}{2}\right)}\to 0.5$$

(149)

and, on the other hand,

$$1-\frac{\mathsf{\Gamma }\left(\frac{m}{2},\frac{e^{1}m}{2}\right)}{\mathsf{\Gamma }\left(\frac{m}{2}\right)}\to 1.$$

(150)

Stated differently, it would seem that it is the implied AIC prior (147) that is optimally parsimonious, rather than Constantineau’s prior (102), as this AIC prior may very well be the uniform proper prior which has the smallest possible parameter space for which requirement (105) will always hold.

We summarize, of the three “stipulation priors”, (102), (140), and (147), the BIC prior is the most conservative in that it has an evidence that penalizes the severest for the number of parameters m, followed by Constantineau’s prior, which, though parsimonious, is not the optimally parsimonious prior, as was initially thought in part II of this research [2]. This honor may very well go to the AIC prior, should it turn out that the value of $e^{1}m/2$ in the second argument of (148) is indeed the exact threshold point above which (119) will always hold.

15. The Corresponding Regression Model

If we combine the prior (18) of the unknown $\sigma$ and the respective priors of the regression coefficients $\mathbf{\beta }$, (78), (89), (95), (102), (140), and (147), with the likelihood model (9), and integrate with respect to the unknown $\sigma$, we obtain the posterior of the unknown $\mathbf{\beta }$s, (21) through (28):

$$p\left(\left.\mathbf{\beta }\right|\mathbf{y},X,M,I\right)=\frac{\mathsf{\Gamma }\left[\left(N+m\right)/2\right]}{\mathsf{\Gamma }\left[\left(N\right)/2\right]{\pi }^{m/2}}\frac{{\left|\frac{1}{N{s}^2}{X}^{T}X\right|}^{1/2}}{{\left[1+{\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)}^T\left(\frac{1}{N{s}^2}{X}^{T}X\right)\left(\mathbf{\beta }-\widehat{\mathbf{\beta }}\right)\right]}^{\left(N+m\right)/2}},$$

(151)

where

$$s^2=\frac{1}{N}{∥\mathbf{y}-\widehat{\mathbf{y}}∥}^2.$$

(152)

Stated differently, as the normalizing constant C of (10) in the priors (78), (89), (95), (102), (140), and (147), is not so much a constant as it is a function of $\sigma$:

$$C\left(\sigma \right)\propto \frac{1}{{\sigma }^m},$$

(153)

we have that the degrees of freedom of the multivariate Student-t distribution (151) and, consequently, the sample error variance (152), are always N, irrespective of the number of predictors m, hence the “seemingly” interjection following (29).

The posterior (151) has a mean of $\widehat{\mathbf{\beta }}={\left(X^{T}X\right)}^{-1}{X}^T\mathbf{y}$, (14), a covariance matrix of ${\left(X^{T}X/s^2\right)}^{-1}$, (152), and a corresponding predictive probability distribution for $\widehat{y}$, given a vector $m\times 1$ vector of predictor values $\mathbf{x}$, [6]:

$$p\left(\left.\widehat{y}\right|\mathbf{x},\mathbf{y},X,M,I\right)=\frac{\mathsf{\Gamma }\left[\left(N+1\right)/2\right]}{\mathsf{\Gamma }\left(N/2\right)\sqrt{\pi }}\frac{\sqrt{\frac{h}{N{s}^2}}}{{\left[1+\frac{h}{N{s}^2}{\left(\widehat{y}-{\mathbf{x}}^T\widehat{\mathbf{\beta }}\right)}^2\right]}^{\left(N+1\right)/2}},$$

(154)

where

$$h=1-{\mathbf{x}}^T{\left(X^{T}X+\mathbf{x}{\mathbf{x}}^T\right)}^{-1}\mathbf{x},$$

(155)

which is in the univariate Student-t form and has expected value, (14), and standard deviation, (152),

$$E\left(\widehat{y}\right)={\mathbf{x}}^T\widehat{\mathbf{\beta }}\mathrm{and}\mathrm{std}\left(\widehat{y}\right)=s\sqrt{\frac{N}{N-2}\left(1+{\mathbf{x}}^T{\left(X^{T}X\right)}^{-1}\mathbf{x}\right)}.$$

(156)

16. Discussion

This research into proper uniform priors was inspired by our research into spline models [5,14]. Spline models may have hundreds of regression coefficients. So, in using these models in an actual data-analysis, one is forced to think about the most suitable bounds of the proper non-informative priors of the unknown regression parameters. Not because this will give us better parameter estimates, but simply because taking a proper prior with overly large bounds will severely punish the larger regression models.

Grappling with the problem of defining a parsimonious proper prior for regression coefficients, it was quickly realized that the proposed priors should include the square root of $\left|X^{T}X\right|$, so that this term could cancel out in the evidence derivations, since this term is not invariant for otherwise equivalent B- and C-spline regression analysis formulations (in which pairs of triangles in the B-spline analysis were forced to connect with continuity orders equal to the polynomial orders in order to merge these paired triangles into squares.) Moreover, it was found that dropping the square root of $\left|X^{T}X\right|$ in an ad-hoc fashion from the regression analysis evidences proposed in [6,7] gives satisfactory results, in terms of (spline) regression model selections that commit neither gross under- nor gross over-fitting. So, the first impetus of this research was the desire to find a principled argument by which we would be allowed to drop the square root of $\left|X^{T}X\right|$ from the evidence, a term which was problematic in that it is non-invariant under certain transformations of the predictor variables and which seemed to be not that essential for a successful model selection.

Apart from the need to include the square root of $\left|X^{T}X\right|$ in the proper priors, or, equivalently, the need to drop this term from the evidences, it was also realized that regression coefficients are bounded by certain aspects of the predictor matrix X and the dependent variable vector $\mathbf{y}$. This second realization led to the finding that the prior accessible space of regression coefficients is ellipsoid in form, which then provided us in the first part of this research [4] with the sought for rational of the inclusion of the square root of $\left|X^{T}X\right|$ in the proper priors.

Now, in the first part of this research it was implicitly assumed that the predictor matrix X is part of the data, which forced us to make a prior estimate of the (scalar) value of the square root of $\left|X^{T}X\right|$. This estimated value then would be weighted by the actual observed value of the square root of $\left|X^{T}X\right|$. But as this prior estimation is a non-trivial task [4], we were forced to think on how to justify the use of the actual observed values of the square root of $\left|X^{T}X\right|$, rather than the prior estimates of these values. This then led us to the second part of this research [2], in which it was observed that X may very well in practicality be obtained during the data-gathering phase, but that X formally is not part of the data, as it admits no likelihood function in ordinary regression analysis. Also, in the second part of this research there was presented a suite of proper uniform priors for the regression coefficients proper $\mathbf{\beta }$, rather than, as was realized in hindsight, a single proper uniform prior for the estimated regression coefficients $\widehat{\mathbf{\beta }}$ given in [4].

It was found in the second part of this research that if the actual observed value of the square root of $\left|X^{T}X\right|$ is used in the construction of the proper prior for regression coefficients, then the user only needs to assign prior values to either the maximum absolute value, or the minimum and maximum, or the standard deviation of the dependent variable y, in order to construct his cruelly realistic priors. Alternatively, if the user is willing to accept empirical overtones in his prior, by way of the stipulation that the proper uniform prior is to be centered at the to be estimated regression coefficients $\widehat{\mathbf{\beta }}$, the need for prior value assignments to the characteristics of the dependent variable y may be circumvented, as we construct Constantineau’s scrupulously fair stipulation prior.

In the third part of this research it has now been checked analytically that the accessible parameter space of the in [2] proposed priors cover the true values of $\mathbf{\beta }$ with a probability that tends to one. It has also been found that the implied AIC prior is a viable stipulation prior, as its accessible parameter space covers the true values of $\mathbf{\beta }$ with a probability one. Moreover, it may very well be that the AIC stipulation prior is optimally parsimonious as it may represent the inverse of the smallest prior volume which covers the true value of $\mathbf{\beta }$ with a probability one, when centered at $\widehat{\mathbf{\beta }}$. It follows that Constantineau’s stipulation prior takes the middle position in terms of conservativeness, as the implied BIC stipulation prior is more conservative in terms of the penalizing for the number of parameters m, whereas the implied AIC stipulation prior is more liberal.

Also, there are given, in Appendix A below, two Monte Carlo studies on the performance of the discussed priors, in terms of their implied evidences, in C-spline regression model selection problems. It is found in these studies that, depending on the accuracy of the prior assessments of the characteristics of the dependent variable y, the priors that were proposed in the second part of this research fill in the space between the BIC and AIC on a continuum of conservativeness, in terms of the number of parameters chosen.