The Singular Value Expansion for Arbitrary Bounded Linear Operators

Abstract

The singular value decomposition (SVD) is a basic tool for analyzing matrices. Regarding a general matrix as defining a linear operator and choosing appropriate orthonormal bases for the domain and co-domain allows the operator to be represented as multiplication by a diagonal matrix. It is well known that the SVD extends naturally to a compact linear operator mapping one Hilbert space to another; the resulting representation is known as the singular value expansion (SVE). It is less well known that a general bounded linear operator defined on Hilbert spaces also has a singular value expansion. This SVE allows a simple analysis of a variety of questions about the operator, such as whether it defines a well-posed linear operator equation and how to regularize the equation when it is not well posed.

1. Introduction

One of the most powerful ideas in linear algebra is diagonalization, which renders many problems completely transparent. For example, if $A\in {\mathbb{R}}^{n\times n}$ is a symmetric matrix, the spectral theorem implies that there exists an orthogonal matrix $V\in {\mathbb{R}}^{n\times n}$ and a diagonal matrix $D\in {\mathbb{R}}^{n\times n}$ such that

$$A=VD{V}^T$$

(1)

(see, for instance, ([1], Section 7.1)). To say that V is orthogonal means that $V^{T}V=I$, the $n\times n$ identity matrix, which implies that the columns $v_1,v_2,\dots ,v_n$ of V form an orthonormal basis for ${\mathbb{R}}^n$. If D has diagonal entries ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_n$, then we can also write

$$A=VD{V}^T=\sum _{i=1}^n{\lambda }_i{v}_i{v}_i^T.$$

If $A\in {\mathbb{R}}^{m\times n}$ is a general matrix (not assumed to be symmetric or even square), the singular value decomposition (SVD) allows us to diagonalize the matrix, at the cost of using two different orthonormal bases. There exist orthogonal matrices $U\in {\mathbb{R}}^{m\times m}$ and $V\in {\mathbb{R}}^{n\times n}$ and a diagonal matrix $S\in {\mathbb{R}}^{m\times n}$ such that

$$A=US{V}^T$$

and $S=\mathrm{diag}({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_n)$, with ${\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_n\ge 0$ (see ([2], Lectures 4–5) or ([1], Chapter 8)). This decomposition follows immediately from the spectral theorem for symmetric matrices; in fact, $A^{T}A=V{S}^{T}S{V}^T$ is the spectral decomposition of the symmetric matrix $A^{T}A$.

Among its other virtues, the SVD reveals the rank of A and bases for the four fundamental subspaces associated with A. If A has rank r, then exactly r of the singular values ${\sigma }_1,{\sigma }_2,\dots ,{\sigma }_n$ are positive, and we can write the SVD in the reduced form

$$A=\widehat{U}\widehat{S}{\widehat{V}}^T=\sum _{i=1}^r{\sigma }_i{u}_i{v}_i^T,$$

where $u_1,u_2,\dots ,u_m\in {\mathbb{R}}^m$, $v_1,v_2,\dots ,v_n\in {\mathbb{R}}^n$ are the columns of U, V, respectively, $\widehat{U}=[u_1|u_2|\cdots |u_r]$, $\widehat{V}=[v_1|v_2|\cdots |v_r]$, and $\widehat{S}=diag({\sigma }_1,{\sigma }_2,\dots ,{\sigma }_r)$. The four fundamental subspaces are represented by orthonormal bases as follows:

$$\begin{array}{cc}\hfill \mathrm{col}\left(A\right)& =\mathrm{sp}\{u_1,u_2,\dots ,u_r\},\hfill \\ \hfill \mathcal{N}\left(A^T\right)=\mathrm{col}{\left(A\right)}^{\perp }& =\mathrm{sp}\{u_{r+1},u_{r+2},\dots ,u_m\},\hfill \\ \hfill \mathcal{N}\left(A\right)& =\mathrm{sp}\{v_{r+1},v_{r+2},\dots ,v_n\},\hfill \\ \hfill \mathrm{col}\left(A^T\right)=\mathcal{N}{\left(A\right)}^{\perp }& =\mathrm{sp}\{v_1,v_2,\dots ,v_r\}.\hfill \end{array}$$

It is well known that spectral theory is considerably more complicated for linear operators defined on infinite-dimensional spaces. However, for compact operators, the finite-dimensional theory carries through almost unchanged. Throughout the rest of this paper, X and Y will denote real Hilbert spaces. If $T:X\to Y$ is linear, then T is called compact if and only if $\left\{T{x}_n\right\}$ has a convergent subsequence in Y for every bounded sequence $\left\{x_n\right\}$ in X (this is equivalent to T’s being continuous when the weak topology is imposed on X and the norm topology on Y).

The spectral theorem for self-adjoint compact operators (see ([3], Section 4.3) or ([4], Section 8.2)) says that if $T:X\to X$ is compact and self-adjoint, then there exists an orthonormal sequence $\left\{{\varphi }_k\right\}$ in X and a sequence $\left\{{\lambda }_k\right\}$ of nonzero real numbers such that

$$T=\sum _{k=1}^{\infty }{\lambda }_k{\varphi }_k\otimes {\varphi }_k,$$

(2)

where the outer product ${\varphi }_k\otimes {\varphi }_k:X\to X$ is the bounded linear operator defined by

$$({\varphi }_k\otimes {\varphi }_k)x={\langle {\varphi }_k,x\rangle }_X{\varphi }_k\mathrm{for} \mathrm{all}x\in X.$$

For ease of exposition, we will assume that the sequences $\left\{{\varphi }_k\right\}$ and $\left\{{\lambda }_k\right\}$ are infinite sequences; in the contrary case, T is a finite-rank operator and the infinite series becomes a finite sum.

For a general (that is, not necessarily self-adjoint) compact operator $T:X\to Y$, we can derive the singular value expansion (SVE) of T by applying the spectral theorem for self-adjoint compact operators to $T^{\ast }T$ (see [3], Section 4.4). The result is

$$T=\sum _{k=1}^{\infty }{\sigma }_k{\psi }_k\otimes {\varphi }_k,$$

where $\left\{{\varphi }_k\right\}$ is an orthonormal sequence in X, $\left\{{\psi }_k\right\}$ is an orthonormal sequence in Y, and ${\sigma }_1\ge {\sigma }_2\ge \cdots$ are positive numbers converging to zero. Moreover, $\left\{{\varphi }_k\right\}$ is a complete orthonormal set for $\mathcal{N}{\left(T\right)}^{\perp }$ and $\left\{{\psi }_k\right\}$ is a complete orthonormal set for $\overline{\mathcal{R}\left(T\right)}$.

The SVE of a compact operator has many applications, particularly in analyzing linear operator equations of the form $Tx=y$ (that is, given $y\in Y$, find or estimate $x\in X$ satisfying $Tx=y$). When T is compact and not of finite rank, this equation is ill-posed in the sense that the solution x (if it exists) does not depend continuously on the data y; in this case, the equation is often referred to as an inverse problem. The singular value expansion of T is useful in analyzing approaches to regularizing $Tx=y$, that is, to computing a stable (approximate) solution to the equation in the presence of noisy data.

When $T:X\to Y$ is not necessarily compact, there still exists a singular value expansion in the following form.

Theorem 1.

Let X and Y be real Hilbert spaces and let $T:X\to Y$ be a bounded linear operator. Then there exist a Borel space $(M,\mathcal{A},\mu )$ with a second-countable topology $\mathcal{T}$, isometries $V:L^2\left(\mu \right)\to X$, $U:L^2\left(\mu \right)\to Y$, and an essentially bounded measurable function $\sigma :M\to \mathbb{R}$ such that

$$T=U{m}_{\sigma }{V}^{†},$$

(3)

where $V^{†}$ is the generalized inverse of V and $m_{\sigma }$ is the multiplication operator defined by σ:

$$\begin{array}{cc}\hfill & m_{\sigma }:L^2\left(\mu \right)\to L^2\left(\mu \right),\hfill \\ \hfill & m_{\sigma }f=\sigma ffor allf\in L^2\left(\mu \right).\hfill \end{array}$$

Moreover, $\sigma >0$ a.e.

By Borel space, we mean a measure space $(M,\mathcal{A},\mu )$ such that there is a topology $\mathcal{T}$ defined on the set M and $\mathcal{A}$ (the collection of measurable subsets of M) is the $\sigma$-algebra of Borel sets of $\mathcal{T}$. As noted in the theorem, the topology is guaranteed to be second-countable, that is, to have a countable base. Furthermore, note that for an isometry V, $V^{†}=V^{\ast }$.

We will call the representation of Theorem 1 the SVE of T. Pietsch ([5], Section D.3) outlines a short proof of Theorem 1 based on the polar decomposition. We give a direct proof in the next section that is analogous to the derivation of the SVD of a matrix A from the spectral decomposition of $A^{T}A$. In Section 3, we derive some basic results about this form of the SVE, including its relationship to the classical SVE of a compact operator and how to recognize from the SVE when $\mathcal{R}\left(T\right)$ fails to be closed. We also include a brief discussion of the relationship of the SVE to notions of s-numbers (the generalization of singular values) that have appeared in the literature. In Section 4, we analyze the inverse problem $Tx=y$, including methods for regularizing the equation, using Theorem 1. The results in Section 4 are not new, but we hope to convince the reader that the analysis based on Theorem 1 is particularly convenient. We conclude with a brief discussion in Section 5.

2. The SVE of a Bounded Linear Operator

As noted above, the SVD of a matrix $A\in {\mathbb{R}}^{m\times n}$ can be derived from the spectral decomposition of the symmetric matrix $A^{T}A\in {\mathbb{R}}^{n\times n}$. In the same way, the SVE described by Theorem 1 can be derived from the following spectral theorem for a bounded, self-adjoint linear operator.

Theorem 2.

Let T be a bounded and self-adjoint linear operator mapping a real Hilbert space X into itself. Then there exists a Borel space $(M,\mathcal{A},\mu )$ with a second-countable topology $\mathcal{T}$, a unitary operator $V:L^2\left(\mu \right)\to X$, and an essentially bounded measurable function $\theta :M\to \mathbb{R}$ such that

$$T=V{m}_{\theta }{V}^{-1},$$

where $m_{\theta }$ is the multiplication operator defined by θ.

This version of the spectral theorem is usually stated in terms of a complex Hilbert space X and complex $L^2\left(\mu \right)$ (see, for instance, [6] for an accessible exposition), but it can be verified that the same proof yields this representation when X is a real Hilbert space and $L^2\left(\mu \right)$ denotes the space of real-valued square-integrable functions.

To derive Theorem 1 from Theorem 2, we require the following preliminary result.

Lemma 1.

([7], Lemma 2.1) Let $(M,\mathcal{A},\mu )$ be a measure space and let $\theta :M\to [0,\infty )$ be a measurable function that is positive a.e. Define

$$S=\{f\in L^2\left(\mu \right):{\theta }^{-1}f\in L^2\left(\mu \right)\}.$$

Then S is dense in $L^2\left(\mu \right)$.

We can now prove a special case of Theorem 1.

Theorem 3.

Let X and Y be real Hilbert spaces and let $T:X\to Y$ be a bounded linear operator with $\mathcal{N}\left(T\right)=\left\{0\right\}$. Then there exist a Borel space $(M,\mathcal{A},\mu )$ with a second-countable topology $\mathcal{T}$, a unitary operator $V:L^2\left(\mu \right)\to X$, an isometry $U:L^2\left(\mu \right)\to Y$, and an essentially bounded measurable function $\sigma :M\to [0,\infty )$ such that

$$T=U{m}_{\sigma }{V}^{-1}.$$

Moreover, $\sigma >0$ a.e. and $\mathcal{R}\left(U\right)=\overline{\mathcal{R}\left(T\right)}$.

Proof. 

By Theorem 2, there exist a measure space $(M,\mathcal{A},\mu )$, a unitary operator $V:L^2\left(\mu \right)\to X$, and a bounded measurable function $\theta :M\to \mathbb{R}$ such that

$$T^{\ast }T=V{m}_{\theta }{V}^{-1}.$$

We first show that $\theta \ge 0$ a.e., which obviously follows if we prove that

$${\langle m_{\theta }f,f\rangle }_{L^2\left(\mu \right)}\ge 0\mathrm{for} \mathrm{all}f\in L^2\left(\mu \right).$$

However, $m_{\theta }=V^{-1}{T}^{\ast }TV$ and hence, for any $f\in L^2\left(\mu \right)$,

$${\langle m_{\theta }f,f\rangle }_{L^2\left(\mu \right)}={\langle V^{-1}{T}^{\ast }TVf,f\rangle }_{L^2\left(\mu \right)}={\langle TVf,TVf\rangle }_Y\ge 0$$

(using the fact that $V^{-1}=V^{\ast }$ because V is unitary). Therefore, $\theta \ge 0$ a.e., as desired, and we can assume that $\theta \ge 0$ everywhere.

Now define $E=\{x\in M:\theta (x)=0\}$. If $\mu \left(E\right)>0$, then ${\chi }_E\ne 0$ in $L^2\left(\mu \right)$, where ${\chi }_E$ is the characteristic function of the set E; this implies that $V{\chi }_E\ne 0$ in X and hence that $T^{\ast }TV{\chi }_E\ne 0$ (since $\mathcal{N}\left(T^{\ast }T\right)=\mathcal{N}\left(T\right)$ is trivial). However,

$$T^{\ast }TV{\chi }_E=V{m}_{\theta }{\chi }_E=V\left(\theta {\chi }_E\right)=0$$

because $\theta =0$ on E and ${\chi }_E=0$ on $M\setminus E$. This contradiction shows that $\mu \left(E\right)$ must be zero, that is, $\theta >0$ a.e. in M.

Therefore, if we define $\sigma =\sqrt{\theta }$ and

$$S=\{f\in L^2\left(\mu \right):{\sigma }^{-1}f\in L^2\left(\mu \right)\},$$

then Lemma 1 applies and we see that S is dense in $L^2\left(\mu \right)$. We define $U:S\to Y$ by $U=TV{m}_{{\sigma }^{-1}}$. Since ${\sigma }^{-1}f\in L^2\left(\mu \right)$ for all $f\in S$, U is well-defined, and we see that it is linear and densely defined. We now show that ${\parallel Uf\parallel }_Y={\parallel f\parallel }_{L^2\left(\mu \right)}$ for all $f\in S$. We have

$$\begin{array}{cc}\hfill {\parallel Uf\parallel }_Y^2={\langle Uf,Uf\rangle }_Y& ={\langle TV{m}_{{\sigma }^{-1}}f,TV{m}_{{\sigma }^{-1}}f\rangle }_Y\hfill \\ \hfill & ={\langle f,m_{{\sigma }^{-1}}{V}^{-1}{T}^{\ast }TV{m}_{{\sigma }^{-1}}f\rangle }_{L^2\left(\mu \right)}\hfill \\ \hfill & ={\langle f,m_{{\sigma }^{-1}}{m}_{{\sigma }^2}{m}_{{\sigma }^{-1}}f\rangle }_{L^2\left(\mu \right)}\hfill \\ \hfill & ={\langle f,f\rangle }_{L^2\left(\mu \right)}={\parallel f\parallel }_{L^2\left(\mu \right)}^2.\hfill \end{array}$$

However, now we see that U is bounded and densely defined and hence it extends to a bounded operator defined on all of $L^2\left(\mu \right)$. We will use U to denote this extension as well (therefore $U:L^2\left(\mu \right)\to Y$ satisfies ${U|}_S=TV{m}_{{\sigma }^{-1}}$). By continuity, we have ${\parallel Uf\parallel }_Y={\parallel f\parallel }_{L^2\left(\mu \right)}$ for all $f\in L^2\left(\mu \right)$ and hence U is an isometry.

Next, we show that $T=U{m}_{\sigma }{V}^{-1}$. For each $x\in X$, $m_{\sigma }{V}^{-1}x\in S$ because $m_{{\sigma }^{-1}}{m}_{\sigma }{V}^{-1}x=V^{-1}x\in L^2\left(\mu \right)$. However, then we see that, for each $x\in X$,

$$U{m}_{\sigma }{V}^{-1}x=TV{m}_{{\sigma }^{-1}}{m}_{\sigma }{V}^{-1}x=TV{V}^{-1}x=Tx.$$

Therefore, $U{m}_{\sigma }{V}^{-1}=T$, as desired.

It remains only to show that $\mathcal{R}\left(U\right)=\overline{\mathcal{R}\left(T\right)}$. Let $y\in \overline{\mathcal{R}\left(T\right)}$. Then there exists $\left\{x_n\right\}\subset X$ such that $T{x}_n\to y$, that is, $U{m}_{\sigma }{V}^{-1}{x}_n\to y$. It follows that $\left\{U{m}_{\sigma }{V}^{-1}{x}_n\right\}$ is a Cauchy sequence and hence, because U is an isometry, $\left\{m_{\sigma }{V}^{-1}{x}_n\right\}$ is a Cauchy sequence in $L^2\left(\mu \right)$. Suppose $m_{\sigma }{V}^{-1}{x}_n\to f\in L^2\left(\mu \right)$. Then

$$Uf=\underset{n\to \infty }{lim}U{m}_{\sigma }{V}^{-1}{x}_n=y,$$

which shows that $y\in \mathcal{R}\left(U\right)$. Since $\mathcal{R}\left(U\right)\subset \overline{\mathcal{R}\left(T\right)}$ by definition of U, it follows that $\mathcal{R}\left(U\right)=\overline{\mathcal{R}\left(T\right)}$. This completes the proof. □

Theorem 1 is an immediate corollary of Theorem 3.

Proof of Theorem 1.

If we apply Theorem 3 to ${T|}_{\mathcal{N}{\left(T\right)}^{\perp }}$, we obtain

$${T|}_{\mathcal{N}{\left(T\right)}^{\perp }}=U{m}_{\sigma }{V}_1^{-1},$$

where $U:L^2\left(\mu \right)\to Y$ is an isometry and $V_1:L^2\left(\mu \right)\to \mathcal{N}{\left(T\right)}^{\perp }$ is unitary. We claim that

$$T=U{m}_{\sigma }{V}^{†},$$

(4)

where $V:L^2\left(\mu \right)\to X$ is defined by $Vf=V_{1}f$ for all $f\in L^2\left(\mu \right)$. Since V is obviously an isometry, proving that (4) holds will complete the proof. By definition, $V^{†}x$ is the minimum-norm least-squares solution of $Vf=x$. Moreover, since $\mathcal{N}\left(V\right)$ is trivial, $V^{†}x$ is the unique least-squares solution of $Vf=x$, which is defined by

$$V^{†}x=\{\begin{array}{cc}\hfill V_1^{-1}x,& \mathrm{if}x\in \mathcal{N}{\left(T\right)}^{\perp },\hfill \\ \hfill 0,& \mathrm{if}x\in \mathcal{N}\left(T\right).\hfill \end{array}$$

However, then

$$x\in \mathcal{N}{\left(T\right)}^{\perp }\Rightarrow U{m}_{\sigma }{V}^{†}x=U{m}_{\sigma }{V}_1^{-1}x=Tx$$

and

$$x\in \mathcal{N}\left(T\right)\Rightarrow U{m}_{\sigma }{V}^{†}x=0=Tx.$$

This proves (4), which completes the proof. □

3. Some Properties of the SVE of a Bounded Linear Operator

3.1. Relationship to the SVE of a Compact Operator

Suppose $T:X\to Y$ is a compact operator with singular value expansion

$$T=\sum _{n=1}^{\infty }{\sigma }_n{\psi }_n\otimes {\varphi }_n.$$

As noted above, $\left\{{\varphi }_n\right\}$ is a complete orthonormal sequence for $\mathcal{N}{\left(T\right)}^{\perp }$ and $\left\{{\psi }_n\right\}$ is a complete orthonormal sequence for $\overline{\mathcal{R}\left(T\right)}$.

Let us define $M={\mathbb{Z}}^{+}$, $\mathcal{A}=\mathcal{P}\left({\mathbb{Z}}^{+}\right)$ (the power set of ${\mathbb{Z}}^{+}$), and $\mu$ to be counting measure (that is, for an $E\subset {\mathbb{Z}}^{+}$, $\mu \left(E\right)$ is the cardinality of E). Then $L^2\left(\mu \right)$ is the space of square summable sequences of real numbers (usually denoted by ${\ell }^2$) and, for $\alpha =\left\{{\alpha }_k\right\}\in L^2\left(\mu \right)$,

$$\int {\alpha }^{2}d\mu =\sum _{k=1}^{\infty }{\alpha }_k^2.$$

We define $V:L^2\left(\mu \right)\to X$ by

$$V\left({\alpha }_k\right)=\sum _{k=1}^{\infty }{\alpha }_k{\varphi }_k.$$

Then it is straightforward to verify that V is an isometry and that $V^{†}=V^{\ast }$ is defined by ${\left(V^{†}\left(x\right)\right)}_k={\langle {\varphi }_k,x\rangle }_X$ for all $k\in {\mathbb{Z}}^{+}$. The sequence $\sigma =\left\{{\sigma }_k\right\}$ is bounded and measurable with respect to the measure space $(M,\mathcal{A},\mu )$ and

$$m_{\sigma }\alpha =\left\{{\sigma }_k{\alpha }_k\right\}.$$

Finally, $U:L^2\left(\mu \right)\to Y$ is defined to be the extension to all of $L^2\left(\mu \right)$ of $TV{m}_{{\sigma }^{-1}}$, which is given, for $\alpha \in L^2\left(\mu \right)$ such that $m_{{\sigma }^{-1}}\alpha$ also lies in $L^2\left(\mu \right)$, by

$$TV{m}_{{\sigma }^{-1}}\alpha =T\left(\sum _{k=1}^{\infty }\frac{{\alpha }_k}{{\sigma }_k}{\varphi }_k\right)=\sum _{k=1}^{\infty }{\sigma }_k\frac{{\alpha }_k}{{\sigma }_k}{\psi }_k=\sum _{k=1}^{\infty }{\alpha }_k{\psi }_k.$$

Clearly this formula extends to every $\alpha \in L^2\left(\mu \right)$.

Therefore, for each $x\in X$, we have

$$\begin{array}{cc}\hfill U{m}_{\sigma }{V}^{-1}x=\sum _{k=1}^{\infty }{\left(m_{\sigma }{V}^{-1}x\right)}_k{\psi }_k& =\sum _{k=1}^{\infty }{\sigma }_k{\langle {\varphi }_k,x\rangle }_X{\psi }_k\hfill \\ \hfill & =(\sum _{k=1}^{\infty }{\sigma }_k{\psi }_k\otimes {\varphi }_k)x\hfill \\ \hfill & =Tx.\hfill \end{array}$$

This shows that $T=U{m}_{\sigma }{V}^{-1}$ and also that $U{m}_{\sigma }{V}^{-1}$ is just another way of writing the usual singular value expansion of T.

3.2. The SVE of Operators Related to T

Throughout the rest of the paper, we assume that $T:X\to Y$ is a bounded linear operator from one real Hilbert space X to another such space Y, and that $T=U{m}_{\sigma }{V}^{†}$ is the SVE of Theorem 1. The associated Borel space is denoted by $(M,\mathcal{A},\mu )$ and the topology of M is denoted by $\mathcal{T}$.

Since $V^{\ast }=V^{†}$ and $V^{\ast }V=I$ hold for any isometry V, we immediately have the following:

$$\begin{array}{cc}\hfill T^{\ast }& =V{m}_{\sigma }{U}^{†},\hfill \\ \hfill T^{\ast }T& =V{m}_{{\sigma }^2}{V}^{†},\hfill \\ \hfill T{T}^{\ast }& =U{m}_{{\sigma }^2}{U}^{†}.\hfill \end{array}$$

3.3. Inverse Problems and the SVE

It is well known that the equation $Tx=y$ represents a true inverse problem if and only if $\mathcal{R}\left(T\right)$ fails to be closed (see ([3], Section 2.3)). In this case, the solution x (if it exists) does not depend continuously on the data vector y. One way to state this precisely is to note that the generalized inverse $T^{†}$ is unbounded if and only if $\mathcal{R}\left(T\right)$ fails to be closed. We will study $T^{†}$ below in Section 4; for now, we prove the following necessary and sufficient condition for $\mathcal{R}\left(T\right)$ to be closed.

Theorem 4.

The range of T is closed if and only if the function σ is bounded away from zero, that is, if and only if there exists $\gamma >0$ such that $\sigma \left(t\right)\ge \gamma$ for almost all $t\in M$.

Proof. 

It is a standard result that $\mathcal{R}\left(T\right)$ is closed if and only if there exists $\gamma >0$ such that

$${\parallel Tx\parallel }_Y\ge \gamma {\parallel x\parallel }_X\forall x\in \mathcal{N}{\left(T\right)}^{\perp }$$

(5)

(see ([3], Theorem 2.20)). If there exists $\gamma >0$ such that $\sigma \left(t\right)\ge \gamma$ for almost all $t\in M$, then

$$\parallel m_{\sigma }{f\parallel }_{L^2\left(\mu \right)}\ge \gamma {\parallel f\parallel }_{L^2\left(\mu \right)}\forall f\in L^2\left(\mu \right).$$

We have ${\parallel Vf\parallel }_X={\parallel f\parallel }_{L^2\left(\mu \right)}$ for all $f\in L^2\left(\mu \right)$; moreover, V defines an isomorphism from $L^2\left(\mu \right)$ to $\mathcal{N}{\left(T\right)}^{\perp }$. Therefore,

$${\parallel Tx\parallel }_Y=\parallel U{m}_{\sigma }{V}^{†}{x\parallel }_Y=\parallel m_{\sigma }\left(V^{†}x\right){\parallel }_{L^2\left(\mu \right)}\ge \gamma \parallel V^{†}{x\parallel }_{L^2\left(\mu \right)}=\gamma {\parallel x\parallel }_X\forall x\in \mathcal{N}{\left(T\right)}^{\perp }.$$

Conversely, suppose $\sigma$ is not bounded away from zero. It follows that

$$S_k=\{t\in M:\sigma \left(t\right)<1/k\}$$

has positive measure for each $k\in {\mathbb{Z}}^{+}$. Therefore, with $f_k={\chi }_{S_k}$, we have

$${\parallel m_{\sigma }{f}_k\parallel }_{L^2\left(\mu \right)}<\frac{1}{k}{\parallel f_k\parallel }_{L^2\left(\mu \right)}\forall k\in {\mathbb{Z}}^{+}$$

and hence

$$\parallel T{x}_k{\parallel }_Y<\frac{1}{k}{\parallel x_k\parallel }_X\forall k\in {\mathbb{Z}}^{+},$$

where $x_k=V{f}_k$. This shows that $\mathcal{R}\left(T\right)$ fails to be closed in this case, and the proof is complete. □

3.4. s-Numbers

Given the utility of singular values for matrices and compact operators, it is natural to try to extend the concept to more general operators. This can be done in various ways. The Courant-Fischer characterization [8,9] of the singular values ${\sigma }_1,{\sigma }_2,\dots$ of a compact operator $T:X\to Y$ is the following:

$${\sigma }_k=\underset{\begin{array}{c}S\subset X\\ dim\left(S\right)<k\end{array}}{min}\underset{\begin{array}{c}x\in S^{\perp }\\ x\ne 0\end{array}}{max}\frac{{\parallel Tx\parallel }_Y}{{\parallel x\parallel }_X},k=1,2,\dots .$$

(6)

Equation (6) can be taken as the definition of the s-numbers of an arbitrary bounded linear operator $T:X\to Y$ by replacing “min” with “inf” and “max” with “sup.”

Alternatively, the singular values of a compact operator can be characterized [10] as

$${\sigma }_k=\underset{\begin{array}{c}\tau :X\to Y\\ \mathrm{rank}\left(\tau \right)<k\end{array}}{min}\parallel \tau -T\parallel ,$$

(7)

which again extends to arbitrary bounded linear operators T by replacing “min” with “inf”. It can be shown that (6) and (7) are equivalent for operators on Hilbert space. In fact, Pietsch [11] formulated a list of five axioms that characterize s-numbers on Hilbert space (in the sense that two definitions, such as (6) or (7), that satisfy the axioms are equivalent).

The above definitions of s-numbers are limited, in that they may not give much information if the continuous spectrum of $T^{\ast }T$ is nonempty. Fack and Kosaki [12] defined generalized s-numbers for certain operators in a von Neumann algebra, and their techniques allow for a nonempty continuous spectrum. In the context of the SVE presented in this paper, it would be natural to define the set of s-numbers of a bounded linear operator $T:X\to Y$ as the essential range of $\sigma$ (where $T=U{m}_{\sigma }{V}^{†}$); we refer the reader to the first author’s PhD dissertation [13] for a discussion. The relationship between these two approaches remains to be investigated.

4. The SVE and Tikhonov Regularization

We believe that Theorem 1 will prove to be useful in a variety of applications. Here we show that it can be used to give transparent proofs of convergence theorems in the theory of Tikhonov regularization, the most popular method for addressing inverse problems.

We consider an equation of the form $Tx=y$. We are given $y\in Y$ and wish to compute or estimate $x\in X$ satisfying the equation. The problem is well-posed if there exists a unique solution x for each $y\in Y$, where x depends continuously on y. Existence fails, at least for some $y\in Y$, if $\mathcal{R}\left(T\right)$ is a proper subspace of Y. However, in that case, it is common to settle for a least-squares solution of the equation, that is, an $x\in X$ that minimizes ${\parallel Tx-y\parallel }_Y^2$. Uniqueness fails to hold if $\mathcal{N}\left(T\right)$ is nontrivial, but we can select a unique (least-squares) solution by choosing the unique solution lying in $\mathcal{N}{\left(T\right)}^{\perp }$, which is equivalent to choosing the minimum-norm least-squares solution. The interesting case occurs when $\mathcal{R}\left(T\right)$ fails to be closed. In that case,

• Least-squares solutions exists only for y in the dense subspace $\mathcal{R}\left(T\right)\oplus \mathcal{R}{\left(T\right)}^{\perp }$ of Y;
• For each $y\in \mathcal{R}\left(T\right)\oplus \mathcal{R}{\left(T\right)}^{\perp }$, there exists a unique minimum-norm least-squares solution $x\in \mathcal{N}{\left(T\right)}^{\perp }$, but x does not depend continuously on y.

The generalized inverse $T^{†}:D\left(T^{†}\right)\to X$, where $D\left(T^{†}\right)=\mathcal{R}\left(T\right)\oplus \mathcal{R}{\left(T\right)}^{\perp }$, is defined by the condition that $T^{†}y$ is the minimum-norm least-squares solution of $Tx=y$. It follows from the above discussion that, when $\mathcal{R}\left(T\right)$ fails to be closed, then $T^{†}$ is a densely defined unbounded linear operator. In this case, the problem $Tx=y$, even when interpreted as asking for the minimum-norm least-squares solution, is ill-posed in that the solution does not depend continuously on the data. In this case, we call $Tx=y$ a (linear) inverse problem.

Many regularization techniques for solving $Tx=y$ approximate $T^{†}$ by a family $\{R_{\lambda }:\lambda >0\}$ of bounded operators. Here $\lambda$ is called the regularization parameter and it is required that $R_{\lambda }\to T^{†}$ pointwise as $\lambda \to 0^{+}$. The most popular regularization method is Tikhonov regularization, in which $R_{\lambda }={(T^{\ast }T+\lambda I)}^{-1}{T}^{\ast }$. This operator arises from solving

$$\underset{x\in X}{min}{\parallel Tx-y\parallel }_Y^2+\lambda {\parallel x\parallel }_X^2.$$

We first show that $R_{\lambda }y\to T^{†}y$ for all $y\in D\left(T^{†}\right)$. For convenience, we will write $x_{\lambda ,y}=R_{\lambda }y$.

By definition, $T^{†}y\in \mathcal{N}{\left(T\right)}^{\perp }$. Furthermore, $x_{\lambda ,y}$ is defined by the equation

$$(T^{\ast }T+\lambda I)x_{\lambda ,y}=T^{\ast }y,$$

which implies that

$$x_{\lambda ,y}={\lambda }^{-1}(T^{\ast }y-T^{\ast }T{x}_{\lambda ,y})\in \mathcal{R}\left(T^{\ast }\right)\subset \mathcal{N}{\left(T\right)}^{\perp }.$$

We will write $\overline{y}={\mathrm{pro}\mathrm{j}}_{\overline{\mathcal{R}\left(T\right)}}y$, and notice that $T^{\ast }y=T^{\ast }\overline{y}$ since $\overline{y}-y\in \mathcal{R}{\left(T\right)}^{\perp }=\mathcal{N}\left(T^{\ast }\right)$. It follows that $x_{\lambda ,y}={(T^{\ast }T+\lambda I)}^{-1}{T}^{\ast }y={(T^{\ast }T+\lambda I)}^{-1}{T}^{\ast }\overline{y}$. Moreover, since the least-squares solutions of $Tx=y$ are precisely the solutions of $T^{\ast }Tx=T^{\ast }y$, it also follows that $T^{†}y=T^{†}\overline{y}$. These two facts (that $T^{†}y,x_{\lambda ,y}\in \mathcal{N}{\left(T\right)}^{\perp }$ and that $T^{†}y$, $x_{\lambda ,y}$, can be defined by $\overline{y}$ in place of y) make it convenient to use the singular value expansion as expressed in Theorem 3 (as opposed to the version of Theorem 1).

Suppose that, using the notation of Theorem 3,

$${T|}_{\mathcal{N}{\left(T\right)}^{\perp }}=U{m}_{\sigma }{V}^{-1},$$

and recall that $\sigma >0$ a.e. in M. Since $x=T^{†}y$ satisfies $Tx=\overline{y}$,

$$U{m}_{\sigma }{V}^{-1}{T}^{†}y=\overline{y}\to T^{†}y=V{m}_{{\sigma }^{-1}}{U}^{†}\overline{y}.$$

Furthermore,

$$T^{\ast }T+\lambda I=V{m}_{{\sigma }^2}{V}^{-1}+\lambda V{V}^{-1}=Vm[{\sigma }^2+\lambda ]V^{-1}$$

(where we write $m_{\theta }$ as $m\left[\theta \right]$ when it is convenient to do so), and hence

$${(T^{\ast }T+\lambda I)}^{-1}=Vm\left[{({\sigma }^2+\lambda )}^{-1}\right]V^{-1}.$$

This leads to

$$x_{\lambda ,y}={(T^{\ast }T+\lambda I)}^{-1}{T}^{\ast }\overline{y}=Vm\left[\frac{\sigma }{{\sigma }^2+\lambda }\right]U^{†}\overline{y}.$$

However, then

$$\begin{array}{cc}\hfill T^{†}y-x_{\lambda ,y}& =V{m}_{{\sigma }^{-1}}{U}^{†}\overline{y}-Vm\left[\frac{\sigma }{{\sigma }^2+\lambda }\right]U^{†}\overline{y}\hfill \\ \hfill & =Vm[\frac{1}{\sigma }-\frac{\sigma }{{\sigma }^2+\lambda }]U^{†}\overline{y}\hfill \\ \hfill & =Vm\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]m_{{\sigma }^{-1}}{U}^{†}\overline{y}.\hfill \end{array}$$

To show that $T^{†}y-x_{\lambda ,y}\to 0$ as $\lambda \to 0$, it suffices to show that

$$m\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]m_{{\sigma }^{-1}}{U}^{†}\overline{y}\to 0as\lambda \to 0.$$

Moreover, since $\overline{y}\in \mathcal{R}\left(T\right)$ (as opposed to merely belonging to $\overline{\mathcal{R}\left(T\right)}$—this follows from the fact that $y\in D\left(T^{†}\right)=\mathcal{R}\left(T\right)\oplus \mathcal{R}{\left(T\right)}^{\perp }$), it follows that $\overline{f}=m_{{\sigma }^{-1}}{U}^{†}\overline{y}\in L^2\left(\mu \right)$. However, then, since $\lambda /({\sigma }^2+\lambda )$ is bounded on M and goes to 0 pointwise as $\lambda \to 0$, it follows that

$${\parallel m\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]m_{{\sigma }^{-1}}{U}^{†}\overline{y}\parallel }_{L^2\left(\mu \right)}=\int \frac{{\lambda }^2}{{({\sigma }^2+\lambda )}^2}{\overline{f}}^2\to 0\mathrm{as}\lambda \to 0$$

by the dominated convergence theorem. This shows that $x_{\lambda ,y}\to T^{†}y$ as $\lambda \to 0$. Henceforth, we will write $x_{0,y}=T^{†}y$.

We will prove two other results to demonstrate the usefulness of the singular value expansion. The result that we just defined shows that, for each $y\in D\left(T^{†}\right)$, $x_{\lambda ,y}\to x_{0,y}$ as $\lambda \to 0$. However, the result says nothing about the rate of convergence and, in fact, the convergence can be arbitrarily slow depending on the data $y\in D\left(T^{†}\right)$ (or, equivalently, on the solution $x_{0,y}$). For certain $x_{0,y}$, though, we can bound the rate of convergence. We will not attempt to prove the most general theorem, but rather just consider what turns out to be the optimal rate of convergence. We will show that if $x_{0,y}\in \mathcal{R}\left(T^{\ast }T\right)$, then

$${\parallel x_{0,y}-x_{\lambda ,y}\parallel }_X=O\left(\lambda \right).$$

From above, we have

$$x_{0,y}-x_{\lambda ,y}=Vm\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]m_{{\sigma }^{-1}}{U}^{†}\overline{y},$$

and $\overline{y}=T{x}_{0,y}=U{m}_{\sigma }{V}^{-1}{x}_{0,y}$. Therefore,

$$x_{0,y}-x_{\lambda ,y}=Vm\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]V^{-1}{x}_{0,y}.$$

If we now assume that $x_{0,y}\in \mathcal{R}\left(T^{\ast }T\right)$, say $x_{0,y}=T^{\ast }T{v}_0=V{m}_{{\sigma }^2}{V}^{-1}{v}_0$ for some $v_0\in \mathcal{N}{\left(T\right)}^{\perp }$, then we obtain

$$x_{0,y}-x_{\lambda ,y}=Vm\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]m_{{\sigma }^2}{V}^{-1}{v}_0=\lambda Vm\left[\frac{{\sigma }^2}{{\sigma }^2+\lambda }\right]V^{-1}{v}_0.$$

Since V is an isometry and $\parallel m\left[\frac{{\sigma }^2}{{\sigma }^2+\lambda }\right]\parallel$ is bounded by 1 (in general, $\parallel m_{\theta }\parallel =\mathrm{ess}\sup \theta$), it follows that $\parallel x_{0,y}-x_{\lambda ,y}{\parallel }_X=O\left(\lambda \right)$, as desired.

We can also prove the following converse result, namely, that if $y\in D\left(T^{†}\right)$ and ${\parallel x_{0,y}-x_{\lambda ,y}\parallel }_X=O\left(\lambda \right)$, then $x_{0,y}\in \mathcal{R}\left(T^{\ast }T\right)$. We will use the fact, easily verified, that $x\in \mathcal{R}\left(T^{\ast }T\right)$ if and only if $m_{{\sigma }^{-2}}{V}^{-1}x\in L^2\left(\mu \right)$, that is, if and only if $V^{-1}x$ belongs to the domain of the densely defined operator $m_{{\sigma }^{-2}}$. Let us write $f_0=V^{-1}{x}_{0,y}$; then we must show that

$$\int \frac{f_0^2}{{\sigma }^4}d\mu <\infty .$$

We have

$$x_{0,y}-x_{\lambda ,y}=Vm\left[\frac{\lambda }{{\sigma }^2+\lambda }\right]V^{-1}{x}_{0,y}=\lambda Vm\left[\frac{1}{{\sigma }^2+\lambda }\right]V^{-1}{x}_{0,y},$$

which implies that

$$\frac{\parallel x_{0,y}-x_{\lambda ,y}{\parallel }_X^2}{{\lambda }^2}=\parallel Vm\left[\frac{1}{{\sigma }^2+\lambda }\right]V^{-1}{x}_{0,y}{\parallel }_X^2=\parallel m\left[\frac{1}{{\sigma }^2+\lambda }\right]f_0{\parallel }_{L^2\left(\mu \right)}^2.$$

Since ${\parallel x_{0,y}-x_{\lambda ,y}\parallel }_X=O\left(\lambda \right)$ by assumption, there exists a constant $C>0$ such that

$$\parallel m\left[\frac{1}{{\sigma }^2+\lambda }\right]f_0{\parallel }_{L^2\left(\mu \right)}^2\le C\mathrm{for} \mathrm{all}\lambda >0,$$

that is,

$$\int \frac{f_0^2}{{({\sigma }^2+\lambda )}^2}d\mu \le C\mathrm{for} \mathrm{all}\lambda >0.$$

Since $f_0^2/{({\sigma }^2+\lambda )}^2$ converges monotonically to $f_0^2/{\sigma }^4$ a.e. in M as $\lambda \to 0$, it follows from the monotone convergence theorem that

$$\int \frac{f_0^2}{{\sigma }^4}d\mu \le C<\infty ,$$

as desired.

5. Discussion

The proofs of the last section are offered as an illustration of the power of the singular value expansion. The reader can compare these proofs to other treatments of the same results that can be found in the literature on inverse problems. In Groetsch’s monograph [14], the analysis is restricted to compact operators and Theorem 2.1.1, Corollary 3.1.2, and Theorem 3.2.2 correspond to our results; Groetsch’s proofs use the singular value expansion (2) for compact operators. The reader will see that our proofs are direct generalizations of the derivations given there, and also that there is no difficulty in extending his other conclusions to general bounded linear operators. Groetsch does present his theory in greater generality, with much of the analysis applying to a certain family of regularization operators $R_{\lambda }$, as opposed to just the Tikhonov approach. We restricted our presentation to Tikhonov regularization simply for convenience of exposition; there would be no difficulty in reproducing his results in the same level of generality.

To extend the results of [14] to general operators, the standard approach is to use the spectral representation of $T^{\ast }T$ in the form

$$T=\int \alpha d{E}_{\alpha },$$

where $\left\{E_{\alpha }\right\}$ is the spectral resolution of $T^{\ast }T$, and apply the so-called functional calculus, which allows the representation of functions of $T^{\ast }T$ via

$$f\left(T^{\ast }T\right)=\int f\left(\alpha \right)d{E}_{\alpha }.$$

It can be shown, for example, that

$$T^{†}=\int {\alpha }^{-1}d{E}_{\alpha }.$$

A good reference for this approach is the book [15] by Engl, Hanke, and Neubauer, which (among other things) extends the results of [14] to general bounded linear operators. There is no intrinsic difficulty in doing so, but it may be argued that the arguments are less intuitive and therefore harder to follow. For instance, it is necessary to work with integrals of the following types:

$$\int (\cdots )d{E}_{\alpha },\int (\cdots )d{\parallel E_{\alpha }x\parallel }_X^2,\int (\cdots )d{\langle E_{\alpha }x,y\rangle }_X.$$

As Halmos stated in his popular expository article [6] on the spectral theorem (one of the most-downloaded articles from the American Mathematical Monthly),

The result (namely, the spectral theorem for Hermitian matrices, when expressed using a resolution $\left\{E_{\alpha }\right\}$ of the identity) is not intuitive in any language; neither Stieltjes integrals with unorthodox multiplicative properties, nor bounded operator representations of function algebras, are in the daily toolbox of every working mathematician. In contrast, the formulation of the spectral theorem given below uses only the relatively elementary concepts of measure theory.

We believe that the singular value expansion for general bounded linear operators, as described above, offers a similarly intuitive tool that can replace the standard use of the functional calculus in many contexts.