Motion Parallax Holograms Generated from an Existing Hologram

Abstract

Generating new motion parallax holograms is required for holographic head-mounted displays when the head moves. Additionally, it is required for hologram generation from light field data that consist of a number of motion parallax images. However, re-rendering three-dimensional (3D) scenes and re-calculating holograms are computationally complex. Therefore, we propose a generation strategy of holograms with different motion parallax from an existing hologram without re-rendering 3D scenes and re-calculating holograms. The proposed method employs Fourier band-pass filtering and the simple relation of trigonometric functions, which makes it capable of skipping the computationally complex processes.

1. Introduction

Recently, holographic displays have been attracting considerable attention because they can realize special projectors and three-dimensional (3D) displays satisfying well human depth cues [1,2,3] compared with stereoscopic displays. There are mainly two categories of holographic 3D displays, naked-eye displays [4,5,6] and head-mounted displays (HMDs) [7]. The former does not need special glasses to observe 3D scenes with wide viewing areas; however, spatial light modulators (SLMs) should have ultra-high-definition and large display area. Thus, very large spatial bandwidth products are required. Meanwhile, the latter does not require SLMs with large spatial bandwidth products because the position of the human eyes are fixed and placed near SLMs.

The fixed eye position helps to reduce the spatial bandwidth products. However, if we observe 3D scenes from different view positions, we need to re-render the 3D scenes from new positions and then re-calculate the holograms from the re-rendered 3D scenes. The calculations of the 3D rendering and holograms are time-consuming. Therefore, it is ideal to generate holograms at different view positions directly from an existing hologram without the re-rendering 3D scenes and re-calculation of holograms.

Hologram generation from light field data [8] requires a number of motion parallax images. It first divides a whole hologram plane into sub-holograms, and the multiple 3D scenes are rendered viewing from the center of each sub-hologram; subsequently, the sub-holograms are calculated. The light field-based hologram generation benefits from the following—complex processing [9] for hidden surface removal, lighting and shading do not need, unlike conventional hologram calculations [9,10,11,12] abovementioned time-consuming problem.

To circumvent the time-consuming process, we propose a generation strategy of motion parallax holograms from an existing hologram without re-rendering and recalculation. The proposed method is based on method demonstrated in [13] which uses trigonometric addition formulas to generate two-dimensional (2D) motion parallax images. Further, the proposed method employs a simple Fourier band-pass filtering and expands it to motion parallax hologram generation. A relighting to 3D scenes based on the wavefront recording plane method [14] without the re-rendering and recalculation of holograms has been proposed [15]. However, to our best knowledge, this is the first approach to generating motion parallax holograms from an existing hologram. The main purpose of this study is to confirm the principle of the proposed method for generating motion parallax holograms. Applications of the proposed method to holographic HMDs and light field-based hologram generation will be discussed in the future. The rest of this paper is organized as follows: Section 2 describes the proposed method, Section 3 shows the results of the proposed method, and Section 4 concludes this study.

2. Proposed Method

The proposed method generates motion parallax holograms from an existing hologram. Here, we employ a layer-based hologram calculation [16,17,18], given as

$$u(m_2,n_2)=\sum _z^{N_d}{\mathcal{P}}_z\left[u(m_1,n_1){\mathcal{M}}_z(m_1,n_1)\right],$$

(1)

where $u(m_2,n_2)$ is the complex amplitudes on a hologram plane, ${\mathcal{P}}_z$ is the operator of a diffraction calculation at a depth z, $N_d$ is the number of layers, $u(m_1,n_1)$ is one color component of an RGB image, and ${\mathcal{M}}_z(m_1,n_1)$ is a binary function [16] for extracting $u(m_1,n_1)$ pixels at the depth z. This study employed the angular spectrum method [12] for diffraction calculation. First, the hologram calculation divides the 3D scene into multiple layers using depth image information. Then, we perform diffraction calculations from the multiple layers to the hologram; subsequently, we accumulate all diffracted results to obtain the final hologram.

The proposed method is based on [13], called Hidden Stereo, which yields motion parallax images directly from RGB and depth images. This technique has been developed for switchable stereoscopic displays of 2D and 3D display modes, not for holographic displays. The principle is simple. As well-known, an image can be decomposed into a combination of sinusoidal waves according to the Fourier theorem. Let us consider a sinusoidal wave $\sin \left(\omega x\right)$, where $\omega$ is an angular frequency and x is a spatial variable. If we can generate the orthogonal sinusoidal wave of the original sinusoidal wave, we can derive the following simple relation:

$$\sin \left(\omega x\right)\pm A\sin (\omega x+\pi /2)=\sqrt{1+A^2}\sin (\omega x\pm \varphi ),$$

(2)

where $\tan \varphi =A$ [13]. This relation indicates that we can generate a right- or left-shifted image with the phase $\varphi$ by adding or subtracting the orthogonal image with an amplitude A to the original image. Although, in [13], a complex steerable pyramid [20] was used to generate orthogonal images, it is computationally complex. Thus, our proposed method uses simple Fourier band-pass filtering.

Figure 1 shows the outline of the proposed method. First, We generate a hologram from 3D scenes with an RGB image and depth image using Equation (1); subsequently, we generate a disparity hologram from the calculated hologram using Fourier band-pass filtering. By adding or subtracting the disparity hologram to the calculated hologram, we obtain new holograms viewed at a right or left position. The newly generated holograms do not experience computational complexity because it requires only addition or subtraction operations. As the number of the bands increases, we obtain finer motion parallax.

Figure 2 shows the generation of disparity holograms. We first perform Fourier band-pass filtering to obtain quasi-orthogonal signals of the original hologram. The band-pass filtering consists of a fast Fourier transform (FFT) and band-pass masks ${\mathcal{B}}_j$. Subsequently, we perform the inverse FFTs (IFFTs) of the filtered spectra and take only the imaginary parts of the IFFT results to obtain the quasi orthogonal signals. That is, we calculate the following equation:

$${\mathcal{O}}_j(m_2,n_2)=\Im \left[\mathrm{IFFT}\left[\mathrm{FFT}\left[u_2(m_2,n_2)\right]{\mathcal{B}}_j(m,n)\right]\right],$$

(3)

where ℑ denotes the operator to take the imaginary part of the IFFT and then multiply the results with $\pm 1$. Holograms are in genral expressed as $\cos \left(\theta \right)$ where $\theta$ is arbitrary phases. Due to the relation of $\cos (\theta +\pi /2)=-\sin \left(\theta \right)$, the quasi-orthogonal signal can be obtained by taking the imaginary part of the IFFT result. ${\mathcal{O}}_j$ corresponds to $\sin (\omega x+\pi /2)$ in Equation (2).

We calculate $A_j$ of each frequency band, corresponding to A in Equation (2), from a depth image using $A_j=\tan ({\omega }_j{d}^{\prime }(m_1,n_1)/2)$, where ${\omega }_j$ is the center frequency of ${\mathcal{B}}_j$ and $d(m_1,n_1)$ denotes a depth image. In this study, a depth image $d(m_1,n_1)$ ranges from 0 to 255 values, larger values indicate farther distances from the hologram plane. Note that we convert the depth image $d(m_1,n_1)$ as $d^{\prime }(m_1,n_1)=(128-d(x,y))/255$, where the value 128 means the middle depth. The motion parallax is generated, centering at the middle depth.

After multiplying $A_j$ with ${\mathcal{O}}_j$, we obtain the disparity hologram $\mathcal{D}$ by summing all the multiplied results as

$$\mathcal{D}(m_2,n_2)=\sum _{j=1}^{N_B}{A}_j{\mathcal{O}}_j(m_2,n_2),$$

(4)

where $N_B$ is the number of frequency bands. In this study, we used $N_B=4$ because it was empirically enough for the image quality of reconstructed images from motion parallax holograms.

When we want to obtain a left-viewed hologram $u_L(m_2,n_2)$, we simply calculate $u_L(m_2,n_2)=u(m_2,n_2)+\mathcal{D}(m_2,n_2)$. Likewise, for a right-viewed hologram, we subtract the disparity hologram to the original hologram as $u_R(m_2,n_2)=u(m_2,n_2)-\mathcal{D}(m_2,n_2)$. By adding and subtracting the disparity hologram continuously, we can obtain changing motion parallax holograms continuously.

3. Results

Figure 3 shows the numerically color-reconstructed images from motion parallax holograms from left and right view positions. They are complex amplitude holograms [12]. We used the 3D model provided from [19]. We simply calculated the color holograms by applying Equations (1), (3) and (4) for red (633 nm), green (532 nm) and blue (430 nm) wavelengths, respectively. The resolution of holograms is $512\times 512$ pixels. The pixel pitch of the holograms is 10 μm. The near and far distances of 3D scenes are 3 and 5 cm from the holograms, respectively. It is an image hologram setup [21]. We used this setup to confirm the effectiveness of the proposed method, but the proposed method works well at long distances as well. Figure 3a,b show the reconstructed images of “Antinous.” We confirmed that the proposed method can generate motion parallax from left and right views. Likewise, the proposed method can generate motion parallax of reconstructed images of “Greek,” as shown in Figure 3c,d.

Additionally, likewise Figure 4, we present three videos in the Supplementary Section (Video S1: reconstructed video of “antinous,” Video S2: reconstructed video of “Greek,” and Video S3: reconstructed video of “Papillon” whose original 3D scene are provided in [22]) to confirm the effectiveness of the proposed method. These videos present reconstructed images with motion parallax using the proposed method, while continuously changing the viewing positions and reconstruction focuses.

We compared the computational complexity between the proposed method and the conventional method using Equation (1) alone. The conventional method requires the re-rendering of 3D scenes and re-calculating of holograms. If we need $N_v$ holograms with different positions, the computational complexity is given by $O((R+N_d{N}^2\log N)\times N_v)$, where R denotes the computational complexity of 3D scene rendering, and N and $N_v$ denote the size of a hologram, and the number of motion parallax, respectively. $N_d{N}^2\log N$ indicates the computational complexity of the layer-based hologram calculation of Equation (1). The computational complexity is proportional to $N_v$. By contrast, the computational complexity of the proposed method is expressed as $O((R+N_d{N}^2\log N)+N_B{N}^2\log N+N^2\times N_v)$. Notably, the computational complexity of the proposed method does not depend on the number of motion parallax $N_v$. $N_B{N}^2\log N$ indicates the computational complexity of the IFFT in Equation (3). $N^2{N}_v$ indicats the computational complexity of the addition or subtraction for the disparity hologram.

Figure 5 shows the calculation times of the proposed method and the conventional method as the function of the number of motion parallax. We used a central processing unit (CPU) and graphics processing unit (GPU) of Intel Core i7-4790 (3.60 GHz) (Santa Clara, CA, USA) and Nvidia GeForce RTX 2080 SUPER (Santa Clara, CA, USA), respectively, on Windows 10 (64 bit) operating system. We generated holograms of Equation (1) on the GPU using our wave optics library [23], and the motion parallax holograms of Equations (3) and (4) on the CPU. The hologram size is $2048\times 2048$ pixels. Other calculation parameters are the same as Figure 3.

The horizontal axes of “zero” include only the hologram calculation time using Equation (1). Besides, the horizontal axes of ‘one” in the proposed method include only the calculation times of Equations (3) and (4). The calculation time of the conventional method increases linearly as the motion parallax increase, whereas the calculation time of the proposed method does not increase after the number of motion parallax of “one,” because it requires only simple additions or subtractions.

If the proposed method is used in a naked-eye holographic display, there will be some holes in reconstructed images when viewed from the side, because we used RGB and depth images. This problem can be solved, for example, by using holographic stereograms [8]. Since holographic stereograms need to generate holograms of multi-view images, the proposed method will be useful for interpolating multi-view image holograms.

Because we used complex holograms, no direct or conjugate light is generated in the reconstructed images. When using amplitude- or phase-based SLMs, it is necessary to convert the complex hologram into an amplitude hologram or phase-only hologram using the single-sideband method [24] or double phase encoding [25]. We obtained the RGB and depth images from [19,22]. If we want to generate RGB and depth images, we can use a 3D graphics library such as OpenGL.

4. Conclusions

We proposed the generation method of motion parallax holograms from an existing hologram. The proposed method can circumvent the time-consuming processes of re-rendering 3D scenes and recalculating holograms. In future work, we will discuss the applications of the proposed approach to light field based-hologram generation and holographic HMDs.