Re-Calibration and Lens Array Area Detection for Accurate Extraction of Elemental Image Array in Three-Dimensional Integral Imaging

Abstract

This paper presents a new method for extracting an elemental image array in three-dimensional (3D) integral imaging. To reconstruct 3D images in integral imaging, as the first step, a method is required to accurately extract an elemental image array from a raw captured image. Thus, several methods have been discussed to extract an elemental image array. However, the accuracy is sometimes degraded due to inaccurate edge detection, image distortions, optical misalignment, and so on. Especially, small pixel errors can deteriorate the performance of an integral imaging system with a lens array. To overcome the problem, we propose a postprocessing method for the accurate extraction of an elemental image array. Our method is a unified version of an existing method and proposed postprocessing techniques. The proposed postprocessing consists of re-calibration and lens array area detection. Our method reuses the results from an existing method, and it then improves the results via the proposed postprocessing techniques. To evaluate the proposed method, we perform optical experiments for 3D objects and provide the resulting images. The experimental results indicate that the proposed postprocessing techniques improve an existing method for extracting an elemental image array in integral imaging. Therefore, we expect the proposed techniques to be applied to various applications of integral imaging systems

1. Introduction

Integral imaging was initially introduced by G. Lippmann in 1908, which is one of the techniques for recording the information of 3D objects and displaying 3D images [1]. The technique has the advantages of full-color 3D images without eyeglasses, horizontal and vertical parallaxes, and continuous viewpoints. The technique not only displays 3D images in physical space but also reconstructs 3D images in computer space through computational integral imaging reconstruction (CIIR). These merits enable integral imaging to be employed in various applications using real 3D images, such as 3D imaging, depth extraction, medical imaging, 3D games, and automation systems. Thus, many studies on integral imaging have been actively conducted [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23].

Integral imaging consists of a pickup process and a reconstruction process, as depicted in Figure 1. A pickup process captures the light photons from 3D objects and records them as an image array that is called the elemental image array (EIA). Optical devices to pick up an elemental image array are classified into a camera array, a moving camera, and a lens array. A camera array requires huge space and high cost. A moving camera has limitations such that it requires scanning time and a motorized device, thus it is hard to pick up a moving object. A lens array with a camera requires less cost and space than the two devices. Moreover, it can capture moving objects without time delay. Accordingly, a lens array is likely employed as a pick-up method in integral imaging [17,18,19,20,21,22,23].

In integral imaging, the setup of a pick-up process consists of a lens array and a capturing device such as a camera, as shown in Figure 1a. The process converts a raw image from a capturing device into an elemental image array (EIA). To reconstruct 3D images effectively and to improve the accuracy of the reconstruction, the process requires an accurate method to extract elemental images without distortion or loss of data. Thus, various methods to extract an elemental image array in 3D integral imaging have been discussed [24,25,26,27,28,29,30]. Among them, a technique using projection profiles was introduced to extract the two-dimensional (2D) lattice line structure, where the boundaries of the element images are estimated through projection profiles [24,25]. A projective transformation was applied to reduce errors by correcting the perspective distortion [26]. Moreover, a calibration pattern image was addressed to extract an elemental image array [27]. To increase the robustness of detecting the lens array lattice, a line segment detection algorithm was studied [28]. In addition, some methods using markers attached to the surface of a lens array were discussed to detect an elemental image array [29,30]. These existing methods can extract an elemental image array from a raw image of a lens array. However, the methods can suffer from inaccurate edge detection, image distortions, and optical misalignment, thus the performance of extracting elemental images can be degraded. Especially, the lens distortion or the inaccurate edge detection likely occurs on the boundaries of each lens within EIA. In this case, the size and location of each elemental image are inaccurate. Therefore, a study is still required for improving the accuracy of extracting elemental images.

In this paper, we propose a postprocessing method to accurately extract an elemental image array in 3D integral imaging. The proposed method improves the accuracy of an existing method by use of re-calibrating the results of the existing method. The proposed re-calibration algorithm calibrates the size and location of each elemental image by estimating the number of lenses accurately. In addition, we propose an algorithm to detect the lens array area via reusing intermediate data of an existing method. Thus, the extraction performance of an existing method can be improved by adding the proposed postprocessing techniques. To evaluate the proposed method, optical experiments are conducted. According to the results of the experiments, it turns out that our method improves the accuracy of extracting an elemental image array.

2. Existing Method of Extracting Elemental Image Array

Figure 2 shows an existing method of extracting an elemental image array using Canny edge detection and edge map projection [24]. The method is chosen as our reference method to improve its accuracy since it is one of the state-of-the-art techniques in terms of accuracy in the literature. The method employed Canny edge detection [31] to find a candidate set of the lens boundaries, as with other methods. Then, the edge map is separated into the horizontal and vertical edges through horizontal and vertical one-dimensional median filters, as shown in Figure 2. Using those edge map images, the projection profiles are calculated in each direction, as depicted in the middle of Figure 2. Those projection profiles have many periodical peaks which are very likely to be locations of lens boundaries. Some peaks are selected by thresholding with a threshold of 20% of the maximum value of the profile. Through the selected peaks, candidate locations for the lens lattice lines in the horizontal direction {l_h} = <l_h1, l_h2, …, l_hN> and in the vertical direction {l_v} = <l_v1, l_v2, …, l_vN> are obtained. Next, the positions of the candidate lattice lines are sequentially subtracted to obtain $\left\{\mathrm{dih}\right\}$ and $\left\{\mathrm{div}\right\}$, which are the horizontal and vertical distances between the expected lattice lines. Since the lenses are square, one joint histogram can be prepared by combining $\left\{\mathrm{dih}\right\}$ and $\left\{\mathrm{div}\right\}$ in the horizontal and vertical directions. Then, the most accumulated peak in the histogram is most likely to be the size of each elemental image. With the elemental image size w₀, the locations of the lattice lines can be expressed as

$$l_i=p_0+i{w}_0=\backslash r) (3$$

(1)

Here, p₀ is an integer between [0, w₀). For each horizontal and vertical direction, {l_best,h} and {l_best,v} can be obtained by setting p₀ to an integer between [0, w₀) so that the matching error between $l_i$ and the projection profile is minimized.

3. Proposed Method of Extracting Elemental Image Array

The proposed method consists of an existing method and two postprocessing techniques for extracting an EIA. Moreover, the method can utilize any existing method to improve its accuracy. Normally, existing methods detect the size of an elemental image and the lattice structure or locations of the lens boundaries in EIA effectively [24,25,26,27,28]. However, the accuracy is sometimes deteriorated by the lens distortion and the inaccurate edge detection at the boundary of each lens. Even a small misalignment such as 1 or 2 pixels negatively affects the image quality in computational reconstruction. Thus, the size and location of each elemental image can be contaminated by errors from those problems. To minimize the errors, postprocessing is very useful to improve an existing method for EIA extraction since any state-of-the-art method can be integrated with postprocessing.

Here, we propose two postprocessing techniques that consist of re-calibration and lens array area detection, as shown in Figure 3. Our re-calibration technique is carried out independently using the size and locations of elemental images from an existing EIA extraction method. An existing method provides the size and location of each elemental image, which should be real numbers to be accurate. Some problems mentioned above can cause the size of each elemental image to be inaccurate. Moreover, the locations of elemental images can take the same problems. However, the number of elemental images is quite robust since it should be an integer value. In our re-calibration, the number is calculated by quantizing its intermediate value. Here, the quantizing process can eliminate the error in the intermediate value. Thus, the number is more accurate than the size and the locations. This property is therefore utilized in our recalibration technique. In our lens array area (LAA) detection, the intermediate edge map is reused in our method. Usually, some existing methods employ markers to determine the lens array area [29,30]. The existing methods without markers use a cropped raw image so that the lens array area is the whole area of the input raw image. This may be a limitation to set up the optical configuration of the pickup process. On the other hand, our LAA detection technique can provide the lens array area without markers, and it can offer an efficient optical configuration.

3.1. Re-Calibration

The proposed re-calibration algorithm is shown in Figure 3. First, the reference lattice lines are sorted in the order of the largest projection profile value. This is because the greater the value obtained from the projection profile, the higher the probability that an accurate line is detected. Let the reference lines be $R_i$ and the number of reference lines be $N$. Then, the possible distances between two chosen reference lines D_k are defined as

$$D_k=R_m-R_n \left(k=1,2,\dots ,\frac{N\left(N-1\right)}{2},=\backslash r\right) \left(3\right),$$

(2)

where the maximum of $k$ is calculated by N-choose-2 and $R_m>R_n$.

The locations of reference lattice lines obtained from larger projection profiles are relatively robust, and there should be an integer number of lenses between the lattice lines. The number of lenses between the reference lines can be easily estimated. The distance D_k is divided by the initial value of the elemental image size, w₀, obtained by the existing method. By rounding this value, the number of lenses is estimated. The estimated number of lenses is very accurate since some errors in w₀ can be eliminated by the rounding or quantizing process. Thus, a new size of the elemental image can be calculated by dividing the distance D_k by the number of lenses. An updated size of the elemental image w_k can be written as

$$w_{k }=\frac{D_k}{Round \left(\frac{D_k}{ w_0}\right)}$$

(3)

Here, it can be said that w_k is more accurate than w₀ since the number is more accurate and the distances D_k are more accurately chosen than the existing method. Note that the ideal distance of D_k is a multiple of the ideal size of w_k. Moreover, w_k can still have an error due to an inaccurate D_k. Thus, we introduce a method to extract the robust size from a set of w_k.

To determine the final version of the elemental image size w_f, we employ a continuous probability density function along the size space. The data w_k in the size space are discrete and sparse, thus a continuous window is required to prepare a continuous probability density function [32]. Here, we use a Gaussian window which can be written by

$$G\left(w_k,\sigma \right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{\frac{-{\left(w-w_k\right)}^2}{2{\sigma }^2}}$$

(4)

where the sigma σ is a parameter of the Gaussian function and we fixed the sigma to be 2 in our method. Now, a linear combination of the Gaussian function for all w_k provide us with the continuous probability function or likelihood function for w_k. Taking the argument of maximizing the probability yields the final version of elemental image size, which is written by

$$w_{f }=\arg \max {\sum }_{k=1}^{M}G\left(w_k,\sigma \right)$$

(5)

For example, a set of Gaussian density functions for w_k = {216, 216.5, 217.625, 219.875, 228.25, 240} using Equation (4) and their linear combination are depicted in Figure 4. Moreover, the argument of maximizing the likelihood function is calculated as w_f =217.03 by using Equation (5). It is seen that a group of w_k which is the set of candidates of the accurate size provides the final version of w_f whereas some of w_k which is considered to be an inaccurate size are discarded. Therefore, our re-calibration technique is an effective method to suppress the noise due to inaccuracy in the size of the elemental image.

3.2. Lens Array Area Detection

The second postprocessing technique is an algorithm for detecting the boundary of the lens array. As shown in Figure 3, a raw image obtained from a capture device can include the area of the elemental image array. In this case, false elemental images affect the reconstruction images, thus a detection method for the lens array area is required to utilize 3D applications in integral imaging.

To localize a lens array area, we reuse the horizontal and vertical edge images which are intermediate images as shown in Figure 2. For the vertical edge image, the horizontal direction scan is performed to find position indices of the first and last edges. Among those indices, the most frequently occurring index is very likely to be the boundary line of the lens array. To find the index, the histogram of the indices is useful, as shown in Figure 5. Here, the index of the peak in the histogram is set to the initial value of the start and end boundaries of the lens array. Similarly, in the case of the horizontal edge image, the process above is repeated with a vertical direction scan to set the initial values of the top and bottom boundaries.

To improve the accuracy of the initial value of the boundary obtained from the histogram, we reuse the lattice line locations detected previously. The location of the lattice line closest to the initial value of the boundary index is determined to be the boundary of the lens array. Thus, the vertical and horizontal processing can finally determine the area of a lens array in the raw image.

4. Experimental Results and Discussion

In this paper, an experiment was conducted to evaluate the performance of the proposed method. The lens array used in the experiment is a 17 × 20 array of square lenses with a focal length of 73 mm and a size of 7.47 mm × 7.47 mm. As shown in Figure 6, three test images such as a diagonal pattern, car and bear toys, and a ruler were used for the experiment. A DSLR camera with pixels of 4000 × 3000 was employed to capture a raw image, which is also shown in Figure 6. Typical images seen through the lens array are prepared as the inputs of proposed and existing methods. To evaluate objectively, we determined the ground truth of the sizes and positions through careful manual operations. We compared the size and position results detected by each method through the ground truth of the elemental images. The ground truth is set by hand to determine the exact size and position. For the ground truth size, we manually counted the number of lenses in the picked-up image. Then four corner points are also manually determined by careful visual inspection of the raw image. The number and four corner points yield the size of the elemental image by dividing the distances of corner points by the number. Moreover, the ground truth positions can be determined from the positions of the counted segment lines between corner points.

Table 1. Experimental results in terms of accuracy.

Input Image	Elemental Image Size (Pixel)			Size Error (Pixel)		Average of Position Error (Pixel)
	Ground Truth	Existing	Proposed	Existing	Proposed	Existing	Proposed
Diagonal pattern	187.43	181	187.72	−6.43	+0.29	35.87	1.62
Car and bear	217.38	216	217.03	−1.38	−0.35	7.37	1.87
Ruler	223.2	222	223.54	−1.20	+0.34	10.10	2.86

and Figure 7 show the experimental results in terms of accuracy. Table 1 says that some poor results can be made from the existing method, especially the diagonal pattern object. The size error is 6.43 pixels for this object image, compared with the ground truth. The strong edges inside of elemental images from the diagonal pattern affect the projection profiles. However, our method improved the existing extraction method by adding our postprocessing technique so that the size error is improved to 0.29 pixels. Thus, the poor results of the existing method were converted to acceptable results with high accuracy, as shown in Figure 7a. Here, the ground truth is highlighted in green, the grid extracted by the existing method is in blue, and the grid is in red for the proposed method.

Similarly, for the car and bear and ruler images, the element image size errors from the existing method are 1.38 and 1.20, respectively. Moreover, the left images of Figure 7b,c shows the moderate results of the existing method. However, the position errors cannot be acceptable since their averages of position errors are 7.37 and 10.10, respectively. On the other hand, the proposed processing method turned the averages of position errors into 1.87 and 2.86, respectively, which are much smaller than the existing method. Moreover, Figure 7b,c depicted this improvement of our method visually.

Figure 8 shows the detected area of the lens array to bound EIA. In the experimental images, some regions are not in the lens array area. These regions should be discarded in detected EIA. The proposed method for a lens array area provides acceptable results according to Figure 8 which shows three resulting images from the detection of the lens array areas. Here, the detected EIA area is highlighted in the red box. The zoomed areas in Figure 8 clearly show the performance of the lens array area detection. These results also indicate our method is very effective in taking elemental images maximally.

Based on the experimental results, we verified that the proposed method can be applied to the extraction of an elemental image array, and it improved the existing method significantly by suppressing errors in the results of the existing method. Moreover, it is confirmed that the proposed lens array area detection works well.

As shown in Figure 9, we reconstructed 3D images for integral image display to show the effectiveness of the misaligned size of elemental images. The standard computational integral imaging reconstruction (CIIR), which is used in [9,16,23], is employed to show integral imaging. The method reconstructs 2D images according to depth based on back projection. Figure 9 shows the reconstructed images with different elemental image sizes. These resulting images were focused on the bear object. It is seen that the bear object is well focused with the correct elemental image size, as shown in Figure 9a, whereas imaging for the object fails with the misaligned elemental image size, as shown in Figure 9b.

5. Conclusions

In this paper, we have proposed an accurate extraction method for an elemental image array. The proposed method consists of an existing method and two postprocessing techniques of recalibration and lens array area detection. The recalibration technique can update the results of an existing method; thus, the size and locations of elemental images can be more accurate than before. In addition, the lens array area detection technique can detect the lens array boundaries in a raw image captured through a lens array. The experimental results indicated the proposed method improves an existing method for extracting an elemental image array. The proposed postprocessing techniques can be integrated with an existing method to improve the accuracy of an integral imaging system. This has the advantage of improving an existing integral imaging system with minimal cost. Therefore, it is expected that the proposed method can be applied to various integral imaging systems.