Image Deblurring Aided by Low-Resolution Events

Abstract

Due to the limitation of event sensors, the spatial resolution of event data is relatively low compared to the spatial resolution of the conventional frame-based camera. However, low-spatial-resolution events recorded by event cameras are rich in temporal information which is helpful for image deblurring, while intensity images captured by frame cameras are in high resolution and have potential to promote the quality of events. Considering the complementarity between events and intensity images, an alternately performed model is proposed in this paper to deblur high-resolution images with the help of low-resolution events. This model is composed of two components: a DeblurNet and an EventSRNet. It first uses the DeblurNet to attain a preliminary sharp image aided by low-resolution events. Then, it enhances the quality of events with EventSRNet by extracting the structure information in the generated sharp image. Finally, the enhanced events are sent back into DeblurNet to attain a higher quality intensity image. Extensive evaluations on the synthetic GoPro dataset and real RGB-DAVIS dataset have shown the effectiveness of the proposed method.

1. Introduction

In contrast to the conventional frame-based camera which represents a dynamic scenario with a sequence of still images, an event-based vision sensor [1,2,3,4] tends to detect per-pixel brightness changes in microsecond resolution. Once the logarithm of the intensity changes exceeds a preset threshold c in a given pixel (x, y), an event will be triggered which can be formulated as

$$log({\mathbf{I}}_{\mathbf{xy}}\left(\mathbf{t}\right)+b)-log({\mathbf{I}}_{\mathbf{xy}}(\mathbf{t}-\Delta \mathbf{t})+b)=p·c$$

(1)

where ${\mathbf{I}}_{\mathbf{xy}}\left(\mathbf{t}\right)$ and ${\mathbf{I}}_{\mathbf{xy}}(\mathbf{t}-\Delta \mathbf{t})$ denote the intensities at time t and $t-\Delta t$, respectively. $\Delta t$ is the time since the last event happened in location (x, y). b is a small constant used for avoiding ${\mathbf{I}}_{\mathbf{xy}}\left(\mathbf{t}\right)$ to be zero. $p\in \{+1,-1\}$ is the polarity representing the direction (increase or decrease) of the intensity change. Finally, an event is represented as $e=(x,y,t,p)$. Since event data are discrete and own high temporal resolution, a high dynamic range, and a low motion blur, the event camera has shown its potential in several robotic and computer vision tasks, such as image reconstruction [5,6,7,8,9,10,11,12], image deblurring [13,14,15], object detection [16,17], and SLAM [18].

However, the spatial resolution of existing event cameras is still not larger than 1 megapixel [1,2,3,4,17], which is far from the 12 or 24 megapixel spatial resolution of existing frame-based cameras. Considering the high temporal events recorded by event cameras and the high-resolution images captured by conventional frame-based cameras, constructing a mixed imaging system with an event camera and a frame-based camera with different spatial resolutions may be a means of achieving high-quality images. A similar imaging system was suggested in [19], where an algorithm is provided for calibrating these two kinds of cameras.However, this work assumes that the intensity images captured by the frame-based camera are sharp, or the intensity image will not produce effectiveness in their method. Since motion in the world is very common and that high-speed frame-based cameras are expensive, it is difficult to acquire sharp intensity images in all scenarios. Therefore, our research is based on low-spatial-resolution events and high-spatial-resolution images with blurry intensity, in the aim of deblurring the image with the help of low-resolution events.

There are some representative works related to event-based image deblurring [14,15]. Pan et al. [15] restored a high frame-rate sharp video by modeling the relationship between the events and the blurry intensity images as a double integral, while Lin et al. [14] implemented this physical model with a neural network and attained high performance in video deblurring and interpolation. However, these methods aim to process the blurry intensity images and events which are in the same spatial resolution. We find experimentally that once the spatial resolution of events degrades, these methods’ performance rapidly declines (as shown in Figure 1).

Although both events and intensity images are degraded in our research, they still hold significant information. Specifically, on the one hand, the events are considered as changes between a sequence of latent images that lead to the blurring of the final image. Although they are in low spatial resolution, they still own high-resolution temporal information (as shown in Figure 2b), and the high-resolution temporal information is rather helpful for image deblurring. On the other hand, events are triggered by the local edges of the scene and it is possible to reconstruct high-resolution events with the assistance of high-resolution intensity images. Therefore, we propose an alternately performed model for a deblurring image aided by low-resolution events. This model mainly consists of two components: a physical model-based Deblur network (DeblurNet) which takes events and blurry image as inputs and aims to output a sharp high-resolution image; an EventSR network (EventSRNet) which takes low-resolution events and the high-resolution intensity images attained by DeblurNet as inputs and aims to derive the corresponding high-resolution events. Since the spatial resolution of events affects the performance of image deblurring (as shown in Figure 2a, the higher the spatial resolution of events, the better of image deblurring), we sought to attain an even better sharp image by another DeblurNet. At this time, DeblurNet takes the high-quality events generated by EventSRNet as input rather than the original low-resolution events.

The main contributions of this method are as follows.

• It aims to deblur images with the help of by low-resolution events, which is more practical;
• It proposes a model to explicitly restore the degraded intensity image and events by considering the complementarity between them, and an alternated strategy is applied to progressively promote the quality of the intensity image;
• Extensive evaluations of the synthetic GoPro [20] dataset and real RGB-DAVIS [19] dataset show the effectiveness of the proposed method.

2. Related Works

2.1. Event Enhancement

Since events captured by neuromorphic vision sensors are brightness changes of scenes, they are highly sensitive to noise. In order to enhance the quality of events, some previous works [22,23,24] have added an additional pre-processing operation implemented by a spatiotemporal filter into sensors. Other works such as [25] preferred to fuse it into a neural network optimized with motion consistency. Although all these works have shown their advantages in event enhancement, they only focus on denoising and do not consider the situation that the events are in a low spatial resolution. In contrast to previous works, Wang et al. [19] proposed to obtain high-resolution and noise-robust events by joint filtering low-resolution events with high-resolution intensity images. However, the high-resolution intensity images required here are supposed to be very clear and sharp, or they will be disabled in the algorithm. In reality, the assumption is too strong to implement because of the existing low-frame rate cameras and moving scenes. Unlike these methods, our work is based on low-spatial-resolution events and high-spatial-resolution images with blurry intensity and aims to iteratively update events and intensity images with the help of one another.

2.2. Event-Based Image Enhancement

Image enhancement aims to improve the image quality by removing blur, noise, or increasing its resolution. Since event cameras have the advantages of a high dynamic range, high temporal resolution, and low motion blur, there is increasing interest in enhancing the quality of images by event data. The method proposed by Wang et al. [26] could directly reconstruct, restore, and super-resolve images from events by three GANs. Mostafavi et al. [27] proposed to reconstruct high-resolution intensity images from events with the help of a flownet. The information used in both of these works are just events that are brightness changes, without a base brightness, and the reconstructed images tend to be miss-matched with the real world. Therefore, DAVIS, an event camera that can not only capture events at a high temporal rate but also attain a sequence of low-frame-rate-intensity images (denoted as APS), appears, and a lot of works started based on these events and APS. Pan et al. [15] proposed an event-based double integral model for obtaining a high-frame-rate video from events and a blurry intensity image and Lin et al. [14] implemented the physical model proposed by [15] as a neural network, which achieved a high performance in terms of video deblurring and interpolation. Moreover, Wang et al. [28] unified denoising, deblurring, and super-resolution in one model by an event-enhanced degeneration model and Zhang et al. [29] proposed a hybrid deblur net for image deblurring with learned event representation. These methods have shown their advantage of image enhancement. However, they are all adapted to the intensity images and the events that are in the same and low spatial resolution. They have ignored the advantages (high spatial resolution) of the frame-based camera and once the spatial resolution of events is lower than that of the intensity images, the performance of these methods rapidly declines. Our method in this work is based on the promising premise that events captured by event camera which are in low-resolution and images captured by the conventional frame-based camera which are blurry but have high spatial resolution can function alternately to provide a solution to enhance the blurry intensity image and low-resolution events.

3. Method

This paper aimed to attain a blur-less and high-resolution intensity image with a sequence of low-spatial-resolution events recorded by an event camera and a high-spatial-resolution blurry intensity image captured by a frame-based camera. In order to make full use of the complementarity between these two kinds of data, an alternately performed model consisting of a Deblur network (DeblurNet) and an EventSR network (EventSRNet) is proposed here. DeblurNet is used for deblurring with the help of events data that own temporal information and an EventSR network (EventSRNet) used for enhancing event data assisted by high spatial resolution intensity images with enriched structure information. DeblurNet first produces a preliminary sharp image with original low-resolution events. Then, EventSRNet enhances the quality of events with the generated preliminary sharp image which potentially owns the rich structure information needed by low-resolution events. Since high-quality events yield high-quality intensity images (as shown in Figure 2a), another DeblurNet inputted with the enhanced events generated by EventSRNet is applied for further attaining a better sharp intensity image.

This section aims to introduce the details of the proposed method, and at the beginning, we will introduce the stacking method used in this paper in Section 3.1. Then, the details of the architecture and its learning method will be introduced in Section 3.2 and Section 3.3, respectively. The final part (Section 3.4) is the training procedure.

3.1. Representation of Event Data

In order to adapt the discrete event data to the conventional convolutional neural network, we stack event data based on time which is similar to the SBT proposed in [12]. Specifically, the exposure time for collecting a blurry intensity image ${\mathbf{I}}_{\mathbf{b}}$ is denoted as ${\mathbf{t}}_{\mathbf{b}}$. Then, events triggered during ${\mathbf{t}}_{\mathbf{b}}$ are separated into $\mathbf{n}$ parts. For the ith ($i=1,2,\cdots ,\mathbf{n}$) part, negative and positive events in the time interval [$\frac{(i-1){\mathbf{t}}_{\mathbf{b}}}{\mathbf{n}}$, $\frac{i{\mathbf{t}}_{\mathbf{b}}}{\mathbf{n}}$] are accumulated into two channels ${\mathbf{E}}_{-}^i(x,y)$ and ${\mathbf{E}}_{+}^i(x,y)$ for each pixel $(x,y)$, respectively. That is, while $e=(x,y,t,p)$:

$$\begin{array}{cc}\hfill {\mathbf{E}}_{-}^i(x,y)& =\sum _{t=\frac{(i-1){\mathbf{t}}_{\mathbf{b}}}{\mathbf{n}}}^{\frac{i{\mathbf{t}}_{\mathbf{b}}}{\mathbf{n}}}p,\hspace{1em}p=-1,\hfill \\ \hfill {\mathbf{E}}_{+}^i(x,y)& =\sum _{t=\frac{(i-1){\mathbf{t}}_{\mathbf{b}}}{\mathbf{n}}}^{\frac{i{\mathbf{t}}_{\mathbf{b}}}{\mathbf{n}}}p,\hspace{1em}p=1.\hfill \end{array}$$

(2)

In conclusion, events for ${\mathbf{I}}_{\mathbf{b}}$ can be expressed as $\mathbf{E}=\left[{\mathbf{E}}_{-}^1,{\mathbf{E}}_{+}^1,\dots ,{\mathbf{E}}_{-}^n,{\mathbf{E}}_{+}^n\right]$, and $\mathbf{E}\in {\mathbf{R}}^{2\mathbf{n}\times H\times W}$, where H and W are the height and width of image ${\mathbf{I}}_{\mathbf{b}}$.

3.2. Architecture

The overview of the architecture is shown in Figure 3. It contains two DeblurNet and one EventSRNet. The first DeblurNet aims to yield a preliminary sharp image with the low-resolution events. This preliminary sharp image will be sent into the EventSRNet to guide the enhancement of events with its rich structure information. Then, another DeblurNet takes the enhanced events generated by EventSRNet to further enhance the quality of the sharp image. Both DeblurNet and EventSRNet are implemented with neural networks. The following was detailed in the introduction.

3.2.1. DeblurNet

The method proposed by Lin et al. [14] has shown the efficiency of the event-based convolutional neural network implemented with a physical model for video deblurring. We tend to adopt this method for event-based image deblurring. Since the original model is designed for video deblurring and interpolation, by abandoning the video-related components, the IntegralNet implemented with an Unet [30] and a dynamic filter network [31] remained as our DeblurNet. Only a blurry intensity image ${\mathbf{I}}_{\mathbf{b}}$ and its events $\mathbf{E}$ which occurred during ${\mathbf{t}}_{\mathbf{b}}$ are required as inputs. Therefore, the DeblurNet can be formulated as

$${\mathbf{I}}_{\mathbf{s}}^{\mathbf{0}}=DeblurNet({\mathbf{I}}_{\mathbf{b}},\mathbf{E}).$$

(3)

3.2.2. EventSRNet

EventSRNet is implemented with a Unet with skip connections. For making full use of the complementarity between intensity images and events, EventSRNet takes the combination of a blurry intensity image ${\mathbf{I}}_{\mathbf{b}}$, events $\mathbf{E}$, and the sharp image ${\mathbf{I}}_{\mathbf{s}}$ generated by DeblurNet as input. Its goal is to generate a sequence of high spatial resolution events ${\mathbf{E}}_{\mathbf{sr}}$. This network can be formulated as

$${\mathbf{E}}_{\mathbf{sr}}=EventSRNet\left([{\mathbf{I}}_{\mathbf{s}},{\mathbf{I}}_{\mathbf{b}},\mathbf{E}]\right).$$

(4)

More details of these two networks are provided in Appendix A.

3.2.3. Deblurring Refinement

For further achieving a high-quality intensity image, the super-resolved events and blurry intensity image will be sent back into the DeblurNet, and it can be formulated as

$${\mathbf{I}}_{\mathbf{s}}^{\mathbf{1}}=DeblurNet({\mathbf{I}}_{\mathbf{b}},{\mathbf{E}}_{\mathbf{sr}}).$$

(5)

3.3. Learning

3.3.1. DeblurNet Loss

DeblurNet is constrained with MSE loss:

$${\mathcal{L}}_d=\frac{1}{HW}(\alpha \parallel {\mathbf{I}}_{\mathbf{s}}^{\mathbf{0}}-\widehat{\mathbf{I}}{\parallel }^2+\beta \parallel {\mathbf{I}}_{\mathbf{s}}^{\mathbf{1}}-\widehat{\mathbf{I}}{\parallel }^2)$$

(6)

where H and W are the height and width of high-resolution intensity images, respectively. $\widehat{\mathbf{I}}$ is the groundtruth of intensity images. $\alpha$ and $\beta$ are the loss weight of the first deblurring and the second deblurring.

3.3.2. EventSRNet Loss

Since the events are the records of brightness changes, they tend to be sparse, especially when they have been stacked. Most values in $\mathbf{E}$ are zeros. Standard regression loss is not appropriate for the learning of EventSRNet. For attaining a better convergence, we adopt an asymmetric L1 loss rather than the standard L1 loss here. The asymmetric L1 loss will pay more attention to the place where occurs events and pay less attention to the place that has no brightness change. It can be formulated as

$${\mathcal{L}}_e=\frac{1}{HW}({\mathcal{L}}_{e+}+\gamma {\mathcal{L}}_{e-})$$

(7)

where:

$$\begin{array}{cc}{\mathcal{L}}_{e+}=|{\mathbf{E}}_{\mathbf{sr}}(x,y)-{\mathbf{E}}_{\mathbf{hr}}(x,y)|,\hfill & {\mathbf{E}}_{\mathbf{hr}}(x,y)>0\hfill \\ {\mathcal{L}}_{e-}=|{\mathbf{E}}_{\mathbf{sr}}(x,y)|,\hfill & {\mathbf{E}}_{\mathbf{hr}}(x,y)==0\hfill \end{array}$$

(8)

where $(x,y)$ is the coordination of stacked events and ${\mathbf{E}}_{\mathbf{hr}}$ is the high spatial resolution events. While $\gamma <1$, the place where no event has happened will apply less effect on ${\mathcal{L}}_e$. In our experiments, $\gamma =0.1$.

In conclusion, the whole loss function is:

$$\mathcal{L}={\mathcal{L}}_d+\theta {\mathcal{L}}_e$$

(9)

3.4. Training Procedure

There are three steps for training:

• Training the first DeblurNet with low-resolution events upsampled by bilinear interpolation and blurry intensity images. Here, $\alpha =1.0$, $\beta =0$, and $\theta =0$;
• Training EventSRNet with low-resolution events upsampled by bilinear interpolation, blurry intensity image, and the preliminary sharp intensity image generated by the first DeblurNet. Here, $\alpha =0$, $\beta =0$, and $\theta =1.0$;
• Training the second DeblurNet with super-resolved events enhanced by EventSRNet and blurry intensity images. Here, $\alpha =0.1$, $\beta =1.0$, and $\theta =1.0$.

3.5. Experimental Settings

The proposed method is implemented on the basis of PyTorch [32] and optimized with AdamW [33] whose momentum, momentum2, and weight decay are 0.9, 0.999, and 1 $\times {10}^{-4}$. The first DeblurNet and EventSRNet are trained with a learning rate of 1 $\times {10}^{-4}$ for 200 epochs, and then the learning rate descends to 1 $\times {10}^{-5}$ for continually fine-tuning for another 100 epochs. As for the second DeblurNet, since its parameters are initially with the first DeblurNet, its learning rate is set to ${10}^{-5}$ and it only needs to be trained for 80 epochs. In addition, our model is trained with the batch size and patch size of the training dataset as 16 and $256\times 256$.

3.6. Datasets

We evaluate the proposed model with two datasets: GoPro [20] and RGB-DAVIS [19].

GoPro [20] is a widely used dataset for dynamic scene deblurring. It contains a total of 33 scenes that have been split into a training set (22 scenes) and a test set (11 scenes). As for every scene, it consists of a sequence of sharp images. To adapt it to our task which requires a sequence of low-resolution events, a sequence of high-resolution events, a blurry high-resolution intensity image, a sharp high-resolution intensity image, and a series of operations have been applied to this dataset. Firstly, for attaining finer events and blurring images, we increase the frame-rate of videos from 240 fps to 960 fps by an existing high-performance video frame interpolation method proposed by Niklaus et al. [34]. Secondly, $(m-1)\times 4+1$ (m is the number of sharp images before interpolation and $m=11$ in this paper) sharp images are fused into the blurry intensity image with the methods mentioned in [20]. Thirdly, events between every two interpolated frames are simulated by an event simulator ESIM [35]. The events generated with the original intensity images are considered high-resolution events, while the low-resolution events are synthetic with the intensity images degraded via bilinear interpolation. As such, we finally 2040 and 1101 samples in the training set and test set, respectively.

RGB-DAVIS [19] is a real event dataset. It contains low-spatial-resolution events captured by event cameras and high-spatial-resolution intensity images attained by conventional frame-based camera. We used the same pipeline (including the video frame interpolation, sharp image fusion, and ESIM simulation) described above to acquire the additional high spatial resolution events and blurry intensity images. After simulating, we split this dataset into a training set and a test set based on scenes. The training set contains 918 samples while the test set owns 230 samples. More details are shown in Appendix C.

4. Results and Analysis

4.1. Comparison with State-of-the-Art Methods

To validate the effectiveness of our proposed method aiming to perform image deblurring aided by low-resolution events, we conducted experiments on the GoPro dataset and the RGB-DAVIS dataset and compared them with state-of-the-art works about image deblurring, including EDI [24] and SRN [21]. As for the metric, we used PSRN and SSIM [36] for the quantitative comparison of intensity images.

Note that EDI [24] only works for the events that have the same spatial resolution with the intensity image. Therefore, the low-resolution events will be upsampled into the spatial resolution of the intensity image by bilinear interpolation before sending them into this model. Since this work aimed to perform video deblurring and interpolation, for adapting the events to image deblurring, we choose the center frame as its deblurring output. As for SRN [21], it was originally designed for image deblurring without events and the synthetic blurry images here are a little bit different from those released in [20] (the synthetic blurry images here are harder than before for deblurring). For a fair comparison, we fine-tuned SRN with our synthetic data to obtain a comparable result. In addition, for demonstrating the basic ability of low-resolution events in the image deblurring task, we conducted an experiment that simultaneously sends the combination of upsampled events and blurry intensity images into SRN, and named it as SRN${}^{+}$. Since SRN is a relative heavy model that aims to process three scale inputs, it is time-consuming. In order to save time, we adopted the physical model-based methods proposed by [14] for DeblurNet. It is approximately 4-fold smaller than SRN, and it can achieve a comparable or even better performance for event-based image deblurring. Here, we name the results generated in the first DeblurNet as Ours_D, while we name the results generated in the second DeblurNet as Ours_DED.

The results of these methods on the GoPro dataset and RGB-DAVIS dataset are shown in

Table 1. Image deblurring results of GoPro in terms of average PSNR and SSIM. SRN and SRN${}^{+}$ represent the results of image deblurring without and with low-resolution events, respectively, while Ours_D and Ours_DED represent the results of our method without and with EventSRNet, respectively. This shows that the proposed method outperforms the methods from previous works and image deblurring can benefit from low-spatial resolution events and event-enhancing operations.

Method	EDI [15]	SRN [21]	SRN${}^{+}$	Ours_D	Ours_DED
2×
PSRN	22.14	30.99	34.13	34.26	35.73
SSIM	0.8029	0.9446	0.9651	0.9630	0.9742
4×
PSRN	21.61	30.99	32.49	32.58	33.92
SSIM	0.7997	0.9446	0.9568	0.9570	0.9651

and

Table 2. Image deblurring results of RGB_DAVIS in terms of average PSNR and SSIM. SRN and SRN${}^{+}$ represent the results of image deblurring without and with low-resolution events, respectively, while Ours_D and Ours_DED represent the results of our method without and with EventSRNet, respectively. This shows that the proposed method outperforms the methods from previous works and image deblurring can benefit from the -spatial resolution events and event-enhancing operations.

Method	EDI [15]	SRN [21]	SRN${}^{+}$	Ours_D	Ours_DED
2×
PSRN	20.16	26.96	27.60	27.73	27.98
SSIM	0.7636	0.8792	0.8901	0.8927	0.9033
4×
PSRN	19.79	26.51	27.27	27.33	27.65
SSIM	0.7582	0.8756	0.8851	0.8874	0.8907

, respectively. Figure 4 and Figure 5 are the qualitative visualizations of some examples in their test set. Each dataset contains two event settings. ×2 means that the spatial resolution of the intensity image is two times that of its events and ×4 means that the spatial resolution of the intensity image is four times that of its events. The proposed method performs favorably against the existing methods on both settings. Since EDI is optimized with a traditional algorithm, it cannot borrow useful information from sharp intensity images or high-resolution events which are considered the supervisor in deep-learning-based methods, regardless of the fact that its performance in PSRN or SSIM is relatively low. The visualized results tend to be artifacts, especially when the resolution of events is four times lower than the intensity images. We then observe SRN and SRN${}^{+}$. The performance of SRN${}^{+}$ is greater than that of SRN by more than 1 dB in PSNR for both the GoPro dataset and RGB-DAVIS dataset, which proves the effectiveness of the low-resolution events for image deblurring. Ours_D, the results of our first DeblurNet, despite being implemented with a smaller model, achieves a comparable or even better performance than that of the deblurring image. Furthermore, by adding additional EventSRNet and DeblurNet, Ours_DED attains even further performance enhancement. The visualized results also appear more smooth and clear. We made a comparison between the different event scale settings. It is obvious that a smaller resolution gap between the intensity image and events means a higher image deblurring performance. This phenomenon reflects the possibility of the performance promotion of image deblurring while enhancing the quality of events. This also proves the effectiveness of our EventSRNet.

4.2. Ablation Study

4.2.1. Effectiveness of EventSRNet

EventSRNet takes low-resolution events, blurry image, and sharp image as inputs. It not only tends to reconstruct high spatial resolution events from temporal information in low-spatial-resolution events, but also from the structure information in the blurry image and sharp image based on the assumption that local events are triggered with edges. Therefore, we conduct experiments with EventSRNet’s inputs in three settings: only low-resolution events, low-resolution events and blurry image, and low-resolution events, blurry image, and sharp images. The metric used for evaluating the quality of events is MSE (mean squared error), which is better the smaller it is. The corresponding results are shown in

Table 3. The results of EventSRNet with different input settings and learning losses in terms of MSE. These are tested on GoPro with the ×4 spatial resolution gap between intensity the image and events. The guides from the sharp images and blurry images and the asymmetric L1 loss benefit the enhancement of events.

Events	Blurry Image	Sharp Image	L1	AL1	MSE
√	-	-	-	√	0.01964
√	√	-	-	√	0.01065
√	√	√	-	√	0.00706
√	√	√	√	-	0.01478

and Figure 6. We can see that both the blurry image and sharp image are useful for the enhancement of events. However, with only blurry images, the generated events also tend to be blurry, while the sharp image can guide the events to be more clear.

4.2.2. Effectiveness of Asymmetric L1 Loss

Since the data structure of events is sparse, we adopted asymmetric L1 loss for the optimization of EventSRNet. To evaluate the usefulness of this setting, we replace the asymmetric L1 loss with normal L1 loss. Table 3 and Figure 6 are the results of event enhancement learning with L1 loss (L1) and asymmetric L1 loss (AL1), in terms of MSE. With low-resolution events, blurry image, and sharp image as inputs, the enhanced events learning by L1 are clear but lose a lot of information that is not near the edges in the intensity image. Asymmetric L1 loss can learn this sparse information well.

5. Conclusions

In contrast to previous single image deblurring works or low-resolution-event-based image deblurring works, this work takes both the advantages and disadvantages of the conventional frame-based camera and event camera into consideration and proposes to deblur images with the help of low-resolution events. Although both events and intensity images are degraded, the low spatial resolution events are still rich in temporal information that is useful for image deblurring, and the intensity images have potential use in the reconstruction of events. Therefore, this paper proposes an alternately performed model for enhancing the quality of intensity images and events by exploiting the complementarity between them. By firstly deblurring the image with state-of-the-art image deblurring methods, we can obtain a reasonably sharp image for providing rich structure information for the enhancement of events. The temporal information in the processed events will further promote the quality of the image. Extensive experiments show the effectiveness of the proposed method.

This work can be extended in several directions. The enhanced events can be further used for many downstream tasks, such as object recognition, object detection, and segmentation. Additionally, it can also be used for the frame interpolation of high-resolution video. We will thus consider further improving the quality of the events to make a contribution to downstream tasks.