Gyro-based Neural Single Image Deblurring

In this paper, we present GyroDeblurNet, a novel single-image deblurring method that utilizes a gyro sensor to resolve the ill-posedness of image deblurring. The gyro sensor provides valuable information about camera motion that can improve deblurring quality. However, exploiting real-world gyro data is challenging due to errors from various sources. To handle these errors, GyroDeblurNet is equipped with two novel neural network blocks: a gyro refinement block and a gyro deblurring block. The gyro refinement block refines the erroneous gyro data using the blur information from the input image. The gyro deblurring block removes blur from the input image using the refined gyro data and further compensates for gyro error by leveraging the blur information from the input image. For training a neural network with erroneous gyro data, we propose a training strategy based on the curriculum learning. We also introduce a novel gyro data embedding scheme to represent real-world intricate camera shakes. Finally, we present both synthetic and real-world datasets for training and evaluating gyro-based single image deblurring. Our experiments demonstrate that our approach achieves state-of-the-art deblurring quality by effectively utilizing erroneous gyro data.

GyroDeblurNet takes a blurred image and its corresponding gyro data with errors that are collected during the exposure time. The input gyro data are first transformed into our novel gyro data representation, camera motion field. The motion information in the camera motion field may not match with that of the blurred image due to errors in the gyro data.

To handle such gyro errors, GyroDeblurNet is equipped with two novel neural network blocks: the gyro refinement block and the gyro deblurring block. The gyro refinement block takes a gyro feature computed from the gyro data and the image feature as its inputs. Then it extracts meaningful feature from the erroenous gyro feature with the help of the image feature. The gyro deblurring block utilizes the refined gyro feature in the deblurring process using a deformable convolution layer. To consider remaining spatial artifacts, spatial refinement of deblurred feature is applied.

The GyroBlur-Synth dataset consists of synthetically generated blurred images and its corresponding gyro data. The blurred images are generated by warping a sharp image using homographies computed from gyro data. Each sample in the dataset consists of a blurred image, corresponding sharp image, and the gyro data sequence. The dataset consists of 14,600 and 640 samples in the training set and the test set respectively. The GyroBlur-Real dataset consists of blurred images and its corresponding gyro data collected from a real-world smartphone with no ground-truth sharp image. The dataset provides 100 pairs of real-world blurred images and gyro data.