Neural Global Shutter

Learn to Restore Video from a Rolling Shutter Camera with Global Reset Feature

CVPR 2022

Zhixiang Wang1,2,3     Xiang Ji1     Jia-Bin Huang4      Shin'ichi Satoh3,1      Xiao Zhou5      Yinqiang Zheng1
1The University of Tokyo     2RIISE     3National Institute of Informatics    
4University of Maryland College Park     5Hefei Normal University

TL;DR: We turn the traditional rolling shutter (RS) rectification problem, i.e., translating geometrically dististored RS images/videos to undistorted global shutter (GS) counterparts, into a deblur-like one. Our key idea is to exploit a widely ignored hardware feature of RS sensors: global reset (RSGR).

Abstract

Most computer vision systems assume distortion-free images as inputs. The widely used rolling-shutter (RS) image sensors, however, suffer from geometric distortion when the camera and object undergo motion during capture. Extensive researches have been conducted on correcting RS distortions. However, most of the existing work relies heavily on the prior assumptions of scenes or motions. Besides, the motion estimation steps are either oversimplified or computationally inefficient due to the heavy flow warping, limiting their applicability. In this paper, we investigate using rolling shutter with a global reset feature (RSGR) to restore clean global shutter (GS) videos. This feature enables us to turn the rectification problem into a deblur-like one, getting rid of inaccurate and costly explicit motion estimation. First, we build an optic system that captures paired RSGR/GS videos. Second, we develop a novel algorithm incorporating spatial and temporal designs to correct the spatial-varying RSGR distortion. Third, we demonstrate that existing image-to-image translation algorithms can recover clean GS videos from distorted RSGR inputs, yet our algorithm achieves the best performance with the specific designs. Our rendered results are not only visually appealing but also beneficial to downstream tasks. Compared to the state-of-the-art RS solution, our RSGR solution is superior in both effectiveness and efficiency. Considering it is easy to realize without changing the hardware, we believe our RSGR solution can potentially replace the RS solution in taking distortion-free videos with low noise and low budget.

Results

RSGR input

Our output

GS ground-truth

BibTeX

@inproceedings{Wang_2022_CVPR,
author = {Wang, Zhixiang and Ji, Xiang and Huang, Jia-Bin and Satoh, Shin'ichi and Zhou, Xiao and Zheng, Yinqiang},
title = {Neural Global Shutter: Learn to Restore Video from a Rolling Shutter Camera with Global Reset Feature},
booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2022}
}

Acknowledgements

This research is supported in part by the JSPS KAKENHI Grant Numbers 20H05951, 20H04215, the Key Project of Natural Science Research of Universities in Anhui (KJ2017A934), and the Value Exchange Engineering, a joint research project between Mercari, Inc. and RIISE. ZW also thanks the MEXT Scholarship.