End-to-end Learned, Optically Coded Super-resolution SPAD Camera

Qilin Sun, Jian Zhang, Xiong Dun, Bernard Ghanem, Yifan Peng, Wolfgang Heidrich,
ACM Transactions on Graphics, 2020

Abstract
Resources

Figure 1: Framework for our joint learning of imaging model and reconstruction.

Abstract

Single Photon Avalanche Photodiodes (SPADs) have recently received a lot of attention in imaging and vision applications due to their excellent performance in low-light conditions, as well as their ultra-high temporal resolution. Unfortunately, like many evolving sensor technologies, image sensors built around SPAD technology currently suffer from a low pixel count. In this work, we investigate a simple, low-cost, and compact optical coding camera design that supports high-resolution image reconstructions from raw measurements with low pixel counts. We demonstrate this approach for regular intensity imaging, depth imaging, as well transient imaging. Our method uses an end-to-end framework to simultaneously optimize the optical design and a reconstruction network for obtaining super-resolved images from raw measurements. The optical design space is that of an engineered point spread function (implemented with diffractive optics), which can be considered an optimized anti-aliasing filter to preserve as much high-resolution information as possible despite imaging with a low pixel count, low fill-factor SPAD array. We further investigate a deep network for reconstruction. The effectiveness of this joint design and reconstruction approach is demonstrated for a range of different applications, including high-speed imaging, and time of flight depth imaging, as well as transient imaging. While our work specifically focuses on low-resolution SPAD sensors, similar approaches should prove effective for other emerging image sensor technologies with low pixel counts and low fill-factors.

Resources

Paper_fullres: [Sun2019SingleShotSPAD.pdf (~20.3MB)]

SuppDoc_fullres: [Sun2019SingleShotSPAD-supp.pdf(~7.7MB)]

Code&Data will be available on github: [Coming Soon]

Citation

@article{10.1145/3372261,
author = {Sun, Qilin and Zhang, Jian and Dun, Xiong and Ghanem, Bernard and Peng, Yifan and Heidrich, Wolfgang},
title = {End-to-End Learned, Optically Coded Super-Resolution SPAD Camera},
year = {2020},
issue_date = {April 2020},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {39},
number = {2},
issn = {0730-0301},
url = {https://doi.org/10.1145/3372261},
doi = {10.1145/3372261},
journal = {ACM Trans. Graph.},
month = mar,
articleno = {9},
numpages = {14},
keywords = {depth/transient imaging, super-resolution, diffractive optics, SPAD}
}