搜索

x

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于超分辨深度学习单光子阵列超快曝光增强成像

张智杰 郭龑强 郭晓丽 张丽 宋铠炜 张明江

引用本文:
Citation:

基于超分辨深度学习单光子阵列超快曝光增强成像

张智杰, 郭龑强, 郭晓丽, 张丽, 宋铠炜, 张明江

Ultra-fast exposure enhanced imaging with SPAD arrays based on super-resolution deep learning

ZHANG Zhijie, GUO Yanqiang, GUO Xiaoli, ZHANG Li, SONG Kaiwei, ZHANG Mingjiang
Article Text (iFLYTEK Translation)
PDF
HTML
导出引用
  • 为突破单光子成像在极短曝光条件下成像质量与速率受限的问题, 本文提出一种融合超分辨深度学习的单光子成像增强神经网络, 实验上对单光子阵列在超快曝光时间下所获取的低信噪比图像进行增强重构, 实现了在低于0.5个光子通量条件下的高保真成像. 实验上实现了对微秒至毫秒曝光时间范围内512×512像素的单光子阵列高分辨成像, 并对μs量级短曝光时间下的目标图像进行重构. 通过构建的单光子增强深度神经网络可显著提升图像的峰值信噪比(PSNR+7.21 dB)和结构相似性指数(SSIM+0.34), 远高于传统超分辨方法(Bicubic)对低曝光图像超分辨增强重构(PSNR+0.49 dB/SSIM+0.05), 并有效补偿了超快曝光下的成像细节丢失. 并在成像曝光时间5 μs时, 成功实现了对5.19 km处无人机512×512像素的高分辨率弱光成像, 表明该方法在远距离目标成像良好的应用潜力.
    In recent years, with wide spread applications of high-sensitivity single-photon detectors, especially in the fields of quantum imaging and optical imaging, many important achievements have been made. and micro light imaging technology based on single-photon level has gradually become an important branch of high-resolution imaging systems. At present, the main single-photon detectors are single-photon counting avalanche diode (SPAD) sensors and support pixel arrays of different sizes, ranging from single-pixel detector sizes to tens of thousands of pixel SPAD arrays. The process structure of single-pixel SPAD detectors is relatively simple, and they are often used as the first choice for low-light imaging due to their high sensitivity, small size, and low cost. However, due to the lack of spatial resolution, single-pixel SPADs can only detect signals at a single location and cannot provide spatial information, and they are usually used in conjunction with a spatial light modulator DMD or SLM with spatial resolution to reconstruct two-dimensional (2D) images through compressed sensing or quantum correlation. Although single-pixel detectors can provide ns-level or even ps-level temporal resolution, they are limited by the frame rate of the spatial light modulator (SLM). For example, the fastest digital micromirror device (DMD) is a type of SLM with a frame rate of 22 kHz, which means that the imaging rate of a single-pixel camera is typically limited to seconds, and this camera typically uses an SPAD and an SLM for single-photon imaging tasks. This makes it a challenge to significantly improve the imaging speed, especially when higher imaging resolution is required, such as those exceeding hundreds of thousands of pixels. Assuming that the imaged object is a fast-moving dynamic target, a few seconds of imaging rate will inevitably lead to dynamic blurring, which also poses a challenge to the fast real-time performance of single-photon imaging systems.The SPAD array sensor retains the excellent sensitivity, low dark count rate, and high temporal resolution of single-pixel SPAD sensors. Due to the improvement of the fabrication process, multiple sensors and readout circuits are fabricated on the same chip, thus leading to the development of spatially resolved SPAD array camera. However, the integrated design of SPAD arrays with multiple pixels and circuits inevitably leads to cross-crosstalk between pixels. This crosstalk can significantly affect the accuracy of the signal. Additionally, the fill factor of such array cameras is typically low. Although the fill factor can be improved by methods such as three-dimensional (3D) stacking and microlens arrays, the spatial utilization is still to be improved in comparison with single-pixel SPADs. However, it is undeniable that SPAD arrays perform well in high dynamic range photon flux detection and high frame rate photon counting measurements due to the parallel processing of multiple detectors. Currently, commercial SPAD arrays integrate hundreds of thousands of detector pixel units, thereby providing excellent spatial resolution. Unfortunately, due to manufacturing processes and various challenges, the SPAD array cameras have been used in high-quantification bit deep sampling mode to acquire high-resolution single-photon intensity imaging. Its exposure time is limited to milliseconds. It is difficult to avoid dynamic blurring during the imaging exposure time when the motion frequency of the dynamic target reaches kHz or higher. Although the quantification bit depth can be sacrificed to shorten the minimum exposure time of array camera to the ns level, too short an exposure time can result in the SPAD array capturing the sparse photon data contaminated by a large amount of noise. Therefore, reliable photon denoising methods need to be developed. These methods are essential for effectively separating background noise from the actual signals, thereby improving the signal-to-noise ratio of the imaging system. Therefore, the real-time performance of the imaging system at the expense of quantification sampling accuracy still needs to be further optimized.In order to solve the problem of limited imaging quality and rate of SPAD arrays under very short exposure times, we propose a single-photon imaging enhanced deep neural network combined with super-resolution deep learning in this work. By constructing a single-photon image dataset with dynamic exposure times and conducting adaptive training, high-fidelity reconstruction of low signal-to-noise ratio single-photon images can be achieved under ultra-short exposure time. In the experiments, the enhanced reconstruction of low-quality fan images (PSNR/SSIM, 6.54 dB/0.18) under very low-light conditions is achieved, with an exposure time of only 1 μs and an average photon number of less than 0.5 photons (PNSR/SSIM, 13.21 dB/0.34). And the images are effectively improved by +7.21 dB/+0.16 for PSNR and SSIM. The passive remote enhanced reconstruction is performed on the drone at a distance of 5.19 km, with an imaging exposure time of 5 μs, and an effective PSNR and SSIM enhancement of +4.78 dB/+0.2. This method provides a new technical solution for SPAD arrays for achieving ultra-fast-exposure high-quality imaging.
  • 图 1  基于单光子阵列的单光子增强成像系统实验装置, 其中NDF为中性密度滤波片, DMD为数字微镜器件, HWP为半波片, PBS为偏振分束器, QWP为四分之一波片, NBF为窄带滤波片, TDC为时间数字转换器

    Fig. 1.  Experimental measurement device for single photon enhanced imaging based on SPAD array. NDF is a neutral density filter, DMD is a digital micromirror device, HWP is a half-wave plate, PBS is a polarization beam splitter, QWP is a quarter-wave plate, NBF is a narrow-band filter, and TDC is a time-to-digital converter.

    图 2  在不同采样位数条件下, 单光子阵列探测平均光子数随曝光时间变化的实验结果

    Fig. 2.  Experimental results of the average number of photons detected by SPAD array with exposure time under different sampling bits.

    图 3  单光子阵列光子数校正偏差随曝光时间的变化

    Fig. 3.  Results of photon number correction bias with exposure time in SPAD array.

    图 4  单光子阵列信噪比测试结果

    Fig. 4.  SNR test results for SPAD array.

    图 5  不同量化位深条件下, 单光子阵列实验成像的(a) PSNR和(b) SSIM随曝光时间的变化

    Fig. 5.  The PSNR (a) and SSIM (b) of SPAD array imaging change with exposure time under different sampling bit depth.

    图 6  单光子阵列超短曝光成像结果与经本文超分辨深度学习处理后的结果

    Fig. 6.  Ultra-short exposure imaging results of SPAD array and the results processed by super-resolution deep learning.

    图 7  基于SwinIR模型的单光子增强深度神经网络结构示意图

    Fig. 7.  Schematic diagram of single-photon enhanced deep neural network architecture based on SwinIR model.

    图 8  使用传统方法和3种不同深度模型的单光子增强成像重构结果

    Fig. 8.  Results of single-photon enhanced imaging reconstruction using conventional methods and three different depth models.

    图 9  传统方法和3种不同深度模型对单光子增强重构成像评价指标对比

    Fig. 9.  Comparison of evaluation indexes between traditional methods and three different depth models in single-photon enhanced reconstruction imaging.

    图 10  短曝光时间下远距离探测无人机单光子阵列增强重构成像结果

    Fig. 10.  Enhanced imaging reconstruction results of SPAD array of long range detection UAV under short exposure time.

    表 1  不同单光子阵列性能指标对比

    Table 1.  Comparison of the single-photon avalanche diode arrays.

    Types Array size Peak PDE Fill factor/% Frame rate/(frames·s–1) DCR/(counts·s–1) SNR/dB
    This work 512×512 50%@520 nm 30—40 100000@1 bit 25 65
    [28] 512×512 5.2%@520 nm 10.5 97.7000@1 bit 0.3
    [33] 180×148 0.8 53.8
    [34] 1024×1000 3.6%@520 nm 13.4 24000@1 bit 0.065
    [35] 32×32 75.4 39.7
    下载: 导出CSV

    表 2  不同超分尺度下重构成像的详细指标对比

    Table 2.  Comparison of detailed indexes of reconstructed imaging at different super-resolution scales.

    SPAD
    exposure
    time/μs
    SR scale Metric Bicubic U-net RCAN SwinIR
    1 2 PSNR 6.79 11.42 12.34 13.11
    SSIM 0.16 0.26 0.28 0.31
    4 PSNR 7.03 11.78 12.17 13.33
    SSIM 0.18 0.28 0.29 0.32
    40 2 PSNR 10.64 12.36 13.14 13.42
    SSIM 0.24 0.30 0.33 0.34
    4 PSNR 11.36 13.12 13.33 13.68
    SSIM 0.27 0.31 0.34 0.36
    200 2 PSNR 12.11 14.23 15.02 15.31
    SSIM 0.28 0.39 0.41 0.42
    4 PSNR 12.53 15.28 15.34 15.33
    SSIM 0.31 0.42 0.43 0.42
    1000 2 PSNR 13.29 16.41 16.62 16.80
    SSIM 0.32 0.44 0.46 0.48
    4 PSNR 13.58 17.16 17.54 17.66
    SSIM 0.35 0.52 0.55 0.57
    下载: 导出CSV
  • [1]

    Zia D, Dehghan N, D’Errico A, Sciarrino F, Karimi E 2023 Nat. Photonics 17 1009Google Scholar

    [2]

    Wang G Q, Madonini F, Li B N, Li C H, Xiang J G, Villa F, Cappellaro P 2023 Adv. Quantum Technol. 6 2300046Google Scholar

    [3]

    Vaidya S, Gao X, Dikshit S, Aharonovich I, Li T 2023 Adv. Phys. X 8 2206049

    [4]

    Altmann Y, McLaughlin S, Padgett M J, Goyal V K, Hero A O, Faccio D 2018 Science 361 2298Google Scholar

    [5]

    李明飞, 杨然, 霍娟, 赵连洁, 杨文良, 王俊, 张安宁 2015 物理学报 64 224208Google Scholar

    Li M F, Yang R, Huo J, Zhao L J, Yang W L, Wang J, Zhang A N 2015 Acta Phys. Sin. 64 224208Google Scholar

    [6]

    Jiang P Y, Li Z P, Xu F 2021 Opt. Lett. 46 1181Google Scholar

    [7]

    Ru S Y, Hao H, Zhao Q Y, Li Z J, Liu H, Liu Z, Deng J, Huang Y H, Yang F, Liu N T, Wan C, Tu X C, Zhang L B, Jia X Q, Chen J, Kang L, Wu P H 2024 Laser Photonics Rev. 18 2400483Google Scholar

    [8]

    李明飞, 阎璐, 杨然, 刘院省 2019 物理学报 68 064202Google Scholar

    Li M F, Yan L, Yang R, Liu Y X 2019 Acta Phys. Sin. 68 064202Google Scholar

    [9]

    李昌恒, 刘璠, 王小庆, 朱露洁, 刘雪峰 2024 光学学报 44 911002Google Scholar

    Li C H, Liu P, Wang X Q, Zhu L J, Liu X F 2024 Acta Opt. Sin. 44 911002Google Scholar

    [10]

    Shin D, Xu F, Venkatraman D, Lussana R, Villa F, Zappa F, Goyal V K, Wong F N C, Shapiro J H 2016 Nat. Commun. 7 12046Google Scholar

    [11]

    李薇薇, 张同意, 康岩, 薛瑞凯, 王晓芳, 梁锦涛, 李力飞 2024 光子学报 53 1111001

    Li W W, Zhang T Y, Kang Y, Xue R K, Wang X F, Liang J T, Li L F 2024 Acta Photonica Sin. 53 1111001

    [12]

    Wang H, Wang X Q, Gao C, Liu X, Wang Y, Zhao H, Yao Z H 2024 Opt. Laser Technol. 170 110196Google Scholar

    [13]

    Li Z P, Ye J T, Huang X, Jiang P Y, Cao Y, Hong Y, Yu C, Zhang J, Zhang Q, Peng C Z, Xu F, Pan J W 2021 Optica 8 344Google Scholar

    [14]

    Duarte M F, Davenport M A, Takhar D, Laska J N, Sun T, Kelly K F, Baraniuk R G 2008 IEEE Signal Process. Mag. 25 83Google Scholar

    [15]

    Meng H Y, Gao Y, Wang X H, Li X Y, Wang L L, Zhao X, Sun B Q 2024 Light Sci. Appl. 13 121Google Scholar

    [16]

    Al Abbas T, Dutton N A W, Almer O, Pellegrini S, Henrion Y, Henderson R K 2016 2016 IEEE International Electron Devices Meeting IEDM San Francisco CA, USA, December 3—7, 2016 pp8.1. 1-8.1. 4

    [17]

    Xu H, Pancheri L, Betta G F D, Stoppa D 2017 Opt. Express 25 12765Google Scholar

    [18]

    Tontini A, Gasparini L, Perenzoni M 2020 Sensors 20 5203Google Scholar

    [19]

    Ghioni M, Gulinatti A, Rech I, Zappa F, Cova S 2007 IEEE J. Sel. Top. Quant. Electron 13 852Google Scholar

    [20]

    Itzler M A, Ben-Michael R, Hsu C F, Slomkowski K, Tosi A, Cova S, Zappa F, Ispasoiu R 2007 J. Mod. Opt. 54 283Google Scholar

    [21]

    Liu F, Liu X F, Yao X R, Dou S C, Li H, Zhai G J 2022 Opt. Express 30 22608Google Scholar

    [22]

    Li S, Liu X Y, Xiao Y, Ma Y, Yang J, Zhu K N, Tian X 2023 Opt. Express 31 4712Google Scholar

    [23]

    Yang Y, Shi J H, Cao F, Peng J Y, Zeng G H 2015 Appl. Opt. 54 9277Google Scholar

    [24]

    Bromberg Y, Katz O, Silberberg Y 2009 Phys. Rev. A 79 053840Google Scholar

    [25]

    Zhang Y, Gibson G M, Edgar M P, Hammond G, Padgett M J 2020 Opt. Express 28 18180Google Scholar

    [26]

    Liu X Y, Ma Y, Li S, Yang J, Zhang Z Y, Tian X 2021 Opt. Express 29 37945Google Scholar

    [27]

    Sun M J, Edgar M P, Gibson G M, Sun B, Radwell N, Lamb R, Padgett M J 2016 Nat. Commun. 7 12010Google Scholar

    [28]

    李明飞, 莫小范, 赵连洁, 霍娟, 杨然, 李凯, 张安宁 2016 物理学报 65 64201Google Scholar

    Li M F, Mo X F, Zhao L J, Huo J, Yang R, Li K, Zhang A N 2016 Acta Phys. Sin. 65 64201Google Scholar

    [29]

    Laurenzis M 2019 Opt. Express 27 38391Google Scholar

    [30]

    Yao G X, Chen Y W, Jiang C, Xuan Y X, Hu X M, Liu Y, Pan Y 2022 Opt. Express 30 37323Google Scholar

    [31]

    Zarghami M, Gasparini L, Perenzoni M, Pancheri L 2019 Instruments 3 38Google Scholar

    [32]

    Ulku A C, Bruschini C, Antolović I M, Kuo Y, Ankri R, Weiss S, Michalet X, Charbon E 2019 IEEE J. Sel. Top. Quant. Electron 25 1

    [33]

    Vargas-Sierra S, Linán-Cembrano G, Rodríguez-Vázquez Á 2015 IEEE Sens. J. 15 180Google Scholar

    [34]

    Morimoto K, Ardelean A, Wu M L, Ulku A C, Antolovic I M, Bruschini C, Charbon E 2020 Optica 7 346Google Scholar

    [35]

    Yu Zou, Bronzi D, Villa F, Weyers S 2014 2014 10th Conference on Ph. D. Research in Microelectronics and Electronics (PRIME) Grenoble, France, June 30-July 03, 2014 pp1–4

  • [1] 朱顺凯, 袁方, 胡凯, 皮涛涛, 朱熙铖, 李程. 基于卷积神经网络的双站雷达散射截面减缩超表面设计. 物理学报, doi: 10.7498/aps.74.20250109
    [2] 王帅, 李希, 姚星灿. 基于增强条纹去除的超冷原子成像. 物理学报, doi: 10.7498/aps.73.20240570
    [3] 黄宇航, 陈理想. 基于未训练神经网络的分数傅里叶变换成像. 物理学报, doi: 10.7498/aps.73.20240050
    [4] 施岳, 欧攀, 郑明, 邰含旭, 王玉红, 段若楠, 吴坚. 基于轻量残差复合增强收敛神经网络的粒子场计算层析成像伪影噪声抑制. 物理学报, doi: 10.7498/aps.73.20231902
    [5] 刘金品, 王秉中, 陈传升, 王任. 基于深度物理启发神经网络的微波波导器件逆设计方法. 物理学报, doi: 10.7498/aps.72.20230031
    [6] 付亚鹏, 孙乾东, 李博艺, 他得安, 许凯亮. 基于RCA阵列三维超快超声血流成像方法仿真研究. 物理学报, doi: 10.7498/aps.72.20222106
    [7] 娄月申, 郭文军. 贝叶斯深度神经网络对于核质量预测的研究. 物理学报, doi: 10.7498/aps.71.20212387
    [8] 黄颖, 顾长贵, 杨会杰. 神经网络超参数优化的删除垃圾神经元策略. 物理学报, doi: 10.7498/aps.71.20220436
    [9] 朱琦, 许多, 张元军, 李玉娟, 王文, 张海燕. 基于卷积神经网络的白蚀缺陷超声探测. 物理学报, doi: 10.7498/aps.71.20221504
    [10] 张航, 胡月姣, 陈嘉文, 修龙汪. 程能映射下配光平移群的深度神经网络实现. 物理学报, doi: 10.7498/aps.71.20220178
    [11] 向鹏程, 蔡聪波, 王杰超, 蔡淑惠, 陈忠. 基于深度神经网络的时空编码磁共振成像超分辨率重建方法. 物理学报, doi: 10.7498/aps.71.20211754
    [12] 臧佳琦, 许凯亮, 韩清见, 陆起涌, 梅永丰, 他得安. 无造影剂增强的超快超声脊髓微血管成像方法. 物理学报, doi: 10.7498/aps.70.20201878
    [13] 徐启伟, 王佩佩, 曾镇佳, 黄泽斌, 周新星, 刘俊敏, 李瑛, 陈书青, 范滇元. 基于深度卷积神经网络的大气湍流相位提取. 物理学报, doi: 10.7498/aps.69.20190982
    [14] 郭永峰, 谭建国. 一类非线性神经网络系统的超阈值随机共振现象. 物理学报, doi: 10.7498/aps.61.170502
    [15] 李海燕, 胡云安, 任建存, 朱敏, 刘亮. 非匹配不确定交叉严反馈超混沌系统神经网络反演同步. 物理学报, doi: 10.7498/aps.61.140502
    [16] 冯朝文, 蔡 理, 李 芹. 基于单电子器件的细胞神经网络实现及应用研究. 物理学报, doi: 10.7498/aps.57.2462
    [17] 董继扬, 张军英, 陈 忠. 自动波竞争神经网络及其在单源最短路问题中的应用. 物理学报, doi: 10.7498/aps.56.5013
    [18] 周小荣, 罗晓曙, 蒋品群, 袁五届. 小世界神经网络的二次超谐波随机共振. 物理学报, doi: 10.7498/aps.56.5679
    [19] 吴忠强, 谭拂晓, 王绍仙. 基于无源化的细胞神经网络超混沌系统同步. 物理学报, doi: 10.7498/aps.55.1651
    [20] 蔡 理, 马西奎, 王 森. 量子细胞神经网络的超混沌特性研究. 物理学报, doi: 10.7498/aps.52.3002
计量
  • 文章访问数:  331
  • PDF下载量:  9
  • 被引次数: 0
出版历程
  • 收稿日期:  2025-04-03
  • 修回日期:  2025-05-12
  • 上网日期:  2025-06-11

/

返回文章
返回