-
Synchrotron radiation-based X-ray nano-imaging is a powerful tool for non-destructively studying the internal nano-scale structure of matter. Here in this paper, we review the state-of-the-art image alignment technology in the field of nano-resolution imaging, and classify and analyze the technology according to the research stage. First, through the publications of image alignment algorithm, the development direction of future research is analyzed. Then, the most effective image alignment application in the field of nano imaging based on classic image alignment algorithms is summarized. The paper also presents the feature detection operators that are useful for nano-scale image registration selected from recent feature detection research, which has important guiding significance for the specific application and optimization of nano-imaging image registration. Finally, the state-of-the-art image registration method based on deep learning is introduced, the applicability and potential of deep learning in nano-imaging registration technology are discussed, and future research directions and challenges are prospected based on different neural network characteristics.
-
Keywords:
- image alignment /
- image registration /
- nano-imaging /
- deep learning
[1] 袁清习, 邓彪, 关勇, 张凯, 刘宜晋 2019 物理 48 205
Google Scholar
Yuan Q X, Deng B, Guan Y, Zhang K, Liu Y J 2019 Physics 48 205
Google Scholar
[2] Xie H L, Deng B, Du G H, Fu Y N, Guo H, Xue Y L, Peng G Y, Tao F, Zhang L, Xiao T Q 2020 Nucl. Sci. Tech. 31 102
Google Scholar
[3] Jun K, Yoon S 2017 Sci. Rep. 7 41218
Google Scholar
[4] [5] 程甲一, 张祥志, 邰仁忠 2013 核技术 36 1
Cheng J Y, Zhang X Z, Tai R Z 2013 Nucl. Tech. 36 1
[6] Viergever M A, Maintz J B A, Klein S, Murphy K, Staring M, Pluim J P W 2016 Med. Image Anal. 33 140
Google Scholar
[7] Maintz J A, Viergever M A 1998 Med. Image Anal. 2 1
Google Scholar
[8] Leese J A, Novak C S, Clark B B 1922 J. Appl. Meteorol. 10 118
[9] Barnea D I, Silverman H F 2009 IEEE Transact. Comput. C-21 179
[10] Pratt W K 1974 IEEE Trans. Aes. 10 353
[11] Guckenberger R 1982 Ultramicroscopy 9 167
Google Scholar
[12] Lewis J P 1995 Circuits, Syst. Sig. Process. 82 144
[13] Yoo J C, Han T H 2009 Circuits, Syst. Sig. Process. 28 819
Google Scholar
[14] Viola P, Wells W M 1995 Fifth International Conference on Computer Vision Cambridge, MA, USA, 20−23 June 1995 p16
[15] 李巧, 周光照, 肖体乔 2016 光学学报 36 108
Li Q, Zhou G Z, Xiao T Q 2016 Acta Optica Sin. 36 108
[16] Holland, John H 1973 SIAM J. Comput. 2 88
Google Scholar
[17] Kennedy J, Eberhart R 1995 IEEE ICNN'95 - International Conference on Neural Networks Perth, WA, Australia, 27 Nov.−1 Dec. 1995 p1942
[18] Dorigo M, Maniezzo V, Colorni A 1996 IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics: A Publication of the IEEE Systems, Man, and Cybernetics Society 26 29
Google Scholar
[19] Keller Y, Averbuch A 2007 Signal Processing 87 124
Google Scholar
[20] Liu Y, Meirer F, Williams P A, Wang J, Pianetta P 2012 J. Synchrotron Radiat. 19 281
Google Scholar
[21] Moravec H P 1977 Proceedings of the 5 th international joint conference on Artificial intelligence Cambridge, MA, USA, August 22−25, 1977 p584
[22] Harris C, Stephens M 1988 Proceedings 4 th Alvey Vision Conference Manchester, UK, 31 August−2 September 1988 p147
[23] Lowe D G 1999 Proceedings of the Seventh IEEE International Conference on Computer Vision Kerkyra, Greece 20−27 Sept. 1999 p1150
[24] Yan K, Sukthankar R 2004 Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, Washington, DC, USA 27 June-2 July 2004 pII-506
[25] Bay H 2006 Comput. Vis. Ima. Und. 110 404
[26] Abdel-Hakim A, Farag A 2006 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition New York, NY, USA, June 17−22, 2006 p1978
[27] Rosten E, Drummond T 2006 9th European Conference on Computer Vision–ECCV 2006 Graz, Austria, May 7−13, 2006 p430
[28] Calonder M, Lepetit V, Strecha C, Fua P 2010 11th European Conference on Computer Vision–ECCV 2010 Heraklion, Crete, Greece, September 5−11, 2010 p778
[29] Rublee E, Rabaud V, Konolige K, Bradski G 2011 2011 International Conference on Computer Vision (ICCV) Barcelona, Spain, 6−13 November, 2011 p2564
[30] Leutenegger S, Chli M, Siegwart R Y 2011 2011 International Conference on Computer Vision(ICCV) Barcelona, Spain, 6−13 November, 2011 p2548
[31] Alahi A, Ortiz R, Vandergheynst P 2012 2012 IEEE Conference on Computer Vision and Pattern Recognition Providence, RI, USA, June 16−21, 2012 p510
[32] Ulupinar F, Medioni G 1990 Comput. Vis. Graph. Ima. Process. 51 275
Google Scholar
[33] Chaple G N, Daruwala R D, Gofane M S 2015 2015 International Conference on Technologies for Sustainable Development (ICTSD) Mumbai, India, Feb. 4−6, 2015 p1
[34] Guizar-Sicairos M, Thurman S T, Fienup J R 2008 Opt. Lett. 33 156
Google Scholar
[35] Gürsoy D, Hong Y P, He K, Hujsak K, Yoo S, Chen S, Li Y, Ge M, Miller L M, Chu Y S, De Andrade V, He K, Cossairt O, Katsaggelos A K, Jacobsen C 2017 Sci. Rep. 7 11818
Google Scholar
[36] Odstrcil M, Holler M, Raabe J, Guizar-Sicairos M 2019 Opt. Express 27 36637
Google Scholar
[37] Yu H, Xia S, Wei C, Mao Y, Larsson D, Xiao X, Pianetta P, Yu Y S, Liu Y 2018 J. Synchrotron. Radiat. 25 1819
Google Scholar
[38] Wang C-C 2020 Sci. Rep. 10 7330
Google Scholar
[39] Wang S, Liu J, Li Y, Chen J, Guan Y, Zhu L 2019 J. Synchrotron. Radiat. 26 1808
Google Scholar
[40] He K, Zhang X, Ren S, Sun J 2016 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Las Vegas, NV, USA, June 27−30, 2016 p770
[41] Sokooti H, de Vos B, Berendsen F, Lelieveldt B P F, Išgum I, Staring M 2017 Medical Image Computing and Computer Assisted Intervention–MICCAI 2017 Quebec City, QC, Canada, September 11−13, 2017 p232
[42] Miao S, Wang Z J, Zheng Y, Liao R 2016 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) Prague, Czech Republic, April 13−16, 2016 p1430
[43] Fu Y, Lei Y, Wang T, Curran W J, Liu T, Yang X 2020 Phys. Med. Biol. 65 20TR01
Google Scholar
[44] Krizhevsky A, Sutskever I, Hinton G 2017 Commun. ACM 60 84
Google Scholar
[45] Cao X, Yang J, Zhang J, Nie D, Kim M, Wang Q, Shen D 2017 Medical Image Computing and Computer Assisted Intervention–MICCAI 2017 Quebec City, QC, Canada, September 11−13, 2017 p300
[46] 曹晓欢 2018 博士论文 (西安: 西北工业大学)
Cao X H 2018 Ph.D. Dissertation (Xian: Northwestern Polytechnical University) (in Chinese)
[47] Krebs J, Mansi T, Delingette H, Zhang L, Ghesu F C, Miao S, Maier A K, Ayache N, Liao R, Kamen A 2017 Medical Image Computing and Computer Assisted Intervention–MICCAI 2017 Quebec City, QC, September 11−13, 2017 p344
[48] Rohé M M, Datar M, Heimann T, Sermesant M, Pennec X 2017 Medical Image Computing and Computer Assisted Intervention – MICCAI 2017 Quebec City, QC, Canada, September 11−13, 2017 p266
[49] Wu G, Kim M, Wang Q, Munsell B C, Shen D 2016 IEEE Transact. Biomed. Engineer. 63 1505
Google Scholar
[50] Fang Q, Gu X, Yan J, Zhao J, Li Q 2019 2019 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC) Manchester, UK, 26 Oct.−2 Nov. 2019 p1
[51] Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y 2014 Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 Montreal, Canada, December 8−13, 2014 p2672
[52] 张程程 2020 硕士论文 (太原: 中北大学)
Zhang C C 2020 M.S. Dissertation (Taiyuan: North University of China) (in Chinese)
[53] Mahapatra D, Antony B, Sedai S, Garnavi R 2018 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) Washington, DC, USA, April 4-7, 2018 p1449
[54] Toriya H, Dewan A, Kitahara I 2019 IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium Yokohama, Japan, 28 July−2 Aug. 2019 p923
[55] Zhu J, Park T, Isola P, Efros A A 2017 2017 IEEE International Conference on Computer Vision (ICCV) Venice, Italy, Oct. 22−29, 2017 p2242
-
图 2 在线机械校准 (a)激光干涉仪[4]; (b)激光干涉仪记录旋转目标偏移轨迹[4]; (c) 上海光源软X光谱学显微实验站样品台[5]
Figure 2. Online mechanical calibration: (a) Laser interferometer[4]; (b) laser interferometer records the deviation trajectory of the rotating target[4]; (c) SSRF soft X spectroscopy microscope experimental station sample stage[5].
图 3 图像配准领域文章发表情况 (a) 近30年图像配准论文发表情况; (b) 近10年深度学习图像配准论文增长趋势
Figure 3. Publication of papers in the field of image registration: (a) Publication of research papers on image registration in the past 30 years; (b) percentage of papers on deep learning image registration in the past 10 years.
图 4 图像配准算法分类概述导图
Figure 4. Classification overview map of image registration algorithms.
图 5 图像配准的常规流程图
Figure 5. General flow chart of image registration.
图 6 基于傅里叶变换的图像配准流程图
Figure 6. Flow chart of image registration based on Fourier transform.
图 9 通过MR-PMA方法配准的FBP法重建质量[36] (Dataset 1为标准数据集, Dataset2为欠采样并包含161个噪点的数据集、Dataset 3在Dataset 2基础上添加了10%的高斯噪声. 分别对每个数据集进行32—1的降采样法配准, 最后一列的插图显示了全分辨率数据集的重建质量)
Figure 9. FBP reconstruction quality after alignment by our MR-PMA method[36] (Columns show reconstruction at different downsampling levels from 32 up to 1, and rows correspond to different synthetic datasets. Insets in the last column show detail of the reconstruction quality for the full resolution dataset).
图 12 配准后页岩投影重建切片对比: (a) 原始数据, (b) 手动配准, (c) 自动配准; 重投影后页岩投影数据对比: (d) 实验数据, (e) 手动重投影, (f) 自动重投影[37]
Figure 12. Reconstructed slices through the center of the shale sample without alignment (a) and with manual (b) or automatic (c) alignment. Panel (d) is the experimentally measured projection image. Panels (e) and (f) are the numerically reprojected images, calculated from the manual and auto-aligned 3D matrixes, respectively[37].
图 13 Ji-Faproma算法不同噪声水平下的配准精度对比[38] (a)投影内旋转误差的平均根方差; (b)投影垂直误差和水平误差的平均根方差; (c)配准后面对不同噪声时的重建质量
Figure 13. Performance of the JI-Faproma evaluated by comparing the alignment accuracy under different noise levels[38]: (a) Root-mean-square error of the in-plane rotational error correction; (b) root-mean-square errors of the vertical and horizontal error corrections; (c) reconstruction quality after JI-Faproma alignment for test phantoms containing different levels of noise.
图 16 采用不同矫正方法的重构结果[39] (图中: a列—d列分别是原始数据、手动对齐、重投影法、GMs法的切片; 第1—第3行是从x–z平面重建的切片, 在y方向上均匀间隔了50个像素; 第4行是x–z平面中的重构切片)
Figure 16. Sinograms (top row) and reconstructed slices of a chlorella cell using different methods[39] (Row 2: reconstructed slices of row 1, displayed in the x–y plane. Row 3: reconstructed slices in the x–z plane. Columns from (a) to (d) are results of TXM without jitter correction, corrected by a re-projection-based method, by manual alignment and by the proposed GM method).
图 17 深度学习研究论文发表情况统计 (a) 2014年至2020年多种深度学习模式图像配准论文发表数量统计; (b) 2014年至2020年多种深度学习模式的图像配准文章百分比图; (c) 2014年至2020年无监督与监督深度学习图像配论文发表百分比图
Figure 17. A survey of the publication of deep learning research papers: (a) Number of published image registration papers based on multiple deep learning methods since 2014; (b) percentage of image registration papers based on multiple deep learning methods since 2014; (c) The percentage of published image registration papers based on unsupervised and supervised deep learning since 2014.
图 19 基于GAN研究的论文发表情况
Figure 19. Number of papers published related to GAN research.
-
[1] 袁清习, 邓彪, 关勇, 张凯, 刘宜晋 2019 物理 48 205
Google Scholar
Yuan Q X, Deng B, Guan Y, Zhang K, Liu Y J 2019 Physics 48 205
Google Scholar
[2] Xie H L, Deng B, Du G H, Fu Y N, Guo H, Xue Y L, Peng G Y, Tao F, Zhang L, Xiao T Q 2020 Nucl. Sci. Tech. 31 102
Google Scholar
[3] Jun K, Yoon S 2017 Sci. Rep. 7 41218
Google Scholar
[4] [5] 程甲一, 张祥志, 邰仁忠 2013 核技术 36 1
Cheng J Y, Zhang X Z, Tai R Z 2013 Nucl. Tech. 36 1
[6] Viergever M A, Maintz J B A, Klein S, Murphy K, Staring M, Pluim J P W 2016 Med. Image Anal. 33 140
Google Scholar
[7] Maintz J A, Viergever M A 1998 Med. Image Anal. 2 1
Google Scholar
[8] Leese J A, Novak C S, Clark B B 1922 J. Appl. Meteorol. 10 118
[9] Barnea D I, Silverman H F 2009 IEEE Transact. Comput. C-21 179
[10] Pratt W K 1974 IEEE Trans. Aes. 10 353
[11] Guckenberger R 1982 Ultramicroscopy 9 167
Google Scholar
[12] Lewis J P 1995 Circuits, Syst. Sig. Process. 82 144
[13] Yoo J C, Han T H 2009 Circuits, Syst. Sig. Process. 28 819
Google Scholar
[14] Viola P, Wells W M 1995 Fifth International Conference on Computer Vision Cambridge, MA, USA, 20−23 June 1995 p16
[15] 李巧, 周光照, 肖体乔 2016 光学学报 36 108
Li Q, Zhou G Z, Xiao T Q 2016 Acta Optica Sin. 36 108
[16] Holland, John H 1973 SIAM J. Comput. 2 88
Google Scholar
[17] Kennedy J, Eberhart R 1995 IEEE ICNN'95 - International Conference on Neural Networks Perth, WA, Australia, 27 Nov.−1 Dec. 1995 p1942
[18] Dorigo M, Maniezzo V, Colorni A 1996 IEEE transactions on systems, man, and cybernetics. Part B, Cybernetics: A Publication of the IEEE Systems, Man, and Cybernetics Society 26 29
Google Scholar
[19] Keller Y, Averbuch A 2007 Signal Processing 87 124
Google Scholar
[20] Liu Y, Meirer F, Williams P A, Wang J, Pianetta P 2012 J. Synchrotron Radiat. 19 281
Google Scholar
[21] Moravec H P 1977 Proceedings of the 5 th international joint conference on Artificial intelligence Cambridge, MA, USA, August 22−25, 1977 p584
[22] Harris C, Stephens M 1988 Proceedings 4 th Alvey Vision Conference Manchester, UK, 31 August−2 September 1988 p147
[23] Lowe D G 1999 Proceedings of the Seventh IEEE International Conference on Computer Vision Kerkyra, Greece 20−27 Sept. 1999 p1150
[24] Yan K, Sukthankar R 2004 Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, Washington, DC, USA 27 June-2 July 2004 pII-506
[25] Bay H 2006 Comput. Vis. Ima. Und. 110 404
[26] Abdel-Hakim A, Farag A 2006 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition New York, NY, USA, June 17−22, 2006 p1978
[27] Rosten E, Drummond T 2006 9th European Conference on Computer Vision–ECCV 2006 Graz, Austria, May 7−13, 2006 p430
[28] Calonder M, Lepetit V, Strecha C, Fua P 2010 11th European Conference on Computer Vision–ECCV 2010 Heraklion, Crete, Greece, September 5−11, 2010 p778
[29] Rublee E, Rabaud V, Konolige K, Bradski G 2011 2011 International Conference on Computer Vision (ICCV) Barcelona, Spain, 6−13 November, 2011 p2564
[30] Leutenegger S, Chli M, Siegwart R Y 2011 2011 International Conference on Computer Vision(ICCV) Barcelona, Spain, 6−13 November, 2011 p2548
[31] Alahi A, Ortiz R, Vandergheynst P 2012 2012 IEEE Conference on Computer Vision and Pattern Recognition Providence, RI, USA, June 16−21, 2012 p510
[32] Ulupinar F, Medioni G 1990 Comput. Vis. Graph. Ima. Process. 51 275
Google Scholar
[33] Chaple G N, Daruwala R D, Gofane M S 2015 2015 International Conference on Technologies for Sustainable Development (ICTSD) Mumbai, India, Feb. 4−6, 2015 p1
[34] Guizar-Sicairos M, Thurman S T, Fienup J R 2008 Opt. Lett. 33 156
Google Scholar
[35] Gürsoy D, Hong Y P, He K, Hujsak K, Yoo S, Chen S, Li Y, Ge M, Miller L M, Chu Y S, De Andrade V, He K, Cossairt O, Katsaggelos A K, Jacobsen C 2017 Sci. Rep. 7 11818
Google Scholar
[36] Odstrcil M, Holler M, Raabe J, Guizar-Sicairos M 2019 Opt. Express 27 36637
Google Scholar
[37] Yu H, Xia S, Wei C, Mao Y, Larsson D, Xiao X, Pianetta P, Yu Y S, Liu Y 2018 J. Synchrotron. Radiat. 25 1819
Google Scholar
[38] Wang C-C 2020 Sci. Rep. 10 7330
Google Scholar
[39] Wang S, Liu J, Li Y, Chen J, Guan Y, Zhu L 2019 J. Synchrotron. Radiat. 26 1808
Google Scholar
[40] He K, Zhang X, Ren S, Sun J 2016 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Las Vegas, NV, USA, June 27−30, 2016 p770
[41] Sokooti H, de Vos B, Berendsen F, Lelieveldt B P F, Išgum I, Staring M 2017 Medical Image Computing and Computer Assisted Intervention–MICCAI 2017 Quebec City, QC, Canada, September 11−13, 2017 p232
[42] Miao S, Wang Z J, Zheng Y, Liao R 2016 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI) Prague, Czech Republic, April 13−16, 2016 p1430
[43] Fu Y, Lei Y, Wang T, Curran W J, Liu T, Yang X 2020 Phys. Med. Biol. 65 20TR01
Google Scholar
[44] Krizhevsky A, Sutskever I, Hinton G 2017 Commun. ACM 60 84
Google Scholar
[45] Cao X, Yang J, Zhang J, Nie D, Kim M, Wang Q, Shen D 2017 Medical Image Computing and Computer Assisted Intervention–MICCAI 2017 Quebec City, QC, Canada, September 11−13, 2017 p300
[46] 曹晓欢 2018 博士论文 (西安: 西北工业大学)
Cao X H 2018 Ph.D. Dissertation (Xian: Northwestern Polytechnical University) (in Chinese)
[47] Krebs J, Mansi T, Delingette H, Zhang L, Ghesu F C, Miao S, Maier A K, Ayache N, Liao R, Kamen A 2017 Medical Image Computing and Computer Assisted Intervention–MICCAI 2017 Quebec City, QC, September 11−13, 2017 p344
[48] Rohé M M, Datar M, Heimann T, Sermesant M, Pennec X 2017 Medical Image Computing and Computer Assisted Intervention – MICCAI 2017 Quebec City, QC, Canada, September 11−13, 2017 p266
[49] Wu G, Kim M, Wang Q, Munsell B C, Shen D 2016 IEEE Transact. Biomed. Engineer. 63 1505
Google Scholar
[50] Fang Q, Gu X, Yan J, Zhao J, Li Q 2019 2019 IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC) Manchester, UK, 26 Oct.−2 Nov. 2019 p1
[51] Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y 2014 Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 Montreal, Canada, December 8−13, 2014 p2672
[52] 张程程 2020 硕士论文 (太原: 中北大学)
Zhang C C 2020 M.S. Dissertation (Taiyuan: North University of China) (in Chinese)
[53] Mahapatra D, Antony B, Sedai S, Garnavi R 2018 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018) Washington, DC, USA, April 4-7, 2018 p1449
[54] Toriya H, Dewan A, Kitahara I 2019 IGARSS 2019 - 2019 IEEE International Geoscience and Remote Sensing Symposium Yokohama, Japan, 28 July−2 Aug. 2019 p923
[55] Zhu J, Park T, Isola P, Efros A A 2017 2017 IEEE International Conference on Computer Vision (ICCV) Venice, Italy, Oct. 22−29, 2017 p2242
DownLoad:
Catalog
Metrics
- Abstract views: 7882
- PDF Downloads: 214
- Cited By: 0
