-
The detection of intensity peaks, which correspond to atom positions, in high-resolution (scanning) transmission electron microscopy images is of great practical significance. By quantitatively determining the locations of these peaks, it is possible to obtain important information such as the structural deformation and the electric dipole distribution inside a material on the nanoscale. The detection of the peak positions in image processing can be regarded as a target detection problem, for which breakthroughs have been made with deep-learning neural networks. Comparing to the traditional target detection algorithms, which are based on specifically designed feature extractor and classifier, the deep-learning approach can obtain the features at multiple levels of abstraction automatically, thus improving the robustness of the detection process. In this paper, we realize the automatic detection of the intensity peaks in high-resolution electron microscopy images by building a high-quality atomic image sample set and using the YOLOv3 target detection framework. With its accuracy and speed, which are superior over other target detection neural networks, the YOLOv3 is suitable for image processing as the number of images increases explosively. The YOLOv3 network converges well in the training process using our atomic image sample set, with the loss function reaching a minimum after 500 epoches; the trained neural network can detect almost all the major atoms in the images, showing its excellent ability. With the aid of YOLOv3, we also develop a program to organize the detected atoms, enabling the detection of all the other atoms within each unit cell. It is found that, combining YOLOv3 with the newly developed algorithm, almost all the atoms can be successfully determined, showing its advantages over previous algorithms based on bravis lattice construction, especially for high-resolution transmission electron microscopy images with lattice defects. Our results show that this image processing technique has the potential in overcoming the bottleneck in the fast processing of high resolution electron microscopy images.
[1] Urban K W 2008 Science 321 506
Google Scholar
[2] Falke U, Bleloch A, Falke M, Teichert S 2004 Phys. Rev. Lett. 92 116103
Google Scholar
[3] Nellist P D, Chisholm M F, Dellby N, Krivanek O L, Murfitt M F, Szilagyi Z S, Lupini A R, Borisevich A, Sides W H, Pennycook S J 2004 Science 305 1741
Google Scholar
[4] Jia C L, Urban K 2004 Science 303 2001
Google Scholar
[5] Mkhoyan K A, Batson P E, Cha J, Schaff W J, Silcox J 2006 Science 312 1354
Google Scholar
[6] Varela M, Findlay S D, Lupini A R, Christen H M, Borisevich A Y, Dellby N, Krivanek O L, Nellist P D, Oxley M P, Allen L J, Pennycook S J 2004 Phys. Rev. Lett. 92 095502
Google Scholar
[7] Bosman M, Keast V J, Garcia-Munoz J L, D'Alfonso A J, Findlay S D, Allen L J 2007 Phys. Rev. Lett. 99 086102
Google Scholar
[8] Muller D A, Kourkoutis L F, Murfitt M, Song J H, Hwang H Y, Silcox J, Dellby N, Krivanek O L 2008 Science 319 1073
Google Scholar
[9] Scott J F 2007 Science 315 954
Google Scholar
[10] Nakagawa N, Hwang H Y, Muller D A 2006 Nat. Mater. 5 204
Google Scholar
[11] Reyren N, Thiel S, Caviglia A D, Kourkoutis L F, Hammerl G, Richter C, Schneider C W, Kopp T, Ruetschi A S, Jaccard D, Gabay M, Muller D A, Triscone J M, Mannhart J 2007 Science 317 1196
Google Scholar
[12] Jia C L, Urban K W, Alexe M, Hesse D, Vrejoiu I 2011 Science 331 1420
Google Scholar
[13] Lu L, Nahas Y, Liu M, Du H, Jiang Z, Ren S, Wang D, Jin L, Prokhorenko S, Jia C L, Bellaiche L 2018 Phys. Rev. Lett. 120 177601
Google Scholar
[14] Tang Y L, Zhu Y L, Ma X L, Borisevich A Y, Morozovska A N, Eliseev E A, Wang W Y, Wang Y J, Xu Y B, Zhang Z D, Pennycook S J 2015 Science 348 547
Google Scholar
[15] Nelson C T, Winchester B, Zhang Y, Kim S J, Melville A, Adamo C, Folkman C M, Baek S H, Eom C B, Schlom D G, Chen L Q, Pan X Q 2011 Nano Lett. 11 828
Google Scholar
[16] Catalan G, Lubk A, Vlooswijk A H G, Snoeck E, Magen C, Janssens A, Rispens G, Rijnders G, Blank D H A, Noheda B 2011 Nat. Mater. 10 963
Google Scholar
[17] Li S, Wang Y J, Zhu Y L, Tang Y L, Liu Y, Ma J Y, Han M J, Wu B, Ma X L 2019 Acta Mater. 171 176
Google Scholar
[18] Gao P, Kumamoto A, Ishikawa R, Lugg N, Shibata N, Ikuhara Y 2018 Ultramicroscopy 184 177
Google Scholar
[19] Sun Y, Abid A Y, Tan C, Ren C, Li M, Li N, Chen P, Li Y, Zhang J, Zhong X, Wang J, Liao M, Liu K, Bai X, Zhou Y, Yu D, Gao P 2019 Sci. Adv. 5 eaav4355
Google Scholar
[20] Chen P, Zhong X, Zorn J A, Li M, Sun Y, Abid A Y, Ren C, Li Y, Li X, Ma X, Wang J, Liu K, Xu Z, Tan C, Chen L, Gao P, Bai X 2020 Nat. Commun. 11 1840
Google Scholar
[21] Du H, Jia C J, Mayer J 2016 Chem. Mater. 28 650
Google Scholar
[22] Galindo P L, Kret S, Sanchez A M, Laval J Y, Yanez A, Pizarro J, Guerrero E, Ben T, Molina S I 2007 Ultramicroscopy 107 1186
Google Scholar
[23] 南虎, 卢江波, 刘明, 井红梅, 汤少杰, 王大威, 贾春林 2016 电子显微学报 035 191
Google Scholar
Nan H, Lu J B, Liu M, Jing H M, Tang S J, Wang D W, Jia C L 2016 J. Chin. Electron. Microsc. Soc. 035 191
Google Scholar
[24] Gong Y J, Liu Z, Lupini A R, Shi G, Lin J H, Najmaei S, Lin Z, Elias A L, Berkdemir A, You G, Terrones H, Terrones M, Vajtai R, Pantelides S T, Pennycook S J, Lou J, Zhou W, Ajayan P M 2014 Nano Lett. 14 442
Google Scholar
[25] Taigman Y, Yang M, Ranzato M, Wolf L 2014 Proc CVPR IEEE 1701
Google Scholar
[26] LeCun Y, Bengio Y, Hinton G 2015 Nature 521 436
Google Scholar
[27] Redmon J, Farhadi A 2018 arXiv: 1804.02767 [cs.CV]
[28] Tzutalin. LabelImg. Git code (2015). https://github.com/tzutalin/labelImglabelImg [2020-12-3]
-
图 1 样本图像标记示例 (a), (b), (d)包含大尺寸原子峰, 做到了100%标记; (c)包含小尺寸原子峰, 做到90%左右的标记.
Figure 1. Sample images with the intensity peaks labeled. The images (a), (b) and (d) contain a relatively small number of atom peaks, which are all labeled. On the other hand, (c) has too many atom peaks, which are partially labeled (about 90%).
图 2 最大训练批次为3000时网络的训练情况 (a)网络训练过程的误差函数, 在500批次前后收敛, 直至训练结束一直保持稳定; (b)网络训练过程的交并比数值, 在500批次前快速上升, 之后一直缓慢上升, 没有下降趋势
Figure 2. The YOLOv3 training process with an epoch number of 3000: (a) The loss function converges after 500 epochs, which remains stable until the end of the training; (b) the IOU value increases quickly during the first 500 epochs, and then slowly increases until the end of the training.
图 3 设置置信度为0.25时YOLOv3网络检测结果
Figure 3. The detection results obtained with a confidence threshold of 0.25.
图 4 设置置信度为0.1的YOLOv3网络检测结果, 各图像的检测时间如下: (a) 7.40 s; (b) 7.38 s; (c) 7.2 s; (d) 7.24 s; (e) 7.28 s; (f) 7.25 s; (g) 7.24 s; (h) 7.24 s; (i) 7.22 s
Figure 4. The detection results obtained with a confidence threshold of 0.1. The detection times for these images are: (a) 7.40 s; (b) 7.38 s; (c) 7.2 s; (d) 7.24 s; (e) 7.28 s; (f) 7.25 s; (g) 7.24 s; (h) 7.24 s; (i) 7.22 s.
图 5 (a), (b) 1024 × 1024尺寸图像检测效果; (c) 512 × 512尺寸晶格缺陷图像检测效果, 图(a)—图(c)检测时间均在7.4 s左右; (d), (e), (f)分别是(a), (b), (c)中原子目标检测框内二维高斯拟合结果、单胞构建效果以及B位原子峰的拟合结果, 计算时间分别为27.32 s, 10.51 s, 20.52 s
Figure 5. (a), (b)The detection result of a 1024 × 1024 image; (c) the detection result of a 512 × 512 lattice distortion image. The detection time for each of them is about 7.4 s; (d), (e), (f) two-dimensional gauss fitting result within each detected window in (a), (b) and (c), constructed frames of unit cells, and fitting result of corresponding B site atom peaks. Computing times are 27.32 s, 10.51 s, 20.52 s respectively.
-
[1] Urban K W 2008 Science 321 506
Google Scholar
[2] Falke U, Bleloch A, Falke M, Teichert S 2004 Phys. Rev. Lett. 92 116103
Google Scholar
[3] Nellist P D, Chisholm M F, Dellby N, Krivanek O L, Murfitt M F, Szilagyi Z S, Lupini A R, Borisevich A, Sides W H, Pennycook S J 2004 Science 305 1741
Google Scholar
[4] Jia C L, Urban K 2004 Science 303 2001
Google Scholar
[5] Mkhoyan K A, Batson P E, Cha J, Schaff W J, Silcox J 2006 Science 312 1354
Google Scholar
[6] Varela M, Findlay S D, Lupini A R, Christen H M, Borisevich A Y, Dellby N, Krivanek O L, Nellist P D, Oxley M P, Allen L J, Pennycook S J 2004 Phys. Rev. Lett. 92 095502
Google Scholar
[7] Bosman M, Keast V J, Garcia-Munoz J L, D'Alfonso A J, Findlay S D, Allen L J 2007 Phys. Rev. Lett. 99 086102
Google Scholar
[8] Muller D A, Kourkoutis L F, Murfitt M, Song J H, Hwang H Y, Silcox J, Dellby N, Krivanek O L 2008 Science 319 1073
Google Scholar
[9] Scott J F 2007 Science 315 954
Google Scholar
[10] Nakagawa N, Hwang H Y, Muller D A 2006 Nat. Mater. 5 204
Google Scholar
[11] Reyren N, Thiel S, Caviglia A D, Kourkoutis L F, Hammerl G, Richter C, Schneider C W, Kopp T, Ruetschi A S, Jaccard D, Gabay M, Muller D A, Triscone J M, Mannhart J 2007 Science 317 1196
Google Scholar
[12] Jia C L, Urban K W, Alexe M, Hesse D, Vrejoiu I 2011 Science 331 1420
Google Scholar
[13] Lu L, Nahas Y, Liu M, Du H, Jiang Z, Ren S, Wang D, Jin L, Prokhorenko S, Jia C L, Bellaiche L 2018 Phys. Rev. Lett. 120 177601
Google Scholar
[14] Tang Y L, Zhu Y L, Ma X L, Borisevich A Y, Morozovska A N, Eliseev E A, Wang W Y, Wang Y J, Xu Y B, Zhang Z D, Pennycook S J 2015 Science 348 547
Google Scholar
[15] Nelson C T, Winchester B, Zhang Y, Kim S J, Melville A, Adamo C, Folkman C M, Baek S H, Eom C B, Schlom D G, Chen L Q, Pan X Q 2011 Nano Lett. 11 828
Google Scholar
[16] Catalan G, Lubk A, Vlooswijk A H G, Snoeck E, Magen C, Janssens A, Rispens G, Rijnders G, Blank D H A, Noheda B 2011 Nat. Mater. 10 963
Google Scholar
[17] Li S, Wang Y J, Zhu Y L, Tang Y L, Liu Y, Ma J Y, Han M J, Wu B, Ma X L 2019 Acta Mater. 171 176
Google Scholar
[18] Gao P, Kumamoto A, Ishikawa R, Lugg N, Shibata N, Ikuhara Y 2018 Ultramicroscopy 184 177
Google Scholar
[19] Sun Y, Abid A Y, Tan C, Ren C, Li M, Li N, Chen P, Li Y, Zhang J, Zhong X, Wang J, Liao M, Liu K, Bai X, Zhou Y, Yu D, Gao P 2019 Sci. Adv. 5 eaav4355
Google Scholar
[20] Chen P, Zhong X, Zorn J A, Li M, Sun Y, Abid A Y, Ren C, Li Y, Li X, Ma X, Wang J, Liu K, Xu Z, Tan C, Chen L, Gao P, Bai X 2020 Nat. Commun. 11 1840
Google Scholar
[21] Du H, Jia C J, Mayer J 2016 Chem. Mater. 28 650
Google Scholar
[22] Galindo P L, Kret S, Sanchez A M, Laval J Y, Yanez A, Pizarro J, Guerrero E, Ben T, Molina S I 2007 Ultramicroscopy 107 1186
Google Scholar
[23] 南虎, 卢江波, 刘明, 井红梅, 汤少杰, 王大威, 贾春林 2016 电子显微学报 035 191
Google Scholar
Nan H, Lu J B, Liu M, Jing H M, Tang S J, Wang D W, Jia C L 2016 J. Chin. Electron. Microsc. Soc. 035 191
Google Scholar
[24] Gong Y J, Liu Z, Lupini A R, Shi G, Lin J H, Najmaei S, Lin Z, Elias A L, Berkdemir A, You G, Terrones H, Terrones M, Vajtai R, Pantelides S T, Pennycook S J, Lou J, Zhou W, Ajayan P M 2014 Nano Lett. 14 442
Google Scholar
[25] Taigman Y, Yang M, Ranzato M, Wolf L 2014 Proc CVPR IEEE 1701
Google Scholar
[26] LeCun Y, Bengio Y, Hinton G 2015 Nature 521 436
Google Scholar
[27] Redmon J, Farhadi A 2018 arXiv: 1804.02767 [cs.CV]
[28] Tzutalin. LabelImg. Git code (2015). https://github.com/tzutalin/labelImglabelImg [2020-12-3]
DownLoad:
Catalog
Metrics
- Abstract views: 5434
- PDF Downloads: 122
- Cited By: 0
