-
The dominant challenge of vital signal monitoring based on facial video is to eliminate the interference of motion artifacts. In this paper, we propose a non-contact heart rate detection method based on an adaptive filter constructed by head movement information to tackle the noise of motion artifacts caused by the rigid rotation of the subject's head. The two-dimensional and three-dimensional feature points of the subject’s face are used to calculate the yaw and pitch Euler angles of the head movement, then the yaw and pitch Euler angles are used as a novel signal quality index (SQI) for modulating process noise covariance to construct an adaptive Kalman filter, and finally robust heart rate is estimated by this method. The experimental results show that the proposed method can effectively suppress the noise caused by the head rigid rotation with an average absolute error of 2.22 beat/min and a root mean square error of 2.76 beat/min, which are statistically significant with an accuracy improvement of 9% and 24.6%, respectively, compared with the existing methods. The adaptive non-contact robust heart rate detection technique based on head rigid rotation may effectively enhance the accuracy in real-world motion situations, as well as broaden the range of applications for IPPG in the field of the video-based monitoring of health conditions.
-
Keywords:
- facial video /
- non-contact heart rate detection method /
- head rotation /
- adaptive Kalman filter
[1] Franco M, Cooper R S, Bilal U, Fuster V 2011 Am. J. Med. 124 95
Google Scholar
[2] Allen J 2007 Physiol. Meas. 28 R1
Google Scholar
[3] Shelley K H 2007 Anesth. Analg. 105 S31
Google Scholar
[4] Sun Y, Thakor N 2016 IEEE Trans. Biomed. Eng. 63 463
Google Scholar
[5] Hulsbusch M, Blazek V 2002 Medical Imaging San Diego, California, USA, April 24, 2002 110
[6] Verkruysse W, Svaasand L O, Nelson J S 2008 Opt. Express 16 21434
[7] Wieringa F P, Mastik F, Steen A 2005 Ann. Biomed. Eng. 33 1034
Google Scholar
[8] Shao D, Yang Y, Liu C, Tsow F, Yu H, Tao N 2014 IEEE Trans. Biomed. Eng. 61 2760
Google Scholar
[9] Hülsbusch M 2008 Ph. D. Dissertation (Aachen: RWTH Aachen University) (in German)
[10] Poh M Z, McDuff D J, Picard R W 2010 Opt. Express 18 10762
Google Scholar
[11] Poh M Z, McDuff D J, Picard R W 2010 IEEE Trans. Biomed. Eng. 58 7
Google Scholar
[12] Lewandowska M, Ruminski J, Kocejko T, Nowak J 2011 Federated Conference on Computer Science and Information Systems-FedCSIS 2011 Szczecin, Poland, September 18–21, 2011 p405
[13] Haan G D, Jeanne V 2013 IEEE Trans. Biomed. Eng. 60 2878
Google Scholar
[14] Wang W, Brinker A C D, Stuijk S, Haan G D 2017 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) Washington, USA, May 30–June 3, 2017 p71
[15] Wang W, Brinker A C D, Stuijk S, Haan G D 2016 IEEE Trans. Biomed. Eng. 64 1479
Google Scholar
[16] Wang W, Brinker A C D, Stuijk S, Haan G D 2017 Physiol. Meas. 38 1023
Google Scholar
[17] Sun Y, Hu S, Azorin-Peris V, Greenwald S, Chambers J, Zhu Y 2011 J. Biomed. Opt. 16 077010
Google Scholar
[18] Wang W, Stuijk S, Haan G D 2014 IEEE Trans. Biomed. Eng. 62 415
Google Scholar
[19] Wu B F, Huang P W, Lin C H, Chung M L, Tsou T Y, Wu Y L 2018 IEEE Access 6 21621
Google Scholar
[20] Kong L, Wu Y, Zhao Y, Dong L, Hui M, Liu M, Liu X 2020 IEEE Photonics J. 12 1
Google Scholar
[21] Viola P A, Jones M J 2001 Computer Vision and Pattern Recognition Kauai, Hawaii, USA, December 8–14, 2001 p511
[22] Kwon S, Kim J, Lee D, Park K 2015 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) Milan, Italy, August 25–29, 2015 p4938
[23] 杨萍, 侯威, 封国林 2008 物理学报 57 5333
Google Scholar
Yang P, Hou W, Feng G L 2008 Acta Phys. Sin. 57 5333
Google Scholar
[24] Bousefsaf F, Maaoui C, Pruski A 2013 Biomed. Signal Process. Control 8 568
Google Scholar
[25] Haan G D, Leest A V 2014 Physiol. Meas. 35 1913
Google Scholar
[26] 孔令琴 2014 博士学位论文 (北京: 北京理工大学)
Kong L 2014 Ph. D. Dissertation (Beijing: Beijing Institute of Technology) (in Chinese)
[27] Wang W 2017 Ph. D. Dissertation (Eindhoven, The Netherlands: Eindhoven University of Technology)
[28] Smith W J 2008 Modern Optical Engineering: The Design of Optical Systems (4th Ed.) (New York: The McGraw-Hill Companies, Inc) p253
[29] Asthana A, Zafeiriou S, Cheng S, Pantic M 2013 Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on Portland, Oregon, USA, June 23–28, 2013 p3444
[30] Baltrusaitis T, Robinson P, Morency L 2012 IEEE Conference on Computer Vision & Pattern Recognition Providence, Rhode island, USA, June 16–21, 2012 p2610
[31] Baltrusaitis T, Robinson P, Morency L 2016 IEEE Winter Conference on Applications of Computer Vision Lake Placid, New York, USA, March 7–10, 2016 p1
[32] Baltrusaitis T, Zadeh A, Lim Y C, Morency L 2018 IEEE International Conference on Automatic Face & Gesture Recognition Xi’an, China, May 15–19, 2018 p59
[33] Andreotti F, Trumpp A, Malberg H, Zaunseder S 2015 2015 IEEE 35th International Conference on Electronics and Nanotechnology (ELNANO) Kyiv, Ukraine, April 21–24, 2015 p428
[34] 张玉燕, 殷东哲, 温银堂, 罗小元 2021 物理学报 70 118102
Google Scholar
Zhang Y Y, Yin D Z, Wen Y T, Luo S Y 2021 Acta Phys. Sin. 70 118102
Google Scholar
[35] Nemati S, Malhotra A, Clifford G 2010 EURASIP J. Adv. Signal Process. 2010 926305
Google Scholar
[36] Tarvainen M P, Ranta-Aho P O, Karjalainen P A 2002 IEEE Trans. Biomed. Eng. 49 172
Google Scholar
[37] Fallet S, Moser V, Braun F, Vesin J M 2017 Computing in Cardiology Conference Vancouver, BC, Canada, September 11–14, 2016 p341
-
图 1 人脸的3个旋转运动自由度
Figure 1. Three rotational freedom degrees of human head.
图 2 头部旋转角度信息作为自适应滤波器质量控制的流程图
Figure 2. Flow chart of head rotation angles as a quality control of our proposed adaptive filter.
图 3 实验方案图 (a) 实验装置示意图; (b) 采集到的视频片段((i)头部姿势的欧拉角为0°; (ii) 头部俯仰运动; (iii) 头部偏航)
Figure 3. Experimental plan diagram: (a) Experiment set-up; (b) video clips ((i) the Euler angle of the head pose is 0°; (ii) the head pitch segment; (iii) the head yaw segment).
图 4 矩形ROI区域选取示意图 (a)面部68个关键点检测; (b)脸颊矩形ROI区域; (c)偏航运动时ROI区域
Figure 4. Flow chart of rectangular ROI area selection: (a) 68 feature points detection; (b) rectangular ROI area; (c) ROI area during yaw motion.
图 5 本文提出方法的流程图
Figure 5. Flow chart of method proposed in this paper.
图 6 本文提出的滤波器与CHROM结合的对比结果
Figure 6. Comparison results of the filter proposed in this paper combined with CHROM.
图 7 21组实验结果相关性图(图中红线表示y=x的线性关系) (a) CHROM方法的相关性图; (b)本文提出的滤波器与CHROM结合的相关性图
Figure 7. Correlation plots of 21 groups of experimental results (the red lines in the plots indicate linear relationship of y = x): (a) Correlation plot of CHROM method; (b) correlation plot of combining CHROM with our proposed adaptive filter.
表 1 本文提出的自适应滤波器与不同的传统方法结合前后实验结果对比
Table 1. Comparison of experimental results before and after combining the filter proposed in this paper with different traditional methods.
方法类型 传统的方法 传统方法结合本文的滤波器 MAE/(beat·min–1) RMSE/(beat·min–1) $ {\mathit{R}}^{2} $ MAE/(beat·min–1) RMSE/(beat·min–1) $ {\mathit{R}}^{2} $ GREEN 5.98 8.65 0.5471 3.88 4.95 0.7217 CDF 5.46 9.22 0.5993 3.18 4.16 0.8402 POS 3.22 4.85 0.7644 2.86 3.61 0.8181 POS+CDF 6.69 11.59 0.4774 3.25 4.42 0.8517 CHROM 2.44 3.66 0.8142 2.22 2.74 0.8401 CHROM+CDF 4.71 8.90 0.7514 2.62 3.46 0.8901 
DownLoad: CSV
-
[1] Franco M, Cooper R S, Bilal U, Fuster V 2011 Am. J. Med. 124 95
Google Scholar
[2] Allen J 2007 Physiol. Meas. 28 R1
Google Scholar
[3] Shelley K H 2007 Anesth. Analg. 105 S31
Google Scholar
[4] Sun Y, Thakor N 2016 IEEE Trans. Biomed. Eng. 63 463
Google Scholar
[5] Hulsbusch M, Blazek V 2002 Medical Imaging San Diego, California, USA, April 24, 2002 110
[6] Verkruysse W, Svaasand L O, Nelson J S 2008 Opt. Express 16 21434
[7] Wieringa F P, Mastik F, Steen A 2005 Ann. Biomed. Eng. 33 1034
Google Scholar
[8] Shao D, Yang Y, Liu C, Tsow F, Yu H, Tao N 2014 IEEE Trans. Biomed. Eng. 61 2760
Google Scholar
[9] Hülsbusch M 2008 Ph. D. Dissertation (Aachen: RWTH Aachen University) (in German)
[10] Poh M Z, McDuff D J, Picard R W 2010 Opt. Express 18 10762
Google Scholar
[11] Poh M Z, McDuff D J, Picard R W 2010 IEEE Trans. Biomed. Eng. 58 7
Google Scholar
[12] Lewandowska M, Ruminski J, Kocejko T, Nowak J 2011 Federated Conference on Computer Science and Information Systems-FedCSIS 2011 Szczecin, Poland, September 18–21, 2011 p405
[13] Haan G D, Jeanne V 2013 IEEE Trans. Biomed. Eng. 60 2878
Google Scholar
[14] Wang W, Brinker A C D, Stuijk S, Haan G D 2017 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017) Washington, USA, May 30–June 3, 2017 p71
[15] Wang W, Brinker A C D, Stuijk S, Haan G D 2016 IEEE Trans. Biomed. Eng. 64 1479
Google Scholar
[16] Wang W, Brinker A C D, Stuijk S, Haan G D 2017 Physiol. Meas. 38 1023
Google Scholar
[17] Sun Y, Hu S, Azorin-Peris V, Greenwald S, Chambers J, Zhu Y 2011 J. Biomed. Opt. 16 077010
Google Scholar
[18] Wang W, Stuijk S, Haan G D 2014 IEEE Trans. Biomed. Eng. 62 415
Google Scholar
[19] Wu B F, Huang P W, Lin C H, Chung M L, Tsou T Y, Wu Y L 2018 IEEE Access 6 21621
Google Scholar
[20] Kong L, Wu Y, Zhao Y, Dong L, Hui M, Liu M, Liu X 2020 IEEE Photonics J. 12 1
Google Scholar
[21] Viola P A, Jones M J 2001 Computer Vision and Pattern Recognition Kauai, Hawaii, USA, December 8–14, 2001 p511
[22] Kwon S, Kim J, Lee D, Park K 2015 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) Milan, Italy, August 25–29, 2015 p4938
[23] 杨萍, 侯威, 封国林 2008 物理学报 57 5333
Google Scholar
Yang P, Hou W, Feng G L 2008 Acta Phys. Sin. 57 5333
Google Scholar
[24] Bousefsaf F, Maaoui C, Pruski A 2013 Biomed. Signal Process. Control 8 568
Google Scholar
[25] Haan G D, Leest A V 2014 Physiol. Meas. 35 1913
Google Scholar
[26] 孔令琴 2014 博士学位论文 (北京: 北京理工大学)
Kong L 2014 Ph. D. Dissertation (Beijing: Beijing Institute of Technology) (in Chinese)
[27] Wang W 2017 Ph. D. Dissertation (Eindhoven, The Netherlands: Eindhoven University of Technology)
[28] Smith W J 2008 Modern Optical Engineering: The Design of Optical Systems (4th Ed.) (New York: The McGraw-Hill Companies, Inc) p253
[29] Asthana A, Zafeiriou S, Cheng S, Pantic M 2013 Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on Portland, Oregon, USA, June 23–28, 2013 p3444
[30] Baltrusaitis T, Robinson P, Morency L 2012 IEEE Conference on Computer Vision & Pattern Recognition Providence, Rhode island, USA, June 16–21, 2012 p2610
[31] Baltrusaitis T, Robinson P, Morency L 2016 IEEE Winter Conference on Applications of Computer Vision Lake Placid, New York, USA, March 7–10, 2016 p1
[32] Baltrusaitis T, Zadeh A, Lim Y C, Morency L 2018 IEEE International Conference on Automatic Face & Gesture Recognition Xi’an, China, May 15–19, 2018 p59
[33] Andreotti F, Trumpp A, Malberg H, Zaunseder S 2015 2015 IEEE 35th International Conference on Electronics and Nanotechnology (ELNANO) Kyiv, Ukraine, April 21–24, 2015 p428
[34] 张玉燕, 殷东哲, 温银堂, 罗小元 2021 物理学报 70 118102
Google Scholar
Zhang Y Y, Yin D Z, Wen Y T, Luo S Y 2021 Acta Phys. Sin. 70 118102
Google Scholar
[35] Nemati S, Malhotra A, Clifford G 2010 EURASIP J. Adv. Signal Process. 2010 926305
Google Scholar
[36] Tarvainen M P, Ranta-Aho P O, Karjalainen P A 2002 IEEE Trans. Biomed. Eng. 49 172
Google Scholar
[37] Fallet S, Moser V, Braun F, Vesin J M 2017 Computing in Cardiology Conference Vancouver, BC, Canada, September 11–14, 2016 p341
DownLoad:
Catalog
Metrics
- Abstract views: 3426
- PDF Downloads: 63
- Cited By: 0
