Author(s):
- Sathyanarayana, Aarti
- Ofli, Ferda
- Fernandes-Luque, Luis
- Srivastava, Jaideep
- Elmagarmid, Ahmed
- Arora, Teresa
- Taheri, Shahrad
Abstract:
Human Activity Recognition (HAR) is a powerful tool for understanding human behaviour. Applying HAR to wearable sensors can provide new insights by enriching the feature set in health studies, and enhance the personalisation and effectiveness of health, wellness, and fitness applications. Wearable devices provide an unobtrusive platform for user monitoring, and due to their increasing market penetration, feel intrinsic to the wearer. The integration of these devices in daily life provide a unique opportunity for understanding human health and wellbeing. This is referred to as the “quantified self” movement. The analyses of complex health behaviours such as sleep, traditionally require a time-consuming manual interpretation by experts. This manual work is necessary due to the erratic periodicity and persistent noisiness of human behaviour. In this paper, we present a robust automated human activity recognition algorithm, which we call RAHAR. We test our algorithm in the application area of sleep research by providing a novel framework for evaluating sleep quality and examining the correlation between the aforementioned and an individual’s physical activity. Our results improve the state-of-the-art procedure in sleep research by 15 percent for area under ROC and by 30 percent for F1 score on average. However, application of RAHAR is not limited to sleep analysis and can be used for understanding other health problems such as obesity, diabetes, and cardiac diseases.
Document:
https://doi.org/10.1109/ICDMW.2016.0077
References:
1. H. Wang, D. Oneata, J. Verbeek and C. Schmid, “A robust and efficient video representation for action recognition”, International Journal of Computer Vision, pp. 1-20, 2015. Show Context Google Scholar
2. B. Fernando, E. Gavves, J. Oramas, A. Ghodrati and T. Tuytelaars, “Rank pooling for action recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, no. 99, pp. 1-1, 2016. Show Context Google Scholar
3. I. Atmosukarto, N. Ahuja and B. Ghanem, “Action recognition using discriminative structured trajectory groups”, 2015 IEEE Winter Conference on Applications of ComputerVision, pp. 899-906, Jan 2015. Show Context Google Scholar
4. S. Ma, L. Sigal and S. Sclaroff, “Space-time tree ensemble for action recognition”, The IEEE Conference on ComputerVision and Pattern Recognition (CVPR), June 2015. Show Context Google Scholar
5. Z. Zhou, F. Shi and W. Wu, “Learning spatial and temporal extents of human actions for action detection”, IEEE Transactions on Multimedia, vol. 17, no. 4, pp. 512-525, April 2015. Show Context View Article Full Text: PDF (2517KB) Google Scholar
6. D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks”, 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4489-4497, Dec 2015. Show Context View Article Full Text: PDF (1362KB) Google Scholar
7. V. Kantorov and I. Laptev, “Efficient feature extraction encoding and classification for action recognition”, The IEEE Conference on ComputerVision and Pattern Recognition (CVPR), June 2014. Show Context Google Scholar
8. X. Peng, C. Zou, Y. Qiao and Q. Peng, “Action Recognition with Stacked Fisher Vectors. Cham” in , Springer International Publishing, pp. 581-595, 2014. Show Context Google Scholar
9. H. Kuehne, J. Gall and T. Serre, “An end-to-end generative framework for video segmentation and recognition”, 2016 IEEE Winter Conference on Applications of ComputerVision (WACV), pp. 1-8, March 2016. Show Context Google Scholar
10. J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, et al., “Long-term recurrent convolutional networks for visual recognition and description”, The IEEE Conference on ComputerVision and Pattern Recognition (CVPR), June 2015. Show Context Google Scholar
11. L. Wang, Y. Qiao and X. Tang, “Action recognition with trajectory-pooled deep-convolutional descriptors”, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4305-4314, June 2015. Show Context Google Scholar
12. L. Sun, K. Jia, D. Y. Yeung and B. E. Shi, “Human action recognition using factorized spatio-temporal convolutional networks”, 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4597-4605, Dec 2015. Show Context View Article Full Text: PDF (674KB) Google Scholar
13. S.-R. Ke, H. L. U. Thuc, Y.-J. Lee, J.-N. Hwang, J.-H. Yoo and K.-H. Choi, “A review on video-based human activity recognition”, Computers, vol. 2, no. 2, pp. 88-131, 2013. Show Context CrossRef Google Scholar
14. P. V. K. Borges, N. Conci and A. Cavallaro, “Video-based human behavior understanding: A survey”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 23, no. 11, pp. 1993-2008, Nov 2013. Show Context View Article Full Text: PDF (15637KB) Google Scholar
15. N. Ikizler, R. G. Cinbis, S. Pehlivan and P. Duygulu, “Recognizing actions from still images”, Pattern Recognition 2008. ICPR 2008. 19th International Conference on, pp. 1-4, Dec 2008. Show Context Google Scholar
16. C. Thurau and V. Hlavac, “Pose primitive based human action recognition in videos or still images”, Computer Vision and Pattern Recognition 2008. CVPR 2008. IEEE Conference on, pp. 1-8, June 2008. Show Context Google Scholar
17. F. S. Khan, R. Muhammad Anwer, J. van de Weijer, A. D. Bagdanov, A. M. Lopez and M. Felsberg, “Coloring action recognition in still images”, International Journal of Computer Vision, vol. 105, no. 3, pp. 205-221, 2013. Show Context CrossRef Google Scholar
18. G. Sharma, F. Julie and C. Schmid, “Expanded parts model for semantic description of humans in still images”, IEEE Transactions on Pattern Analysis and Machine Intelligence, no. 99, pp. 1-1, 2016. Show Context Google Scholar
19. “A survey on still image based human action recognition”, Pattern Recognition, vol. 47, no. 10, pp. 3343-3361, 2014. Show Context CrossRef Google Scholar
20. L. Xia, C. C. Chen and J. K. Aggarwal, “View invariant human action recognition using histograms of 3d joints”, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 20-27, June 2012. Show Context Google Scholar
21. X. Yang and Y. L. Tian, “Eigenjoints-based action recognition using naive-bayes-nearest-neighbor”, 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 14-19, June 2012. Show Context Google Scholar
22. F. Ofti, R. Chaudhry, G. Kurillo, R. Vidal and R. Bajesy, “Sequence of the most informative joints (smij): A new representation for human skeletal action recognition”, Journal of Visual Communication and Image Representation, vol. 25, no. 1, pp. 24-38, 2014. Show Context Google Scholar
23. R. Chaudhry, F. Ofti, G. Kurillo, R. Bajesy and R. Vidal, “Bio-inspired dynamic 3d discriminative skeletal features for human action recognition”, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 471-478, June 2013. Show Context Google Scholar
24. J. Wang, Z. Liu, Y. Wu and J. Yuan, “Mining actionlet ensemble for action recognition with depth cameras”, Computer Vision and Pattern Recognition (CVPR) 2012 IEEE Conference on, pp. 1290-1297, June 2012. Show Context Google Scholar
25. R. Vemulapalli, F. Arrate and R. Chellappa, “Human action recognition by representing 3d skeletons as points in a lie group”, 2014 IEEE Conference on ComputerVision and Pattern Recognition, pp. 588-595, June 2014. Show Context Google Scholar
26. M. E. Hussein, M. Torki, M. A. Gowayyed and M. EI-Saban, “Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations”, IJCAI, vol. 13, pp. 2466-2472, 2013. Show Context Google Scholar
27. F. Lv and R. Nevatia, Recognition and Segmentation of 3-D Human Action Using HMM and Multi-class AdaBoost, Berlin, Heidelberg:Springer Berlin Heidelberg, pp. 359-372, 2006. Show Context Google Scholar
28. Y. Sheikh, M. Sheikh and M. Shah, “Exploring the space of a human action”, Tenth IEEE International Conference on Computer Vision (ICCV’05) Volume 1, vol. 1, pp. 144-149, Oct 2005. Show Context Google Scholar
29. J. Wang, Z. Liu, J. Chorowski, Z. Chen and Y. Wu, Robust 3D Action Recognition with Random Occupancy Patterns, Berlin, Heidelberg:Springer Berlin Heidelberg, pp. 872-885, 2012. Show Context Google Scholar
30. A. W. Vieira, E. R. Nascimento, G. L. Oliveira, Z. Liu and M. F. M. Campos, STOP: Space-Time Occupancy Patterns for 3D Action Recognition from Depth Map Sequences, Berlin, Heidelberg:Springer Berlin Heidelberg, pp. 252-259, 2012. Show Context Google Scholar
31. “Human activity recognition from 3d data: A review”, Pattern Recognition Letters, vol. 48, pp. 70-80, 2014. Show Context CrossRef Google Scholar
32. L. Chen, J. Hoey, C. D. Nugent, D. J. Cook and Z. Yu, “Sensor-based activity recognition”, IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews), vol. 42, no. 6, pp. 790-808, Nov 2012. Show Context View Article Full Text: PDF (520KB) Google Scholar
33. A. Bulling, U. Blanke and B. Schiele, “A tutorial on human activity recognition using body-worn inertial sensors”, ACM Comput. Surv., vol. 46, no. 3, pp. 33:1-33:33, Jan. 2014. Show Context Access at ACM Google Scholar
34. A. Calzada, J. Liu, C. D. Nugent, H. Wang and L. Martinez, “Sensor-based activity recognition using extended belief rule-based inference methodology”, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2694-2697, Aug 2014. Show Context View Article Full Text: PDF (735KB) Google Scholar
35. “Recognizing upper limb movements with wrist worn inertial sensors using k-means clustering classification”, Human Movement Science, vol. 40, pp. 59-76, 2015. Show Context CrossRef Google Scholar
36. A. M. Tripathi, D. Baruah and R. D. Baruah, “Acoustic sensor based activity recognition using ensemble of one-class classifiers”, Evolving and Adaptive Intelligent Systems (EAIS) 2015 IEEE International Conference on, pp. 1-7, Dec 2015. Show Context View Article Full Text: PDF (704KB) Google Scholar
37. “On the use of ensemble of classifiers for accelerometer-based activity recognition”, Applied Soft Computing, vol. 37, pp. 1018-1022, 2015. Show Context Google Scholar
38. R. Akhavian and A. Behzadan, “Wearable sensor-based activity recognition for data-driven simulation of construction workers’ activities”, 2015 Winter Simulation Conference (WSC), pp. 3333-3344, Dec 2015. Show Context Google Scholar
39. “Sensor-based human activity recognition system with a multilayered model using time series shapelets”, Knowledge-Based Systems, vol. 90, pp. 138-152, 2015. Show Context CrossRef Google Scholar
40. “Dynamic sensor data segmentation for real-time knowledge-driven activity recognition”, Pervasive and Mobile Computing, vol. 10, no. Part B, pp. 155-172, 2014. Show Context Google Scholar
41. S. C. Mukhopadhyay, “Wearable sensors for human activity monitoring: A review”, IEEE Sensors Journal, vol. 15, no. 3, pp. 1321-1330, March 2015. Show Context View Article Full Text: PDF (959KB) Google Scholar
42. L. M. Zhou, C. Gurrin and Z. Qiu, “Zhiwo: Activity tagging and recognition system for personal lifelogs”, Proceedings of the 3rd ACM Conference on International Conference on Multimedia Retrieval set ICMR ’13, pp. 321-322, 2013. Show Context Access at ACM Google Scholar
43. C. Meurisch, B. Schmidt, M. Scholz, I. Schweizer and M. Mühlhäuser, “Labels: Quantified self app for human activity sensing”, Adjunct Proceedings of the 2015 ACM InternationalJoint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM InternationalSymposium on Wearable Computers ser. UbiComp/ISWC’15 Adjunct, pp. 1413-1422, 2015. Show Context Access at ACM Google Scholar
44. J. Hamm, B. Stone, M. Belkin and S. Dennis, Automatic Annotation of Daily Activity from Smartphone-BasedMultisensory Streams, Berlin, Heidelberg:Springer Berlin Heidelberg, pp. 328-342, 2013. Show Context Google Scholar
45. C. Dobbins and R. Rawassizadeh, “Clustering of physical activities for quantified self and mhealth applications”, Computer and Information Technology; Ubiquitous Computing and Communications; Dependable Autonomic and Secure Computing; Pervasive Intelligence and Computing (CIT/IUCC/DASC/PICOM) 2015 IEEE International Conference on, pp. 1423-1428, Oct 2015. Show Context View Article Full Text: PDF (339KB) Google Scholar
46. M. Uddin, A. Salem, I. Nam and T. Nadeem, “Wearable sensing framework for human activity monitoring”, Proceedings of the 2015 Workshop on Wearable Systems and Applications ser. WearSys ’15, pp. 21-26, 2015. Show Context Access at ACM Google Scholar
47. O. Banos, J. Bang, T. Hur, M. H. Siddiqi, H. T. Thien, L. B. Vui, et al., “Mining human behavior for health promotion”, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 5062-5065, Aug 2015. Show Context Google Scholar
48. P. S. Freedson, E. Melanson and J. Sirard, “Calibration of the computer science and applications inc. accelerometer”, Medicine and Science in Sports and Exercise, vol. 30, no. 5, pp. 777-781, 1998. Show Context CrossRef Google Scholar
49. T. W. Strine and D. P. Chapman, “Associations of frequent sleep insufficiency with health-related quality of life and health behaviors”, Sleep medicine, vol. 6, no. 1, pp. 23-27, 2005. Show Context CrossRef Google Scholar
50. T. Arora, E. Broglia, D. Pushpakumar, T. Lodhi and S. Taheri, “An investigation into the strength of the association and agreement levels between subjective and objective sleep duration in adolescents”, PloS one, vol. 8, no. 8, pp. e72406, 2013. Show Context CrossRef Google Scholar
51. C. Iber, S. Ancoli-Israel, A. Chesson and S. Quan, “The AASM Manual for the Scoring of Sleep and Associated Events: Rules Terminology and Technical Specifications”, American Academy of Sleep Medicine, 2007. Show Context Google Scholar
52. A. Sadeh, A. Raviv and R. Gruber, “Sleep patterns and sleep disruptions in school-age children”, Developmental psychology, vol. 36, no. 3, pp. 291, 2000. Show Context CrossRef Google Scholar
53. J. T. Cacioppo, L. C. Hawkley, G. G. Berntson, J. M. Ernst, A. C. Gibbs, R. Stickgold, et al., “Do lonely days invade the nights? potential social modulation of sleep efficiency”, vol. 13, no. 4, pp. 384-387, 2002. Show Context CrossRef Google Scholar
54. “Normative values of polysomnographic parameters in childhood and adolescence: Quantitative sleep parameters”, Sleep Medicine, vol. 12, no. 6, pp. 542-549, 2011. Show Context CrossRef Google Scholar
55. K. P. Gabriel, J. J. McClain, K. K. Schmid, K. L. Storti, R. R. High, D. A. Underwood, et al., “Issues in accelerometer methodology: the role of epoch length on estimates of physical activity and relationships with health outcomes in overweight post-menopausal women”, International Journal of Behavioral Nutrition and Physical Activity, vol. 7, no. 1, pp. 1-10, 2010. Show Context CrossRef Google Scholar
56. R. P. Troiano, D. Berrigan, K. W. Dodd, L. C. Masse, T. Tilert, M. McDowell et al., “Physical activity in the united states measured by accelerometer”, Medicine and science in sports and exercise, vol. 40, no. 1, pp. 181-188, 2008. Show Context CrossRef Google Scholar
57. N. James and D. Matteson, “ecp: An r package for nonparametric multiple change point analysis of multivariate data”, Journal of Statistical Software, vol. 62, no. 1, pp. 1-25, 2015. Show Context Google Scholar
58. R. Williams, I. Karacan and C. Hursch, Electroencephalography (Eeg) of Human Sleep: Clinical Applications ser, A Wiley biomedical-health publication. Show Context Google Scholar