Author(s):
- Shaukat Ali,
- Shah Khusro,
- Akif Khan,
- Inayat Khan,
- Iftikhar Alam
- Salman Faiz Solehria
Abstract:
The advancements in sensors, processing, storage and networking technologies have turned smartphone into de facto lifelogging device. Realizing the lifelogging potential of smartphone, researchers have postulated with smartphone-based lifelogging systems with varying capabilities of capturing and storing information about peoples’ daily life activities, behaviors, interactions, locations, environments, etc., into their lifelog archives. However, the systems are lacking with several capturing, storing, retrieving and design issues, importantly not organizing the lifelog information with the same semantics as they exist in the real world and encoded in human episodic memory. The semantic organization of lifelog information will develop a true digital prosthetic memory to solve problems in different real-world domains including memory augmentation. In this paper, we have presented the architecture and implementation of Semantic LifelOGging (SLOG) framework for smartphone-based semantic lifelogging using Semantic Web technologies. The SLOG framework incorporates smartphone sensors and processing capabilities, and Semantic Web technologies to develop a semantically enriched digital prosthetic memory on smartphone. The semantic lifelog data model is developed by extending SmartOntoSensor ontology with relevant concepts and properties to represent, store and relate lifelog and contextual information. The semantic lifelog data model will provide contextual semantics to organize, annotate and relate lifelog information like human episodic memory and provide enough contextual cues to recall lifelog information like associative recall in human memory. The proof-of-concept applications are developed to show practicality of the SLOG framework. The quantitative and empirical evaluations have shown the SLOG framework and system has improved features and is acceptable in usability and effectiveness.
Documentation:
https://doi.org/10.1007/s00779-022-01701-0
References:
Gurrin C, Smeaton AF, Doherty AR (2014) LifeLogging: personal big data. Found Trends Inf Retr 8(1):1–125
Ali S (2018) Exploiting sensor data semantics for smartphone-based lifelogging: Towards the development of digital prosthetic memory on smartphone. PhD Dissertation, University of Peshawar, Pakistan
Chen Y, Jones GJF (2010) Augmenting human memory using personal lifelogs. In Proceedings of the 1st Augmented Human International Conference. ACM, pp. 1–9
Hodges S, Williams L, Berr E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood K (2006) SenseCam: A retrospective memory aid. In Proceedings of the 8th international conference on Ubiquitous Computing. Springer, pp. 177–193
Aizenbud-Reshef N, Belinsky E, Jacovi M, Laufer D, Soroka V (2008) Pensieve: augmenting human memory. In Proceedings of the CHI ’08 Extended Abstracts on Human Factors in Computing Systems. ACM, pp 3231–3236. https://doi.org/10.1145/1358628.1358836
Chennuru SK, Chen P-W, Zhu J, Zhang JY (2010) Mobile Lifelogger – recording, indexing, and understanding a mobile user’s life. In Proceedings of the International Conference on Mobile Computing, Applications, and Services. Springer, pp 263–281
Klapperich H, Uhde A, Hassenzahl M (2020) Designing everyday automation with well-being in mind. Pers Ubiquit Comput 24(6):763–779
Belimpasakis P, Roimela K, Yu Y (2009) Experience Explorer: a life-logging platform based on mobile context collection. In Proceedings of the Third International Conference on Next Generation Mobile Applications, Services and Technologies – NGMAST ’09. IEEE, pp 77–82. https://doi.org/10.1109/ngmast.2009.49
Albatal R, Gurrin C, Zhou J, Yang Y, Carthy D, Li N (2013) SenseSeer: mobile-cloud-based lifelogging framework. In Proceedings of the IEEE International Symposium on Technology and Society (ISTAS). IEEE, pp 27–29
Mohamed EST (2012) Designing and evaluating a user interface for continues embedded lifelogging based on physical context. PhD Dissertation, Newcastle University, England
Karkkainen T, Vaittinen T, Vaananen-Vainio-Mattila K (2010) I don’t mind being logged, but want to remain in control: a field study of mobile activity and context logging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 163–172
Bush V (1996) As we may think interactions 3(2):35–46. https://doi.org/10.1145/227181.227186
Oren E (2006) An overview of information management and knowledge work studies: lessons for the semantic desktop. In Proceedings of the 5th International Conference on Semantic Desktop and Social Semantic Collaboration- SemDesk’06. ACM, pp. 14–24
Dodge M, Kitchin R (2007) ‘Outlines of a world coming into existence’: pervasive computing and the ethics of forgetting. Environ Plann B Plann Des 34(3):431–445
Kalnikaite V, Sellen A, Whittaker S, Kirk D (2010) Now let me see where i was: understanding how lifelogs mediate memory. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, pp. 2045–2054
Ali S, Khusro S, Khan A, Khan I, Solheria SF (2019) An insight of smartphone-based lifelogging research: Issues, challenges, and research opportunities. Proceedings of the Pakistan Academy of Sci: A Physical and Computational Sci 56(3):1–16
Sellen A, Whittaker S (2010) Beyond total capture: A constructive critique of lifelogging. Commun ACM 53(5):70–77. https://doi.org/10.1145/1735223.1735243
Rawassizadeh R (2012) A holistic multi-purpose life logging framework. PhD Dissertation, University of Vienna, Austria
Siewiorek D, Smailagic A, Furukawa J, Krause A, Moraveji N, Reiger K, Shaffer J, Wong FL (2003) SenSay: A context-aware mobile phone. In Proceedings of the 7th IEEE International Symposium on Wearable Computers. IEEE, pp. 248–253
Shah M, Mears B, Chakrabarti C, Spanias A (2012) Lifelogging: Archival and retrieval of continuously recorded audio using wearable devices. In Proceedings of the IEEE International Conference on Emerging Signal Processing Applications (ESPA). IEEE, pp 99–102
Wolf K, Schmidt A, Bexheti A, Langheinrich M (2014) Lifelogging: you’re wearing a camera? IEEE Pervasive Comput 13(3):8–12
Gurrin C, Johansen H, Sødring T, Johansen D (2014) Digital chronofiles of life experience. In Proceedings of the International Conference on Social Informatics. Springer, pp 264–273
Rawassizadeh R, Tomitsch M, Wac K, Tjoa AM (2013) UbiqLog: a generic mobile phone-based life-log framework. Pers Ubiquit Comput 17(4):621–637. https://doi.org/10.1007/s00779-012-0511-8
Vemuri S, Schmandt C, Bender W (2006) iRemember: a personal, long-term memory prosthesis. In Proceedings of the 3rd ACM Workshop on Continous Archival and Retrieval of Persoanl Experiences CARPE’ 06. ACM, pp 65–74
Ali S, Khusro S, Rauf A, Mahfooz S (2014) Sensors and mobile phones: Evolution and state-of-the-art. Pak J Sci 66(4):386–400
Ali S, Khusro S, Khan AA, Hassan L (2014) A survey of mobile phones context-awareness using sensing computing research. J Engineering and Applied Sci 33(4):75–93
Myka A (2005) Nokia lifeblog — towards a truly personal multimedia information system. In Proceedings of the Workshop on Mobile Data Banken and Information Systemes – MDBIS’05. pp 21–30
Ahmed M, Hoang HH, Karim MS, Khusro S, Lanzenberger M, Latif K, Michlmayr E, Mustofa K, Nguyen H, Rauber A (2004) ‘SemanticLIFE’-a framework for managing information of a human lifetime. In Proceedings of the Sixth International Conference on Information Integration and Web Based Applications & Services. iiWAS2004, pp. 725–734
Froehlich J, Chen MY, Consolvo S, Harrison B, Landay JA (2007) MyExperience: A system for in situ tracing and capturing of user feedback on mobile phones. In Proceedings of the 5th International Conference on Mobile Systems, Applications and Services. ACM, pp. 57–70
Packer HS, Smith A, Lewis P (2012) MemoryBook: generating narratives from lifelogs. In Proceedings of the 2nd Workshop on Narrative and Hypertext. ACM, pp 7–12
Memon MA, Bhatti S, Mahoto NA (2016) A Digital Diary: Remembering the past using the present context. Mehran University Research J Engineering Technol 35(2):275–286
Sangkeun L, Gihyun G, Inbeom H, Sang-goo L (2010) LifeLogOn: A practical lifelog system for building and exploiting lifelog ontology. In Proceeding of the IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing (SUTC). IEEE, pp 367–373. https://doi.org/10.1109/sutc.2010.33
Zafeiropoulos A, Konstantinou N, Arkoulis S, Spanos DE, Mitrou N A (2008) Semantic-based architecture for sensor data fusion. In Proceedings of the Second Internation Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. IEEE, pp 116–121. https://doi.org/10.1109/ubicomm.2008.67
Tuffield MM, Loizou A, Dupplaw D (2006) The Semantic Logger: Supporting service building from personal context. In Proceedings of the 3rd ACM Workshop on Continuous Archival and Retrival of Personal Experences – CARPE ’06. ACM, pp. 55–64
Song I, Haam D, Kim H, Kim MH (2010) OntoLMS: An ontology-based lifelog management system. In Proceedings of the 12th International Asia-Pacific Web Conference. IEEE, pp 341–343
Ali S, Khusro S (2016) POEM: practical ontology engineering model for semantic web ontologies. Cogent Engineering 3(1):1193959. https://doi.org/10.1080/23311916.2016.1193959
Cohen G, Conway MA (2008) Memory in the real world. Psychology Press, Hove East Sussex, United Kingdom
Tulving E, Murray D (1985) Elements of episodic memory. Can Psychol 26(3):235–238
Tulving E, Thomson DM (1973) Encoding specificity and retrieval processes in episodic memory. Psychol Rev 80(5):359–380
Ali S, Khusro S, Ullah I, Khan A, Khan I (2017) SmartOntoSensor: ontology for semantic interpretation of smartphone sensors data for context-aware applications. Journal of Sensors 17(2017):1–26
Preuveneers D, den Bergh JV, Wagelaar D, Georges A, Rigole P, Clerckx T, Berbers Y, Coninx K, Jonckers V, Bosschere KD (2004) Towards an extensible context ontology for ambient intelligence. In Proceedings of the Second European Symposium on Ambient Intelligence. Springer, pp. 148–159
Kim I-J, Ahn SC, Kim H-G (2007) Personalized life log media system in ubiquitous environment. In: Ubiquitous Convergence Technology. Springer, pp 20–29
Weiser M (1994) The world is not a desktop. interactions 1 (1):7–8
Indulska J, Sutton P (2003) Location management in pervasive systems. In Proceedings of the Australasian Information Security Workshop Conference on ACSW Frontiers. Australian Computer Society Inc, pp 143–151
Ali S, Khusro S (2016) Mobile phone sensing: a new application paradigm. Indian J Sci Technol 9(19):1–42
Wang P (2011) Semantic interpretation of events in lifelogging. PhD Dissertation, Dublin City University, Ireland
Smeaton AF, Over P, Kraaij W (2009) High-level feature detection from video in TRECVid: A 5-year retrospective of achievements. Multimedia Content Analysis, Signals and Communication:151–174
Arndt N (2011) TriplePlace: a flexible triple store for Android with six indices. https://core.ac.uk/reader/226099420#page=15. Accessed 20/03/2020 2020
Möller K, Handschuh S (2007) Towards a light-weight semantic desktop. In Proceedings of the Semantic Desktop Design Workshop (SemDeskDesign 2007) at ESWC2007.
Bell G (2001) A personal digital store. Commun ACM 44(1):86–91
Allen AL (2008) Dredging up the past: Lifelogging, memory, and surveillance. The University of Chicago Law Review 75(1):47–74
Strahilevitz LJ (2005) A social networks theory of privacy. The University of Chicago Law Review:919–988
Langheinrich M (2001) Privacy by design—principles of privacy-aware ubiquitous systems. In Proceedings of the International conference on Ubiquitous Computing. Springer, pp 273–291
Khan I, Khusro S, Ali S, Din AU (2016) Daily life activities on smartphones and their effect on battery life for better personal information management. Proceedings of the Pakistan Academy of Sci: A Physical and Computational Sci 53(1):61–74
Khan I, Ali S, Khusro S (2019) Smartphone-based lifelogging: an investigation of data volume generation strength of smartphone sensors. In, Cham, 2019. Simulation tools and techniques. Springer International Publishing, pp 63–73
Brooke J (1996) SUS: a “quick and dirty” usability. Usability evaluation in indmstry:189
Mazumdar S, Petrelli D, Ciravegna F (2014) Exploring user and system requirements of linked data visualization through a visual dashboard approach. Semantic Web 5(3):203–220
Bangor A, Kortum P, Miller J (2009) Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 4(3):114–123
Zeis C, Regassa H, Shah A, Ahmadain A (2001) Goodness-of-fit test for rating scale data: applying the minimum chi-sqaure method. J Econ Soc Meas 27(1–2):25–39