Myscéal: A Deeper Analysis of an Interactive Lifelog Search Engine

by | Mar 22, 2023 | Research Publications

Author(s):

  • Ly-Duyen Tran
  • Manh-Duy Nguyen
  • Binh T. Nguyen
  • Liting Zhou 

Abstract:

There is a growing number of lifelogging retrieval systems that have been introduced in several lifelogging workshops and sessions. Across all systems at the LSC, which is an annual international challenge about lifelogging retrieval, our Myscéal is currently considered as the state-of-the-art. In this paper, we describe the system in detail and show how it has been upgraded through time since firstly introduced in 2020. In addition, we analyse Myscéal performance not only in the three lifelog retrieval competitions it participated in but also with additional user experiments. The result shows that the fast searching time of Myscéal is the system’s most important feature that helps it get some significant advantages in competitions. On the other hand, the findings from user experiments indicate that Myscéal still needs some improvements for novice users who are unfamiliar with how to interact with the system. Moreover, the user study plays a vital role in the development of Myscéal as many updates of this system came from the feedback of the participating users. We also demonstrate the efficacy of Myscéal as a lifelog retrieval system to help the lifeloggers, who capture their daily life in images, recall memorable moments in their massive lifelog archives.

Documentation:

https://doi.org/10.1007/s11042-023-15078-6

References:
  1. Alam N, Graham Y, Gurrin C (2021) Memento: a prototype lifelog search engine for LSC’21. In: Proceedings of the 4th annual on lifelog search challenge. LSC ’21, Association for Computing Machinery, New York, pp 53–58
  2. Alateeq A, Roantree M, Gurrin C (2021) Voxento 2.0: a prototype voice-controlled interactive search engine for lifelogs. In: Proceedings of the 4th annual on lifelog search challenge. LSC ’21, Association for Computing Machinery, New York, pp 65–70
  3. Anderson P, He X, Buehler C, Teney D, Johnson M, Gould S, Zhang L (2018) Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6077–6086
  4. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818
  5. Dang Nguyen DT, Piras L, Riegler M, Boato G, Zhou L, Gurrin C (2017) Overview of imageclef lifelog 2017: lifelog retrieval and summarization
  6. de Oliveira Barra G, Cartas Ayala A, Bolaños M, Dimiccoli M, Giró Nieto X, Radeva P (2016) Lemore: a lifelog engine for moments retrieval at the ntcir-lifelog lsat task. In: Proceedings of the 12th NTCIR conference on evaluation of information access technologies
  7. Duane A, Gurrin C, Huerst W (2018) Virtual reality lifelog explorer: lifelog search challenge at acm icmr 2018. In: Proceedings of the 2018 ACM workshop on the lifelog search challenge. LSC’18, pp 20–23
  8. Gurrin C, Smeaton AF, Doherty AR et al (2014) Lifelogging: personal big data. Found and Trends®; Inf Retrieval 8(1):1–125Article  Google Scholar 
  9. Gurrin C, Joho H, Hopfgartner F, Zhou L, Albatal R (2016) Overview of ntcir-12 lifelog task. NTCIR
  10. Gurrin C, Schoeffmann K, Joho H, Leibetseder A, Zhou L, Duane A, Nguyen D, Tien D, Riegler M, Piras L et al (2019) Comparing approaches to interactive lifelog search at the lifelog search challenge (LSC 2018). ITE Trans Media Technol Appl 7(2):46–59Article  Google Scholar 
  11. Kovalčík G, Škrhak V, Souček T, Lokoč J (2020) Viret tool with advanced visual browsing and feedback. In: Proceedings of the 3rd annual workshop on lifelog search challenge. LSC ’20, Association for Computing Machinery, New York, pp 63–66
  12. Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li L-J, Shamma DA et al (2017) Visual genome: connecting language and vision using crowdsourced dense image annotations. Int J Comput Vision 123(1):32–73Article  MathSciNet  Google Scholar 
  13. Le N-K, Nguyen D-H, Nguyen V-T, Tran M-T (2019) Lifelog moment retrieval with advanced semantic extraction and flexible moment visualization for exploration. In: CLEF (working notes)
  14. Leibetseder A, Schoeffmann K (2020) Lifexplore at the lifelog search challenge 2020. In: Proceedings of the 3rd annual workshop on lifelog search challenge. LSC ’20, Association for Computing Machinery, New York, pp 37–42
  15. Lokoč J, Mejzlík F, Veselỳ P, Souček T (2021) Enhanced somhunter for known-item search in lifelog data. In: Proceedings of the 4th annual on lifelog search challenge. LSC’21, pp 71–73
  16. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the 7th IEEE international conference on computer vision, vol 2, pp 1150–11572
  17. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110Article  Google Scholar 
  18. Luo H, Wei H, Lai LL (2011) Creating efficient visual codebook ensembles for object categorization. IEEE Trans Syst Man Cybern Part A Syst Humans 41(2):238–253Article  Google Scholar 
  19. Mejzlík F, Veselý P, Kratochvíl M, Souček T, Lokoč J (2020) Somhunter for lifelog search. In: Proceedings of the 3rd annual workshop on lifelog search challenge. LSC ’20, Association for Computing Machinery, New York, pp 73–75
  20. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119
  21. Nguyen T-N, Le T-K, Ninh V-T, Tran M-T, Thanh Binh N, Healy G, Caputo A, Gurrin C (2021) Lifeseeker 3.0: an interactive lifelog search engine for LSC’21. In: Proceedings of the 4th annual on lifelog search challenge, pp 41–46
  22. Nguyen M-D, Nguyen BT, Gurrin C (2021) Graph-based indexing and retrieval of lifelog data. In: International conference on multimedia modeling. MMM’21. Springer, pp 256–267
  23. Oram P (2001) Wordnet: an electronic lexical database. christiane fellbaum (ed.). Cambridge, ma: Mit press, 1998. pp 423. Appl Psycholinguist 22(1):131–134Article  Google Scholar 
  24. Rossetto L, Gasser R, Heller S, Amiri Parian M, Schuldt H (2019) Retrieval of structured and unstructured data with vitrivr. In: Proceedings of the ACM workshop on lifelog search challenge. LSC’19, pp 27–31
  25. Rossetto L, Baumgartner M, Gasser R, Heitz L, Wang R, Bernstein A (2021) Exploring graph-querying approaches in lifegraph. In: Proceedings of the 4th annual on lifelog search challenge. LSC ’21, Association for Computing Machinery, New York, pp 7–10
  26. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
  27. Tran L-D, Nguyen M-D, Binh NT, Lee H, Gurrin C (2020) Myscéal: an experimental interactive lifelog retrieval system for LSC’20. In: Proceedings of the 3rd annual workshop on lifelog search challenge, pp 23–28
  28. Tran L-D, Nguyen M-D, Nguyen BT, Gurrin C (2020) An experiment in interactive retrieval for the lifelog moment retrieval task at imageCLEFlifelog2020. In: CLEF (working notes), p 12
  29. Tran L-D, Nguyen M-D, Thanh Binh N, Lee H, Gurrin C (2021) Myscéal 2.0: a revised experimental interactive lifelog retrieval system for LSC’21. In: Proceedings of the 4th annual on lifelog search challenge. LSC’21, pp 11–16
  30. Tran L-D, Nguyen M-D, Thanh Binh N, Lee H, Gurrin C (2022) E-myscéal: embedding-based interactive lifelog retrieval system for LSC’22. In: Proceedings of the 5th annual lifelog search challenge
  31. Trang-Trung H-P, Le H-A, Tran M-T (2020) Lifelog moment retrieval with self-attention based joint embedding model. In: CLEF (working notes)