FoodScrap: Promoting Rich Data Capture and Reflective Food Journaling Through Speech Input

by | Jun 28, 2021 | Research Publications

Author(s):

  • Yuhan Luo
  • Young-Ho Kim
  • Bongshin Lee
  • Naeemul Hassan
  • Eun Kyoung Choe

Abstract:


The factors influencing people’s food decisions, such as one’s mood and eating environment, are important information to foster self-reflection and to develop personalized healthy diet. But, it is difficult to consistently collect them due to the heavy data capture burden. In this work, we examine how speech input supports capturing everyday food practice through a week-long data collection study (N = 11). We deployed FoodScrap, a speech-based food journaling app that allows people to capture food components, preparation methods, and food decisions. Using speech input, participants detailed their meal ingredients and elaborated their food decisions by describing the eating moments, explaining their eating strategy, and assessing their food practice. Participants recognized that speech input facilitated self-reflection, but expressed concerns around re-recording, mental load, social constraints, and privacy. We discuss how speech input can support low-burden and reflective food journaling and opportunities for effectively processing and presenting large amounts of speech data.

Documentation:

https://doi.org/10.1145/3461778.3462074

References:
  1. Mohd Khanapi Abd Ghani and Ika Novita Dewi. 2012. Comparing speech recognition and text writing in recording patient health records. In 2012 IEEE-EMBS Conference on Biomedical Engineering and Sciences. IEEE, 365–370. https://doi.org/10.1109/IECBES.2012.6498100
  2. Susan Albers. 2012. Eating mindfully: How to end mindless eating and enjoy a balanced relationship with food. New Harbinger Publications.
  3. Ferran Altarriba Bertran, Samvid Jhaveri, Rosa Lutz, Katherine Isbister, and Danielle Wilde. 2019. Making sense of human-food interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3290605.3300908
  4. Amazon.com, Inc.2021. Amazon Alexa. Retrieved April 26, 2021 from https://www.amazon.com/b?node=21576558011
  5. Norman Azah Anir, Hairul Nizam, and Azmi Masliyana. 2008. The users perceptions and opportunities in Malaysia in introducing RFID system for Halal food tracking. WSEAS Transactions on information science and applications 5, 5(2008), 843–852.
  6. Apple, Inc.2021. Siri. Retrieved April 26, 2021 from https://www.apple.com/siri/
  7. Gastón Ares, Franco Mawad, Ana Giménez, and Alejandro Maiche. 2014. Influence of rational and intuitive thinking styles on food choice: Preliminary evidence from an eye-tracking study with yogurt labels. Food Quality and Preference 31 (2014), 28–37. https://doi.org/10.1016/j.foodqual.2013.07.005
  8. Els Bilman, Ellen van Kleef, and Hans van Trijp. 2017. External cues challenging the internal appetite control system—overview and practical implications. Critical reviews in food science and nutrition 57, 13(2017), 2825–2834. https://doi.org/10.1080/10408398.2015.1073140
  9. Johnna Blair, Yuhan Luo, Ning F Ma, Sooyeon Lee, and Eun Kyoung Choe. 2018. OneNote Meal: A photo-based diary study for reflective meal tracking. In AMIA Annual Symposium Proceedings, Vol. 2018. American Medical Informatics Association, 252. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6371351/
  10. Marcela CC Bomfim, Sharon I Kirkpatrick, Lennart E Nacke, and James R Wallace. 2020. Food literacy while shopping: Motivating informed food purchasing behaviour with a situated gameful app. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3313831.3376801
  11. Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
  12. Lora E Burke, Melanie Warziski, Terry Starrett, Jina Choo, Edvin Music, Susan Sereika, Susan Stark, and Mary Ann Sevick. 2005. Self-monitoring dietary intake: current and future practices. Journal of Renal Nutrition 15, 3 (2005), 281–290. https://doi.org/10.1016/j.jrn.2005.04.002
  13. Barbara L Chalfonte, Robert S Fish, and Robert E Kraut. 1991. Expressive richness: a comparison of speech and text as media for revision. In Proceedings of the 1991 CHI Conference on Human Factors in Computing Systems. 21–26. https://doi.org/10.1145/108844.108848
  14. Eun Kyoung Choe, Bongshin Lee, Matthew Kay, Wanda Pratt, and Julie A Kientz. 2015. SleepTight: low-burden, self-monitoring technology for capturing and reflecting on sleep behaviors. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 121–132. https://doi.org/10.1145/2750858.2804266
  15. Eun Kyoung Choe, Bongshin Lee, and m.c. schraefel. 2015. Characterizing visualization insights from quantified selfers’ personal data presentations. IEEE computer graphics and applications 35, 4 (2015), 28–37. https://doi.org/10.1109/MCG.2015.51
  16. Eun Kyoung Choe, Bongshin Lee, Haining Zhu, Nathalie Henry Riche, and Dominikus Baur. 2017. Understanding self-reflection: how people reflect on personal data through visual data exploration. In Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare. 173–182. https://doi.org/10.1145/3154862.3154881
  17. Chia-Fang Chung, Elena Agapie, Jessica Schroeder, Sonali Mishra, James Fogarty, and Sean A Munson. 2017. When personal tracking becomes social: Examining the use of Instagram for healthy eating. In Proceedings of the 2017 CHI Conference on human factors in computing systems. 1674–1687. https://doi.org/10.1145/3025453.3025747
  18. Chia-Fang Chung, Qiaosi Wang, Jessica Schroeder, Allison Cole, Jasmine Zia, James Fogarty, and Sean A Munson. 2019. Identifying and planning for individualized change: patient-provider collaboration using lightweight food diaries in healthy eating and irritable bowel syndrome. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 3, 1 (2019), 1–27. https://doi.org/10.1145/3314394
  19. Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards, 2019. The state of speech in HCI: Trends, themes and challenges. Interacting with Computers 31, 4 (2019), 349–371. https://doi.org/10.1093/iwc/iwz016
  20. Deborah A Cohen and Susan H Babey. 2012. Contextual influences on eating behaviours: heuristic processing and dietary choices. Obesity Reviews 13, 9 (2012), 766–779. https://doi.org/10.1111/j.1467-789X.2012.01001.x
  21. Frederick G Conrad, Michael F Schober, Christopher Antoun, H Yanna Yan, Andrew L Hupp, Michael Johnston, Patrick Ehlen, Lucas Vickers, and Chan Zhang. 2017. Respondent mode choice in a smartphone survey. Public Opinion Quarterly 81, S1 (2017), 307–337. https://doi.org/10.1093/poq/nfw097
  22. Sunny Consolvo, David W McDonald, Tammy Toscos, Mike Y Chen, Jon Froehlich, Beverly Harrison, Predrag Klasnja, Anthony LaMarca, Louis LeGrand, Ryan Libby, 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the 2008 CHI Conference on Human Factors in Computing Systems. 1797–1806. https://doi.org/10.1145/1357054.1357335
  23. Felicia Cordeiro, Elizabeth Bales, Erin Cherry, and James Fogarty. 2015. Rethinking the mobile food journal: Exploring opportunities for lightweight photo-based capture. In Proceedings of the 2015 CHI Conference on Human Factors in Computing Systems. 3207–3216. https://doi.org/10.1145/2702123.2702154
  24. Felicia Cordeiro, Daniel A Epstein, Edison Thomaz, Elizabeth Bales, Arvind K Jagannathan, Gregory D Abowd, and James Fogarty. 2015. Barriers and negative nudges: Exploring challenges in food journaling. In Proceedings of the 2015 CHI Conference on Human Factors in Computing Systems. 1159–1162. https://doi.org/10.1145/2702123.2702155
  25. Marika De Bruijne and Arnaud Wijnant. 2013. Comparing survey results obtained via mobile devices and computers: An experiment with a mobile web survey on a heterogeneous group of mobile devices versus a computer-assisted web survey. Social Science Computer Review 31, 4 (2013), 482–504. https://doi.org/10.1177/0894439313483976
  26. Yaron D Derman, Tamara Arenovich, and John Strauss. 2010. Speech recognition software and electronic psychiatric progress notes: physicians’ ratings and preferences. BMC medical informatics and decision making 10, 1 (2010), 1–7. https://doi.org/10.1145/1497185.1497286
  27. Daniel PW Ellis and Keansub Lee. 2004. Minimal-impact audio-based personal archives. In Proceedings of the 1st ACM workshop on Continuous archival and retrieval of personal experiences. 39–47. https://doi.org/10.1145/1026653.1026659
  28. Daniel Epstein, Felicia Cordeiro, Elizabeth Bales, James Fogarty, and Sean Munson. 2014. Taming data complexity in lifelogs: exploring visual cuts of personal informatics data. In Proceedings of the 2014 conference on Designing interactive systems. 667–676. https://doi.org/10.1145/2598510.2598558
  29. Clayton Feustel, Shyamak Aggarwal, Bongshin Lee, and Lauren Wilcox. 2018. People like me: Designing for reflection on aggregate cohort data in personal informatics systems. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 3 (2018), 1–21. https://doi.org/10.1145/3264917
  30. Rowanne Fleck and Geraldine Fitzpatrick. 2010. Reflecting on reflection: framing a design landscape. In Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction. 216–223. https://doi.org/10.1145/1952222.1952269
  31. Genesant Technologies, Inc.2021. Talk-to-Track. Retrieved April 26, 2021 from https://www.talktotrack.com/
  32. Google, Inc.2021. Google Assistant. Retrieved April 26, 2021 from https://assistant.google.com/
  33. Mette Holst, Tina Beermann, Marie Nerup Mortensen, Lotte Boa Skadhauge, Marianne Køhler, Karen Lindorff-Larsen, and Henrik Højgaard Rasmussen. 2017. Optimizing protein and energy intake in hospitals by improving individualized meal serving, hosting and the eating environment. Nutrition 34(2017), 14–20. https://doi.org/10.1016/j.nut.2016.05.011
  34. Amy Leung Hui, Gustaaf Sevenhuysen, Dexter Harvey, and Elizabeth Salamon. 2014. Barriers and coping strategies of women with gestational diabetes to follow dietary advice. Women and Birth 27, 4 (2014), 292–297. https://doi.org/10.1016/j.wombi.2014.07.001
  35. Jisu Jung, Lyndal Wellard-Cole, Colin Cai, Irena Koprinska, Kalina Yacef, Margaret Allman-Farinelli, and Judy Kay. 2020. Foundations for systematic evaluation and benchmarking of a mobile food logger in a large-scale nutrition study. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1–25. https://doi.org/10.1145/3397327
  36. Ravi Karkar, Jessica Schroeder, Daniel A Epstein, Laura R Pina, Jeffrey Scofield, James Fogarty, Julie A Kientz, Sean A Munson, Roger Vilardaga, and Jasmine Zia. 2017. Tummytrials: a feasibility study of using self-experimentation to detect individualized food triggers. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 6850–6863. https://doi.org/10.1145/3025453.3025480
  37. Rohit Ashok Khot, Florian Mueller, 2019. Human-food interaction. Foundations and Trends® in Human-Computer Interaction 12, 4(2019), 238–415. http://dx.doi.org/10.1561/1100000074
  38. Young-Ho Kim, Jae Ho Jeon, Eun Kyoung Choe, Bongshin Lee, KwonHyun Kim, and Jinwook Seo. 2016. TimeAware: Leveraging framing effects to enhance personal productivity. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 272–283. https://doi.org/10.1145/2858036.2858428
  39. Young-Ho Kim, Jae Ho Jeon, Bongshin Lee, Eun Kyoung Choe, and Jinwook Seo. 2017. OmniTrack: A flexible self-tracking approach leveraging semi-automated tracking. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 1–28. https://doi.org/10.1145/3130930
  40. Mandy Korpusik and Jim Glass. 2021. Coco Nutrition. https://www.coco-nutrition.com/.
  41. Mandy Korpusik, Salima Taylor, Sai Krupa Das, Cheryl Gilhooly, Susan Roberts, and James Glass. 2019. A food logging system for iOS with natural spoken language meal descriptions (P21-009-19). Current developments in nutrition 3, Supplement_1 (2019), nzz041–P21. https://doi.org/10.1093/cdn/nzz041.P21-009-19
  42. Brian Y Lim, Xinni Chng, and Shengdong Zhao. 2017. Trade-off between automation and accuracy in mobile photo recognition food logging. In Proceedings of the Fifth International Symposium of Chinese CHI. 53–59. https://doi.org/10.1145/3080631.3080640
  43. Yuhan Luo, Bongshin Lee, and Eun Kyoung Choe. 2020. TandemTrack: shaping consistent exercise experience by complementing a mobile app with a smart speaker. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, 1–13. https://doi.org/10.1145/3313831.3376616
  44. Yuhan Luo, Peiyi Liu, and Eun Kyoung Choe. 2019. Co-Designing food trackers with dietitians: Identifying design opportunities for food tracker customization. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3290605.3300822
  45. Holger Lütters, Malte Friedrich-Freksa, and Marc Egger. 2018. Effects of speech assistance in online questionnaires. In General Online Research Conference, Cologne, Germany.
  46. Mark Mirtchouk, Christopher Merck, and Samantha Kleinberg. 2016. Automated estimation of food type and amount consumed from body-worn audio and motion sensors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 451–462. https://doi.org/10.1145/2971648.2971677
  47. Elizabeth L Murnane, Dan Cosley, Pamara Chang, Shion Guha, Ellen Frank, Geri Gay, and Mark Matthews. 2016. Self-monitoring practices, attitudes, and needs of individuals with bipolar disorder: implications for the design of technologies to manage mental health. Journal of the American Medical Informatics Association 23, 3(2016), 477–484. https://doi.org/10.1093/jamia/ocv165
  48. Shuo Niu, D Scott McCrickard, Timothy L Stelter, Alan Dix, and G Don Taylor. 2019. Reorganize Your Blogs: Supporting Blog Re-visitation with Natural Language Processing and Visualization. Multimodal Technologies and Interaction 3, 4 (2019), 66. https://doi.org/10.3390/mti3040066
  49. Arnel B Ocay, Jane M Fernandez, and Thelma D Palaoag. 2017. NutriTrack: Android-based food recognition app for nutrition awareness. In 2017 3rd IEEE International Conference on Computer and Communications (ICCC). IEEE, 2099–2104. https://doi.org/10.1109/CompComm.2017.8322907
  50. Hyungik Oh, Jonathan Nguyen, Soundarya Soundararajan, and Ramesh Jain. 2018. Multimodal food journaling. In Proceedings of the 3rd International Workshop on Multimedia for Personal Health and Health Care. 39–47. https://doi.org/10.1145/3264996.3265000
  51. Ronaldo Parente, Ned Kock, and John Sonsini. 2004. An analysis of the implementation and impact of speech-recognition technology in the healthcare sector. Perspectives in Health Information Management 1 (2004). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2047322/
  52. Somani Patnaik, Emma Brunskill, and William Thies. 2009. Evaluating the accuracy of data collection on mobile phones: A study of forms, SMS, and voice. In 2009 International Conference on Information and Communication Technologies and Development (ICTD). IEEE, 74–84. https://doi.org/10.1109/ICTD.2009.5426700
  53. Bernd Ploderer, Wolfgang Reitberger, Harri Oinas-Kukkonen, and Julia van Gemert-Pijnen. 2014. Social interaction and reflection for behaviour change. Personal and Ubiquitous Computing 18, 7 (2014), 1667–1676. https://doi.org/10.1007/s00779-014-0779-y
  54. Margaret Raber, Tom Baranowski, Karla Crawford, Shreela V Sharma, Vanessa Schick, Christine Markham, Wenyan Jia, Mingui Sun, Emily Steinman, and Joya Chandra. 2020. The Healthy Cooking Index: Nutrition optimizing home food preparation practices across multiple data collection methods. Journal of the Academy of Nutrition and Dietetics (2020). https://doi.org/10.1016/j.jand.2020.01.008
  55. Melanie Revilla, Mick P Couper, Oriol J Bosch, and Marc Asensio. 2020. Testing the use of voice input in a smartphone web survey. Social Science Computer Review 38, 2 (2020), 207–224. https://doi.org/10.1177/0894439318810715
  56. Sherry Ruan, Jacob O Wobbrock, Kenny Liou, Andrew Ng, and James A Landay. 2018. Comparing speech and keyboard text entry for short messages in two languages on touchscreen phones. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–23. https://doi.org/10.1145/3161187
  57. Samsung, Inc.2021. Samsung Voice Recorder. Retrieved April 26, 2021 from https://www.samsung.com/au/support/mobile-devices/using-voice-recorder/
  58. Kshitij Saxena, Robert Diamond, Reid F Conant, Terri H Mitchell, Guido Gallopyn, and Kristin E Yakimow. 2018. Provider adoption of speech recognition and its impact on satisfaction, documentation quality, efficiency, and cost in an inpatient EHR. AMIA Summits on Translational Science Proceedings 2018 (2018), 186. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5961784/
  59. Michael F Schober, Frederick G Conrad, Christopher Antoun, Patrick Ehlen, Stefanie Fail, Andrew L Hupp, Michael Johnston, Lucas Vickers, H Yanna Yan, and Chan Zhang. 2015. Precision and disclosure in text and voice interviews on smartphones. PloS one 10, 6 (2015), e0128337. https://doi.org/10.1371/journal.pone.0128337
  60. Benjamin Schüz, Natalie Schüz, and Stuart G Ferguson. 2015. It’s the power of food: individual differences in food cue responsiveness and snacking in everyday life. International Journal of Behavioral Nutrition and Physical Activity 12, 1(2015), 149. https://doi.org/10.1186/s12966-015-0312-3
  61. Lisa B Signorello, Heather M Munro, Maciej S Buchowski, David G Schlundt, Sarah S Cohen, Margaret K Hargreaves, and William J Blot. 2009. Estimating nutrient intake from a food frequency questionnaire: incorporating the elements of race and geographic region. American journal of epidemiology 170, 1 (2009), 104–111. https://doi.org/10.1093/aje/kwp098
  62. Lucas M Silva, Yuqi Huai, and Daniel A Epstein. 2020. Exploring Voice Input Opportunities in Multimodal Food Journaling. In CHI’20 Workshop on Conversational Agents for Health and Wellbeing.
  63. Rachel Silver, Mandy Korpusik, Salima Taylor, Sai Das, Cheryl Gilhooly, James Glass, and Susan Roberts. 2019. Testing the Validity of a Natural Spoken Language Application for the Self-Monitoring of Daily Dietary Intake (P13-035-19). Current developments in nutrition 3, Supplement_1 (2019), nzz036–P13. https://doi.org/10.1093/cdn/nzz036.P13-035-19
  64. Jolene D Smyth, Don A Dillman, Leah Melani Christian, and Mallory McBride. 2009. Open-ended questions in web surveys: Can increasing the size of answer boxes and providing extra verbal instructions improve response quality?Public Opinion Quarterly 73, 2 (2009), 325–337. https://doi.org/10.1093/poq/nfp029
  65. Jeffery Sobal and Carole A Bisogni. 2009. Constructing food choice decisions. Annals of Behavioral Medicine 38, suppl_1 (2009), s37–s46. https://doi.org/10.1007/s12160-009-9124-5
  66. Hyewon Suh, Nina Shahriaree, Eric B Hekler, and Julie A Kientz. 2016. Developing and validating the user burden scale: A tool for assessing user burden in computing systems. In Proceedings of the 2016 CHI conference on human factors in computing systems. 3988–3999.
  67. TeamViewer. 2021. TeamViewer QuickSupport. Retrieved April 26, 2021 from https://www.teamviewer.com/en-us/info/quicksupport/
  68. Nađa Terzimehić, Christina Schneegass, and Heinrich Hussmann. 2018. Towards finding windows of opportunity for ubiquitous healthy eating interventions. In International Conference on Persuasive Technology. Springer, 99–112. https://doi.org/10.1007/978-3-319-78978-1_8
  69. WA Van Staveren, JO De Boer, and J Burema. 1985. Validity and reproducibility of a dietary history method estimating the usual food intake during one month. The American journal of clinical nutrition 42, 3 (1985), 554–559. https://doi.org/10.1093/ajcn/42.3.554
  70. Stephen H Walsh. 2004. The clinician’s perspective on electronic health records and how they can affect patient care. Bmj 328, 7449 (2004), 1184–1187. https://doi.org/10.1136/bmj.328.7449.1184
  71. Yunlong Wang, Ulrike Pfeil, and Harald Reiterer. 2016. Supporting self-assembly: The IKEA effect on mobile health persuasive technology. In Proceedings of the 2016 ACM Workshop on Multimedia for Personal Health and Health Care. 19–22. https://doi.org/10.1145/2985766.2985775
  72. Matthias Wenzel, Anja Perlich, Julia PA von Thienen, and Christoph Meinel. 2019. New ways of data entry in doctor-patient encounters. In Design Thinking Research. Springer, 159–177. https://doi.org/10.1007/978-3-319-97082-0_9
  73. Yixuan Zhang and Andrea G Parker. 2020. Eat4Thought: A Design of Food Journaling. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8. https://doi.org/10.1145/3334480.3383044
  74. Zoom Video Communications, Inc.2021. Video Conferencing, Web Conferencing, Webinars, Screen Sharing – Zoom. Retrieved April 26, 2021 from https://zoom.us/