Pearl: A Technology Probe for Machine-Assisted Reflection on Personal Data

by | Mar 1, 2023 | Research Publications

Author(s):

  • Matthew Jörke
  • Yasaman S. Sefidgar
  • Talie Massachi
  • Jina Suh
  • Gonzalo Ramos

Abstract:

Reflection on one’s personal data can be an effective tool for supporting wellbeing. However, current wellbeing reflection support tools tend to offer a one-size-fits-all approach, ignoring the diversity of people’s wellbeing goals and their agency in the self-reflection process. In this work, we identify an opportunity to help people work toward their wellbeing goals by empowering them to reflect on their data on their own terms. Through a formative study, we inform the design and implementation of Pearl, a workplace wellbeing reflection support tool that allows users to explore their personal data in relation to their wellbeing goal. Pearl is a calendar-based interactive machine teaching system that allows users to visualize data sources and tag regions of interest on their calendar. In return, the system provides insights about these tags that can be saved to a reflection journal. We used Pearl as a technology probe with 12 participants without data science expertise and found that all participants successfully gained insights into their workplace wellbeing. In our analysis, we discuss how Pearl’s capabilities facilitate insights, the role of machine assistance in the self-reflection process, and the data sources that participants found most insightful. We conclude with design dimensions for intelligent reflection support systems as inspiration for future work.

Documentation:

https://doi.org/10.1145/3581641.3584054

References:
  1. 2021. Workplace Stress. https://www.stress.org/workplace-stress
  2. Charles Abraham and Susan Michie. 2008. A taxonomy of behavior change techniques used in interventions.Health psychology 27, 3 (2008), 379.
  3. Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza. 2014. Power to the people: The role of humans in interactive machine learning. Ai Magazine 35, 4 (2014), 105–120.
  4. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13.
  5. Bon Adriel Aseniero, Charles Perin, Wesley Willett, Anthony Tang, and Sheelagh Carpendale. 2020. Activity river: Visualizing planned and logged personal activities for reflection. In Proceedings of the International Conference on Advanced Visual Interfaces. 1–9.
  6. Kirstie Ball. 2010. Workplace surveillance: An overview. Labor History 51, 1 (2010), 87–106.
  7. Eric PS Baumer. 2015. Reflective informatics: conceptual dimensions for designing technologies of reflection. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 585–594.
  8. Eric PS Baumer, Sherri Jean Katz, Jill E Freeman, Phil Adams, Amy L Gonzales, John Pollak, Daniela Retelny, Jeff Niederdeppe, Christine M Olson, and Geri K Gay. 2012. Prescriptive persuasion and open-ended social awareness: expanding the design space of mobile health. In Proceedings of the ACM 2012 conference on computer supported cooperative work. 475–484.
  9. Eric PS Baumer, Vera Khovanskaya, Mark Matthews, Lindsay Reynolds, Victoria Schwanda Sosik, and Geri Gay. 2014. Reviewing reflection: on the use of reflection in interactive system design. In Proceedings of the 2014 conference on Designing interactive systems. 93–102.
  10. Marit Bentvelzen, Jasmin Niess, Mikołaj P Woźniak, and Paweł W Woźniak. 2021. The Development and Validation of the Technology-Supported Reflection Inventory. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–8.
  11. Marit Bentvelzen, Paweł W Woźniak, Pia SF Herbes, Evropi Stefanidi, and Jasmin Niess. 2022. Revisiting Reflection in HCI: Four Design Resources for Technologies that Support Reflection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (2022), 1–27.
  12. Kirsten Boehner, William Gaver, and Andy Boucher. 2012. 14 Probes. Inventive Methods: The happening of the social 185 (2012).
  13. Kirsten Boehner, Janet Vertesi, Phoebe Sengers, and Paul Dourish. 2007. How HCI interprets the probes. In Proceedings of the SIGCHI conference on Human factors in computing systems. 1077–1086.
  14. Jenna Butler and Sonia Jaffe. 2021. Challenges and gratitude: A diary study of software engineers working from home during covid-19 pandemic. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 362–363.
  15. Eun Kyoung Choe, Saeed Abdullah, Mashfiqui Rabbi, Edison Thomaz, Daniel A Epstein, Felicia Cordeiro, Matthew Kay, Gregory D Abowd, Tanzeem Choudhury, James Fogarty, 2017. Semi-automated tracking: a balanced approach for self-monitoring applications. IEEE Pervasive Computing 16, 1 (2017), 74–84.
  16. Eun Kyoung Choe, Bongshin Lee, 2015. Characterizing visualization insights from quantified selfers’ personal data presentations. IEEE computer graphics and applications 35, 4 (2015), 28–37.
  17. Eun Kyoung Choe, Bongshin Lee, Matthew Kay, Wanda Pratt, and Julie A Kientz. 2015. SleepTight: low-burden, self-monitoring technology for capturing and reflecting on sleep behaviors. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 121–132.
  18. Eun Kyoung Choe, Bongshin Lee, Haining Zhu, Nathalie Henry Riche, and Dominikus Baur. 2017. Understanding self-reflection: how people reflect on personal data through visual data exploration. In Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare. 173–182.
  19. Thomas W Colligan and Eileen M Higgins. 2006. Workplace stress: Etiology and consequences. Journal of workplace behavioral health 21, 2 (2006), 89–97.
  20. Victor P Cornet and Richard J Holden. 2018. Systematic review of smartphone-based passive sensing for health and wellbeing. Journal of biomedical informatics 77 (2018), 120–132.
  21. Vedant Das Swain, Victor Chen, Shrija Mishra, Stephen M Mattingly, Gregory D Abowd, and Munmun De Choudhury. 2022. Semantic Gap in Predicting Mental Wellbeing through Passive Sensing. In CHI Conference on Human Factors in Computing Systems. 1–16.
  22. Nediyana Daskalova, Eindra Kyi, Kevin Ouyang, Arthur Borem, Sally Chen, Sung Hyun Park, Nicole Nugent, and Jeff Huang. 2021. Self-E: Smartphone-Supported Guidance for Customizable Self-Experimentation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 227, 13 pages. https://doi.org/10.1145/3411764.3445100
  23. Chris Elsden, Abigail C Durrant, and David S Kirk. 2016. It’s Just My History Isn’t It? Understanding smart journaling practices. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2819–2831.
  24. Chris Elsden, David S Kirk, and Abigail C Durrant. 2016. A quantified past: Toward design for remembering with personal informatics. Human–Computer Interaction 31, 6 (2016), 518–557.
  25. Daniel A Epstein. 2015. Personal informatics in everyday life. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers. 429–434.
  26. Jerry Alan Fails and Dan R Olsen Jr. 2003. Interactive machine learning. In Proceedings of the 8th international conference on Intelligent user interfaces. 39–45.
  27. Rowanne Fleck. 2012. Rating reflection on experience: A case study of teachers’ and tutors’ reflection around images. Interacting with computers 24, 6 (2012), 439–449.
  28. Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. Design: cultural probes. interactions 6, 1 (1999), 21–29.
  29. Lars Grammel, Melanie Tory, and Margaret-Anne Storey. 2010. How information visualization novices construct visualizations. IEEE transactions on visualization and computer graphics 16, 6(2010), 943–952.
  30. Hayley Guillou, Kevin Chow, Thomas Fritz, and Joanna McGrenere. 2020. Is your time well spent? reflecting on knowledge work more holistically. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–9.
  31. Lars Hallnäs and Johan Redström. 2001. Slow technology–designing for reflection. Personal and ubiquitous computing 5, 3 (2001), 201–212.
  32. Kristina Höök. 2000. Steps to take before intelligent user interfaces become real. Interacting with computers 12, 4 (2000), 409–426.
  33. Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 159–166.
  34. Dandan Huang, Melanie Tory, and Lyn Bartram. 2016. A Field Study of On-Calendar Visualizations. In Proceedings of the 42nd Graphics Interface Conference. 13–20.
  35. Hilary Hutchinson, Wendy Mackay, Bo Westerlund, Benjamin B Bederson, Allison Druin, Catherine Plaisant, Michel Beaudouin-Lafon, Stéphane Conversy, Helen Evans, Heiko Hansen, 2003. Technology probes: inspiring design for and with families. In Proceedings of the SIGCHI conference on Human factors in computing systems. 17–24.
  36. Jodi Kantor and Arya Sundaram. 2022. The Rise of the Worker Productivity Score. https://www.nytimes.com/interactive/2022/08/14/business/worker-productivity-tracking.html
  37. Ashish Kapoor and Eric Horvitz. 2008. Experience sampling for building predictive user models: a comparative study. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 657–666.
  38. Harmanpreet Kaur, Daniel McDuff, Alex C Williams, Jaime Teevan, and Shamsi T Iqbal. 2022. “I Didn’t Know I Looked Angry”: Characterizing Observed Emotion and Reported Affect at Work. In CHI Conference on Human Factors in Computing Systems. 1–18.
  39. Rafal Kocielnik, Daniel Avrahami, Jennifer Marlow, Di Lu, and Gary Hsieh. 2018. Designing for workplace reflection: a chat and voice-based conversational agent. In Proceedings of the 2018 designing interactive systems conference. 881–894.
  40. Wouter Kool and Matthew Botvinick. 2018. Mental labour. Nature human behaviour 2, 12 (2018), 899–908.
  41. Carol Collier Kuhlthau. 1999. The role of experience in the information search process of an early career information worker: Perceptions of uncertainty, complexity, construction, and sources. Journal of the American Society for information Science 50, 5(1999), 399–412.
  42. Ian Li, Anind Dey, and Jodi Forlizzi. 2010. A stage-based model of personal informatics systems. In Proceedings of the SIGCHI conference on human factors in computing systems. 557–566.
  43. Ian Li, Anind K Dey, and Jodi Forlizzi. 2011. Understanding my data, myself: supporting self-reflection with ubicomp technologies. In Proceedings of the 13th international conference on Ubiquitous computing. 405–414.
  44. Madelene Lindström, Anna Ståhl, Kristina Höök, Petra Sundström, Jarmo Laaksolathi, Marco Combetto, Alex Taylor, and Roberto Bresin. 2006. Affective diary: designing for bodily expressiveness and self-reflection. In CHI’06 extended abstracts on Human factors in computing systems. 1037–1042.
  45. Stephen M Mattingly 2019. The Tesserae Project: Large-Scale, Longitudinal, In Situ, Multimodal Sensing of Information Workers. In CHI Ext. Abstracts.
  46. Daniel McDuff, Amy Karlson, Ashish Kapoor, Asta Roseway, and Mary Czerwinski. 2012. AffectAura: an intelligent system for emotional memory. In Proceedings of the SIGCHI conference on human factors in computing systems. 849–858.
  47. Andre N Meyer, Gail C Murphy, Thomas Zimmermann, and Thomas Fritz. 2017. Design recommendations for self-monitoring in the workplace: Studies in software development. Proceedings of the ACM on Human-Computer Interaction 1, CSCW(2017), 1–24.
  48. Kiran Mirchandani. 1998. Protecting the boundary: Teleworker insights on the expansive concept of “work”. Gender & Society 12, 2 (1998), 168–187.
  49. William C Murray and Adam Rostis. 2007. Who’s running the machine? A theoretical exploration of work stress and burnout of technologically tethered workers. Journal of individual employment rights 12, 3 (2007), 249–263.
  50. Arpit Narechania, Arjun Srinivasan, and John Stasko. 2020. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. IEEE Transactions on Visualization and Computer Graphics 27, 2(2020), 369–379.
  51. Felicia Ng, Jina Suh, and Gonzalo Ramos. 2020. Understanding and supporting knowledge decomposition for machine teaching. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 1183–1194.
  52. Donald A. Norman. 1994. How Might People Interact with Agents. Commun. ACM 37, 7 (jul 1994), 68–71. https://doi.org/10.1145/176789.176796
  53. Afarin Pirzadeh, Li He, and Erik Stolterman. 2013. Personal informatics and reflection: a critical examination of the nature of reflection. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. 1979–1988.
  54. Bernd Ploderer, Wolfgang Reitberger, Harri Oinas-Kukkonen, and Julia van Gemert-Pijnen. 2014. Social interaction and reflection for behaviour change., 1667–1676 pages.
  55. Mashfiqui Rabbi, Shahid Ali, Tanzeem Choudhury, and Ethan Berke. 2011. Passive and in-situ assessment of mental and physical well-being using mobile sensors. In Proceedings of the 13th international conference on Ubiquitous computing. 385–394.
  56. Gonzalo Ramos, Chris Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. 2020. Interactive machine teaching: a human-centered approach to building machine-learned models. Human–Computer Interaction (April 2020). https://www.microsoft.com/en-us/research/publication/interactive-machine-teaching-a-human-centered-approach-to-building-machine-learned-models/
  57. Amon Rapp and Federica Cena. 2016. Personal informatics for everyday life: How users without prior self-tracking experience engage with personal data. International Journal of Human-Computer Studies 94 (2016), 1–17.
  58. Patrice Y Simard, Saleema Amershi, David M Chickering, Alicia Edelman Pelton, Soroush Ghorashi, Christopher Meek, Gonzalo Ramos, Jina Suh, Johan Verwey, Mo Wang, 2017. Machine teaching: A new paradigm for building machine learning systems. arXiv preprint arXiv:1707.06742(2017).
  59. Petr Slovák, Christopher Frauenberger, and Geraldine Fitzpatrick. 2017. Reflective practicum: A framework of sensitising concepts to design for transformative reflection. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2696–2707.
  60. Poorna Talkad Sukumar, Anind Dey, Gloria Mark, Ronald Metoyer, and Aaron Striegel. 2021. Triggers and Barriers to Insight Generation in Personal Visualizations. In Graphics Interface 2022.
  61. Alice Thudt, Dominikus Baur, Samuel Huron, and Sheelagh Carpendale. 2015. Visual mementos: Reflecting memories with personal data. IEEE transactions on visualization and computer graphics 22, 1(2015), 369–378.
  62. Niels Van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2017. The experience sampling method on mobile devices. ACM Computing Surveys (CSUR) 50, 6 (2017), 1–40.
  63. Ariane G Wepfer, Tammy D Allen, Rebecca Brauchli, Gregor J Jenny, and Georg F Bauer. 2018. Work-life boundaries and well-being: Does work-to-life integration impair well-being through lack of recovery?Journal of Business and Psychology 33, 6 (2018), 727–740.