When Fitness Trackers Don’t ‘Fit’: End-User Difficulties in the Assessment of Personal Tracking Device Accuracy

by | Sep 7, 2015 | Research Publications

Author(s):

  • Rayoung Yang
  • Eunice Shin
  • Mark W. Newman
  • Mark S. Ackerman

Abstract:

Personal tracking technologies allow users to monitor and reflect on their physical activities and fitness. However, users are uncertain about how accurately their devices track their data. In order to better understand this challenge, we analyzed 600 product reviews and conducted 24 interviews with tracking device users. In this paper, we describe what methods users used to assess accuracy of their tracking devices and identify seven problems they encountered. We found that differences in users’ expectations, physical characteristics, types of activities and lifestyle led them to have different perceptions of the accuracy of their devices. With the absence of sound mental models and unclear understanding of the concepts of accuracy and experimental controls, users designed faulty tests and came to incorrect conclusions. We propose design recommendations to better support end-users’ efforts to assess and improve the accuracy of their tracking devices as required to suit their individual characteristics and purposes.

Documentation:

https://doi.org/10.1145/2750858.2804269

References:
  1. Victoria Bellotti, Maribeth Back, W. Keith Edwards, Rebecca E. Grinter, Austin Henderson, and Cristina Lopes. 2002. Making sense of sensing systems: five questions for designers and researchers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’02), 415–422. http://dl.acm.org/citation.cfm?id=503450
  2. Victoria Bellotti and Keith Edwards. 2001. Intelligibility and accountability: human considerations in context-aware systems. Human–Computer Interaction 16, 2–4, 193–212.
  3. Eun Kyoung Choe, Nicole B. Lee, Bongshin Lee, Wanda Pratt, and Julie A. Kientz. 2014. Understanding quantified-selfers’ practices in collecting and exploring personal data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), 1143–1152. http://dl.acm.org/citation.cfm?id=2557372
  4. Sunny Consolvo, David W. McDonald, Tammy Toscos, et al. 2008. Activity sensing in the wild: a field trial of ubifit garden. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’08) 1797–1806. http://dl.acm.org/citation.cfm?id=1357335
  5. Cara Bailey Fausset, Tracy L. Mitzner, Chandler E. Price, Brian D. Jones, Brad W. Fain, and Wendy A. Rogers. 2013. Older Adults’ Use of and Attitudes toward Activity Monitoring Technologies. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, SAGE Publications, 1683–1687. http://pro.sagepub.com/content/57/1/1683.short
  6. Robert M. Fein, Gary M. Olson, and Judith S. Olson. 1993. A mental model can help with learning to operate a complex device. INTERACT’93 and CHI’93 conference companion on Human factors in computing systems, 157–158. http://dl.acm.org/citation.cfm?id=260170
  7. Thomas Fritz, Elaine M. Huang, Gail C. Murphy, and Thomas Zimmermann. 2014. Persuasive technology in the real world: a study of long-term use of activity sensing devices for fitness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), 487–496. http://dl.acm.org/citation.cfm?id=2557383
  8. Matthew Kay, Dan Morris, Julie A. Kientz, and others. 2013. There’s no such thing as gaining a pound: reconsidering the bathroom scale user interface. In Proceedings of the ACM international joint conference on Pervasive and ubiquitous computing (UbiComp ’13), 401–410. http://dl.acm.org/citation.cfm?id=2493456
  9. Willett Kempton. 1986. Two Theories of Home Heat Control*. Cognitive Science 10, 1, 75–90.
  10. Joshua Klayman and Young-Won Ha. 1987. Confirmation, disconfirmation, and information in hypothesis testing. Psychological review 94, 2, 211.
  11. Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of Explanatory Debugging to Personalize Interactive Machine Learning. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ’15), 126–137. http://doi.acm.org/10.1145/2678025.2701399
  12. Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan. 2012. Tell me more?: the effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12), 1–10. http://dl.acm.org/citation.cfm?id=2207678
  13. Nicholas D. Lane, Ye Xu, Hong Lu, et al. 2014. Community Similarity Networks. Personal Ubiquitous Comput. 18, 2 (February 2014), 355–368. http://dx.doi.org/10.1007/s00779-013-0655-1
  14. Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. 2015. Working with Machines: The Impact of Algorithmic and Data-Driven Management on Human Workers. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’15), 1603–1612. http://doi.acm.org/10.1145/2702123.2702548
  15. Ian Li, Anind K. Dey, and Jodi Forlizzi. 2011. Understanding my data, myself: supporting self-reflection with ubicomp technologies. In Proceedings of the international conference on Ubiquitous computing (UbiComp ’11), 405–414. http://dl.acm.org/citation.cfm?id=2030166
  16. Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. 2009. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), 2119–2128. http://dl.acm.org/citation.cfm?id=1519023
  17. Molly Mackinlay. 2013. Phases of Accuracy Diagnosis:(In) visibility of System Status in the Fitbit. Intersect: The Stanford Journal of Science, Technology and Society 6, 2. http://ojs.stanford.edu/ojs/index.php/intersect/article/view/555
  18. Antonio Menditto, Marina Patriarca, and Bertil Magnusson. 2007. Understanding the meaning of accuracy, trueness and precision. Accreditation and quality assurance 12, 1, 45–47.
  19. John Rooksby, Mattias Rost, Alistair Morrison, and Matthew Chalmers Chalmers. 2014. Personal tracking as lived informatics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14), 1163–1172. http://dl.acm.org/citation.cfm?id=2557039
  20. Patrick C. Shih, Kyungsik Han, Erika Shehan Poole, Mary Beth Rosson, and John M. Carroll. 2015. Use and Adoption Challenges of Wearable Activity Trackers. Proc. iConference. http://hdl.handle.net/2142/73649
  21. Rayoung Yang and Mark W. Newman. 2013. Learning from a learning thermostat: lessons for intelligent systems for the home. In Proceedings of the international conference on Ubiquitous computing (UbiComp ’13), 93–102. http://dl.acm.org/citation.cfm?id=2493489
  22. Main Report | Pew Research Center’s Internet & American Life Project. Retrieved February 21, 2015 from http://www.pewinternet.org/2013/01/28/main-report-8/#fn-87-3
  23. Fitness Tracker Comparison Chart – Best Fitness Tracker Reviews. Retrieved February 21, 2015 from http://www.bestfitnesstrackerreviews.com/comparison-chart.html
  24. Basis | The Ultimate Fitness and Sleep Tracker. Retrieved February 21, 2015 from https://www.mybasis.com/
  25. For Wearables, Accurate Sensing Is Tricky | MIT Technology Review. Retrieved June 23, 2015 from http://www.technologyreview.com/review/538416/the-struggle-for-accurate-measurements-on-your-wrist/
  26. Sleep-tracking gadgets raise awareness – and skepticism. Retrieved February 21, 2015 http://www.usatoday.com/story/news/nation/2013/03/24/sleep-tracking-devices/2007085/
  27. ISO 5725-1:1994(en) Accuracy (trueness and precision) of measurement methods and results — Part 1: General principles and definitions. Retrieved July 1, 2015 from https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en