Lessons Learned from Two Cohorts of Personal Informatics Self-Experiments

by | Sep 1, 2017 | Research Publications

Author(s):

  • Nediyana Daskalova
  • Karthik Desingh
  • Alexandra Papoutsaki
  • Diane Schulze
  • Han Sha
  • Jeff Huang

Abstract:

Self-experiments allow people to investigate their own individual outcomes from behavior change, often with the aid of personal tracking devices. The challenge is to design scientifically valid self-experiments that can reach conclusive results. In this paper, we aim to understand how novices run self-experiments when they are provided with a structured lesson in experimental design. We conducted a study on self-experimentation with two cohorts of students, where a total of 34 students performed a self-experiment of their choice. In the first cohort, students were given only two restrictions: a specific number of variables to track and a set duration for the study. The findings from this cohort helped us generate concrete guidelines for running a self-experiment, and use them as the format for the next cohort. A second cohort of students used these guidelines to conduct their own self-experiments in a more structured manner. Based on the findings from both cohorts, we propose a set of guidelines for running successful self-experiments that address the pitfalls encountered by students in the study, such as inadequate study design and analysis methods. We also discuss broader implications for future self-experimenters and designers of tools for self-experimentation.

Documentation:

https://doi.org/10.1145/3130911

References:
  1. 2015. Fitbit. (2015). https://www.fitbit.com/
  2. Shipra Agrawal and Navin Goyal. 2012. Analysis of Thompson Sampling for the Multi-armed Bandit Problem. In Proceedings of COLT. 39–1.
  3. David H. Barlow, Nock K. Matthew, and Michel Hersen. 2008. Single case experimental designs: Strategies for studying behavior for change. Pearson.
  4. Colin Barr, Maria Marois, Ida Sim, Christopher H. Schmid, Barth Wilsey, Deborah Ward, Naihua Duan, Ron D. Hays, Joshua Selsky, Joseph Servadio, and others. 2015. The PREEMPT study-evaluating smartphone-assisted n-of-1 trials in patients with chronic pain: study protocol for a randomized controlled trial. Trials 16, 1 (2015), 67.
  5. Frank Bentley, Konrad Tollmar, Peter Stephenson, Laura Levy, Brian Jones, Scott Robertson, Ed Price, Richard Catrambone, and Jeff Wilson. 2013. Health Mashups: Presenting Statistical Patterns Between Wellbeing Data and Context in Natural Language to Promote Behavior Change. ACM Trans. Comput.-Hum. Interact. 20, 5, Article 30 (Nov. 2013), 27 pages.
  6. Olivier Chapelle and Lihong Li. 2011. An empirical evaluation of thompson sampling. In Advances in neural information processing systems. 2249–2257.
  7. Eun Kyoung Choe, Nicole B. Lee, Bongshin Lee, Wanda Pratt, and Julie A. Kientz. 2014. Understanding Quantified-selfers’ Practices in Collecting and Exploring Personal Data. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’14). 1143–1152.
  8. Nediyana Daskalova, Danaë Metaxa-Kakavouli, Adrienne Tran, Nicole Nugent, Julie Boergers, John McGeary, and Jeff Huang. 2016. SleepCoacher: A Personalized Automated Self-Experimentation System for Sleep Recommendations. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 347–358.
  9. Joan Didion. 1968. On Keeping a Notebook. (1968). Retrieved September 24, 2015 from https://penusa.org/sites/default/files/didion.pdf.
  10. Naihua Duan, Ian Eslick, N Gabler, Heather Kaplan, Richard Kravitz, Eric Larson, Wilson Pace, Christopher Schmid, Ida Sim, and Sunita Vohra. 2014. Design and Implementation of N-of-1 Trials: A User’s Guide. Agency for Healthcare Research and Quality. www.effectivehealthcare.ahrq.gov/N-1-Trials.cfm
  11. Eugene S. Edgington. 1987. Randomized single-subject experiments and statistical tests. Journal of Counseling Psychology 34, 4 (1987), 437–442.
  12. Garabed Eknoyan. 1999. Santorio Sanctorius (1561–1636)-founding father of metabolic balance studies. American journal of nephrology 19, 2 (1999), 226–233.
  13. Daniel A. Epstein, An Ping, James Fogarty, and Sean A. Munson. 2015. A lived informatics model of personal informatics. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 731–742.
  14. Deborah Estrin and Ida Sim. 2010. Open mHealth Architecture: An Engine for Health Care Innovation. Science 330, 6005 (2010), 759–760. arXiv:http://www.sciencemag.org/content/330/6005/759.full.pdf
  15. BJ Fogg. 2015. Tiny Habits. (2015). Retrieved September 24, 2015 from http://tinyhabits.com/.
  16. Martin J. Gardner and Douglas G. Altman. 1986. Confidence intervals rather than P values: estimation rather than hypothesis testing. Br Med J (Clin Res Ed) 292, 6522 (1986), 746–750.
  17. Anne Germain, Robin Richardson, Douglas E Moul, Oommen Mammen, Gretchen Haas, Steven D Forman, Noelle Rode, Amy Begley, and Eric A Nofzinger. 2012. Placebo-controlled comparison of prazosin and cognitive-behavioral treatments for sleep disturbances in US Military Veterans. Journal of psychosomatic research 72, 2 (2012), 89–96.
  18. Andria Hanbury, Katherine Farley, Carl Thompson, Paul M. Wilson, Duncan Chambers, and Heather Holmes. 2013. Immediate versus sustained effects: interrupted time series analysis of a tailored intervention. Implementation Science 8, 1 (2013), 130–147.
  19. Eric B Hekler, Winslow Burleson, and Jisoo Lee. 2013. A DIY Self-Experimentation Toolkit for Behavior Change. In Personal Informatics in the Wild: Hacking Habits for Health 8 Happiness at the ACM-CHI Conference. ACM.
  20. Mieke Heyvaert and Patrick Onghena. 2014. Randomization tests for single-case experiments: State of the art, state of the science, and state of the application. Journal of Contextual Behavioral Science 3, 1 (2014), 51–64.
  21. Jim A. Horne and Olov Ostberg. 1976. A self-assessment questionnaire to determine morningness-eveningness in human circadian rhythms. International Journal of Chronobiology 4, 2 (1976), 97–110.
  22. Hsiu-Fang Hsieh and Sarah E Shannon. 2005. Three approaches to qualitative content analysis. Qualitative health research 15, 9 (2005), 1277–1288.
  23. Bradley E. Huitema, Ron Van Houten, and Hana Manal. 2014. Time-series intervention analysis of pedestrian countdown timer effects. Accident Analysis 8 Prevention 72 (2014), 23–31.
  24. W Paul Jones. 2003. Single-case time series with Bayesian analysis: a practitioner’s guide. (Methods, Plainly Speaking). Measurement and evaluation in counseling and development 36, 1 (2003), 28–40.
  25. Ravi Karkar, Jasmine Zia, Roger Vilardaga, Sonali R. Mishra, James Fogarty, Sean A. Munson, and Julie A. Kientz. 2015. A framework for self-experimentation in personalized health. Journal of the American Medical Informatics Association (2015), ocv150.
  26. Matthew Kay, Gregory L Nelson, and Eric B Hekler. 2016. Researcher-centered design of statistics: Why Bayesian statistics better fit the culture and incentives of HCI. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 4521–4532.
  27. Thomas R. Kratochwill, John H. Hitchcock, Robert H. Horner, Joel R. Levin, Samuel L. Odom, David M. Rindskopf, and William R. Shadish. 2013. Single-Case Intervention Research Design Standards. Remedial and Special Education 34, 1 (2013), 26–38. arXiv:http://rse.sagepub.com/content/34/1/26.full.pdf+html
  28. Quantified Self Labs. 2012. About the Quantified Self. (2012). Retrieved September 08, 2015 from http://quantifiedself.com/about/.
  29. Gary P Latham. 2003. Goal Setting:: A Five-Step Approach to Behavior Change. Organizational Dynamics 32, 3 (2003), 309–318.
  30. Jisoo Lee, Erin Walker, Winslow Burleson, Matthew Kay, Matthew Buman, and Eric B Hekler. 2017. Self-experimentation for behavior change: Design and formative evaluation of two approaches. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 6837–6849.
  31. Ian Li, Anind Dey, and Jodi Forlizzi. 2010. A Stage-based Model of Personal Informatics Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10). 557–566.
  32. Allen Neuringer. 1981. Self-experimentation: A call for change. Behaviorism 9, 1 (1981), 79–94.
  33. Gemma Phillips, Lambert Felix, Leandro Galli, Vikram Patel, and Philip Edwards. 2010. The effectiveness of M-health technologies for improving health and health services: a systematic review protocol. BMC Research Notes 3, 1 (2010), 250.
  34. Seth Roberts. 2004. Self-experimentation as a source of new ideas: Ten examples about sleep, mood, health, and weight. Behavioral and Brain Sciences 27, 2 (2004), 227 — 288.
  35. Seth Roberts. 2010. The unreasonable effectiveness of my self-experimentation. Medical hypotheses 75, 6 (2010), 482–489.
  36. Seth Roberts. 2012. The reception of my self-experimentation. Journal of Business Research 65, 7 (2012), 1060–1066.
  37. Steven L Scott. 2010. A modern Bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry 26, 6 (2010), 639–658.
  38. William R. Shadish, David M. Rindskopf, and Larry V. Hedges. 2008. The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention 2, 3 (2008), 188–196.
  39. Justin D. Smith. 2012. Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods 17, 4 (2012), 510–550.
  40. John B. Todman and Pat Dugard. 2001. Single-case and small-n experimental designs: A practical guide to randomization tests. Psychology Press.
  41. Allen B. Weisse. 2012. Self-experimentation and its role in medical research. Texas Heart Institute Journal 39, 1 (2012), 51–54.
  42. Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z Gajos, Walter S Lasecki, and Neil Heffernan. 2016. Axis: Generating explanations at scale with learnersourcing and machine learning. In Proceedings of the Third (2016) ACM Conference on Learning@ Scale. ACM, 379–388.