Using narratives and numbers in performance prediction: Attitudes, confidence, and validity

Author(s):

• A. Susan M. Niessen
• Edgar E. Kausel
• Marvin Neumann

Abstract:

In a preregistered prediction-task experiment, we investigated the effect of narrative versus quantified information on decision-maker perceptions, confidence, predictor weighting, and predictive accuracy when making performance predictions. We also investigated the effect of who quantifies information (the decision maker or someone else). As expected, we found higher perceived informativeness and use intentions for narrative than quantified information. Information presented narratively was also weighted somewhat more heavily than quantified information. Using quantitative information quantified by decision makers themselves yielded higher perceived autonomy and use intentions than quantitative information quantified by someone else. However, no differences in prediction confidence were found and self- and other-produced quantifications received identical weight. Moreover, unexpectedly, differences in weighting did not translate to differences in predictive accuracy.

Documentation:

https://doi.org/10.1111/ijsa.12364

References:

• Allen, M., & Preiss, R. W. (1997). Comparing the persuasiveness of narrative and statistical evidence using meta-analysis. Communication Research Reports, 14, 125– 131. https://doi.org/10.1080/08824099709388654CrossrefGoogle Scholar
• Anderson, N., Salgado, J. F., & Hülsheger, U. R. (2010). Applicant reactions in selection: Comprehensive meta-analysis into reaction generalization versus situational specificity. International Journal of Selection and Assessment, 18, 291– 304. https://doi.org/10.1111/j.1468-2389.2010.00512.xWiley Online LibraryPubMedWeb of Science®Google Scholar
• Armstrong, J. S. (2001). Judgmental bootstrapping: Inferring experts’ rules for forecasting. In J. S. Armstrong (Ed.), Principles of Forecasting. International series in operations research & management science (pp. 171– 192). Springer. https://doi.org/10.1007/978-0-306-47630-3_9CrossrefGoogle Scholar
• Beach, L. R. (2010). The psychology of narrative thought. Xlibris. Google Scholar
• Bonaccio, S., & Dalal, R. S. (2006). Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences. Organizational Behavior and Human Decision Processes, 101, 127– 151. https://doi.org/10.1016/j.obhdp.2006.07.001CrossrefWeb of Science®Google Scholar
• Boswell, W. R., Roehling, M. V., LePine, M. A., & Moynihan, L. M. (2003). Individual job-choice decisions and the impact of job attributes and recruitment practices: A longitudinal field study. Human Resource Management, 42, 23– 37. https://doi.org/10.1002/hrm.10062Wiley Online LibraryWeb of Science®Google Scholar
• Brunswik, E. (1956). Perception and the representative design of psychological experiments ( 2nd ed.). University of California Press. CrossrefGoogle Scholar
• Dalal, D. K., Sassaman, L., & Zhu, X. (2020). The impact of nondiagnostic information on selection decision making: A cautionary note and mitigation strategies. Personnel Assessment and Decisions, 6, 54– 64. https://doi.org/10.25035/pad.2020.02.007CrossrefGoogle Scholar
• Dalal, R. S., & Bonaccio, S. (2010). What types of advice do decision-makers prefer? Organizational Behavior and Human Decision Processes, 112, 11– 23. https://doi.org/10.1016/j.obhdp.2009.11.007CrossrefWeb of Science®Google Scholar
• Dana, J., Dawes, R., & Peterson, N. (2013). Belief in the unstructured interview: The persistence of an illusion. Judgment and Decision Making, 8, 512– 520. Web of Science®Google Scholar
• Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571– 582. https://doi.org/10.1037/0003-066X.34.7.571CrossrefWeb of Science®Google Scholar
• Deci, E. L., & Ryan, R. M. (1991). A motivational approach to self: Integration in personality. In R. A. Dienstbier (Ed.), Nebraska symposium on motivation, 1990: Perspectives on motivation (pp. 237– 288). University of Nebraska Press. Web of Science®Google Scholar
• Dipboye, R. L. (1997). Structured selection interviews: Why do they work? Why are they underutilized? In N. Anderson, & P. Herriott (Eds.), International handbook of selection and assessment (pp. 455– 474). Wiley. Google Scholar
• Dipboye, R. L., & Jackson, S. L. (1999). Interviewer experience and expertise effects. In W. Eder, & M. M. Harris (Eds.), The employment interview handbook (pp. 259– 278). Sage Publications Inc. CrossrefGoogle Scholar
• Goldberg, L. R. (1970). Man versus model of man: A rationale, plus some evidence, for a method of improving on clinical inferences. Psychological Bulletin, 73, 422– 432. https://doi.org/10.1037/h0029230CrossrefWeb of Science®Google Scholar
• Grömping, U. (2006). Relative importance for linear regression in R: The package relaimpo. Journal of Statistical Software, 17, 1– 27. https://doi.org/10.18637/jss.v017.i01CrossrefWeb of Science®Google Scholar
• Hall, C. C., Ariss, L., & Todorov, A. (2007). The illusion of knowledge: When more information reduces accuracy and increases confidence. Organizational Behavior and Human Decision Processes, 103, 277– 290. https://doi.org/10.1016/j.obhdp.2007.01.003CrossrefWeb of Science®Google Scholar
• Highhouse, S. (1997). Understanding and improving job-finalist choice: The relevance of behavioral decision research. Human Resource Management Review, 7, 449– 470. https://doi.org/10.1016/S1053-4822(97)90029-2CrossrefWeb of Science®Google Scholar
• Highhouse, S. (2008). Stubborn reliance on intuition and subjectivity in employee selection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 333– 342. https://doi.org/10.1111/j.1754-9434.2008.00058.xWiley Online LibraryWeb of Science®Google Scholar
• Highhouse, S., Brooks, M. E., Nesnidol, S., & Sim, S. (2017). Is a .51 validity coefficient good? Value sensitivity for interview validity. International Journal of Selection and Assessment, 25, 383– 389. https://doi.org/10.1111/ijsa.12192Wiley Online LibraryWeb of Science®Google Scholar
• Highhouse, S., Freier, L. M., Stevenor, B. A., Shea, M. A., Childers, M., & Melick, S. R. (2021). Failure to replicate the basic dilution effect in performance prediction. International Journal of Selection and Assessment. Published online September 20, 2021. https://doi.org/10.1111/ijsa.12344Wiley Online LibraryWeb of Science®Google Scholar
• Highhouse, S., & Kostek, J. A. (2013). Holistic assessment for selection and placement. In K. F. Geisinger, B. A. Bracken, J. F. Carlson, J.-I. C. Hansen, N. R. Kuncel, S. P. Reise, & M. C. Rodriguez (Eds.), APA handbook of testing and assessment in psychology. Test theory and testing and assessment in industrial and organizational psychology (Vol. 1, pp. 565– 577). American Psychological Association. https://doi.org/10.1037/14047-031CrossrefGoogle Scholar
• Hsee, C. K., Loewenstein, G., Blount, S., & Bazerman, M. (1999). Preference reversals between joint and separate evaluations of options: A theoretical analysis. Psychological Bulletin, 125(5), 576– 590. https://doi.org/10.1037/0033-2909.125.5.576CrossrefWeb of Science®Google Scholar
• Hsee, C. K., & Zhang, J. (2004). Distinction bias: Misprediction and mischoice due to joint evaluation. Journal of Personality and Social Psychology, 86, 680– 695. https://doi.org/10.1037/0022-3514.86.5.680CrossrefPubMedWeb of Science®Google Scholar
• Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2014). Moving forward indirectly: Reanalyzing the validity of employment interviews with indirect range restriction methodology. International Journal of Selection and Assessment, 22, 297– 309. https://doi.org/10.1111/ijsa.12078Wiley Online LibraryWeb of Science®Google Scholar
• Kahneman, D. (2011). Thinking fast and slow. Farrar, Straus and Giroux. Google Scholar
• Karelaia, N., & Hogarth, R. M. (2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychological Bulletin, 134, 404– 426. https://doi.org/10.1037/0033-2909.134.3.404CrossrefPubMedWeb of Science®Google Scholar
• Kausel, E. E., Culbertson, S. S., & Madrid, H. P. (2016). Overconfidence in personnel selection: When and why unstructured interview information can hurt hiring decisions. Organizational Behavior and Human Decision Processes, 137, 27– 44. https://doi.org/10.1016/j.obhdp.2016.07.005CrossrefWeb of Science®Google Scholar
• Kemmelmeier, M. (2004). Separating the wheat from the chaff: Does discriminating between diagnostic and nondiagnostic information eliminate the dilution effect? Journal of Behavioral Decision Making, 17, 231– 243. https://doi.org/10.1002/bdm.473Wiley Online LibraryWeb of Science®Google Scholar
• Kuncel, N. R. (2008). Some new (and old) suggestions for improving personnel selection. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 343– 346. https://doi.org/10.1111/j.1754-9434.2008.00059.xWiley Online LibraryWeb of Science®Google Scholar
• Kuncel, N. R. (2018). Judgment and decision making in staffing research and practice. In D. S. Ones, N. Anderson, C. Viswesvaran, & H. K. Sinangil (Eds.), The SAGE handbook of industrial, work & organizational psychology: Personnel psychology and employee performance (Vol. 1, 2nd ed., pp. 474– 488). Sage Reference. CrossrefGoogle Scholar
• Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical versus clinical data combination in selection and admissions decisions: A meta-analysis. Journal of Applied Psychology, 98, 1060– 1072. https://doi.org/10.1037/a0034156CrossrefPubMedWeb of Science®Google Scholar
• Lievens, F., & De Paepe, A. (2004). An empirical investigation of interviewer-related factors that discourage the use of high structure interviews. Journal of Organizational Behavior, 25, 29– 46. https://doi.org/10.1002/job.246Wiley Online LibraryWeb of Science®Google Scholar
• Longoni, C., Bonezzi, A., & Morewedge, C. K. (2019). Resistance to medical artificial intelligence. Journal of Consumer Research, 46, 629– 650. https://doi.org/10.1093/jcr/ucz013CrossrefWeb of Science®Google Scholar
• Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press. https://doi.org/10.1037/11281-000CrossrefGoogle Scholar
• Montoya, A. K., & Hayes, A. F. (2017). Two-condition within-participant statistical mediation analysis: A path-analytic framework. Psychological Methods, 22, 6– 27. https://doi.org/10.1037/met0000086CrossrefPubMedWeb of Science®Google Scholar
• Morris, S. B., Daisley, R. L., Wheeler, M., & Boyer, P. (2015). A meta-analysis of the relationship between individual assessments and job performance. Journal of Applied Psychology, 100, 5– 20. https://doi.org/10.1037/a0036938CrossrefPubMedWeb of Science®Google Scholar
• Neumann, M., Hengeveld, M., Niessen, A. S. M., Tendeiro, J. N., & Meijer, R. R. (2021). Education increases decision-rule use: An investigation of education and incentives to improve decision making. Journal of Experimental Psychology: Applied, 1– 13. https://doi.org/10.1037/xap0000372PubMedWeb of Science®Google Scholar
• Neumann, M., Niessen, A. S. M., & Meijer, R. R. (2021). Implementing evidence-based assessment and selection in organizations: A review and an agenda for future research. Organizational Psychology Review, 11, 205– 239. https://doi.org/10.1177/2041386620983419CrossrefWeb of Science®Google Scholar
• Newman, D. T., Fast, N. J., & Harmon, D. J. (2020). When eliminating bias isn’t fair: Algorithmic reductionism and procedural justice in human resource decisions. Organizational Behavior and Human Decision Processes, 160, 149– 167. https://doi.org/10.1016/j.obhdp.2020.03.008CrossrefWeb of Science®Google Scholar
• Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2017). Applying organizational justice theory to admission into higher education: Admission from a student perspective. International Journal of Selection and Assessment, 25, 72– 84. https://doi.org/10.1111/ijsa.12161Wiley Online LibraryWeb of Science®Google Scholar
• Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2018). Admission testing for higher education: A multi-cohort study on the validity of high-fidelity curriculum-sampling tests. PLOS One, 13(6), e0198746. https://doi.org/10.1371/journal.pone.0198746CrossrefPubMedWeb of Science®Google Scholar
• Nisbett, R. E., Zukier, H., & Lemley, R. E. (1981). The dilution effect: Nondiagnostic information weakens the implications of diagnostic information. Cognitive Psychology, 13, 248– 277. https://doi.org/10.1016/0010-0285(81)90010-4CrossrefWeb of Science®Google Scholar
• Nolan, K. P., Dalal, D. K., & Carter, N. (2020). Threat of technological unemployment, use intentions, and the promotion of structured interviews in personnel selection. Personnel Assessment and Decisions, 6, 38– 53. https://doi.org/10.25035/pad.2020.02.006CrossrefGoogle Scholar
• Nolan, K. P., & Highhouse, S. (2014). Need for autonomy and resistance to standardized employee selection practices. Human Performance, 27, 328– 346. https://doi.org/10.1080/08959285.2014.929691CrossrefWeb of Science®Google Scholar
• Pennington, N., & Hastie, R. (1992). Explaining the evidence: Tests of the story model for juror decision making. Journal of Personality and Social Psychology, 62, 189– 206. https://doi.org/10.1037/0022-3514.62.2.189CrossrefWeb of Science®Google Scholar
• Preacher, K. J., & Kelley, K. (2011). Effect size measures for mediation models: Quantitative strategies for communicating indirect effects. Psychological Methods, 16, 93– 115. https://doi.org/10.1037/a0022658CrossrefPubMedWeb of Science®Google Scholar
• Revelle, W. (2019). Psych: Procedures for psychological, psychometric, and personality research. R package version 1.9.12. https://CRAN.R-project.org/package=psychGoogle Scholar
• Ryan, A. M., & Sackett, P. R. (1987). A survey of individual assessment practices by I/O psychologists. Personnel Psychology, 40, 455– 488. https://doi.org/10.1111/j.1744-6570.1987.tb00610.xWiley Online LibraryWeb of Science®Google Scholar
• Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals’ beliefs about effective human resource practices: Correspondence between research and practice. Human Resource Management, 41, 149– 174. https://doi.org/10.1002/hrm.10029Wiley Online LibraryWeb of Science®Google Scholar
• Sackett, P. R., Shewach, O. R., & Keiser, H. N. (2017). Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity. Journal of Applied Psychology, 102, 1435– 1447. https://doi.org/10.1037/apl0000236CrossrefPubMedWeb of Science®Google Scholar
• Sanfey, A., & Hastie, R. (1998). Does evidence presentation format affect judgment? An experimental evaluation of displays of data for judgments. Psychological Science, 8, 99– 103. https://doi.org/10.1111/1467-9280.00018CrossrefWeb of Science®Google Scholar
• Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262– 274. https://doi.org/10.1037/0033-2909.124.2.262CrossrefWeb of Science®Google Scholar
• Slaughter, J. E., & Kausel, E. E. (2013). Employee selection decisions. In S. Highhouse, R. S. Dalal, & E. Salas (Eds.), Judgment and decision making at work (pp. 57– 79). Routledge/Taylor & Francis Group. Google Scholar
• Steiger, J. H. (1980). Testing pattern hypotheses on correlation matrices: Alternative statistics and some empirical results. Multivariate Behavioral Research, 15, 335– 352. https://doi.org/10.1207/s15327906mbr1503_7CrossrefCASPubMedWeb of Science®Google Scholar
• Tonidandel, S., & LeBreton, J. M. (2015). RWA Web: A free, comprehensive, web-based, and user-friendly tool for relative weight analyses. Journal of Business and Psychology, 30, 207– 216. https://doi.org/10.1007/s10869-014-9351-zCrossrefWeb of Science®Google Scholar
• Troutman, C. M., & Shanteau, J. (1977). Inferences based on nondiagnostic information. Organizational Behavior & Human Performance, 19, 43– 55. https://doi.org/10.1016/0030-5073(77)90053-8CrossrefWeb of Science®Google Scholar
• Van Iddekinge, C. H., & Ployhart, R. E. (2008). Developments in the criterion-related validation of selection procedures: A critical review and recommendations for practice. Personnel Psychology, 61, 871– 925. https://doi.org/10.1111/j.1744-6570.2008.00133.xWiley Online LibraryWeb of Science®Google Scholar
• Waller, W. S., & Zimbelman, M. F. (2003). A cognitive footprint in archival data: Generalizing the dilution effect from laboratory to field settings. Organizational Behavior and Human Decision Processes, 91, 254– 268. https://doi.org/10.1016/S0749-5978(03)00024-4CrossrefWeb of Science®Google Scholar
• Winterbottom, A., Bekker, H. L., Conner, M., & Mooney, A. (2008). Does narrative information bias individual’s decision making? A systematic review. Social Science & Medicine, 67, 2079– 2088. https://doi.org/10.1016/j.socscimed.2008.09.037CrossrefPubMedWeb of Science®Google Scholar
• Witt, J. K. (2019). Graph construction: An empirical investigation on setting the range of the y-axis. Meta-Psychology, 3, 1– 20. https://doi.org/10.5626/MP.2018.895CrossrefGoogle Scholar
• Yaniv, I. (2004). The benefit of additional opinions. Current Directions in Psychological Science, 13, 75– 78. https://doi.org/10.1111/j.0963-7214.2004.00278.xCrossrefWeb of Science®Google Scholar
• Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision making: Egocentric discounting and reputation formation. Organizational Behavior and Human Decision Processes, 83, 260– 281. https://doi.org/10.1006/obhd.2000.2909CrossrefCASPubMedWeb of Science®Google Scholar
• Yu, M. C. (2018). Viewing expert judgment in individual assessments through the Lens Model: Testing the limits of expert information processing (Doctoral dissertation). Minneapolis, MN: University of Minnesota. Google Scholar
• Zhang, D. C., Zhu, X., Ritter, K. J., & Thiele, A. (2019). Telling stories to communicate the value of the pre-employment structured job interview. International Journal of Selection and Assessment, 27, 299– 314. https://doi.org/10.1111/ijsa.12264Wiley Online LibraryWeb of Science®Google Scholar