2021- Reprints: Technology and Information Systems
Who is a better decision maker? Data-driven expert ranking under unobserved quality, Production and Operations Management, 30(1), 127-144, 2021
T. Geva and M. Saar-Tsechansk
(Reprint no. 376)
Research no.: 00970100
The capacity to rank expert workers by their decision quality is a key managerial task of substantial significance to business operations. However, when no ground truth information is available on experts’ decisions, the evaluation of expert workers typically requires enlisting peer-experts, and this form of evaluation is prohibitively costly in many important settings. In this work, we develop a data-driven approach for producing effective rankings based on the decision quality of expert workers; our approach leverages historical data on past decisions, which are commonly available in organizational information systems. Specifically, we first formulate a new business data science problem: Ranking Expert decision makers’ unobserved decision Quality (REQ) using only historical decision data and excluding evaluation by peer experts. The REQ problem is challenging because the correct decisions in our settings are unknown (unobserved) and because some of the information used by decision makers might not be available for retrospective evaluation. To address the REQ problem, we develop a machine-learning-based approach and analytically and empirically explore conditions under which our approach is advantageous. Our empirical results over diverse settings and datasets show that our method yields robust performance: Its rankings of expert workers are consistently either superior or at least comparable to those obtained by the best alternative approach. Accordingly, our method constitutes a de facto benchmark for future research on the REQ problem.
Modeling social distancing strategies to prevent SARS-CoV-2 spread in Israel: A cost-effectiveness analysis, Value in Health, 24(5), 607-614, 2021
A. Shlomai, A. Leshno, E. H. Sklan and M. Leshno
(Reprint no. 379)
Research no. : 05121100
Objectives: While highly effective in preventing SARS-CoV-2 spread, national lockdowns come with an enormous economic price. Few countries have adopted an alternative “testing, tracing, and isolation” approach to selectively isolate people at high exposure risk, thereby minimizing the economic impact. To assist policy makers, we performed a cost-effectiveness analysis of these 2 strategies.
Methods: A modified Susceptible, Exposed, Infectious, Recovered, and Deceased (SEIRD) model was employed to assess the situation in Israel, a small country with ~9 million people. The incremental cost-effectiveness ratio (ICER) of these strategies as well as the expected number of infected individuals and deaths were calculated.
Results: A nationwide lockdown is expected to save, on average, 274 (median 124, interquartile range: 71-221) lives compared to the “testing, tracing, and isolation” approach. However, the ICER will be, on average, $45 104 156 (median $49.6 million, interquartile range: 22.7-220.1) to prevent 1 case of death.
Conclusion: A national lockdown has a moderate advantage in saving lives with tremendous costs and possible overwhelming economic effects. These findings should assist decision makers dealing with additional waves of this pandemic.
The impact of social vs. nonsocial referring channels on online news consumption, Management Science, 67(4), 2420-2447, April 2021
S. Bar-Gill, Y. Inbar and S. Reichman
(Reprint no. 383)
Research no. : 01990100
The digitization of news markets has created a key role for online referring channels. This research combines field and laboratory experiments and analysis of large-scale clickstream data to study the effects of social versus nonsocial referral sources on news consumption in a referred news website visit. We theorize that referrer-specific browsing modes and referrer-induced news consumption thresholds interact to impact news consumption in referred visits to an online newspaper and that news sharing motivations invoked by the referral source impact sharing behavior in these referred visits. We find that social media referrals promote directed news consumption—visits with fewer articles, shorter durations, yet higher reading completion rates—compared with nonsocial referrals. Furthermore, social referrals invoke weaker informational sharing motivations relative to nonsocial referrals, thus leading to a lower news sharing propensity relative to nonsocial referrals. The results highlight how news consumption changes when an increasing amount of traffic is referred by social media, provide insights applicable to news outlets’ strategies, and speak to ongoing debates regarding biases arising from social media’s growing importance as an avenue for news consumption.
The design of reciprocal learning between human and artificial intelligence, Proceedings of the ACM, CSCW, 2021
A. Zagalsky, D. Te’eni, I. Yahav, D. Schwartz, G. Silverman, D. Cohen, Y. Mann & D. Lewinsky
(Reprint No. 387)
Research no. : 05621100
The need for advanced automation and artificial intelligence (AI) in various fields, including text classification, has dramatically increased in the last decade, leaving us critically dependent on their performance and reliability. Yet, as we increasingly rely more on AI applications, their algorithms are becoming more nuanced, more complex, and less understandable precisely at a time we need to understand them better and trust them to perform as expected. Text classification in the medical and cybersecurity domains are good examples of this. Human experts lack the capacity to deal with the high volume and velocity of data that needs to be classified, and ML techniques are often unexplainable and lack the ability to capture the required context needed to make the right decision and take action. We propose a new abstract configuration of Human-Machine Learning (HML) that focuses on reciprocal learning, where the human and the AI are collaborating partners. We employ design-science research (DSR) to learn and design an HML configuration, which incorporates software to support combining human and artificial intelligences. We define the HML configuration by its conceptual components and their function. We then describe the development of a system called Fusion that supports human-machine reciprocal learning. Using two case studies of text classification from the cyber domain, we evaluate Fusion and the proposed HML approach, demonstrating benefits and challenges. Our results show a clear ability of domain experts to improve the ML classification performance over time, while both human and machine, collaboratively, develop their conceptualization, i.e., their knowledge of classification. We generalize our insights from the DSR process as actionable principles for researchers and designers of ’human in the learning loop’ systems. We conclude the paper by discussing HML configurations and the challenge of capturing and representing knowledge gained jointly by human and machine, an area we feel has great potential.
