2023 - Working Papers: Technology and Information Systems
Responsible AI reviewing and evaluation, 18 pp.
I. Drori, K. Tyser, A. Shporer, M. Udell and D.Te’eni,
(Working Paper No. 2/2023)
Can large language models (LLMs) be used responsibly to accelerate and improve the peer review process? This paper explores the feasibility, opportunities, and risks of using LLMs to review academic submissions. Humans remain in the loop, but reviewing tasks are performed by an LLM. The analysis focuses on tasks designed to evaluate submissions and provide authors with constructive feedback according to predefined criteria, including novelty, soundness, and presentation. We demonstrate the feasibility of using LLMs for reviewing, examine the opportunities and risks, and conclude with recommendations for mitigating these risks. Using the GPT-4 system role as different personas, we simulate the human editorial process, reducing the process time from human months to machine minutes without compromising the evaluation quality and its usefulness. A blind human evaluation of paper reviews by area chairs of the same conference demonstrates that human and LLM reviews are indistinguishable in how well their reviews explain their ratings, how well their reviews guide the authors to improve their paper, and that human and LLM reviews are both highly content specific. This work highlights the potential of LLMs to revolutionize the review process by automation but also underscores the limitations, risks, and challenges, including the potential for misuse and the need for value alignment between humans and LLMs.
Willingness to Use Privacy-Enhancing Technologies (PETs) Under Different Privacy Concerns, 15 pp.
S. Katalan and S. Reichman
(Working Paper No. 5/2023)
Research No.: 01922100
In this study we offer insights into users’ efforts to implement their willingness to protect their privacy, which is process that is typically concealed and unobservable. Specifically, we analyze individuals’ actual use of a genuine privacy-protecting tool that is made available to them, in order to protect what they intentionally or unintentionally disclose. Thus, we extend current privacy research by analyzing behavioral evidence and not only self-reported intentions. We designed a field experiment that raises participants’ privacy awareness, provides access to a privacy-enhancing technology (PET) in the form of a genuine free privacy-protecting browser called “Epic1”, and lower the barriers that might impede their use of such tools.
Anyone can be a micro-influencer: The impact of perceived influence on online content exploration and satisfaction, 8 pp.
S. Bar-Gill and G. Oestreicher-Singer
(Working Paper No. 6/2023)
Research Nos.: 06023100; 05023100
In this work we study the impact of awareness of social media influencer status, and its interaction with popularity information on online content exploration, choice, and satisfaction. We find that participants in the perceived micro-influence condition have a higher exploration intensity and a stronger reliance on topic relative to popularity information, than participants in the perceived follower condition. The introduction of popularity information has stronger effects on perceived followers, who tend to follow the crowd, increasing their exploration intensity and reliance on topic, more so than perceived influencers. Interestingly, popularity information reduces perceived influencers’ satisfaction with, and tendency to share and recommend their chosen content, while increasing these for perceived followers.
