Notes
How credible is ChatGPT? An experimental comparison of ChatGPT, Alexa, and Wikipedia
92Theoretical background: Language models such as ChatGPT have recently improved considerably. However, these models currently often “hallucinate,” that is, they make up information. Prior research has shown that people perceive information as more credible and that people are less able to distinguish between high and low-accuracy information when it is read by a voice assistant vs. presented as a search engine snippet (Gaiser & Utz, in press). The underlying processes are, however, unclear. Elaborate processing of auditive (vs. written) information might be lower, or the dialogic nature of talking with an agent might trigger (para)social processes, thereby increasing perceived credibility. ChatGPT provides written text but shares the dialogic style with voice assistants.
93Overall Research question: How are credibility judgments and identifying low-accuracy information affected by the medium? What are the underlying processes?
94H1: Perceived information credibility is higher when information is received from Alexa compared to Wikipedia.
95RQ1: Is there a difference in perceived information credibility between receiving information from ChatGPT and receiving information from a) Alexa b) Wikipedia?
96H2: The differences in perceived information credibility between low and high-accuracy information are higher when individuals receive information from Wikipedia compared to Alexa. Further RQs: Do social presence, social attraction, elaborate processing, and perceived enjoyment mediate the effects of the medium on credibility?
97Methodological approach: A preregistered online experiment (aim: n = 600) with a 3 (medium: ChatGPT vs. Alexa vs. Wikipedia) x 2 (accuracy: low vs. high) x 2 (set: counterbalancing answers with low/high accuracy)-between-subjects design will be conducted in May 2023. Participants read/listen to answers to six questions and judge the perceived credibility after each presentation. At the end, perceived social presence, enjoyment, processing effort, and social attraction (only for Alexa and ChatGPT) are assessed.
98Credibility / smart speaker / ChatGPT / information processing / social presence