Understanding LLM Usage Behavior of Online Survey Participants
Authors: Ruijie Sophia Huang, Joey Li, Monica P. Van, Nayeli Suseth Bravo, Jonathan Q. Li, Matthew L. Lee Published in: Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems (CHI EA ’26), 2026
As large language models (LLMs) become widely accessible, online study participants increasingly use LLM tools such as ChatGPT when completing surveys, especially for open-ended questions. Such use raises concerns about response quality, as LLM-generated text may not accurately reflect participants’ actual lived experiences and opinions. In this pilot study, we capture how participants use AI chatbots when answering survey questions by instrumenting an online survey with a custom AI chatbot side-by-side with survey questions. We investigate how different question types, as well as the effort and personalness of questions, impact participants’ propensity to use the AI chatbot. Preliminary findings suggest that when the AI chatbot is used, participants use initial prompts very similar to the question text and their question responses are very similar to the LLM output. Higher perceived effort is associated with increased odds of LLM use, and anecdotal questions show lower overlap with LLM-generated content than generative questions.