Using LLMs to Measure Sensitive Topics

Abstract

Measuring sensitive topics remains a challenge in marketing research. Researchers often rely on surveys, but these may yield biased data when participants refuse to participate, skip sensitive questions, or provide socially desirable answers because of privacy concerns. Large Language Models (LLMs) might offer a new avenue to conduct sensitive marketing research, since LLM-generated survey responses have shown considerable promise in the social sciences in terms of validity and cost effectiveness. Thus, we investigate whether LLM-generated responses to sensitive questions can serve as an effective alternative to human responses. For both approaches, responses were collected via direct and indirect questioning techniques. Three studies evaluate the distributions of simulated responses, their covariate relationships, and their alignment with empirical estimates from the human responses.

Our findings reveal significant limitations of simulated responses from the leading off-the-shelf LLMs. First, these responses display social desirability bias, which can be observed both in prevalence estimates, and in their relationships with covariates. Second, applying indirect questioning techniques in the prompts does little to reduce this bias. Additional strategies, such as conditioning on respondent background variables in the prompt, chain-of-thought reasoning, few-shot learning, using an ensemble of prompts or a suite of LLMs, proved ineffective as well. In light of these limitations, we suggest several new dimensions that affect the usefulness of LLMs for sensitive topics, and we propose a future research agenda.