Assessing self-selection biases in Facebook-recruited online surveys: Evidence from the COVID-19 Health Behavior Survey.

During the COVID-19 pandemic, many primary data collection efforts relied on online surveys via social media recruitment. According to the leverage-salience theory, respondents' varying interest in the survey topic can lead to differential survey responses, potentially introducing biases. In this study, we investigate the potential impact of displaying the survey topic in the survey recruitment materials on survey responses. We use data from the "COVID-19 Health Behavior Survey", a cross-national online survey that we ran between March and August 2020 in eight countries in Europe and North America (N=120,184). Respondents were recruited via targeted advertisements placed on Facebook with varying degrees of reference to the survey topic of COVID-19. The aim of our study is to assess whether stronger (or weaker) topic salience in the ad images is associated with higher (or lower) threat perceptions of COVID-19 and the adoption of preventive behaviors, including face mask use and increased hand-washing. Regression analyses show that in 20 of the 32 models, ad images had no significant effect on the survey outcomes. Factors like the month of survey participation or respondents' age were more influential. In the remaining models, where unexplained image effects persisted, the impact was minimal. While mask-wearing images were generally associated with lower threat perceptions of COVID-19 to oneself and the family, we found no consistent pattern for the adoption of protective behaviors. Overall, our findings do not provide consistent evidence that higher topic salience in our Facebook-based recruitment materials systematically influenced survey responses. However, in specific countries, certain recruitment images were linked to variations in COVID-19 threat perception and uptake of preventive behaviors. These context-specific effects highlight the importance of careful recruitment design for Facebook-based surveys during health crises.