2021 Innovations in Online Research Conference

Ensuring high-quality data has become more challenging over time. ‘Bad-faith responses’, may include intentionally careless responses (e.g., participants responding without reading items or randomly), fraudulent participants (e.g., individuals lying about who they are on the study inclusion items), the same person completing multiple surveys, and bots (computer algorithms or programs) completing multiple surveys or otherwise producing fake data. We discuss management scholars’ perceived prevalence of, and experiences with, bad-faith survey data.

Participants are assumed to Read instructions, Understand them, and also Believe them (RUB assumption). We investigated the validity and necessity of the RUB assumption in online experiments. Specifically, whether responses from participants who read the instructions differed from those who do not read instructions. Based on ten studies (N=2500) we show that the attentive participants behave very similarly to physical lab participants, while the inattentive majorly differ. Interestingly, even using highly qualified participants does not eliminate this bias.

Maintaining data quality in online studies has always been a concern for researchers. This concern has been exacerbated in recent years due to bots and problematic respondents on platforms like Mechanical Turk and Lucid. In this talk Leib describes tools that virtually eliminate problematic data on Mechanical Turk and significantly reduce problematic data on online market research platforms like Lucid and Qualtrics.

When collecting data online, an automated method to quantify relative centrality of participant responses can provide insights for quality assurance. Given a set of textual responses to a survey question, which responses are most normative vs. others? Which represent the greatest outliers? Over a set of questions, which participants provide the most normative or outlying responses overall? How might such automated quantitative assessment inform analysis of responses and participants? In this talk I will present our open source library that enables such centrality measures to be computed in a general way across arbitrary question types.

For online survey and task systems such as Mechanical Turk and CloudResearch, data quality is a hard requirement. A psychological survey is less than worthless if the answers were filled out randomly or in an inattentive manner. If an open-ended question is answered cursorily, with plagiarized answers, or with gibberish (so as to finish a survey and collect payment), the answer is not so useful. We will discuss some of the machine learning/NLP approaches we have applied to free-form responses, to predict respondent quality.

Several existing approaches to mitigating careless responding rely on response content (e.g., bogus items, semantic or psychometric synonyms or antonyms, Mahalanobis distance) yet may introduce systematic biases. I present data from a behavioral approach, measuring individual item timing and persistent, willful rushing, and find support for carelessness as a broader personality trait that should be measured and controlled for, not discarded.

Our most common habits and routines are repeated so often that they demand little thought or reflection of us. This can be a challenge for researchers who often must unpack the most minute everyday detail in the search for opportunity. In this talk, I will discuss how deprivation, in combination with remote research methods, can be used to dislodge people from their routines and thus give us a view into the ‘why’ of today’s most rote routines and the keys to nudging people towards adopting new behaviors tomorrow.

To capture trolling behavior, we created an interactive chat-bot. Participants, who were told they would be interacting with another person, were given three options per interaction cycle, a positive response, a neutral response, and a trolling response. We measured objective trolling as the number of trolling responses selected in the chat-bot. We found evidence of three different types of trolls: (1) Self-aware trolls, (2) fake trolls, and (3), secret trolls. To the best of our knowledge, this is the first study to comprehensively examine trolling from multiple points of view and one of the first to employ the use of interactive chat-bots. Finally, we discuss the next steps of this line of work, and ways to better organically capture instances of online antisocial behaviors.

What problems and opportunities exist in scaling remote research? We’ll answer this question with four unique, post-pandemic case studies.

Reducing face touching could help slow COVID-19’s spread, but it can be difficult to do. We tested whether implementation intentions to reduce either the frequency or duration of face-touching effectively reduced the behavior. In this pre-registered online study, we utilized a novel way to collect behavioral data during a pandemic by using the participants’ webcams to obtain video recordings of them performing three engaging tasks for four minutes each. This procedure allowed us to observe (face-touching) behavior remotely.

We utilized a novel online interview method to assist in validating a new clinical tool for the measurement of trauma symptoms arising from the race-based maltreatment of people of color. We will discuss how we worked with CloudResearch to maximize and prioritize diversity and representation in our racial trauma study for a stronger real-world impact.

Looking behaviour has been a valuable measure in infant studies in various domains of development. Recently, there has been an increasing interest in conducting such studies online as opposed to in-lab. There are many benefits to acquiring infant data via online studies, but the method has unique challenges that have to be addressed. In this presentation, I will talk about the benefits, challenges and potential solutions to acquiring infant data via online measurements.

Positivity resonance, an interpersonal experience characterized by shared positive affect, is hypothesized to build social resources that potentially support health behavior change. This research examines how perceptions of positivity resonance emerge and influence consumers’ health behaviors in a commercial online weight-loss community.

What motivates people to join an environmentally sustainable bank? Drawing from social identity theory and cognitive dissonance theory, three online experiments examined how webpage messaging could be made more persuasive. Studies 1 & 2 demonstrated that using meta-contrast (contrasting our bank’s environmental values vs other bank’s environmental values) was more effective than only stating our bank’s environmental values. Study 3 demonstrated that using induced hypocrisy was more effective for those with stronger vs. weaker environmental identities.

In the last decade, MTurk emerged as a flexible, affordable, and reliable source of human participants and was widely adopted. Yet some have questioned whether researchers should continue using the platform on ethical grounds. Their concern is that people on MTurk are financially insecure, subject to abuse, and earn inhumane wages. We conducted two representative probability surveys of the U.S. MTurk population. The surveys revealed that the financial situation of people on MTurk mirrors the general population, most participants do not find MTurk stressful or requesters abusive, and MTurk offers flexibility and benefits that most people value above other options.

We measured cognitive performance in participants from three self-recruitment websites (MTurk, CloudResearch MTurk Toolkit, and Prolific) and three panel recruitment websites (Lucid, CloudResearch PrimePanels, and Qualtrics Panels). Consistent with established norms, on all six platforms, aging was positively correlated with vocabulary performance, agreeableness, conscientiousness, and conservatism, and negatively related to processing speed and neuroticism. There were some differences between the self-recruited and panel samples.

Over 15 million individuals in the US will need to find a path towards recovery following a suicide attempt. The VA has been at the forefront of integrating theory into mental health care, and is developing a novel recovery-oriented psychotherapy, Continuous Identity Cognitive Therapy (CI-CT), which aims to treat veterans struggling with suicidality. We will discuss our collaboration with CloudResearch to identify a group of Post-Acute Suicidal Episode veterans and pay them to collaborate on CI-CT treatment development.

We use a completely gender-blind online work setting to examine the effect of a covert source of gender inequality: differential pay expectations. Despite the absence of many traditional barriers to gender equality in this online work setting, female workers earn less than male workers. Our findings further reveal that differences in earnings are largely driven by differential pay expectations: Women’s salary expectations are lower than those of men, and these lower pay expectations lead to reduced earnings.

COVID-19 has made the world seem unpredictable. We investigate paranoia and belief updating in an online sample in the U.S. We demonstrate the pandemic increased individuals’ self-rated paranoia and rendered their task-based belief updating more erratic. State-mandated mask-wearing increased paranoia and induced erratic behavior, particularly in states where adherence to mask-wearing rules was poor but where rule following is typically more common. This paranoia may explain the lack of compliance with masks.

In this talk, I will discuss methodological strategies for conducting rigorous and efficient survey research with older adults recruited using online panel aggregators. This talk will focus on: (1) the design of a survey instrument both comprehensible and usable by older adults; (2) rapid collection (within hours) of data from a large number of older adults; and (3) validation of data using attention checks, independent validation of age, and detection of careless responses to ensure data quality.

This talk will cover data collection using Reddit, specifically my experiences 1) assessing the data quality of r/SampleSize relative to Amazon Mechanical Turk and 2) actually using r/SampleSize for my own applied research study (currently under review). Beyond the different strategies that I used to assess data quality, I will also discuss some of the demographic advantages for sampling from r/SampleSize, practical advice for optimizing data collection on Reddit based on experience/literature, and mistakes that I made in my own experiences.

Using social media to find research participants is not the first sampling method research firms think to utilize. However, from our experience, there are several instances where using Facebook, Instagram, or LinkedIn to find qualified participants is better than using a panel or call list. In this presentation, we will share best practices and real-world examples of using paid Facebook ads to find the most difficult to source audiences in market research.

While we can see comment threads under most online news these days, it is difficult to guess how well they represent the whole audience’s ideas. This study addresses the representativeness of online comments employing natural language processing (NLP). Two types of text data sets were collected: real online comments and comments written by survey participants which were assumed to represent the whole audience group’s ideas. The extent of similarity between the real comments and the survey comments was calculated for each political group using a state-of-the-art NLP technique BERT.

In two large-scale longitudinal datasets we investigated ability-related political polarization in responses to the COVID-19 pandemic. We observed more polarization with greater ability in emotional responses and risk perceptions across five waves of data collection with a diverse, high-quality convenience sample of MTurk workers recruited with CloudResearch. The results from our study suggest that polarization may be a function of the amount and/or application of verbal knowledge rather than selective application of quantitative reasoning skills. The results also demonstrate that convenience samples of MTurk workers recruited with CloudResearch are fast, cost-effective, and comparable to larger, representative samples.

Intensive longitudinal methods (e.g., experience sampling or daily diary studies that involve collecting repeated measurements from participants over several days) offer notable benefits over cross-sectional designs: reducing retrospective recall bias, distinguishing within-person and between-person processes, and examining predictors of within-person change. Compared to typical convenience samples, crowdsourcing services can afford greater confidentiality while providing access to a relatively large and diverse participant population. This talk will discuss the benefits and challenges of collecting intensive longitudinal data through crowdsourcing, including strategies to promote compliance and enhance data quality while protecting participant confidentiality. It will also cover an example study of daily social interaction in which 200 participants recruited from MTurk completed daily surveys over one week, yielding more than 1,300 daily responses.

During 2019-2020, Emerson College Polling conducted a series of monthly national polls tracking the presidential nomination and eventual race of the White House. Throughout the year, ECP used a mixed-mode approach, combining IVR, online panels, and SMS-to-web data at varying proportions. ECP ended the election cycle with one of the most accurate national polls, with the October national poll consisting of IVR and online panel provided by Amazon MTurk and predicting a 4.2 percent (D) magnitude in the popular vote, in comparison to the final magnitude of 4.4 (D) percent.

This talk will discuss the key challenges for pollster accuracy in the 2020 presidential cycle, drawing on AtlasIntel’s methodological and practical insights. It will also explain how AtlasIntel worked to ensure sample representativity and correct sources of potential bias.