Last month, we published a blog titled, “Five Things You Should Not Be Doing in Online Data Collection.” Among the things we identified that you should not be doing was launching your study without piloting it first. As a way to reiterate how important we think this issue is, we describe in this blog how to easily conduct a pilot study using CloudResearch.
Why Should I Run a Pilot Study?
Piloting your study benefits both you and your research participants. For you, piloting is a chance to catch any errors in the study instructions, question wording, study programming, or technical details (e.g., redirecting workers to an external web page). For workers, piloting ensures that researchers can accurately estimate how long the study takes to complete and therefore how much the study should pay. A common complaint among workers on MTurk is that study tasks often take much longer than researchers estimate. Without a pilot study, it’s hard to know what is a reasonable amount of time to complete the study.
How Can I Run a Pilot Study?
“Piloting” your study may be as simple as walking through the survey yourself, asking a friend or research assistant to take the survey, or running a small portion of the study (say 10%) before launching the full version. Collecting a small portion of your results before launching the full study is the most rigorous form of piloting and so it is the one we describe how to do here.
If you were to conduct a pilot study using only MTurk, you might find the process complicated and uncompromising. For example, once you launch a study on MTurk, you cannot pause or edit the study. You must wait for the study to complete and then incorporate any changes or feedback into your next HIT. In addition, once your pilot is complete, you need to create your full study essentially from scratch. Even when using the copy feature on MTurk, most of your settings do not save or carry over to the new HIT. Finally, the process of excluding people who took your pilot from taking your second study is not easy. Excluding workers requires creating and managing csv files and uploading a list of workers who need excluded from your full study.
On CloudResearch, however, conducting pilot studies is easy. If you need to edit your pilot once it is live, you can simply pause the study and edit your instructions, edit the number of participants you want, edit the expected time to complete the study, or even edit the pay workers receive. Once it is time to launch your full study, you can use the Copy HIT button to quickly copy all of your study settings and automatically exclude people who took your first study from taking your second study. Alternatively, you can simply raise the total number of participants after your pilot is complete and then proceed to collect the rest of your data. Regardless of how exactly you run your pilot study, planning time to pilot is essential.
At CloudResearch, we’ve made running pilot studies and conducting other study-related tasks easy because we’re researchers who often use these tools ourselves. If you have ideas for tools that would make your online research easier, let us know with our suggestion box.
