Several years ago, we introduced the option for researchers to create a dynamic completion code when setting up MTurk Toolkit studies. The dynamic completion code would take each participant’s Worker ID and generate a unique completion code. When using the dynamic completion code researchers could also set their studies to automatically approve each participant who submitted the correct code. While this system has worked well in the past, we are discontinuing its use for two reasons.
1. Security concerns
We’ve recently learned of some vulnerabilities in the system that creates each participants’ dynamic completion code, vulnerabilities that may present a security concern. Specifically, savvy MTurk workers may be able to trick the system into generating a completion code without actually progressing through a study. Because some researchers pair the dynamic completion code with automatic participant approval there is a risk that workers could submit their code and be automatically approved without providing any data.
To be clear, we have only seen this behavior in a few instances, but we are taking action now to prevent a larger problem in the future.
2. Obsolescence
One reason the dynamic approval code became popular among researchers was because it enabled automatic approval. In an environment where most participants provided quality data, checking each person’s data before approving their submission was time consuming and unnecessary. That is not the environment behavioral research operates in today.
Since 2018, researchers have had to take a much more active role in managing data quality. CloudResearch has led these efforts by vetting and maintaining an Approved Group of MTurk participants. Even so, we recommend checking participants’ data before approving them.
MTurk Approval Codes You Can Use Instead
Depending on your situation and objectives, there are still several approval codes and methods of approval you can use.
Use a Static Code with Automatic or Manual Approval
Static codes present the same alphanumeric string to all participants. You can create your own code or use a randomly generated code from CloudResearch.
When you use a static code, you can choose between automatic or manual approval methods.
Use a Custom Code with Manual Approval
A custom secret code allows you to present each participant with a unique or custom code. You can do this using your survey platform, JavaScript, or any other method you choose. Because custom codes are housed within your survey, CloudResearch has no access to the code. This means when using custom codes you must manually approve workers.
While manual approval may sound tedious, the MTurk Toolkit actually allows you to approve participants in bulk. By choosing to “select all” participants within a category, you can choose to Approve or Reject multiple participants at the same time.
Note: If you have a dynamic completion code set up with Qualtrics, you can still use this system to generate a unique code for each participant. However, you will need to select “custom completion code” during study set up and use manual approval.
