A research team led by Kinnon Ross MacKinnon, a professor in York University's School of Social Work, has introduced a new protocol to identify and remove fraudulent responses from web-based surveys - a move that may set a new standard for studies involving hard-to-reach populations.
The protocol was developed after MacKinnon's team noticed a surge in suspicious activity while recruiting participants for a study on detransition experiences among LGBTQ2S+ people in Canada and the U.S.
"Early in the recruitment process for the DARE (Dean's Award for Research Excellence) study, we noticed suspicious survey responses. To ensure that we had a reliable dataset, we developed a protocol to identify and remove scam, bot and nonsense responses," says MacKinnon, a professor in the Faculty of Liberal Arts & Professional Studies.
"If a researcher applied online recruitment methods with no way of verifying the identities and eligibility of participants, there is a good chance that a significant portion of the sample would be bots or scam responses. So this poses a real threat to our knowledge production and the reliability and trustworthiness of research findings."
The multi-step protocol excluded responses based on several criteria, including duplicate IP addresses, geolocation outside Canada or the U.S., reCAPTCHA scores below 0.5 and inconsistent or nonsensical answers. The team also inserted fictitious medical conditions - "chekalism" and "syndomitis" - to help flag inattentive or fraudulent respondents. Survey responses completed in less than 12 minutes were removed and participants who gave suspicious or conflicting information were invited to complete a Zoom screening interview. Only those who could verify their eligibility were included in the final sample.
After applying these measures, 69 per cent of the 1,377 completed survey responses - 957 in total - were deemed eligible and included in the analysis. The final sample was diverse: most participants were living in the U.S., a majority were assigned female at birth and more than half identified as bisexual or pansexual. The study's recruitment strategy used three different flyers, distributed in English, French and Spanish, to attract a broad range of experiences and identities related to detransition, including those who stopped or reversed transition for reasons ranging from identity shifts to loss of access to gender-affirming care.
MacKinnon emphasizes that the protocol's value extends beyond this particular study. "The protocol for removing scam, fraud, nonsense or bots could be applied to any study population, or community, being studied," he says.
With online recruitment becoming more common, he points out, "the risk in not doing this today is that if a researcher applied online recruitment methods with no way of verifying the identities and eligibility of participants, there is a good chance that a significant portion of the sample would be bots or scam responses."
Despite the challenges - including public debate and some skepticism within online communities - the study was able to recruit a large and diverse sample, offering new opportunities for research on detransition and gender fluidity.
MacKinnon's approach, now published and openly detailed in the Journal of Medical Internet Research, provides a template for researchers facing similar challenges in the digital era.
MacKinnon is also behind the research website Detrans Support, which provides resources, support and data-driven information about identity evolution and detransition/retransition. He will also share recent findings from the DARE (Detransition Analysis, Representation, and Exploration) study - the largest 2SLGBTQ+ community-engaged project to ever examine detransition, retransition and shifts in identity post-transition in Canada and the United States - during a June 17 webinar.
This story was originally featured in YFile, York University's community newsletter.
