Picker is a health and social care research charity who design and run a multitude of patient and service user experience survey programmes, including the Under 16 Cancer Patient Experience Survey on behalf of NHS England. Children and young people who have been diagnosed with cancer, along with their parents, are invited to provide valuable feedback on their treatment and care. An important part of the methodology is thoroughly testing the survey with cognitive interviews to assess question interpretation, comprehension and ensure it is fit for purpose. These are conducted remotely via video (Microsoft Teams) or phone call.
To recruit participants for cognitive interviews, we use social media platforms like Twitter, Facebook and LinkedIn alongside more traditional methods such as sharing posters and leaflets with patients and families in treatment centres. Short advertisements detailing the study, eligibility criteria and details about the gift voucher incentive are posted to social media channels, with a link to further information about the study.
Image 1: Example of recruitment advert posted on social media (Twitter)
Participants opt into cognitive testing for the Under 16 Cancer Patient Experience Survey by registering interest via an online screening questionnaire, which determines suitability and eligibility to take part (they are eligible if they are a child or a parent or carer of a child aged under 16 who has been diagnosed with a cancer or tumour and received NHS care in the last year).
The shift to online research methods has been beneficial for:
- Recruitment: by providing access to a broader group of potential participants, creating an opportunity for increased diversity.
- Participation: Caring for a child with cancer can be extremely challenging for families and having the flexibility of an online interview without the need to travel may give them more opportunity to participate.
However, we have recently noticed many instances of ingenuine participants when recruiting for cognitive interviews for the Under 16 Cancer Patient Experience Survey, which might relate to our online recruitment and methodology approaches.
Types of ingenuine contact
There are different types of ingenuine participants to be aware of:
- People who provide fake responses in a screening questionnaire to appear eligible.
- People who complete a screening questionnaire more than once (i.e., using a different name and/or contact details). This could be a genuine participant trying to increase their chances of being selected, or a single ingenuine person making different applications under multiple aliases.
- Spam bots – automated computer programmes designed to send bulk spam emails to users (for example, claiming to be interested in participating in research). These bots can be designed to either contact the researchers directly via email or in some cases can be programmed to complete an online survey. For the Under 16 Cancer Patient Experience Survey, we received a flurry of seemingly ingenuine emails shortly after we shared recruitment adverts via social media channels.
It is likely that such activity arises due to people’s interest in a monetary or gift voucher incentive, leading them to be untruthful about their circumstances to appear eligible, or to log their interest multiple times to increase their chances of selection. We soon began to recognise certain characteristics associated with fraudulent applications and introduced a more effective screening process. Below, we share some of our learnings and tips.
Identifying ingenuine participants during screening
There are various ways to identify instances of ingenuine participation in recruitment:
- Spam bot activity is often evident by receiving scores of emails sent within seconds of each other at odd times (e.g., through the night), often with no subject line and very vague sentences e.g. “I am interested in your research, want to take part”.
- Many of our fraudulent emails were from a Gmail account in the format of a name followed by a string of random numbers.
- Multiple online responses to screening surveys with the same IP address or contact information indicates repeat offenders.
- Incompatible answers to screening questions (for example receiving palliative care for their cancer whilst also being in remission).
Minimising the selection of ineligible participants
We introduced some measures to minimise the likelihood of ineligible participants being invited to the interview stage:
Introduce measures to make it more difficult to be untruthful
- Depending on what survey platform is used, online screening questionnaires can record IP address and make it impossible for someone with the same IP address to respond more than once.
- Removing back buttons on screener questionnaires prevents respondents from going backwards to change a previous answer if a response makes them ineligible. However, there is a risk that this might rule out eligible respondents who make a genuine mistake on their response, and therefore researchers should weigh up the pros and cons of this when recruiting.
Conduct screening calls
- Set up a phone call to ask people to verify information they included in their screening response (for us, this included the name and age of their child, what type of cancer they had been diagnosed with and where they received care and treatment). Where there is a substantial mismatch in information without reason, we would advise not to progress with their participation.
Targeted recruitment
- Target online adverts to specific groups and organisations that will successfully reach your required population, instead of blanket and widespread messaging. For our work, we reached out to NHS organisations providing children’s cancer care, and children’s cancer care charities and support networks.
- When using social media to support recruitment, avoid reference to monetary incentives in posts, as these are likely to be picked up by bots or individuals scouring for a pound sign! We would often receive a flurry of suspicious emails after such posts. Although offering incentives can encourage participation, this could be detailed on a separate information page or only on advert images and removed from written posts.
How to deal with an ineligible participant during an interview
It is always possible that an ingenuine participant slips through the net, even after introducing measures to identify them. It is often easy to identify when this happens, for example someone becomes uncomfortable or is unable to answer specific questions about their care pathway during an interview, or provides inconsistent answers. However, interviewers should be briefed on the possibility of ingenuine participants and advised how they might be identified and dealt with. In our research, when it is clear a participant was not genuine, we politely ended the interview explaining we were unable to progress it further due to the answers provided.
If the interview is ended due to suspected ineligibility, the decision on incentive payment is a difficult one. Whilst it is possible to include a clause in the consent or screening phase indicating that if deemed ingenuine then payment will not be made, this may deter genuine participants. We recommend that research teams define the procedures surrounding suspected ineligibility for each project to ensure there is a consistent plan and approach in place.
Conclusion
As we shift to online methods for recruitment and interviewing in social research, ingenuine participants may become more common, particularly when incentives are involved. Researchers should remain vigilant, and we hope that this blog has provided some insightful guidance and recommendations.
Author Bios:
Amy Tallett is Head of Research at Picker Institute Europe. Amy is an experienced healthcare researcher dedicated to understanding people’s experiences of receiving and delivery care to drive improvements in care quality. Particular areas of interest include researching specific care services and patient groups, including children and those living with particular long-term health conditions.
Caroline Hancock is a Research Associate at Picker Institute Europe. Caroline relocated to the UK in 2022 to focus her varied project and research management experience towards the healthcare industry. As a Research Associate at Picker, she plays an integral role within a team who produce the annual Cancer Patient Experience Surveys, which aim to improve the quality of care received by patients and their families.