Tilburg University
Patient involvement in the development of patient-reported outcome measures
Wiering, B.M.; De Boer, D.; Delnoij, D.
Published in:
BMC Health Services Research DOI:
10.1186/s12913-017-2582-8 Publication date:
2017
Document Version
Publisher's PDF, also known as Version of record
Link to publication in Tilburg University Research Portal
Citation for published version (APA):
Wiering, B. M., De Boer, D., & Delnoij, D. (2017). Patient involvement in the development of patient-reported outcome measures: The developers’ perspective. BMC Health Services Research, 17(1), [635].
https://doi.org/10.1186/s12913-017-2582-8
General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal
Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.
R E S E A R C H A R T I C L E
Open Access
Patient involvement in the development of
patient-reported outcome measures: The
developers
’ perspective
Bianca Wiering
1*, Dolf de Boer
2and Diana Delnoij
1Abstract
Background: Patient-reported outcome measures (PROMs) are increasingly used in health care. To capture the patient’s perspective, patient involvement in PROM development is needed. As earlier research showed varying degrees of patient involvement in PROM development, this study aimed to investigate why PROM developers do or do not involve patients, how patients can be successfully involved and what the negative aspects and benefits of patient involvement are.
Methods: PROM developers who, according to an earlier scoping review, involved patients in at least two phases of PROM development or did not involve patients at all, were contacted for a telephone interview. The interviews were recorded, transcribed and analysed using a general inductive approach.
Results: From the PROM developers who involved patients, 21 developers were interviewed and three answered questions via e-mail. Most developers considered patient involvement necessary to create a valid questionnaire and relied on guidelines, personal experience and practical considerations for choosing a qualitative method. Negative aspects of patient involvement were mainly time investment and budget impact. One developer who did not involve patients was interviewed. Two developers sent back answers via e-mail. These developers did not involve patients because of limited resources or because no benefits were expected.
Conclusion: Although PROM developers agree that patient involvement is necessary, a lack of resources can be a stumbling block. Most developers rely on guidelines, personal experience or practical considerations for choosing a qualitative method. Although this may be a good place to start, to optimize patient involvement developers should explicitly think about which methods would suit their study.
Keywords: Patient-reported outcome measures, Patient participation, Questionnaire development procedures, Qualitative research
Background
Patient-reported outcome measures (PROMs) are ques-tionnaires which measure patients’ perspectives on health outcomes [1, 2]. PROMs may be used to inform patients of the performance of treatments and health care professionals, improve the care of individual pa-tients, assist health care purchasers and reward perform-ance in pay for performperform-ance schemes [2–5]. PROMs are increasingly assigned these important tasks of measuring
performance, treatment progress and quality of care in health care systems such as the English National Health Service (NHS) [2, 4], the American Centers for Medicare and Medicaid Services (CMS) [6], the Dutch and the Swedish health care system [7].
As PROMs are meant to reflect the patient’s perspec-tive, the aim should be to use valid PROMs which truly reflect this perspective. Patient involvement throughout the development of PROMs is therefore essential [8–13]. Patient involvement can have a great impact on the questionnaire as the relevance of outcomes and the comprehensibility of the questionnaire can only be de-termined by patients [12, 14–19]. Furthermore, patients * Correspondence:B.M.Wiering@uvt.nl
1Tranzo (Scientific Centre for Transformation in Care and Welfare), Tilburg
University, PO Box 90153, 5000 LE Tilburg, The Netherlands Full list of author information is available at the end of the article
© The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Wieringet al. BMC Health Services Research (2017) 17:635
may fail to complete the questionnaire if the question-naire does not sufficiently reflect the patient’s perspec-tive. A lack of patient involvement may have a negative influence on the validity, sensitivity and response of a questionnaire [13, 20].
However, although patient involvement in PROM development has been recommended by researchers [8–12, 21], and has also become a requirement of the US Food and Drug Administration (FDA) [22], many PROMs appear not to have been developed with pa-tient input [9, 23, 24]. A scoping review of 193 PROMs looking at whether and to what extent pa-tients are involved in the development of PROMs, suggests that in over a quarter of the included papers describing the development of a PROM, no patient involvement was recorded. Furthermore, the level of patient involvement in the remaining PROMs varied greatly, from feedback forms to focus groups, inter-views and cognitive interinter-views [24]. Another review of 42 PROMs for rheumatoid arthritis showed that PROMs only minimally covered personal factors which determine functioning and the individuals’ ex-perience of disability according to the International Classification of Functioning, Disability and Health (ICF) and which were deemed important by patients [25]. Other studies which investigated patient involve-ment in a small number of PROMs suggest a complete lack of patient involvement [23], conceptual difficulties and questionnaire design problems. These problems might have been prevented by involving patients in the development process [9].
Apparently, not all PROM developers choose to in-volve patients in the development of their PROM. Be-sides, if a developer does decide to involve patients, there is a great variety in the levels and methods of in-volving patients. Although many benefits may be gained by involving patients [26], research suggests that devel-opers also need to consider slightly less positive conse-quences such as the costs associated with patient involvement [12, 26]. Additionally, there is no perfect method to involve patients, as fairly little is known about which methods are the most successful and which cir-cumstances are needed for achieving optimal outcomes [27]. Decisions that need to be made, such as whether patients should be involved, to what extent patients should be involved, and which methods should be used to involve patients, appear to be mostly based on the ex-periences and opinions of the researchers, the organisa-tion they belong to, or the regulaorganisa-tions or guidelines they adhere to such as the US Food and Drug Administration [22] or PROMIS (Patient-Reported Outcomes Measure-ment Information System) [28].
However, as most professionals involved in PROM de-velopment deal with similar decisions, it is necessary
that more insight is gained in how patients can be suc-cessfully involved in PROM development and what kind of benefits and costs of patient involvement are experienced by PROM developers. By looking at suc-cessful involvement, we are not necessarily interested in how patients were involved, as this was also re-corded in an earlier scoping review [24]. The focus lies instead on how developers chose their methods, and how they experienced using these methods. Such an overview may help PROM developers make a more in-formed decision regarding patient involvement. This qualitative study therefore aimed to gather the views of PROM developers, who did and did not involve pa-tients, regarding patient involvement and its negative aspects and benefits. Our research questions are:
According to the interviewed PROM developers:
1 Why did certain developers involve patients and others did not?
2 How are patients successfully involved in the development of PROMs?
3 What do patients contribute to the development of PROMs?
4 What are negative aspects of patient involvement?
Methods
Participants
Procedure
A list of questions was drafted to guide the interview process (Tables 1 and 2). The questions were tested in three pilot interviews with researchers who had experience
with patient involvement during questionnaire develop-ment. No changes were made to the list of questions.
PROM developers were sent an e-mail asking whether we could contact them via telephone for some questions regarding patient involvement during the development of a certain questionnaire. Reminders were sent a few weeks later. If PROM developers agreed to a telephone interview, a questionnaire (Tables 1 and 2) was sent and an appointment was scheduled. If PROM developers answered the questions via e-mail, an e-mail was sent informing the developers of the purpose of our study. In this e-mail we also asked them to consent to the use of the information for a paper. PROM developers who agreed to a telephone appointment were phoned at the scheduled time and date and were informed of the pur-pose of our study. They were asked to consent to the re-cording of the interview and the use of the information given during the interview for a paper. The conversation and informed consent were recorded. A semi-structured approach [29] was used for the interviews. Although the questions were used to guide the conversation, room was created for other topics. Transcripts of the interviews were sent to the PROM developers for approval. 17 of the 22 developers returned their transcript with no or small text-ual alterations. One developer added some text further explaining his/her methods.
Analysis
The interviews were transcribed verbatim, after which the transcriptions were analysed using a general induct-ive approach for analysis of qualitatinduct-ive evaluation data [30]. Segments of the interview text were coded closely following the meaning of the segments, after which the segments were grouped into subthemes and themes. For example, when asked for the difficulties of patient
Table 1 Questions used as interview guide for interviews with developers who involved patients
Questions for developers who involved patients Patient recruitment
-Why did you decide to involve patients in the development of your questionnaire?
-How many patients were involved in your development process? -How did you decide how many patients you wanted to involve? -How did you recruit the patients who were involved in your study? -Did you have any difficulties with the recruitment of patients? -Would you consider the patients who were involved a good
representation of the patient group? Methods of involving patients
-How many methods did you use to involve patients in the development of your questionnaire?
-During which development phases did you involve patients? -Which methods did you use to involve patients?
-How did you decide which methods to use?
-The participation ladder of decision making in health care consists of three steps. At the first step patients are involved as consultants. At the second step patients are considered partners and at the third step patients have a dominant role. Which step of the ladder describes the position of the patients in your development process the most accurate?
-Did you have any difficulties with involving patients in your development process?
-If focus groups were used, how many focus groups were organised and how many patients were involved per focus group?
-If interviews were used, how many interviews took place? -Was a specific method used to interview the patients?
-If cognitive interviews were used, how many interviews took place? -If another method was used; how many patients were involved? -Which method was the most successful and why?
-Which method was the least successful and why? Benefits and negative aspects of patient involvement
-Did patient involvement influence the questionnaire in any way? -If you could go through the development process again, would you
change anything concerning patient involvement?
-Were there any benefits from involving patients? What kind of benefits? -Did you perceive any negative aspects concerning patient
involvement? What kind of negative aspects?
-In hindsight, was there anything that could have been done to prevent these negative aspects?
-Did the benefits outweigh the costs?
-Would you recommend involving patients during the development process?
Table 2 Questions used as interview guide for interviews with developers who did not involve patients
Questions for developers who did not involve patients -Was involving patients in the development of the questionnaire actively considered?
-Was there anything you expected to gain from patient involvement? -Where there any negative consequences you expected to experience regarding patient involvement?
-What were the main reasons for not involving patients? -Would you consider the questionnaire relevant for patients? -Could the questionnaire have been improved by involving patients? -If you would develop a new questionnaire, would you involve patients? -How did you make sure that patients interpreted the questions as they were intended?
-How did you make sure that patients could complete the questionnaire?
involvement, a developer explained that patient involve-ment is costly and takes time. The subthemes would be ‘costs’ and ‘time’. All subthemes describing difficulties or challenges would be grouped under the theme ‘chal-lenges of patient involvement’. The transcripts were ana-lysed using ATLAS.ti, version 7 [31]. All interviews were analysed by author BW. The first three interviews were checked by author DD to ensure that no relevant infor-mation was missed and the coding matched her under-standing of the content. Five interviews were double coded by authors BW and DB to check the reliability and transparency of the coding. Any differences were discussed until agreement was reached. Finally, a content check was performed where authors DD and DB each read 10 interviews and made sure that the interviews were well reflected in the results section.
Results
Response
Forty-one PROM developers who involved patients and 16 PROM developers who did not involve patients were invited to participate. Initially 22 PROM developers who involved patients agreed to an interview. Seventeen de-velopers who involved patients did not respond to the invitation. One interview took place at the developer’s place of work, the other interviews were held via tele-phone. One interview was joined by another developer involved with the instrument. This resulted in 21 inter-views. Additionally, three PROM developers who in-volved patients sent back the completed questionnaire. This is a response rate of 58.5%. Of the 16 PROM devel-opers who did not involve patients only two develdevel-opers sent back a completed questionnaire and two developers agreed to an interview. Twelve developers did not re-spond to the invitation. After the interviews had taken place one PROM developer withdrew his/her consent. This is a response rate of 18.8%. Most participants were native English speakers from the United Kingdom, the United States, Canada and New Zealand. One partici-pant came from Germany. Most non-respondents also came from the United States or the United Kingdom. However, as ten out of eleven non-native English speakers declined our invitation, there was a difference in countries of residence between respondents and non-respondents. The average publication date of re-spondents was 2011, which was similar to the average publication date of non-respondents.
Themes
The major themes that were identified during the ana-lysis of the interviews concerned the motivation for in-volving patients, motivation for not inin-volving patients, requirements for patient involvement, the benefits of patient involvement, the overall challenges of involving
patients, recruitment methods, recruitment challenges, methods to involve patients, method motivation, chal-lenges with methods, and future plans for patient involve-ment. Although many hours of interview recordings were collected, in this paper only the interview sections are dis-cussed which are considered highly relevant and essential to answer our research questions. The selection of interview sections was based on the themes which were derived from the interviews.
Why did developers involve patients?
Motivation for involving patients There are many rea-sons why developers felt that they had to include pa-tients. Some developers involved patients because this was required by the FDA. Many developers wanted to understand how it feels to live with a certain disease or condition. Furthermore, developers wanted to be sure that their PROM covered all the important issues, was relevant to patients and could easily be completed by pa-tients. One developers said that it is the academic’s role to voice what patients want and experience.
“I don't know what it is to live with rheumatoid arthritis. They have fatigue caused by rheumatoid arthritis, so I could not do it better on my own. I would say that we are the voice of the patient. The patients know what they want and we as academics have to be their voice and develop it. But it has to come from them.”
One researcher gave an example illustrating why it is important that PROMs are relevant to patients:
“It is called a generic measure (EQ-5D), but of course the five aspects of health that are selected are more relevant to some conditions than they are to others. So someone with macular disease, whose sight is severely impaired and registered as what used to be called blind may appear to be in perfect health on the EQ-5D, because it doesn't affect the five aspects of health that are being asked about. Yet their quality of life is shot to pieces, because they can't see. And yet, NICE is demanding EQ-5D data from pharmaceutical companies evaluating new treatments.”
Future plans for patient involvement Regardless of how beneficial patient involvement was or how many difficulties related to patient involvement PROM devel-opers experienced, all PROM develdevel-opers would recom-mend patient involvement and wished to involve patients again during their next PROM development.
How are patients successfully involved in the development of PROMs?
Requirements for patient involvement Successful pa-tient involvement requires trained and/or experienced staff, good communication, a clear understanding of roles and expectations and awareness of a patient’s limitations.
“I think the key thing once again is being clear from the outset about roles and expectations around how things work. We were very clear about that.”
Recruitment methods Most developers recruited pa-tients in clinics or other health care centres. Other more common options were charities and recruitment com-panies. Most developers recruited until data saturation was reached.
Recruitment challenges The biggest challenge of recruiting patients was to achieve diversity, as finding patients with certain characteristics could be difficult. Examples of these characteristics were different cultures, age groups or patients with disabilities. However, most developers were confident that they had achieved a good representation of the patient population.
“Well obviously by the end you knew you need to find a 25 year old man and that is quite difficult in arthritis. It is mostly middle-aged women and by the end of it you are always looking for a young man at the end.”
Another difficulty was getting patients to actually par-ticipate. As patients are managing a personal life, work life and an often time and energy consuming disease, pa-tients are not always able to or interested in attending interviews or focus groups. Developers therefore tried to encourage participation by fitting appointments between hospital appointments, offering lunch and keeping pa-tients updated. One developer mentioned difficulties with people lying about their condition as they then got paid for participating. Another developer encountered problems when he/she wanted to investigate an issue that is taboo among a certain culture. Even if patients were willing to participate, family members sometimes kept them from participating.
Methods used to involve patients Most developers in-volved patients during item development and while test-ing the PROM for comprehensibility. The most used methods to do this were interviews and/or focus groups. Sometimes focus groups were organised online. Slightly less common but still much used methods were cognitive interviews and feedback questionnaires. Some developers
involved patients as patient research partners. When asked to choose their preferred method, most developers said that this is impossible as the methods serve different purposes and complement each other.
Method motivation Many researchers used scientific guidelines or standards as a guide for patient involvement.
“I mean, our whole development process has been informed by FDA guidelines, so that is how we decided to use the methods we have used. The FDA guidelines are exactly that. They are guidelines. They are not strict rules so they are open to certain amount of interpretation. But certainly our development process complies with their guidelines.”
Literature, personal preferences and experiences and practical considerations such as how much time or money it costs and how difficult it is to organise also influenced the method of patient involvement. Focus groups were sometimes chosen because they were an easier and cheaper way to receive the input from many patients at once.
“The reason why we choose focus groups for […] was in part driven by cost implications, because it makes it easier to include a larger number of patients for less cost.”
Additionally, the interaction between patients was greatly appreciated as developers thought that it would result in different or more information than interviews. Interviews, however, were chosen because they offered a one on one opportunity with the patient. This not only made it more suitable for private subjects, but it also resulted in richer data.
“We’ve always done individual interviews because we find we get a lot better, or more information, rich information.”
Furthermore, developers considered interviewing an easier method to involve patients.
Challenges with methods Although developers re-ported few difficulties with interviewing, a developer did mention that it can be challenging if a patient is not very reflective or talkative and that patients can get off-track. However, developers experienced more difficulties with focus groups. Focus groups mainly suffered under dom-inant patients taking over the focus group and patients getting off-track. These problems were solved by re-moving a patient, intervene verbally or use experienced moderators.
“We would actually move that patient out of the focus group, if we couldn’t stop them any other way. Sometimes these are personality traits, and it’s not that we’re there to blame that person, so we might then have to help to say this, very difficult patient, mister X, you’ve got such comprehensive views on this, I think it would be very helpful if you came with me and sat with me in a separate room and tell me all about this.”
What do patients contribute to the development of PROMs?
Patients mainly contributed to the questionnaire devel-opment by helping to increase the validity and compre-hensibility of the PROM. Patients were able to offer a different perspective than physicians.
“In diabetes care until five years ago if you asked any clinician or diabetologist:“How important are non-severe hypoglycaemic events?” They would say: “Not important at all. Drink a glass of orange juice. You are fine.” We don’t even think about it. But by talking to patients we found out that non-severe events had a huge impact on patients’ functioning and well-being. Because after they drink the orange juice they still feel terrible for 20 h.”
Involving patients ensured that the PROM was rele-vant to patients, and that patients were more likely to complete the PROM.
“We hadn't asked any questions about changes in medication in the PROM before and after treatment. Some patients put notes on their responses saying:“I have been able to stop taking my medication”, which was something we hadn't considered.”
Many developers also highlighted that it is a great ex-perience for patients as patients can share exex-periences with each other, learn from each other, and also feel valued by being involved.
“Yeah, you know, I think especially for the focus groups. We found that people tended to have a very positive experience. We definitely had people who even, after the groups would exchange contact information with each other.”
What are negative aspects of patient involvement?
On the downside involving patients is time consum-ing, logistically challenging and impacts the budget. It can also add to the complexity of the research project and can be difficult to report. Furthermore, involving patients may be upsetting for both patient and developer.
“So again from the practical viewpoint it is financially and time consuming. And as I said it was upsetting in some cases, but I think overall we felt that patients were getting a lot out of it.”
Some developers also mentioned that patients were only able to reflect their own, personal perspective, which could influence the relevance of the research if only a few patients were involved and/or if a patient was involved extensively. These challenges however, would not deter most developers from involving at least a few patients.
Why did some developers choose not to involve patients?
The PROM developers who refrained from involving patients mainly did so because they either did not have the time and resources or did not expect to gain any-thing. Even though their aim was to develop a PROM, one developer said that they did not aim to capture the patients’ perspective. Another developer considered pa-tient involvement unnecessary as the disease was well documented. Additionally, he/she felt, based on earlier experiences, that patients mostly just added subjective impressions of single patients.
“From earlier questionnaire developmental processes we got the impression that patient involvement introduced several additional aspects. However, many of those aspects were very subjective impressions of single patients without significant impact on the questionnaire.”
Plans for patient involvement in future developments Even though not all developers were convinced of the need to involve patients during PROM development, all developers thought that patient involvement could be beneficial, dependent upon circumstances such as the aim of the questionnaire and the availability of resources. Therefore they would consider involving patients next time.
Discussion
aimed to give more insight into PROM developers’ ex-periences with patient involvement by first investigating how patients are involved in the development process. Patients were mostly recruited via clinics or health pro-fessionals until data saturation was reached. Recruiting until data saturation has been reached is a method in qualitative research that is widely accepted and used by many researchers [32, 33]. The developers mostly in-volved patients using interviews and focus groups. These methods are mentioned by several scientific standards or guidelines as fitting methods to involve patients [28, 34]. Although some developers indicated that they adapted the recommended methods to their patient group or to how they thought it would work better, many followed the guidelines or used methods which were the most practical and/or cheapest, or based their choice on per-sonal experiences and preferences.
Besides the more common methods used to involve patients, several developers involved patients especially in their more recent PROM developments as patient research partners or included patients in steering or ad-visory groups. Even though this more collaborative form of patient involvement may have some issues that need to be dealt with, such as the amount of experience and skills patients would need to actively participate [35] and whether patients who are able to participate to such an extent are good representatives of the patient group [36], it is considered an important step forward for patient involvement in PROM development [12].
Another less common but increasingly used method concerns conducting qualitative research on the internet. As many people use the internet, conducting qualitative research online can be helpful in gathering the opinions of many patients [37]. Patients with rare conditions can also be more easily reached [38] and communicating online may offer patients more freedom to express themselves [39]. Online qualitative research also has some drawbacks. For example, you are only able to study a select group of patients as patients need to have access to internet and certain skills to participate on the internet [40, 41]. Additionally, it is much less clear what the rules are regarding informed consent and what is considered a private conversation and what is not [37]. As both online and face to face qualitative methods have several pros and cons, a decision for ei-ther of these methods should be made separately for each study.
Besides how patients are involved in PROM develop-ment, the present study also investigated what patient involvement contributes towards the PROM develop-ment. Patients contributed by offering their experiences and views regarding living with a disease. These views not only helped increase the validity and comprehensi-bility of the PROM, but also helped to ensure that the
PROM was relevant for other patients. The benefits of patient involvement were very important to PROM de-velopers as most would not consider to refrain from in-volving patients, even if they did not really have the budget or the time to do it.
Although most PROM developers were overwhelmingly enthusiastic about patient involvement, negative aspects and costs of patient involvement were encountered at sev-eral levels. Most developers experienced some difficulties finding enough diverse patients and getting patients to fit research into their lives could be challenging. Further-more, actually involving patients could be challenging as most developers encountered patients who dominated the conversation or easily went off-track. Overall, patient involvement was time consuming, made an impact on the budget and was logistically challenging. Although the costs of patient involvement are sometimes recog-nised in the literature as a challenge of qualitative re-search [12], the challenge of actually recruiting and involving patients and the time it takes to do it well should also be acknowledged.
It should therefore come as no surprise that a lack of resources was one of the reasons why not all PROM de-velopers involved patients. There were however also a few PROM developers who did not see any added bene-fit of patient involvement for their PROM. As patient involvement is increasingly required by organisations such as the FDA [34], these PROM developers may not have this choice for much longer.
Limitations and strengths
Some limitations need to be taken into account. First, this study aimed to give insight into the views of both developers who involved patients and developers who did not involve patients. However, only a low number of PROM developers who did not involve patients were willing to answer our questions.
Second, only 58.5% of the invited PROM developers who did involve patients agreed to participate in our study. As we indicated in our invitation that we wished to discuss patient involvement in PROM development, it is possible that the PROM developers who agreed to participate were perhaps more enthusiastic about pa-tient involvement than the PROM developers who did not reply. Whether we received a reply was in any case not related to how long ago the paper was published, as no difference was found between the publication dates of developers who participated and developers who did not participate. However, developers may have felt unsure about discussing their research in a lan-guage other than their native lanlan-guage, as all but one non-native English speakers declined our invitation. This makes it difficult to generalise the results beyond English speaking countries.
Third, results may be influenced by a confirmation bias. As patient involvement costs time, resources and effort, developers must have been very motivated to in-volve patients. This may also mean that they are more focused on the positives.
Fourth, although by including both co-authors in the analysing process efforts were made to guard the trans-parency and reliability, results may be coloured by views held by the first author who interviewed the developers and was the main coder.
An important strength of this study is that we were able to offer an insight into the views of many both ex-perienced and less exex-perienced PROM developers who worked with a number of patient groups and developed a PROM fairly recently. This study was therefore able to not only give insight into the methods used to recruit and involve patients, the many pitfalls and benefits of patient involvement, but also into the general opinion held by PROM developers regarding patient involvement in PROM development.
Implications
The results of this study have several implications for patient involvement in PROM development. First, this study indicates that patient involvement requires the availability of resources such as time and money. A lack of or limited resources may keep developers from opti-mally involving patients or it may keep them from involv-ing patients at all. As patient involvement is included in some scientific guidelines [28] and the Food and Drug Administration even requires it [34], it is essential that the necessary resources are included in the costs and the duration of the study when applying for subsidies so this can no longer be a stumbling block.
This is especially important as not involving patients may have implications for the use of the PROM. It may impact the response and the validity of the PROM [13, 20]. Perhaps more importantly, PROMs are used for important purposes, such as measuring the performance of health care providers [2–5]. Not involving patients may mean that health care providers are judged on as-pects of care which may not be important to patients, while important aspects may be forgotten. Furthermore, research shows that health care providers especially try to improve on the performance aspects they scored less well on [42]. If performance measures are used which do not reflect what patients consider important, health care providers may neglect to improve on aspects of care which are important to patients. Because of the possible consequences of using PROMs which do not reflect the patient’s perspective adequately, patient involvement should be taken very seriously.
Furthermore, the results suggest that many developers keep to scientific guidelines for methodological decisions
or alternatively pick methods to involve patients which are most practical to them or which suit their personal preferences. Guidelines may be a good place to start as it guarantees a certain level of patient involvement and practical solutions should certainly not be completely disregarded as it helps developers involve patients within their limitations. However, as there is no perfect method for involving patients [27], it may be more beneficial that pros and cons of the available methods are considered. For example, based on our results, interviews may be es-pecially fitting for discussing intimate or taboo subjects. Focus groups may be helpful to learn about the majority view or if you have a very tight budget. Both also have drawbacks such as the difficulty of keeping a focus group on track, or the single subjective view of a patient during interviews. It is therefore important that a method is picked based on what is most suitable for the patient group and the purpose of the study.
Fourth, several PROM developers mentioned the im-portance of good qualitatively trained and/or experi-enced staff for moderating focus groups or conducting interviews. They also highlighted the importance of good communication, a clear understanding of roles and expectations and awareness of patient’s limitations. These requirements combined with the difficulties one may experience while interviewing or conducting focus groups asks a lot of the persons who are hired to do so. Adequately training staff to deal with patients and con-duct qualitative research may be imperative for success-ful patient involvement and optimal patient benefits for the PROM development.
Conclusion
Abbreviations
CMS:Centers for Medicare and Medicaid Services; EQ-5D: EuroQol five dimension scale; FDA: Food and Drug Administration; ICF: International Classification of Functioning, Disability and Health; NHS: National Health Service; NICE: The National Institute for Health and Care Excellence; PROM: Patient-reported outcome measure; PROMIS: Patient-Reported Outcomes Measurement Information System
Acknowledgements
We wish to thank all PROM developers who participated in this study for their time, honesty and cooperation.
Funding
This study was funded by the National Health Care Institute. The National Health Care Institute did not have any role in the study design; in the collection, analysis and interpretation of data; in the writing of the manuscript; and in the decision to submit the manuscript for publication.
Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
Authors’ contributions
BW conducted the interviews, analysed the interviews and wrote the draft and final version of the manuscript. DD analysed part of the interviews, checked the analyses to ensure that the content of the interviews was adequately represented and helped write the paper. DB analysed part of the interviews for inter-rater reliability, checked the analyses to ensure that the content of the interviews was adequately represented and helped write the paper. All authors read and approved the final manuscript.
Ethics approval and consent to participate
As our study concerned no personal issues, participants were not subjected to acts and the participants were consenting adults, no ethics approval was needed in the Netherlands. Participants were informed of the purpose of our study and were asked to consent to the recording of the interview and the use of the information they gave for a paper. The consent was audio recorded.
Consent for publication
Participants were informed of the purposes of our study. They were also asked to consent to the recoding of the interview and the use of the information for a paper. Participants were told that they could withdraw their consent at any time and that their information will be stored anonymously.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Author details
1Tranzo (Scientific Centre for Transformation in Care and Welfare), Tilburg
University, PO Box 90153, 5000 LE Tilburg, The Netherlands.2NIVEL
(Netherlands institute for health services research), Utrecht, the Netherlands.
Received: 1 March 2017 Accepted: 30 August 2017
References
1. Brédart A, Marrel A, Abetz-Webb L, Lasch K, Acquadro C. Interviewing to develop Patient-Reported Outcome (PRO) measures for clinical research: eliciting patients’ experience. Health Qual Life Outcomes. 2014;12(1):15. 2. Black N: Patient-reported outcome measures could help transform healthcare.
BMJ (Clinical research ed) 2013, 346:f167.
3. Burge P, Devlin N, Appleby J, Gallo F, Nason E, Ling T: Understanding patients’ choices at the point of referral. 2006.
4. Devlin NJ, Parkin D, Browne J. Patient-reported outcome measures in the NHS: new methods for analysing and reporting EQ-5D data. Health Econ. 2010;19(8):886–905.
5. Wu AW, Snyder C, Clancy CM, Steinwachs DM. Adding the patient perspective to comparative effectiveness research. Health Aff. 2010;29(10):1863–71. 6. https://www.cms.gov/Regulations-and-Guidance/Legislation/
EHRIncentivePrograms/index.html. Accessed 31 Dec 2014.
7. Nelson EC, Eftimovska E, Lind C, Hager A, Wasson JH, Lindblad S. Patient-reported outcome measures in practice. BMJ. 2015;350:g7818. 8. Fitzpatrick R, Davey C, Buxton MJ, Jones DR. Evaluating patient-based
outcome measures for use in clinical trials. Health Technol Assess. 1998;2(14):i-iv, 1-74.
9. Paterson C. Seeking the patient's perspective: a qualitative assessment of EuroQol, COOP-WONCA charts and MYMOP. Qual Life Res. 2004;13(5):871–81. 10. Kirwan JR, Fries JF, Hewlett S, Osborne RH. Patient perspective: choosing or
developing instruments. J Rheumatol. 2011;38(8):1716–9.
11. McKenna SP. Measuring patient-reported outcomes: moving beyond misplaced common sense to hard science. BMC Med. 2011;9(1):86. 12. Staniszewska S, Adebajo A, Barber R, Beresford P, Brady LM, Brett J, Elliott J,
Evans D, Haywood KL, Jones D. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J Consum Stud. 2011;35(6):628–32.
13. Meadows KA. Patient-reported outcome measures: an overview. Br J Community Nurs. 2011;16(3):146–51.
14. Trujols J, Portella MJ, Iraurgi I, Campins MJ, Siñol N. Cobos JPdL: Patient-reported outcome measures: Are they patient-generated, patient-centred or patient-valued? J Ment Health. 2013;22(6):555–62.
15. Haywood KL. Patient-reported outcome II: selecting appropriate measures for musculoskeletal care. Musculoskeletal care. 2007;5(2):72–90. 16. Turner RR, Quittner AL, Parasuraman BM, Kallich JD, Cleeland CS.
Patient-Reported Outcomes: Instrument Development and Selection Issues. Value Health. 2007;10(s2):S86–93.
17. Coyne KS, Tubaro A, Brubaker L, Bavendam T. Development and validation of patient-reported outcomes measures for overactive bladder: a review of concepts. Urology. 2006;68(2):9–16.
18. Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, Ring L. Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO good research practices task force report: part 1—eliciting concepts for a new PRO instrument. Value Health. 2011;14(8):967–77. 19. Patrick DL, Burke LB, Gwaltney CJ, Leidy NK, Martin ML, Molsen E, Ring L.
Content validity—establishing and reporting the evidence in newly developed patient-reported outcomes (PRO) instruments for medical product evaluation: ISPOR PRO Good Research Practices Task Force report: part 2—assessing respondent understanding. Value Health. 2011;14(8):978–88.
20. Fossey EM, Harvey CA. A conceptual review of functioning: implications for the development of consumer outcome measures. Aust N Z J Psychiatry. 2001;35(1):91–8.
21. Staniszewska S, Haywood KL, Brett J, Tutton L. Patient and public involvement in patient-reported outcome measures. The Patient-Patient-Centered Outcomes Research. 2012;5(2):79–87.
22. Administration. USFaD: Guidance for industry: patient-reported outcome measures: use in medical product development to support labeling claims. Washington; 2009. https://www.fda.gov/downloads/drugs/guidances/ ucm193282.pdf. Accessed 31 Dec 2014.
23. Haywood KL, Staniszewska S, Chapman S. Quality and acceptability of patient-reported outcome measures used in chronic fatigue syndrome/ myalgic encephalomyelitis (CFS/ME): a systematic review. Qual Life Res. 2012;21(1):35–52.
24. Wiering B, Boer D, Delnoij D. Patient involvement in the development of patient-reported outcome measures: a scoping review. Health Expect. 2017;20(1):11–23.
25. Dür M, Coenen M, Stoffer MA, Fialka-Moser V, Kautzky-Willer A, Kjeken I, Drăgoi RG, Mattsson M, Boström C, Smolen J. Do patient-reported outcome measures cover personal factors important to people with rheumatoid arthritis? A mixed methods design using the International Classification of Functioning, Disability and Health as frame of reference. Health Qual Life Outcomes. 2015;13(1):27.
26. Staniszewska S. Patient and public involvement in health services and health research: A brief overview of evidence, policy and activity. J Res Nurs. 2009;14(4):295–8.
27. Staniszewska S, Herron-Marx S, Mockford C. Measuring the impact of patient and public involvement: the need for an evidence base. Int J Qual Health Care. 2008;20(6):373–4.
28. Health UDo, Services H: PROMIS: Instrument Development and Psychometric Evaluation Scientific Standards 2012.
29. Fylan F. Semi structured interviewing. A handbook of research methods for clinical and health psychology. 2005:65–78.
30. Thomas DR. A general inductive approach for analyzing qualitative evaluation data. Am J Eval. 2006;27(2):237–46.
31. Muhr T. Atlas.ti. Berlin, Germany: Scientific Software Development; 1993. 32. Pope C, Ziebland S, Mays N. Analysing qualitative data. BMJ. 2000;
320(7227):114–6.
33. Francis JJ, Johnston M, Robertson C, Glidewell L, Entwistle V, Eccles MP, Grimshaw JM. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229–45.
34. Food and Drug Administration US. Guidance for industry patient-related outcome measures: use in medical product development to support labeling claims. Washington: US Department of Health and Human Services Food and Drug Administration; 2009.
35. Conway J. Patients and families: powerful new partners for healthcare and caregivers. Healthc Exe. 2008;23:60–2.
36. Groene O. Patient and Public Involvement in Developing Patient-Reported Outcome Measures. The Patient: Patient-Centered Outcomes Research. 2012;5(2):75–7.
37. Eysenbach G, Till JE. Ethical issues in qualitative research on internet communities. BMJ. 2001;323(7321):1103–5.
38. Lasker JN, Sogolow ED, Sharim RR. The role of an online community for people with a rare disease: content analysis of messages posted on a primary biliary cirrhosis mailinglist. J Med Internet Res. 2005;7(1):e10. 39. Campbell MK, Meier A, Carr C, Enga Z, James AS, Reedy J, Zheng B. Health
Behavior Changes after Colon Cancer: A Comparison of Findings from Face to Face and On Line Focus Groups. Family & community health. 2001;24(3):88–103.
40. Synnot A, Hill S, Summers M, Taylor M. Comparing face-to-face and online qualitative research with people with multiple sclerosis. Qual Health Res. 2014;24(3):431-8. doi:10.1177/1049732314523840.
41. Egan J, Chenoweth L, McAuliffe D. Email-facilitated qualitative interviews with traumatic brain injury survivors: A new and accessible method. Brain Inj. 2006;20(12):1283–94.
42. Hibbard JH, Stockard J, Tusler M. Does publicizing hospital performance stimulate quality improvement efforts? Health Aff. 2003;22(2):84–94.
• We accept pre-submission inquiries
• Our selector tool helps you to find the most relevant journal
• We provide round the clock customer support
• Convenient online submission
• Thorough peer review
• Inclusion in PubMed and all major indexing services
• Maximum visibility for your research Submit your manuscript at
www.biomedcentral.com/submit
