Patient-driven decision support for the detection and correction of inappropriate prescribing

 

Patient-driven decision support for

the detection and correction of

inappropriate prescribing

 

Scientific Research Project ­ Medical Informatics 

 

    Student  R.S. Schermer  Waaloord 101  3448 BJ, Woerden  Student number: 10667628  E­mail: _​r.s.schermer@amc.uva.nl    Mentor  Dr. S.K. Medlock  Department of Medical Informatics  Faculty of Medicine, AMC­UvA        Tutor  Prof. dr. A. Abu­Hanna  Department of Medical Informatics  Faculty of Medicine, AMC­UvA    Location of Scientific Research Project  Department of Medical Informatics  Faculty of Medicine, AMC­UvA  Meibergdreef 15  1105 AZ, Amsterdam    Period  November 2014 ­ November 2015    Keywords  Inappropriate prescribing, adverse drug reactions, decision support, older adults     

 

Table of Contents

Summary 4

Samenvatting 6

General introduction 8

References 9

Chapter 1 - The association between explicit criteria for potentially inappropriate medication and adverse drug reactions: a literature review

10 Introduction 11 Methods 11 Results 12 Discussion 14 References 16

Chapter 2 - Applying the LERM to patient-driven decision support based on the START / STOPP and ACOVE criteria: critical assessment and modifications

19 Introduction 20 Methods 20 Results 22 Discussion 25 ​References 26

Chapter 3 - Detecting and correcting inappropriate prescribing based on patient information entered by patients: a pilot study

28 Introduction 29 Methods 30 Results 33 Discussion 34 _​References 36

Chapter 4 - Usability problems encountered by older adults when using a software tool for the detection and correction of suboptimal prescribing: a think aloud evaluation

Introduction 39 Methods 40 Results 41 Discussion 42 References 43 General d​iscussion 45 References 47 Acknowledgements 48

Appendix A - PubMed MEDLINE search for literature review 49

Appendix B - Critical appraisal forms for literature review 50

Appendix C - LERM formalization process with software tool 59

Appendix D - Non-functional usability requirements for the patient facing user interfaces of the medication screening software prototype

70

Appendix E - A description of the knowledge model and reasoning system implemented by the prototype medication screening software

73

Appendix F - A description of the patient-facing user interfaces for patient information entry

76

Summary

The aim of this project was to investigate the feasibility and effectiveness of applying a patient-driven approach to the detection and correction of inappropriate medication prescribing in older adults. In such a patient-driven approach a computerized decisions support system (CDSS) suggests medication changes by combining on patient-specific information, obtained directly from the patient himself, with explicit criteria for inappropriate medication prescribing: rules describing when a specific drug or class of drugs should or should not be prescribed.

Chapter 1 reports on how we investigated the question: which of the available explicit criteria sets for the identification of potentially inappropriate prescribing can identify prescriptions for which a correlation with adverse drug reactions (ADRs) exists? We performed a systematic review of the literature and identified publications that reported on the association between potentially inappropriate medication - as identified by explicit criteria - and ADRs. We found that the STOPP criteria seem best supported by current evidence in their ability to identify prescriptions with a high risk of ADRs, although the evidence should still be considered preliminary.

In the study described in Chapter 2 we aimed to create formalized versions of two sets of explicit criteria by answering the question: how can we adapt the Logical Element Rule Method (LERM)[10] - a 7-step systematic formalization method - to the formalization of explicit criteria intended for use in a patient-driven CDSS? Step 5 and 7 or the LERM were adapted for use with a decision support system for which the information available for reasoning is not predetermined and extracted from a pre-existing data source, but is instead obtained from the patient himself through questions and tasks. An additional step was added to the formalization process to prevent the system from triggering false positive decision support messages. A software tool was created to assist the formalizer. With the help of the software tool and the modified LERM formalization process, 85% of the 110 selected explicit criteria were partially or fully formalized and implemented as 136 formal logical rules.

In Chapter 3 we attempt to answer the question: what is the accuracy of medication change suggestions generated by a CDSS which uses formalized explicit criteria, when it can only obtain patient information from the patient himself? It describes how we created a prototype version of a decision support system for the detection and correction of inappropriate prescribing in older adults, which uses only patient information obtained from the patient himself. It also describes an initial pilot carried out on an outpatient geriatrics clinic, aimed at evaluating the accuracy - as rated by a geriatrician-clinical pharmacologist - of the suggested medication changes generated by this prototype. We found that the overall correctness of the suggestions was 51.6% and the overall completeness of the suggestions was 84.2%.

Chapter 4 describes an evaluation of the usability of the prototype system that was developed in Chapter 3, aimed at identifying design problems in the patient-facing user interfaces of the system so they may be addressed in a redesigned version of the prototype and will not affect future research. Subjects were given two tasks to be carried out with the system and were asked to “think aloud” while doing so. By recording these verbalizations and the interactions with the system, we identified six design problems that negatively impacted task execution: one problem prevented task completion in a subject and required intervention from the researcher, three problems caused subjects to input incorrect patient information, and three problems delayed task completion.

In conclusion, the STOPP criteria seem best supported by current evidence in their ability to identify prescriptions with a high risk of adverse drug reactions, although the evidence should still be considered preliminary. We were able to adapt the LERM to a patient-driven target system by modifying two steps and adding an additional step, and we successfully used this modified LERM to formalize two sets of explicit criteria. Finally, we found that computer generated medication change suggestions, based on formalized explicit criteria and patient information obtained from the patient himself, can achieve a correctness of 51.6% and a completeness of 84.2%. These were only the first steps in investigating the feasibility and effectiveness of applying a patient-driven approach to medication screening, but hopefully they provide a foundation for further research.

Samenvatting

Dit project had tot doel de haalbaarheid en effectiviteit te onderzoeken van een patïentgedreven benadering voor het detecteren en corrigeren van ongeschikte geneesmiddelen bij ouderen. In zo’n patïentgedreven benadering stelt een een beslissingsondersteunend system medicatiewijzigingen voor, door patïentspecifieke informatie die rechtstreeks is verkregen van de patïent zelf, te combineren met expliciete criteria voor ongeschikte geneesmiddelen: regels die beschrijven wanneer een specifiek geneesmiddel of een geneesmiddelengroep wel of niet voorgeschreven moeten worden.

Hoofdstuk 1 beschrijft hoe we de volgende vraag onderzocht hebben: welk van de bestaande sets expliciete criteria voor het identificeren van potentieel ongeschikte medicatie kan medicatie detecteren waarvoor een correlatie met geneesmiddelgerelateerde problemen bestaat? Hiertoe hebben we publicaties geidentificeerd die rapporteerden over het verband tussen potentieel ongeschikte medicatie (als vastgesteld door expliciete criteria) en geneesmiddelgerelateerde problemen. We vonden dat het beschikbare bewijs de STOPP criteria het best leek te ondersteunen, zij het onder voorbehoud, in afwachting van replicatie in toekomstig onderzoek.

In de studie beschreven in Hoofdstuk 2 hadden we tot doel twee sets expliciete criteria te formaliseren door de volgende vraag te beantwoorden: hoe kunnen we de Logical Elements Rule Method (LERM) - een systematische methode voor de formalisatie van richtlijnregels in zeven stappen - aanpassen aan patïentgedreven beslissingsondersteuning? Stap 5 en 7 van de LERM werden aangepast aan op een beslissingsondersteunend system waarvoor de informatie die beschikbaar is om mee te redeneren niet vooraf bepaald is in een reeds bestaande databron, maar in plaats daarvan actief verkregen wordt van de patïent zelf, door middel van opdrachten en vragen. We voegden ook een extra stap toe om te voorkomen dat het systeem fout-positieve suggesties zou generen. Daarnaast werd er een software tool ontwikkeld om de formalisatie te ondersteunen. Met behulp van deze software tool en het aangepaste LERM process lukte het om 85% van 110 expliciete criteria geheel of partieel te formaliseren tot 136 formele logische regels.

In Hoofdstuk 3 proberen we de volgende vraag te beantwoorden: wat is de juistheid van medicatiewijzigingen voorgesteld door een beslissingsondersteunend systeem dat gebruik maakt van geformaliseerde expliciete criteria, wanneer het de benodigde patïentinformatie uitsluitend kan verkrijgen van de patïent zelf? Dit hoofdstuk beschrijft hoe we een prototype beslissingsondersteunend systeem ontwikkelden voor het detecteren en corrigeren van ongeschikte medicatie bij ouderen, dat uitsluitend gebruik maakt van patïentinformatie ingevoerd door de patïent zelf. Dit hoofdstuk beschrijft ook een initiele pilotstudie die werd uitgevoerd op een polikliniek voor ouderengeneeskunde en tot doel had de juistheid te beoordelen (zoals bepaald door een geriater-klinisch farmacoloog) van de medicatiewijzigingen die dit prototype systeem voorstelde. We vonden dat de mate van correctheid van de voorgestelde medicatiewijzigingen 51.6% was en dat de compleetheid van de voorgestelde medicatiewijzigingen 84.2% was.

Hoofdstuk 4 beschrijft een evaluatie van de gebruiksvriendelijkheid van het prototype dat in Hoofdstuk 3 werd ontwikkeld, met als doel het vaststellen van problemen in het ontwerp van de gebruikersinterface, zodat deze problemen aangepakt kunnen worden in een volgende versie van het prototype, en zo toekomstig onderzoek niet zullen beïnvloeden. We vroegen een aantal ouderen om twee taken uit te voeren met het systeem en hierbij hardop te denken. Opnames

hiervan werden geanalyseerd en zo stelden we zes problemen in het ontwerp vast die het volbrengen van de taken hinderden.

Concluderend, lijkt het huidige bewijs de STOPP criteria het best te ondersteunen in hun vermogen om geneesmiddelen te identificeren die gecorreleerd zijn met geneesmiddelgerelateerde problemen. We waren in staat om de LERM aan te passen aan patïentgedreven beslissingsondersteuning door twee stappen te modificeren en een extra stap toe te voegen, en konden deze aangepaste LERM gebruiken om twee sets expliciete criteria te formaliseren. Tot slot, vonden we dat computergegenereerde suggesties voor medicatiewijzigingen, gegenereerd op basis van geformaliseerde expliciete criteria en patïentinformatie verkregen van de patïent zelf, een correctheid hebben van 51.6% en een compleetheid hebben van 82.4%. Dit waren slechts de eerste stappen bij het vaststellen of deze patïentgedreven benadering kan bijdragen aan medicatiescreening, maar hopelijk vormen zij een basis voor verder onderzoek op dit gebied. 

General introduction

Medication can provide us with substantial health benefits. Especially as we grow older, drugs can help increase our life expectancy and improve our quality of life. However, with drug use often comes the risk of adverse drug reactions (ADRs), defined by the World Health Organisation as any noxious, unintended, and undesired effects of drugs, which occur at doses used in humans for prophylaxis, diagnosis, or therapy.[1] Ideally, a prescriber would find an optimal balance between these benefits and risks, avoiding medication for which the risk of an ADR exceeds the expected clinical benefits, while at the same time not omitting medication for which the expected clinical benefits exceed the risk of an ADR.[2] However, striking this balance is difficult, especially when prescribing medication to older patients, for whom multimorbidity and altered pharmacodynamics and pharmacokinetics further complicate prescribing decisions.[2,3] As such, older adults are particularly vulnerable to suboptimal or inappropriate prescribing, which may in part explain why older adults are disproportionately affected by ADRs.[4-6]

Several sets of explicit criteria for identifying potentially inappropriate prescriptions in older patients have been developed.[7] Such sets of explicit criteria could be used to support a medication screening program.[8] Screening patients by manually applying such criteria sets would be very time-consuming. However, as these explicit criteria require little or no clinical judgement, they can potentially be automatically evaluated by a computer. For a computer to be able to interpret these criteria, they would first have to be translated from their original human-oriented natural language format, to a computer-oriented formal logic format; this process is known as formalization. A computer system could then use the formalized criteria to provide clinical decision support to a medication reviewer by suggesting changes to the prescribed medication. Because such a Clinical Decision Support System (CDSS) would need to generate patient-specific advice, it would also need access to patient-specific information regarding prescriptions and relevant medical history.

During a previous internship project, aimed at making the medical knowledge contained in the Assessing Care for Vulnerable Elders (ACOVE) quality indicators[9] directly accessible to older patients themselves, a CDSS was developed that obtained the necessary patient-specific information from the patient himself. The goal of this CDSS was to empower older patients to participate more actively in improving the quality of their own care, by suggesting specific topics that the patient could discuss with his physician. To achieve this, the CDSS would first ask the patient a series of “yes”/”no” questions about his medical history and current medical status. It would then reason about this patient information by evaluating formalized ACOVE quality indicators, and it would add a discussion topic to its list of suggestions if it suspected that a quality indicator might not have been satisfied.

The aim of the current project was to investigate the feasibility and effectiveness of applying a similar patient-driven approach to the detection and correction of inappropriate medication prescribing in older adults, in which, based solely on patient information obtained from the patient himself, a CDSS suggests medication changes by evaluating formalized explicit criteria. In working towards this objective, we investigated three main research questions:

● Which of the available explicit criteria sets for the identification of potentially inappropriate prescribing can identify prescriptions for which a correlation with ADRs exists?

● How can we adapt the Logical Element Rule Method (LERM)[10] - a systematic formalization method - to the formalization of explicit criteria intended for use in a patient-driven CDSS?

● What is the accuracy of medication change suggestions generated by a CDSS which uses formalized explicit criteria, when it can only obtain patient information from the patient himself?

Chapter 1 of this document reports on how we investigated the first question through a systematic review of the literature. In Chapter 2 we describe a study, aimed at answering the second question, in which we applied the LERM to two sets of explicit criteria and attempted to adapt or improve the formalization process if problems were encountered. Chapter 3 reports on how we investigated the third question by developing a prototype patient-driven CDSS, using the explicit criteria we formalized in Chapter 2, and evaluating its accuracy in an initial pilot investigation with older patients at an outpatient geriatrics clinic. Finally, Chapter 4 describes an evaluation of the usability of the prototype system that was developed in Chapter 3, aimed at identifying design problems in the patient-facing user interfaces of the system so they may be addressed in a redesigned version of the prototype and will not affect future research.

References

[1] World Health Organization. International Drug Monitoring: The Role of the Hospital. Geneva, Switzerland: World Health Organization; 1966. Technical Report Series No. 425.

[2] Hanlon JT, Schmader KE, Ruby CM, et al. Suboptimal prescribing in older inpatients and outpatients. J Am Geriatr Soc. 2001 Feb;49(2):200-9.

[3] Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA. 1997 Jan 22-29;277(4):301-6.

[4] Kongkaew C, Noyce PR, Ashcroft DM. Hospital admissions associated with adverse drug reactions: a systematic review of prospective observational studies. Ann Pharmacother. 2008 Jul;42(7):1017-25.

[5] Budnitz DS, Pollock DA, Weidenbach KN, et al. National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006 Oct 18;296(15):1858-66.

[6] Beard K. Adverse reactions as a cause of hospital admission in the aged. Drugs Aging. 1992 Jul-Aug;2(4):356-67.

[7] Chang CB, Chan DC. Comparison of published explicit criteria for potentially inappropriate medications in older adults. Drugs Aging. 2010 Dec 1;27(12):947-57.

[8] Van Marum RJ, Verduijn MM, De Vries-Moeselaar AC, et al. Multidisciplinaire Richtlijn Polyfarmacie bij ouderen. Nederlands Huisartsen Genootschap (NHG). 2012.

[9] Shekelle PG, MacLean CH, Morton SC, et al. ACOVE quality indicators. Ann Intern Med 2001 Oct 16;135(8 Pt 2):653-67.

[10] Medlock S, Opondo D, Eslami S, Askari M, Wierenga P, de Rooij SE, Abu-Hanna A. LERM (Logical Elements Rule Method): a method for assessing and formalizing clinical rules for decision support. Int J Med

Chapter 1 - The association between explicit criteria for

potentially inappropriate medication and adverse drug

reactions: a literature review

Abstract

Objective:​To investigate the association between potentially inappropriate medication, as defined by explicit criteria, and adverse drug reactions in older adults.

Methods: ​A search was performed in the PubMed MEDLINE database. Publications were included if they concerned primary research, were available in English, the study population consisted of older adults, and the association between potentially inappropriate medication and adverse drug reactions was one of the reported outcomes. The search results were supplemented by analysis of the references of the included papers. All relevant publications were appraised using the evidence-based librarianship critical appraisal checklist.

Results:_​The search returned 170 publications, nine of which met the inclusion criteria. Searching references of included papers did not add any further publications. One study evaluated the STOPP criteria, one evaluated both the Beers criteria and the STOPP criteria, and seven studies evaluated the Beers criteria, one of which evaluated the Beers criteria in combination with the McLeod criteria. Both studies evaluating the STOPP criteria had sufficient critical appraisal scores and found a significant association. Of the eight studies evaluating the Beers criteria, only three found a significant association. One of these three had a poor critical appraisal score and another reported an association ten times stronger than any other study.

Conclusion:​Evidence supporting the potential of the Beers criteria as a screening tool for adverse drug reactions is weak; evidence supporting the STOPP criteria is preliminary, but thus far is stronger. There is cause for further research into the association between explicit criteria for potentially inappropriate medication and adverse drug reactions, particularly for the most recent revisions of the Beers and STOPP criteria.

 

Introduction

Adverse drug reactions (ADRs) constitute a significant public health issue. Approximately 5.3% of hospital admissions may be ADR-related.[1] The World Health Organisation (WHO) defines ADRs as: any noxious, unintended, and undesired effect of a drug, which occurs at doses used in humans for prophylaxis, diagnosis, or therapy.[2] ADRs are associated with considerable morbidity, mortality and costs.[3-6] Older adults are disproportionately affected.[1,7,8] Factors contributing to this are thought to be: polypharmacy, multimorbidity, cognitive impairment, assisted living, and altered pharmacokinetics and pharmacodynamics.[9,10]

Approximately 16.5% of ADRs in ambulatory care settings and 52.9% of ADRs in hospital based care may be preventable.[11] Potentially inappropriate medication (PIM) is defined as medication for which the risk of an ADR exceeds the expected clinical benefit when given to elderly adults, and which can be replaced by better-tolerated alternatives.[12] Methods for the detection of PIM might therefore be able to help reduce the number of ADRs. A distinction can be made between implicit PIM detection methods, which rely on clinical judgement, and explicit PIM detection methods, which are criteria-based and require little or no clinical judgement.[13] Because limited clinical judgement is required, explicit criteria for PIM are a potential target for (semi-) automation through information technology. At least 13 sets of explicit criteria have been developed, generally using a Delphi method.[14] Such sets of explicit criteria for PIM usually consist of a list of invalid indications to prescribe a specific drug or class of drugs.[13]

This study aims to review the potential of explicit criteria for PIM as a screening tool for the prevention of ADRs in older adults. A review was performed of primary studies that investigated the association between PIM (defined by explicit criteria) as a risk factor and ADRs as an outcome, in older adults. Older adults are defined as people over the age of 65 for developed countries and over the age of 60 for developing countries.[15]

Methods

Inclusion

PubMed MEDLINE was chosen as the database to perform the search. The search query, along with some explanation of the choice of search terms, is available as appendix A. The search was performed on the 30th of March 2015. All studies published on or before this date were included, including e-publications ahead of print. The reviewer screened the title and abstract of all search results. The inclusion criteria were:

● The publication concerned primary research (not a review or editorial). ● The publication was available in English.

● Older adults were the study population.

● The association between PIM as a risk factor and ADRs was one of the reported outcomes. Articles were provisionally included based on title and abstract. Final inclusion was determined after reviewing the full text of the articles. Studies of older adults were identified by provisionally including articles if the title or abstract stated that the population was over 65 for developed countries or 60 for developing countries, or consisted of older adults. Final inclusion was determined based on the age of the study participants.

The search resulted in a set of included publications. References of included articles were screened for additional potentially relevant titles using the same inclusion criteria. If additional relevant articles were found, these were also included.

Quality assessment

The validity of the publications was assessed using the evidence-based librarianship (EBL) critical appraisal checklist.[16] This checklists consists of 28 questions, divided over four sections: population, data collection, study design, and results. Questions can be answered with ‘yes’ (Y), ‘no’ (N), ‘unclear’ (U), or ‘not applicable’ (NA). Validity scores could then be calculated, both for the publication overall and for the individual sections, by dividing the number of questions answered with ‘yes’ by the total number of applicable questions (Y / (Y + N + U)). The EBL critical appraisal checklist states that for scores equal to or above 0.75, the study can safely be considered valid, whereas for scores below 0.75 validity becomes dubious.

Data extraction

For each of the included publications, the following was recorded: the study design, the set of explicit criteria for PIM used, the study population, the sample size, the method used for detecting ADRs, and the association between PIM and ADRs.

Results

Inclusion  

Figure 1: Literature inclusion flow diagram. The publications included from the search were supplemented with potentially relevant references. Abbreviations: PIM = Potentially Inappropriate Medication, ADR = Adverse Drug

The search resulted in 170 publications, which were screened and assessed for eligibility (figure 1). Screening articles referenced by articles included from the search did not add any additional articles. Nine publications were ultimately included for this review (table 1).

Study Study design

PIM criteria Study population Sampl e size

ADR detection Validit y (Y/T) Outcome Cahir et al. 2014[17] Retrospectiv e cohort

STOPP[26] Irish GP patients

≧ 70 y.o. 931 EPR review + structured patient survey 0.79 1 PIM: OR 1.26 (0.77-2.05) ≧ 2 PIM: OR 2.21(1.02-4.83) Hamilton et al. 2011[18] Prospective cohort STOPP[26] Beers 2003[28] Patients admitted with acute illness

≧ 65 y.o. 600 WHO-UMC criteria[30] (specific data collection) 0.95 STOPP: OR 1.85 (1.51-2.26) Beers: OR 1.28 (0.95-1.72) Chrischille s et al. 2009[19] Prospective cohort Beers 1997[27] + interactions + duplications Medicare clients ≧ 65 y.o. with mobility limitations 626 Patient self-reporting 0.76 OR 2.14 (1.26-3.65) Laroche et al. 2007[20] Prospective cohort Beers 1997[27] adapted for France Patients ≧ 70 y.o. presenting to a geriatrics unit 2018 Probable on French causality scale[31] (EPR review) 0.80 OR 1.0 (0.8 - 1.3) Page et al. 2006[21] Retrospectiv e cohort

Beers 2003[28] Patients ≧ 75 y.o. admitted to internal medicine wards 389 Naranjo score[32] ≧ 3 (EPR review) 0.79 OR 1.51 (0.98-2.35) Rask et al. 2005[22] Retrospectiv e cohort Custom based on Beers 1997[27] and McLeod[29] Medicare clients ≧ 65 y.o. 396 Patient self-reporting 0.89 OR 1.42 (0.90-2.25) Passarelli et al. 2005[23] Prospective cohort

Beers 2003[28] Patients ≧ 60 y.o. admitted to internal medicine wards

186 Naranjo score[32]

≧ 5 (EPR review + patient and staff inquiries) 0.71 OR 2.32 (1.17-4.59) Onder et al. 2005[24] Retrospectiv e cohort

Beers 2003[28] Patients ≧ 65 y.o. admitted to internal medicine wards

5152 Naranjo score[32]

≧ 5 (EPR review + patient and staff inquiries) 0.76 OR 1.20 (0.89-1.61) Chang et al. 2005[25] Prospective cohort

Beers 1997[27] Patients ≧ 65 y.o., first visit to outpatient services 882 Naranjo score[32] ≧ 3 (EPR review + telephone survey) 0.75 RR 15.3 (4.0 - 58.8)

Table 1: overview of publications identified as relevant to this review. Validity scores were calculated using the EBL critical appraisal checklist.[16] Abbreviations: EPR = Electronic Patient Record, PIM = Potentially Inappropriate Medication, OR = Odds Ratio, RR = Relative Risk.

The age of participants in the included studies was 65 years and older, with the exception of Passarelli et al.[23], who defined their study population as adults older than 60. However, they performed their study in Brazil and they refer to the WHO definition for “elderly”[15], which defines older adults for developing countries as 60 years old or older, as opposed to 65 years old or older for developed countries.

Data extraction

A limited number of PIM criteria sets were studied: two studies[17,18] evaluated the STOPP criteria[26]; two studies[20,25] evaluated the 1997 revision of the Beers criteria[27]; four studies[18,21,23,24] evaluated the 2003 revision of the Beers criteria[28]. ​Chrischilles et al.[19] evaluated the _​1997 revision of the Beers criteria, supplemented with criteria for drug-drug

interactions and drug duplications, but they report that these additions added only a relatively small number of PIM to the PIM already identified by the Beers criteria. Rask et al.[22] used a custom set of criteria based on the 1997 revision of the Beers criteria and the McLeod criteria[29].

Six out of the nine studies[18,20,21,23-25] used a sample of patients who presented at a hospital in need of elective or acute care. The remaining three studies used community dwelling patients, two[19,22] through Medicare registrations, and one[17] through registrations with general practitioners. Sample sizes ranged from 186[23] to 5152[24].

For the detection of ADRs, Hamilton et al.[18] used the WHO-UMC criteria[30] in combination with patient data that was specifically collected for this purpose. Laroche et al.[20] used the French causality scale[31] with routine information from the electronic patient record (EPR). Cahir et al.[17] used routine information from the EPR, supplemented with a patient survey, without using specific criteria for ADRs. Chrischilles et al.[19] and Rask et al.[22] used patient self-reporting for the detection of ADRs. The remaining four studies[21,23-25] used the Naranjo algorithm[32] with EPR data; three of these studies[23-25] supplemented the EPR data with patient inquiries.

Association between PIM and ADRs

Three[19,23,25] out of the eight studies[18-25] reporting on the association between PIM defined by the Beers criteria and ADRs found the association to be significant. The included studies report odds ratios ranging from 1.0 (0.8 - 1.3)[20] to 2.32 (1.17-4.59)[23], with the exception of Chang et al.[25] who report a relative risk of 15.3 (4.0 - 58.8). The two studies reporting on the association between PIM defined by the STOPP criteria and ADRs both found a significant result. Hamilton et al.[18] found an odds ratio of 1.85 (1.51-2.26) and Cahir et al.[17] found an odds ratio of 2.21 (1.02-4.83) when using two or more PIM as defined by the STOPP criteria. However, Cahir et al. did not find a significant association when using only one PIM (OR 1.26 (0.77-2.05)). Unlike the other studies, Cahir et al. did not report on the association between any PIM (one or more) and ADRs, which makes it difficult to compare their findings.

Quality assessment

One study, from Passarelli et al.[23], scored 0.71 on EBL critical appraisal, which is below the threshold for which validity can be assumed. They scored poorly (0.6) on the population category, because it is unclear whether the sample size is large enough for accurate estimates, and it is unclear whether or not informed consent was obtained. They also scored poorly (0.67) on the results category, because it is unclear whether or not important confounding variables were accounted for and no suggestions for future research were offered. The complete checklist for this study and the checklists for the other studies are available as appendix B.

Discussion

Of the published lists of PIMs, only studies assessing the Beers criteria, the STOPP criteria and the McLeod criteria were identified. Evidence supporting the potential of the Beers criteria for the prevention of ADRs is weak. Out of the eight studies reporting on the Beers criteria, only three found a significant association, one of which was appraised as being of questionable validity. Evidence supporting the potential of the STOPP criteria for the prevention of ADRs is stronger, but still preliminary; only 2 studies were identified and although both report a significant association, one was by performed the original authors of the STOPP criteria.

Chrischilles et al.[19], Passarelli et al.[23], and Chang et al.[25] found a significant association between Beers criteria PIM and ADRs. However, one of these, Passarelli et al., scored poorly in the EBL critical appraisal. Another, Chang et al., report an association that is ten times stronger than the associations reported by the other studies. Further, Chang et al. report a relative risk while the other studies report an odds ratio. Odds ratios tend to be an overestimation of relative risk [33], meaning that the difference in these results compared to the other included studies may be even more pronounced than it seems. This raises some suspicion about the validity of this study as well, although no methodological issues were evident from the article.

Although the Beers criteria were specifically created with the purpose of reducing the risk of ADRs, less than half the studies that were reviewed found a significant association. This may in part be because these studies lack the power necessary to produce significant results. This is supported by the fact that although most of the reported associations are not significant, they are all greater than one; if there truly is no association, we would have expected at least some of the reported associations to have been smaller than one. It could also be that the methods used for detecting ADRs fail to recognize longer term effects of Beers criteria PIMs. However, this may also be due to deficiencies in the Beers criteria. In 2012 another update on the Beers criteria was published through support from the American Geriatrics Society.[34] This iteration used a more rigorous approach using the Institute of Medicine standards[35] and may address these deficiencies in previous versions of the Beers criteria, but this has not yet been evaluated.

Hamilton et al.[18] show a significant association between PIM defined by the STOPP criteria and ADRs. They also received a high critical appraisal score. However, it should be noted that this study was performed by the same group that was responsible for the creation of the STOPP criteria. The STOPP criteria might fit their local practices and population better than it would elsewhere, and therefore external replication of their results is desirable. Cahir et al.[17] are an external group and they do report a significant association for two or more PIMs. However, they do not find a significant association for exactly one PIM and they do not report on the association for one or more PIMs like the other studies. This makes it difficult to compare their findings with the findings of Hamilton et al. The STOPP criteria were also recently revised[36], which may be a good opportunity for further research into their ability to predict ADRs, preferably not only performed by its creators, but also by unrelated research groups.

The association between PIM and ADRs was stronger in the included studies for the STOPP criteria than for the Beers criteria. However, both the Beers and STOPP criteria are lists of medications without a measure of the probability of an ADR, the time frame in which an ADR is expected to occur, or the expected severity. It may be that the difference in association with ADRs is attributable to only a few medications. For example, it may be that one or a few medications on the STOPP list are responsible for the majority of ADRs, or that the Beers list contains medications with ADRs that are less likely to be detected by the methods used in these studies. Investigation of the association between specific medications and ADRs within these studies may illuminate this finding.

Apart from the Beers and STOPP criteria, the only other criteria evaluated were the McLeod criteria, but these were not evaluated separately. Rask et al.[22] combined them with the Beers criteria and they report that most of the PIMs observed in their sample were Beers criteria PIMs (a 2:1 Beers:McLeod ratio). Chang et al.[14] identified 13 sets of explicit criteria in 2010, although the multiple versions of the Beers criteria (the original version, the 1997 revision, and the 2003 revision) were counted individually. Still, for eight of these criteria sets no studies evaluating their ability to predict ADRs were uncovered by the search for this review. However, Chang et al.

also report that two of these - the Zhan[37] and HEDIS[38] criteria - are reclassifications of the 1997 and 2003 Beers criteria respectively and, similarly, the IPFT[39] are a simplification of the McLeod criteria. The PIMs in the ACOVE quality indicators[40] are also based on the Beers criteria. Therefore, their validity as tools for ADR prevention can to some extent be inferred from the results reported for the Beers and McLeod criteria. However, the remaining four - the Rancourt[41], Laroche[42], Winit-Watjana[43], and NORGEP[44] criteria - are unrelated and thus little can be concluded about their value as ADR prevention tools. This review did uncover a study[45] evaluating the German PRISCUS criteria[46], which were published after the review performed by Chang et al. It was excluded here, because it did not report on the association between PIM and ADRs, but they did report a significant association between PRISCUS PIM and hospitalization, which could be considered a surrogate outcome for ADRs, but as it is further down the causal chain there is greater potential for confounding.

This review has several limitations. First of all, only a single database was searched. It is possible that searching additional databases might have uncovered more publications that matched the inclusion criteria. This risk may be somewhat mitigated by having performed a reference analysis. Secondly, article selection, critical appraisal, and data extraction were done by a single reviewer. Other reviewers might have performed article selection and data extraction differently or uncovered validity issues missed by this reviewer. However, we used explicit criteria for article inclusion, data extraction, and quality assessment to minimize this risk. A limitation of the EBL critical appraisal tool is that all questions receive equal weight in this score, even though in reality one might be considerably more concerned when “Is there face validity?” was answered with “no” than when “Are suggestions provided for further areas to research?” was answered with “no”. Thus, the validity scores calculated using the EBL critical appraisal tool may give a somewhat skewed image of the validity of a publication.

In conclusion, evidence for the validity of the Beers criteria for the prevention of ADRs is weak. Evidence supporting the STOPP criteria is stronger, however, it should be considered preliminary as only two studies have been performed, one of which was by the creators of the STOPP criteria. Both were recently revised, but the search performed for this review did not uncover any studies that have evaluated the association with ADRs for these revisions. There is a need for future research that validates the ability to predict the risk of ADRs, not only for these revisions, but also other criteria sets for which this has not yet been evaluated. We recommend one of the outcome measures future studies report be the association between any (one or more) PIMs and ADRs, to make it easy to compare the study with the previous studies identified here. We also recommend future studies use a systematic method for the detection of ADRs, such as the Naranjo algorithm[32] or the WHO-UMC criteria[30]. Of the currently available sets of explicit criteria, the STOPP criteria seem best supported by current evidence as a tool for ADR prevention in older adults.

References

[1] Kongkaew C, Noyce PR, Ashcroft DM. Hospital admissions associated with adverse drug reactions: a systematic review of prospective observational studies. Ann Pharmacother. 2008 Jul;42(7):1017-25.

[2] World Health Organization. International Drug Monitoring: The Role of the Hospital. Geneva, Switzerland: World Health Organization; 1966. Technical Report Series No. 425.

[3] Classen DC, Pestotnik SL, Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA. 1997 Jan 22-29;277(4):301-6.

[4] White TJ, Arakelian A, Rho JP. Counting the costs of drug-related adverse events. Pharmacoeconomics. 1999 May;15(5):445-58.

[5] Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA. 1998 Apr 15;279(15):1200-5.

[6] Pirmohamed M, James S, Meakin S, et al. Adverse drug reactions as cause of admission to hospital: prospective analysis of 18 820 patients. BMJ. 2004 Jul 3;329(7456):15-9.

[7] Budnitz DS, Pollock DA, Weidenbach KN, et al. National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006 Oct 18;296(15):1858-66.

[8] Beard K. Adverse reactions as a cause of hospital admission in the aged. Drugs Aging. 1992 Jul-Aug;2(4):356-67.

[9] Nolan L, O'Malley K. Prescribing for the elderly. Part I: Sensitivity of the elderly to adverse drug reactions. J Am Geriatr Soc. 1988;36:142-149.

[10] Leendertse AJ, Egberts AC, Stoker LJ, et al. Frequency of and risk factors for preventable

medication-related hospital admissions in the Netherlands. Arch Intern Med. 2008 Sep 22;168(17):1890-6. [11] Taché SV, Sönnichsen A, Ashcroft DM. Prevalence of adverse drug events in ambulatory care: a systematic review. Ann Pharmacother. 2011 Jul;45(7-8):977-89.

[12] Laroche ML, Charmes JP, Bouthier F, Merle L. Inappropriate medications in the elderly. Clin Pharmacol Ther. 2009;85:94–97.

[13] Spinewine A, Schmader KE, Barber N, et al. Appropriate prescribing in elderly people: how well can it be measured and optimised? Lancet. 2007 Jul 14;370(9582):173-84.

[14] Chang CB, Chan DC. Comparison of published explicit criteria for potentially inappropriate medications in older adults. Drugs Aging. 2010 Dec 1;27(12):947-57.

[15] World Health Organization. The uses of epidemiology in the study of the elderly. Report of a WHO scientific group on the epidemiology of aging. Geneva: WHO, 1984: 1-84.

[16] Glynn L. A critical appraisal tool for library and information research. Library Hi Tech. 2006 Jun; 24(3):387-399.

[17] Cahir C, Bennett K, Teljeur C, Fahey T. Potentially inappropriate prescribing and adverse health outcomes in community dwelling older patients. Br J Clin Pharmacol. 2014 Jan;77(1):201-10.

[18] Hamilton H, Gallagher P, Ryan C, Byrne S, O'Mahony D. Potentially inappropriate medications defined by STOPP criteria and the risk of adverse drug events in older hospitalized patients. Arch Intern Med. 2011 Jun 13;171(11):1013-9.

[19] Chrischilles EA, VanGilder R, Wright K. Inappropriate medication use as a risk factor for self-reported adverse drug effects in older adults. J Am Geriatr Soc. 2009 Jun;57(6):1000-6.

[20] Laroche ML, Charmes JP, Nouaille Y, Picard N, Merle L. Is inappropriate medication use a major cause of adverse drug reactions in the elderly? Br J Clin Pharmacol. 2007 Feb;63(2):177-86.

[21] Page RL 2nd, Ruscin JM. The risk of adverse drug events and hospital-related morbidity and mortality among older adults with potentially inappropriate medication use. Am J Geriatr Pharmacother. 2006

Dec;4(4):297-305.

[22] Rask KJ, Wells KJ, Teitel GS, et al. Can an algorithm for appropriate prescribing predict adverse drug events? Am J Manag Care. 2005 Mar;11(3):145-51.

[23] Passarelli MC, Jacob-Filho W, Figueras A. Adverse drug reactions in an elderly hospitalised population: inappropriate prescription is a leading cause. Drugs Aging. 2005;22(9):767-77.

[24] Onder G, Landi F, Liperoti R, et al. Impact of inappropriate drug use among hospitalized older adults. Eur J Clin Pharmacol. 2005 Jul;61(5-6):453-9.

[25] Chang CM, Liu PY, Yang YH, et al. Use of the Beers criteria to predict adverse drug reactions among first-visit elderly outpatients. Pharmacotherapy. 2005 Jun;25(6):831-8.

[26] Gallagher P, Ryan C, Byrne S, Kennedy J, O'Mahony D. STOPP (Screening Tool of Older Person's Prescriptions) and START (Screening Tool to Alert doctors to Right Treatment). Consensus validation. Int J Clin Pharmacol Ther. 2008 Feb;46(2):72-83.

[27] Beers MH. Explicit criteria for determining potentially inappropriate medication use by the elderly. An update. Arch Intern Med. 1997 Jul 28;157(14):1531-6.

[28] Fick DM, Cooper JW, Wade WE, et al. Updating the Beers criteria for potentially inappropriate medication use in older adults: results of a US consensus panel of experts. Arch Intern Med. 2003 Dec 8-22;163(22):2716-24.

[29] McLeod PJ, Huang AR, Tamblyn RM, Gayton DC. Defining inappropriate practices in prescribing for elderly people: a national consensus panel. CMAJ. 1997 Feb 1;156(3):385-91.

[30] Uppsala Monitoring Centre. The use of the WHO-UMC system for standardised case causality assessment.​​http://who-umc.org/Graphics/24734.pdf​, consulted on April 17th, 2015.

[31] Begaud B, Evreux JC, Jouglard J, Lagier G. Imputation of the unexpected or toxic effects of drugs. Actualization of the method used in France. Thérapie 1985;40:111-8.

[32] Naranjo CA, Busto U, Sellers EM, et al. A method for estimating the probability of adverse drug reactions. Clin Pharmacol Ther. 1981 Aug;30(2):239-45.

[33] Schmidt CO, Kohlmann T. When to use the odds ratio or the relative risk? Int J Public Health. 2008;53(3):165-7.

[34] The American Geriatrics Society 2012 Beers Criteria Update Expert Panel. American Geriatrics Society Updated Beers Criteria for Potentially Inappropriate Medication Use in Older Adults. J Am Geriatr Soc. 2012 Apr; 60(4): 616–631.

[35] Graham R, Mancher M, Wolman DM, et al. Institute of Medicine: Clinical Practice Guidelines We Can Trust. National Academies Press; Washington, DC: 2011.

[36] O'Mahony D, O'Sullivan D, Byrne S, et al. STOPP/START criteria for potentially inappropriate prescribing in older people: version 2. Age Ageing. 2015 Mar;44(2):213-8.

[37] Zhan C, SangI J, Bierman AS, et al. Potentially inappropriate medication use in the community-dwelling elderly: findings from the 1996 Medical Expenditure Panel Survey. JAMA 2001 Dec 12;286(22):2823-9.

[38] Pugh MJ, Hanlon JT, Zeber JE, et al. Assessing potentially inappropriate prescribing in the elderly Veterans Affairs population using the HEDIS 2006 quality measure. J Manag Care Pharm 2006 Sep;12(7):537-45. [39] Naugler CT, Brymer C, Stolee P, et al. Development and validation of an improving prescribing in the elderly tool. Can J Clin Pharmacol 2000 Summer;7(2):103-7.

[40] Shekelle PG, MacLean CH, Morton SC, et al. ACOVE quality indicators. Ann Intern Med 2001 Oct 16;135(8 Pt 2):653-67.

[41] Rancourt C, Moisan J, Baillargeon L, et al. Potentially inappropriate prescriptions for older patients in long-term care. BMC Geriatr 2004 Oct 15;4:9.

[42] Laroche ML, Charmes JP, Merle L. Potentially inappropriate medications in the elderly: a French consensus panel list. Eur J Clin Pharmacol 2007 Aug;63(8):725-31.

[43] Winit-Watjana W, Sakulrat P, Kespichayawattana J. Criteria for high-risk medication use in Thai older patients. Arch Gerontol Geriatr 2008 Jul-Aug;47(1);35-51.

[44] Rognstad S, Brekke M, Fetveit A, et al. The Norwegian General Practice (NORGEP) criteria for assessing potentially inappropriate prescriptions to elderly patients: a modified Delphi study. Scand J Prim Health Care 2009;27(3):153-9.

[45] Dormann H, Sonst A, Müller F, et al. Adverse drug events in older patients admitted as an emergency: the role of potentially inappropriate medication in elderly people (PRISCUS). Dtsch Arztebl Int. 2013

Mar;110(13):213-9.

[46] Holt S, Schmiedl S, Thurmann PA. Potentially Inappropriate Medications in the Elderly: The PRISCUS List. Dtsch Arztebl Int. 2010 Aug; 107(31-32): 543–551.

Chapter 2 - Applying the LERM to patient-driven decision

support based on the START / STOPP and ACOVE criteria:

critical assessment and modifications

Abstract

Objective:_​To create a knowledge base for a patient-driven decision support system by formalizing two guidelines consisting of clinical rules using the Logical Elements Rule Method (LERM) and to improve or adapt the LERM formalization process if problems were encountered.

Methods: A software tool was created to improve the consistency the formalization process and to better inform formalization choices. The LERM was subsequently applied to the selected clinical rules with the aid of this tool, in an iterative manner. When a problem was encountered, the process was adapted to resolve this problem. The modified process would then also be retroactively applied to the already formalized rules.

Results:​Step 5 and 7 or the LERM were adapted for use with a decision support system in which the information available for reasoning is not predetermined and extracted from a pre-existing data source, such as an electronic patient record, but is instead actively obtained from the patient himself through questions and tasks. Also, an additional step was added to the formalization process to prevent the system from triggering false positive decision support messages. With the help of the software tool and the modified LERM formalization process, 85% of the 110 selected clinical rules were partially or fully formalized and implemented as 136 logical rules.

Discussion: We successfully adapted the LERM to our specific decision support implementation and created a software tool to improve the consistency the formalization process and to better inform formalization choices. The modifications to step 5 and 7 of the LERM might only be useful to decision support implementations that do not use a pre-existing data source. However, the additional step aimed at preventing false positives could be a general improvement to the LERM.

Introduction

Clinical guidelines are systematically developed statements to assist practitioner and patient decisions about appropriate healthcare for specific clinical circumstances.[1] As such, they are an attractive knowledge base for Clinical Decision Support Systems (CDSSs): computer systems that combine computerized clinical knowledge with patient-specific health information in order to provide patient-specific clinical advice to clinicians.[2] However, clinical guidelines written in natural language are not directly suitable for use in such CDSSs, as computer processing of natural language in biomedicine is a complex and unsolved problem.[3] Therefore, before a guideline can be used as the knowledge base for a CDSS, it needs to be transformed from a document written in natural language, to a formal logical algorithm that can be executed by a computer: a process known as guideline formalization.

The Logical Elements Rule Method (LERM) is a step-by-step method for transforming clinical rules for use in decision support.[4] The LERM was created for the purpose of formalizing guidelines or quality indicators that consist of clinical rules: elementary, isolated care recommendations. It is less suitable for formalizing narrative guidelines, which describe a process of care with branching decisions unfolding over time, although it is possible to extract clinical rules from narrative guidelines.[5]

A new standalone decision support system was being developed, with the purpose of detecting potentially inappropriate prescribing in elderly patients. This CDSS was to be patient-driven, that is, it would have to obtain all of the necessary patient-specific information from the patient himself through tasks and questions. Two sets of guidelines were selected to serve as the knowledge base for this system: the START and STOPP criteria version 1 adapted for the Netherlands[6], and domain I and II of an adaptation of the ACOVE criteria by Bijleveld et al. for the Netherlands.[7] However, this meant these guidelines first had to be formalized. Both of these guidelines define explicit criteria for the identification of potentially inappropriate prescribing: clinical rules describing when certain drugs or classes of drugs should or should not be prescribed. As these guidelines exclusively consist of clinical rules, the LERM was chosen as a framework for this formalization process.

The goals of this study were twofold. The first goal was to create formalized versions of the two selected guidelines, suitable for use in the patient-driven target system. The second goal was to adapt or improve the LERM formalization process if problems were encountered.

Methods

Rule selection

Not all of the rules in the selected guidelines were suitable for this decision support implementation. If a rule matched any of the following conditions, the rule would be excluded:

● None of the actions implied by the rule concern changes to the patient’s medication. The target system was only concerned with providing advice on changes to medication. ● The rule only applies to hospital care. The target system was not intended for use in a

hospital setting.

● The rule only concerns (sub-)acute care. The target system’s intent was to help manage chronic drug therapy, not to provide assistance in acute situations.

● The rule was removed from START and STOPP version 2.[8] This updated version of START/STOPP was published during the course of this research, but unlike the original

START/STOPP version, it had not yet been reviewed for, and adapted to, the Dutch situation. However, since version 2 removed rules where the validity had been brought into doubt, the rules that were elided in version 2 were also excluded from our implementation.

Formalization

Software tool

The original approach was to record the formalization process in text documents or in a tabular format, similar to how a previous study applied the LERM.[9] However, during the planning process concern arose that, as this project included over twice as many rules as the previous study, tracking the relationships between rules would be considerably more difficult and time consuming using unstructured data. Also, having structured input forms was expected to improve the consistency of the formalization procedure. It was therefore decided that a web-based software tool was to be created to assist with formalization. The software tool was created by one of the researchers using the Ruby programming language and the Ruby on Rails web application framework.

Modifications to the LERM

The LERM method consists of seven steps. An overview of these steps is given in figure 1. The LERM process was applied to the clinical rules in an iterative manner: the LERM was followed as closely as possible, until a problem with the process was encountered. When a problem was encountered, the process was adapted to resolve this problem. The modified process would then also be retroactively applied to the already formalized rules.

Figure 1: Overview of the seven steps of the Logical Elements Rule Method (LERM). Figure originally appeared in Medlock et al. 2012[9]

Formalization of selected rules

The formalization process was carried out by a medical informatics student with a background in medicine (bachelor’s degree). The LERM involves making certain decisions that require clinical expertise. If the formalizer was unsure about such a formalization decision, he would make a preliminary decision, but mark it for review. These decisions were then discussed with a geriatrician-clinical pharmacologist at a later time, after which a final decision was made.

Results

Rule selection

A total of 141 clinical rules (START: 27, STOPP: 61, ACOVE: 53) were identified in the selected guidelines. Of these, 31 clinical rules (22%) were excluded (START: 3, STOPP: 11, ACOVE: 17): 3 because none of the actions implied by the rule concerned medication changes, 9 because they only concerned hospital care, 5 because they only concerned (sub-)acute care, and 14 because they were removed from START/STOPP version 2.

Formalization

Software tool

The software tool provides interfaces for creating, updating, removing, and searching clinical rules formalizations. The process of creating new formalized rules has been split into six steps, corresponding to the first six steps of the LERM. A more detailed description of what the creation of new formalized rules using this tool looks like, including some screenshots, is available as appendix C.

Modifications to the LERM process

Modifications were made to steps 5 and 7 of LERM, and one additional step was added.

Step 5: Restating as data elements

Step 5 of the LERM involves restating the relevant phrases of a rule in terms of data elements: the units of clinical information which will be used by the target decision support system. However, for this study’s target system the data elements were not predetermined, and were created as part of the formalization process.

The only data source available to this target system is the patient himself; instead of relying on a medical practitioner for data input, or extracting it from a pre-existing data source such as an electronic patient record, this system relies solely on the patient himself to input all the data it needs for its reasoning. The patient is asked to input his age, input his medication list, and is presented with a set of statements (such as “I have asthma.”), for each of which the patient then has to indicate whether or not the statement applies to him. Consequently, the following four different data element types were available:

1. The “age range” data element type: this type of data element represents whether or not the patient’s age falls within a certain range. This type can be used to restate phrases such as: ​“70 years old or older”​.

2. The “drug compound” data element type: this type of data element represents whether or not one of the drugs entered by the patient contains a particular generic compound. This type can be used to restate phrases such as: ​“uses metoprolol”​.

3. The “drug group” data element type: this type of data element represents whether or not one of the drugs entered by the patient belongs to a certain drug group. This type can be used to restate phrases such as: _​“uses a beta blocker” ​. If possible, data elements of this type were defined based of how drugs are grouped on the Farmacotherapeutisch Kompas website (Dutch drug formulary) in December 2014.[10] If no reasonable group could be found there, the formalizer would compose a custom group, which would later be reviewed by the geriatrician-clinical pharmacologist.

4. The “statement applicability” data element type: this type of data element is associated with a statement. The statement is presented to the patient and the patient has to indicate whether or not the statement applies to him. This type can be used to restate phrases such as: ​“was diagnosed with asthma”​.

Therefore, the following was added as a substep to step 5:

Given a certain phrase, the formalizer first attempted to link it to one of the first three data element types, either by creating a new data element or reusing one if possible. If none of the first three data element types were applicable, the formalizer attempted to link it to a “statement applicability” data element type, by creating a new statement or reusing a previously created statement. Because the applicability of the statement is to be judged by a patient, not by a medical practitioner, care was taken to avoid medical jargon as much as possible when phrasing a statement, which tended to reduce its specificity.

Step 7: Determining data element availability

Step 7 of the LERM involves determining whether or not all the data elements are available in the system and, if a data element is missing, discussing whether or not is would be worthwhile to start recording this data element. However, as explained, the data elements available to our target system were not predetermined. Also, notice the “catch-all” nature of the “statement applicability” data element type: it is relatively straightforward to turn any phrase into a statement. However, the usefulness of this data element type is restricted by a patient’s knowledge. Even if care is taken to avoid medical jargon as much as possible, for certain statements, most patients would still lack the knowledge necessary to determine whether or not the statement is applicable; simply creating statements for all phrases would result in a significant number of mostly useless statements. Therefore, in order to reduce the number of statements that would have to be presented to a patient, step 7 was replaced with the following:

It was decided that phrases dependent on certain types of knowledge would be deemed “unimplementable”, specifically:

● Phrases depending on knowledge of (laboratory) test results, such as glomerular filtration rate, urine albumin, and blood glucose.

● Phrases depending on knowledge of medical physical examination observations, such as blood pressure, BMI, and pulse rate.

● Phrases depending on dosing information.

● Phrases depending on medical jargon that could not be eliminated without losing the intent of the phrase. Whether or not this is the case is subjective and this decision was left to the discretion of the formalizer. If the formalizer was uncertain about this decision, it would be marked for review and discussed with the geriatrician-clinical pharmacologist.

Before declaring such phrases unimplementable, the formalizer would first attempt to translate them to a different kind of knowledge. For example, the phrase _​“has a glomerular filtration rate less than 30 ml/min” could be translated to ​“was diagnosed with renal insufficiency”​. If the formalizer could not come up with a reasonable translation, the phrase was marked as “unimplementable”.

Additional step: Add negation of conclusion to condition

Typically, a rule formalized with the LERM method is of this general form:

IF conditional state THEN desirable state

Where “conditional state” is a logic expression that expresses the combination of conditions that should trigger this rule and “desirable state” is a logic expression that expresses the goal state that the actions triggered by the CDSS should contribute to. However, implementing a rule like this would trigger false-positive actions: some patients may be in the conditional state, thus triggering the rule, but they are also already in the desirable state, thus the triggered actions are unnecessary. To prevent such false-positives an additional step was added to the formalization process, in which the formalizer added the negation of the desirable state to the rule’s condition:

IF conditional state AND NOT desirable state THEN desirable state

Now the rule will only trigger for patients who are not already in the desirable state. A concrete example:

IF [NSAID] AND [70 or older] THEN [proton-pump inhibitor]

A CDSS implementing this rule could trigger an alert telling the physician to prescribe a proton-pump inhibitor when it encounters a patient who uses a drug belonging to the NSAID family of drugs and who is also 70 years old or older. However, this implementation would trigger a lot of false-positive alerts: many patients who use an NSAID are already prescribed a proton-pump inhibitor. To avoid these false-positive alerts, the rule can be modified using the additional step described above:

IF [NSAID] AND [70 or older] AND NOT [proton-pump inhibitor] THEN [proton-pump inhibitor]

Now the rule will only trigger for patients who do not already use a proton-pump inhibitor.

Formalization of selected rules

A rule was considered fully implementable if all logical rules extracted from it could be fully restated in terms of data elements. A rule was unimplementable if none of the logical rules extracted could be fully restated in terms of data elements. A rule as considered partially

implementable if at least one of the logical rules extracted could be fully restated in terms of data elements. Of the 110 included rules, 81 (74%) were fully implementable (START: 16, STOPP: 39, ACOVE: 26), 17 (15%) were unimplementable (START: 2, STOPP: 10, ACOVE: 5), and 12 (11%) were partially implementable (START: 6, STOPP: 1, ACOVE: 5).

Formalization of the 110 included clinical rules resulted in 163 logical rules (START: 47, STOPP: 70, ACOVE: 46). Of the 46 logical ACOVE rules, 26 (57%) were duplicates of logical START or STOPP rules. Duplicate rules were only implemented once in the target CDSS. One ACOVE rule conflicted with a START rule and the START rule was chosen over the ACOVE rule. Eliminating the conflicting rule and the duplications gives a final result of 136 implementable logical rules. This resulted in the creation of 159 data elements to implement these logical rules, specifically: 48 “drug compound” data elements, 41 “drug group” data elements, 5 “age range” data elements, and 65 “statement applicability” data elements. The full formalization results are available online (​http://pim-aid-lerm.herokuapp.com/_​).

Discussion

We modified step 5 and 7 of the LERM to better fit our specific decision support implementation, in which the information available for reasoning is not predetermined and extracted from a pre-existing data source, such as an electronic patient record, but is instead actively obtained from the patient himself through questions and tasks. Also, an extra step was added to the formalization process to prevent the target system from triggering false positive decisions support messages. With the help of a software tool and the modified LERM, 85% of the included clinical rules were partially or fully formalized and implemented as 136 logical rules.

Although evaluation of the software tool that was created to help with the formalization process was not part of this study, the formalizer did perceive some substantial benefits of using this tool. Because the tool made data elements easily searchable, reuse of data elements was promoted. This helped limit the number of data elements that had to be created, which was especially important for “statement applicability” data elements, as this also meant the list of statements that end-users of the target system would need to sift through would be reduced. Also, reusing data elements allowed the tool to automatically track relationships between rules. For certain rules, it identified as many as 20 such relationships; in the time available, it would likely not have been feasible to identify relationships manually in such detail. The conflict between an ACOVE rule and a START rule, as well as the duplications, were all discovered through these automatically detected data element relationships. Although some of these may also have been discovered by hand, a significant portion of them might have gone unnoticed.

As described, the formalization steps suggested by the LERM were extended or replaced on three occasions. Step 5 was extended to include the creation of data elements, step 7 was replaced with a step that involved judging data element implementability, and an additional step was added that added the negation of a rule’s consequence to the rule’s condition in order to reduce false positives. Previously, Medlock et al.[9] also reported on some modifications to the LERM as a result of applying the method to a set of ACOVE rules. First of all, they suggest adding the transformation of rules to a consistent grammar as an additional substep to step 2 (breaking the rules into normal form). This was not included as an explicit step in this study, but due to the strict nature of the input forms provided by the software tool, the formalizer was implicitly forced change all rules to a consistent grammar when performing step 2. Medlock et al. also targeted a decision support platform that did not support a standard terminology and thus created their own set of definitions with which they expressed the data elements. This is analogous to the way in