Guideline representation and execution tools: An evaluation study

(1)

1 |

P a g e

GUIDELINE REPRESENTATION

AND EXECUTION TOOLS

An Evaluation Study

MAGDALENA GAMBA

MASTER THESIS

SUPERVISORS:

Stephanie ‘Ace’ Medlock, DVM, PhD

Danielle Sent, PhD

UNIVERSITY OF AMSTERDAM

(2)

2 |

P a g e

Student:

Magdalena Gamba

Student Number: 10916520

Email:

gamba_m@yahoo.com

SRP Mentor:

Stephanie “Ace” Medlock, DVM, PhD

Department of Medical Informatics, AMC

University of Amsterdam

SRP Tutor:

Danielle Sent, PhD

Department of Medical Informatics, AMC

University of Amsterdam

SRP address:

Department of Medical Informatics, AMC

University of Amsterdam

Meibergdreef 15

1105 AZ Amsterdam

Period:

(3)

3 |

P a g e

1

GENERAL INTRODUCTION

Clinical guidelines are recommendations pertaining to the evaluation, diagnosis, and treatment

of specific medical conditions. They are developed by panels of clinical experts for use in

medical practice, and are based on best evidence obtained from research, as well as literature

and/or consensus. In a 2011 consensus report issued by IoM

1

, guidelines are defined as

“statements that include recommendations intended to optimize patient care that are informed

by a systematic review of evidence and an assessment of the benefits and harms of alternative

care options” [1]. The report goes on to state that in order to optimize patient care, clinical

guidelines need to be trustworthy, having the potential “to reduce inappropriate practice

variation, enhance translation of research into practice, and improve healthcare quality and

safety”.

Clearing houses such as the US Agency for Healthcare Research and Quality Guideline

Clearing House (www.guideline.gov) systematically categorize guidelines into domains,

thereby helping users to identify and choose guidelines pertinent to their field of study, research

or practice [2]. Guidelines can be freely downloaded as text based documents from such

websites, however the provision of guidelines in this format is not conducive to their actual

implementation in practice. In the Netherlands, the Dutch Guideline Database (www.

richtlijnendatabase.nl), a project of the Comprehensive Cancer Centre the Netherlands (IKNL)

and the Knowledge Institute of Medical Specialists (KiMS), comprises of guidelines used

mainly in hospital settings. In addition to text based format, the database presents guidelines as

modules, where each module refers to a different aspect of a guideline. This structure makes it

easier for users to refine their searches, especially through use of keywords and filters.

Guidelines are usually composed of large documents that are cumbersome and time consuming

to read, and are difficult to update when new scientific evidence is made available [3]. Medical

practitioners may not have enough time or skill to find all the information they need or to keep

up with updates, and may not be able to recognize when information is untrustworthy or

reconcile conflicting information [4]. Terms within guidelines may not be clearly defined and

recommendations vaguely worded [5]. They may contain inconsistencies, ambiguities, and

1

_{Institute of Medicine (IoM), currently known as the National Academy of Medicine (NAM), is an American}

non-profit non-governmental organization that provides advice on national issues concerning biomedical

science, medicine and health.

(7)

7 |

P a g e

logical errors which make the implementation of these guidelines into practice difficult [6].

Text based guidelines also contain broad information on diagnoses, diseases and treatments,

which limits their usability at point of care as they are not customized for use with specific

patients [7].

To address the issues above and facilitate the implementation of clinical guidelines into medical

practice, many representation and execution tools have been developed. Computerizing

guidelines eliminates the need for medical practitioners to read text based guidelines in their

entirety, as well as reduces the time needed for the transfer of new medical knowledge into

practice [8]. By integrating them with electronic medical records, patient tailored

recommendations can be given at point of care, increasing the probability of having an impact

on clinician behavior as opposed to using only narrative guidelines [9]. Additionally, the

process helps in the clarification and verification of consistency, logical structure, and

completeness of clinical guidelines [6].

Despite the clear benefits of computerizing guidelines, there are challenges that occur.

Transforming text based guidelines into computerized representations requires medical

knowledge to be manually extracted, formally defined and typed based on the representation

formalism used, and relationships between each defined piece of knowledge explicitly

determined. This is a time consuming activity and requires the cooperation of various domain

experts [10]. As so much time and effort is invested in developing computerized guidelines, it

is important that they can be shared between institutions and integrated into different systems,

which in itself, brings on additional challenges. The formalization process can also cause

variations in guideline interpretation, leaving room for potential errors of omission and biases

[10].

The purpose of our research was twofold. Our first goal was to identify tools that have been and

are currently being used for the representation and execution of guidelines and guideline like

documents, with particular attention to free and open source systems. Our choice to focus on

tools with free and open source licenses was motivated by their advantage over proprietary

software, especially for use within education and research. Free and open source tools can be

used at no cost, and can be freely distributed. Additionally users are given access to the source

code, which gives them the option to modify it in order to make improvements or adapt the tool

to their needs. The identified tools were classified into toolchains comprising of guideline

modeling and execution tools related to a specific representation formalism. Subsequently we

determined their availability, license and critical features such as research group, source of

funding, medical domains in which tools were used, and ascertained their applicability based

on objective evaluation criteria derived from literature. Our second goal was to evaluate freely

available tools, especially free and open source tools, using the aforementioned criteria,

determining their comparative advantages and disadvantages.

(8)

8 |

P a g e

Our research questions are presented below:

1. What tools are used for the representation and execution of guidelines and guideline like

documents, with particular attention to freely available systems, especially free and

open source tools.

2. What are the comparative advantages and disadvantages of at least two of these tools,

in creating an initial guideline representation and updating it with a new version of the

same guideline? Based on experimental results, how can these tools be improved?

3. What tools should be recommended for teaching purposes, research and adoption as a

standard for guideline implementation in clinical practice?

In the following chapter, we report the results of our literature review. Chapter 3 discusses

evaluation criteria derived from literature. In Chapter 4 we evaluate two freely available

systems in their function to represent and execute a portion of a clinical guideline. Chapter 5

presents our general discussion and conclusion.

(9)

9 |

P a g e

2

LITERATURE REVIEW

Introduction

Knowledge representation is defined as ‘the application of logic and an ontology in order to

construct computable models of some domain’[153]. Knowledge representation methods and

tools enable the transformation of text based guidelines into computerized format, making their

tacit knowledge explicit. Formal and expressive representation models provide an in-depth

understanding of clinical procedures, and a precise and unambiguous description of the

guideline itself [140]. To facilitate the implementation of guidelines into clinical practice and

provision of point of care recommendations, the knowledge that has been represented in

computerized format has to be interpreted by a computer program known as a parser, then

applied as an action against patient data. This is known as knowledge execution.

Following the introduction of Arden Syntax in 1989, one of the first methods used to represent

guideline knowledge, research into guideline representation and execution methods and tools

gained momentum in the nineties and early 2000’s. Various research groups developed their

tools based on five main approaches, namely rule-based, document-centric, expression

languages, model-centric, and decision trees. In the rule based approach, formalizations use sets

of if-then statements to create assertions, as well as rules on how to act upon these

assertions[11]. Examples include Arden Syntax [12] and the more recent guideline definition

language (GDL) [13]. Document-centric approaches organize heterogenous information from

text based guidelines into formal models by identifying and tagging components relevant to

their operationalization [14]. GEM [15] is one of the more known document centric

approaches. Approaches using expression languages such as GELLO [16] formalize guidelines

as standard queries and expressions for decision support. In model-centric approaches,

conceptual models (task network models) are used to represent guidelines as hierarchical

structures that contain networks of actions and decisions that unfold over time [17]. Generated

models visually range from simple flowcharts like in GLIF [18] to three dimensional

metaphoric representations that use familiar day to day objects to represent different aspects of

a guideline like in Asbru [19]. The last approach comprises of decision trees, models augmented

with probabilities and utilities to provide an optimal action strategy [9]. They take into

consideration the uncertainty of the outcome of selected treatments by specifying prior

probabilities in the population for different outcomes, and the decision-maker’s preferences by

specifying their utilities for different outcome states [9]. An example of this approach is the

EsPeR system [20]. Fuzzy Cognitive Maps (FCMs) are similar to decision trees. In addition to

(10)

10 |

P a g e

representing guideline knowledge, they enable reasoning with causality and uncertainty. An

example of its use can be seen in [21].

To implement these approaches, a number of tools have been developed under either free and

open source or proprietary licenses. Free and open source licenses allow users to run software

for any purpose, modify original source code, and freely redistribute original or modified copies

[154]. Proprietary software on the other hand, keeps its source code hidden, and does not allow

any modifications or free redistribution.

There is no recent complete overview of guideline representation and execution tools in

literature. In the first part of our study, we address this problem and answer our first research

question through a literature review. Particular attention is given to freely available tools,

especially free and open source systems.

Methods

A comprehensive search of peer review journals, including conference papers and reports, was

conducted using Medline and Embase databases. The search consisted of the following

keywords and their synonyms: ((((<knowledge representation> or <knowledge execution>) and

<guidelines>) or <computerized guidelines>) or (<open source> and <decision support>)). The

search was limited to English language papers with no restrictions on date of publication. The

final search was performed on February 22, 2017 and is shown in Appendix A.

Title and abstract screening, full text screening, and data extraction were performed by one

researcher (M.G.). A second researcher (S.M.) was consulted when a decision could not be

reached after a full text review. A third reviewer was to be consulted in case of disagreement.

Additional papers were selected through a bibliographic search.

The inclusion and exclusion criteria of our literature review were decided by 3 researchers

(M.G., S.M., D.S.) and are presented below.

Inclusion criteria:

1. Papers that describe or mention the use of a tool or a framework for guideline

representation and/or execution.

2. Papers that present specifically developed guideline based decision support systems

built atop general purpose tools such as Protégé and inference or rule engines.

3. Papers that describe the development of a guideline based decision support system built

using tools for guideline representation and/or execution.

(11)

11 |

P a g e

4. Papers that define evaluation criteria or design criteria for guideline representation and

execution tools or systems

2

.

Exclusion criteria:

1. Literature reviews and syntheses.

2. Papers that describe a methodology without mention of the tools used.

3. Papers on proposed languages with no tools developed at the time of writing.

4. Papers that describe decision support systems built using general purpose tools such as

C++, Prolog, CLIPS, etc.

5. Papers that describe tools for building guideline based decision support systems that are

used for purposes other than knowledge representation and execution.

6. Papers describing representation and execution of knowledge other than guidelines such

as care plans, clinical pathways, or knowledge in general.

7. Papers on perception of use of methods and tools.

8. Unpublished papers such as theses.

The resulting papers were grouped into toolchains, and the following information was

extracted:

1. Title, author, date of publication.

2. Name of tool with description of its functionality.

3. Institute or department (including country) where research was done, as well as source

of funding for the research.

4. Focus of research – development of a tool for guideline representation and/or execution

versus development of a guideline based decision support system.

5. Medical domain in which the tool or decision support system was used.

6. Number of patient records used for validating the tool or decision support system.

7. Use of Protégé as an authoring tool in the framework.

8. Possible evaluation criteria.

Results

206 papers were reviewed in full by the first researcher(M.G.), 22 by the second researcher

(S.M.), and as there was no disagreement between the first and second researcher, the third

researcher was not consulted. In total 131 papers identifying 19 toolchains were included.

Majority of the papers presented tools originating from one toolchain. In two papers, more than

one toolchain was discussed. The first paper was a case study that compared and evaluated the

formalization of a guideline using six different methodologies and tools [17]. The second paper

described the development of a decision support system built atop two different toolchains [22].

Figure 1 presents the results of our article screening and inclusion.

(12)

12 |

P a g e

Figure 1: Screening Flow Chart

Table 1 summarizes our literature review results. In the first three columns we present basic

information that includes the name of the tool chain, the year the representation formalism the

toolchain is based on was introduced, and the name of the institute that developed the

formalism. The next two columns present the number of papers referencing a specific toolchain

in our literature review, as well as the year of the most recent publication to help determine if

research on a specific toolchain is ongoing. The next column indicates if Protégé was used as

an authoring tool within a specific toolchain. Protégé is a general purpose knowledge authoring

tool that has been adapted for use by different groups to represent guidelines using different

formalisms. In our last column we identify the domains a toolchain has been tested or used in.

A brief description of each toolchain follows.

(13)

13 |

P a g e

Table 1: Literature Review Results

Toolchain Year Introduced

Founding Institute(s) Last Paper Paper Count Protégé as Authoring Tool

Domain(s) in which toolchain has been applied/tested

Arden Syntax 1989 Columbia Presbyterian Medical Center & IBM, USA

2016 6 No DVT, Lyme Borreliosis, Obstetrics, Screening and Preventative guidelines

PROforma 1992 Advanced Computation Laboratory of Cancer Research, UK

2014 10 No Acute Asthma, ALL, Breast Cancer, CHD, Chronic Cough, Dyspepsia, HTN, IBS, Thyroid Nodules Prestige 1995 Collaboration of 30 organizations

from 8 countries

2000 3 No Angina, Diabetes

EON 1996 Stanford University, USA 2003 7 Yes Asthma, Breast Cancer, Chronic Cough, HIV, HTN, Influenza Vaccine

PRODIGY 1996 University of Newcastle upon Tyne, UK

2004 5 Yes Asthma, Chronic Cough, HTN, Stable Angina GASTON 1997 Eindhoven Technical University &

Maastricht University, The Netherlands

2009 8 Yes Anesthesia, Diabetes, HF, HTN, Leukemia, Psychiatry GLARE 1997 University of Eastern Piedmont &

Azienda Ospedaliera S. Giovanni Battista Hospital, Italy

2013 14 No Bladder Cancer, HF, Management of harmful drinking and alcohol dependence, Non-Hodgkin lymphoma, Reflux

esophagitis, Severe Trauma Management, Stroke Asbru 1998 Vienna University of Technology,

Austria & Stanford University, USA

2009 10 No Breast Cancer, Chronic Cough, Diabetes, HTN, ICU, Jaundice, Mechanical Ventilation of Newborn Infants GLIF 1998 InterMed Collaboratory: Harvard,

Columbia & Stanford Universities, USA

2014 22 Yes Acute Stroke, Acute MI, Angina, Cancer (Breast, Colorectal), CHF, Chronic Cough, Chronic Hepatitis C,

Depression, Dermatology, Diabetes, DTP, Influenza Vaccination, Hypercholesterolemia, Hyperkalemia

screening, HTN, LBP, Sleep Apnea, Pulmonary Embolism, Thyroid screening, Tobacco Dependence GUIDE/

NEWGUIDE

1998 University of Pavia, Italy 2005 9 No Acute Stroke, AML, Breast Cancer, Chronic Cough, HF, HTN, Pressure Ulcer

GEM 2000 Yale University School of Medicine, USA

2010 4 No Extravasation of infused medication, Smoking cessation DeGeL 2001 Ben Gurion University in Beer

Sheva, Israel

2016 14 No Cardiology, CHF, COPD, Endocrinology (Primary Hypothyroidism), Gynecology, HF, Hypercholesterolemia,

HTN, PID, Preeclampsia, Pulmonology, Toxemia Stepper 2001 University of Economics, Czech

Republic

2004 3 No Unknown SAGE 2002 IDX Corporation, Stanford

University, Mayo Clinic Rochester, University of Nebraska, Intermountain Health Care &

Apelon Inc., USA

2015 10 Yes Acute Stroke, CAP, Diabetes, Headache, HTN, Immunization, Lab Alerting, Pressure Ulcer

HELEN 2004 Heidelberg University Medical Center, Germany

2004 1 Yes Apnea (newborns), Hyperbilirubinemia, Small Cell Lung Cancer, Uveitis

Semantic Web Technologies

N/A Various 2016 8 Yes Asthma, ATP, Breast cancer, HF, HTN, Imaging (RPG), Nursing, Ocular Conditions, Pediatric Stoma, PGx guideline, Pressure Ulcers, Stroke, UTI, Upper RTI

OTHER

SIEGFRIED Unknown Duke University Medical Center, USA

1998 2 No Low back pain HGML

Markup Tool

Unknown Rutgers University, USA 2000 1 No Stroke Medical Text

Markup Tool

Unknown University of Pavia, Italy 2003 1 No Hypertension

Arden Syntax

Overview: Arden Syntax is a well-established rule based formalism used to represent medical

knowledge in the form of medical logic modules (MLM’s), where each one houses the

knowledge needed to make one clinical decision. It enables knowledge sharing between

institutions through the use of ‘curly braces’, which separate parts of code specific to a local

information system.

Arden Syntax was first introduced in 1989, and was declared a standard by

Health Level 7 (HLA7) in 1992. It was initially developed to implement simple clinical alerts,

(14)

14 |

P a g e

reminders and recommendations, but has been used for the implementation of clinical practice

guidelines as well.

Tools: Arden2ByteCode [23], ArdenSuite [23, 27]. Both tools are currently available.

Arden2ByteCode is a free and open source tool downloadable from GitHub. ArdenSuite is

available under license from www.medexter.com with an offer of a 30 day free trial.

Decision Support Systems: EGADSS [23], HELP [26], NEXPERT Object [24, 25], Arden

Syntax GBDSS for Lyme Borreliosis and Obstetrics [3, 27].

PROforma

Overview: PROforma was introduced in 1992 by the Advanced Computation Laboratory of

Cancer Research, UK, as a formalism that combines logic programming and object oriented

modeling [17]. It consists of a graphical notation for knowledge design and a formal

representation language that enables execution. PROforma enables the specification of

guidelines in terms of tasks and collection of tasks, with each task defined as being either a

plan, decision, action or an enquiry [28, 29, 30, 31]. PROforma has been used in building

systems for clinical task management and typically presents an authoring system for

formalizing knowledge, and a software engine for enacting the tasks in a clinical setting [31].

Tools: PROforma graphical knowledge editor [29, 30], PROforma engine [29, 30, 32], Arezzo

[28, 31, 35], Tallis [17, 22, 31, 35]. Arezzo is available under commercial license, whereas

Tallis is available upon request from COSSAC

3

.

Decision Support Systems: GBDSS (monitoring and assessing of hypertensive patients by

pharmacists) [35], COGENT [33], HeCaSe2 [34]. Non-guideline based decision support

systems developed using PROforma include CAPSULE, RAGS, CADMIUM, Arno System,

RetroGram, ERA, and LISA [28, 31].

Prestige

Overview: The Prestige project (Guidelines in Healthcare) was a collaboration of about 30

organizations from eight countries started in December of 1995 [36]. Its focus was on the

application of telematics technologies for the dissemination and implementation of clinical

practice guidelines and protocols. The project led to the creation of several clinical applications

that were linked to established IT environments, with programs for validation for use in clinical

practice. One of the aims of the project included the creation of a shared methodology for

conversion of text based clinical guidelines into a formal machine readable structure that could

be integrated with computerized clinical data to generate recommendations for action. Another

aim was to develop a sound and sustainable process for managing, disseminating and locally

adapting computerized versions of clinical guidelines. Each application created within Prestige

contained a common Protocol Manager, a software component to identify clinical guideline

(15)

15 |

P a g e

recommendations relevant to a specific patient, with interfaces to connect it to the applications

operational front end, patient record, and knowledge server [36].

Tools: GAUDI [36, 155], GLEAM [36], PAT [36]

Decision Support Systems: Prototype GBDSS for the management of diabetes [37]

EON

Overview: EON, developed in 1996 at Stanford University, provided a suite of models and

software components that were used to develop guideline based decision support systems [17].

Research on EON ended in 2003 and its results were carried over to the SAGE project.

Eligibility criteria, definitions of abstractions, guideline algorithms, decision models, and

recommended actions were structured within the Dharma guideline model. The execution

system obtained patient data through the Tzolkin temporal data mediator or from user input in

order to generate situation specific recommendations. WOZ provided explanation services for

other components.

Tools: EON uses Protégé 2000 for authoring guidelines. Padda execution server generates

situation specific recommendations based on patient data [17, 39, 40, 41, 43].

Decision Support Systems: ATHENA [38, 40, 42], T-HELPER [43].

PRODIGY

Overview: PRODIGY (Prescribing RatiOnally with Decision Support In General Practice

study) was introduced in 1996 by the University of Newcastle upon Tyne, UK. It was a

guideline based decision support system, developed in several phases, and used mainly in

chronic disease management. Phase I and II were implemented as modules by system vendors,

extending their existing EPR systems and accessing the coded information to help direct choices

in the guidelines [44, 45]. Phase II was introduced for use in acute disease management. Phase

III focused on the integration of the system with legacy systems through the introduction of

standardized interfaces, terminology mapping and numeric unit conversion knowledge bases

[46]. PRODIGY structured guidelines as a set of choices (patient scenarios) that were to be

made by the physician, and enabled the synchronization of patient management with respective

guideline recommendations.

Tools: PRODIGY used Protégé as its authoring tool [17, 44, 45]; KWIZ [47]

Decision Support Systems: PRODIGY was integrated with two primary care clinical systems

in the UK [46].

GASTON

Overview: GASTON was introduced in 1997, developed jointly by Eindhoven Technical

University and Maastricht University in the Netherlands. The framework consists of a

formalism that uses primitives (based on version 2.0 of GLIF), problem solving methods and

ontologies, for the purpose of representing guidelines varying in complexity and granularity, as

well as in different application domains. GASTON was later extended to include the use of

(16)

16 |

P a g e

intentions, as they made the developed systems more flexible [48]. A guideline authoring

environment allowed authors to define guidelines using the developed representation

formalism, and an execution environment translated the defined guidelines to a symbol level

representation that could be read and processed by an execution engine [50].

Tools: GASTON framework consists of a Protégé based editor and an execution engine [48,

50, 55]; GASTINE (GASTon INtentional Expressions) is built atop GASTON and focuses on

intention based guidelines [54].

Decision Support Systems: CritICIS [49, 50, 51, 52, 53, 54], GRIF [49, 50, 51],

Multidisciplinary psychoactive drug selection advisor system (M-PADS) [49, 50, 51],

TANDEM project [50, 51], Medical Guideline Technology (MGT) project [50, 51].

GLARE

Overview: GLARE is a domain independent system used to acquire, represent and execute

clinical guidelines. The system was introduced in 1997 in a long-term cooperation between the

Dipartimento di Informatica of Università del Piemonte Orientale, Alessandria, Italy, and

Azienda Ospedaliera S. Giovanni Battista, Torino, one of the largest hospitals in Italy. It

presents a limited set of clear representation primitives that cover most of the relevant aspects

of a guideline, and provides facilities to deal with its context dependent character, taking into

account resource availability, times and cost. It allows users to deal with the temporal aspects

of guidelines and is able to automatically detect different forms of syntactic and semantic

inconsistencies in the guidelines being acquired. The execution module also incorporates a

decision support facility which allows users to compare alternative paths while navigating

through the guideline. Over the years GLARE has evolved to provide users with extended

functionalities such as managing of exceptions, enabling semi-automatic adaptation to specific

execution contexts, automatic treatment of temporal constraints, decision making based on

decision theory, model based verification, and execution in a distributed environment.

Tools: GLARE comprises of the following modules: CG_KRM (Clinical Guidelines

Knowledge Representation Manager) [58, 59, 62, 63, 64, 160], CG_AM (Clinical Guidelines

Acquisition Manager) [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 160], CG_EM (Clinical

Guidelines Execution Module) [56, 57, 60, 61, 62, 63, 64, 65, 66, 67, 68, 160], CG_IM (Clinical

Guidelines Interface Module) [56, 63, 64], CG_SS (Clinical Guidelines Support System [58,

59], CG_ES (Clinical Guidelines Evaluation System) [58, 59].

Asbru

Overview: Asbru was developed in 1998 within the Asgaard project by researchers at the

Vienna University of Technology and Stanford University Medical Informatics department.

Asbru is a time oriented and intention based language defined in Backus-Naur form (BNF),

used to represent clinical guidelines and protocols as time oriented skeletal plans [70]. A

hybrid-Asbru representation format is used within the DeGeL framework.

(17)

17 |

P a g e

Tools: AsbruView

[17, 19, 69, 70], Asbru Knowledge Acquisition Tool [70, 71], Asbru

Interpreter [72, 75], Guideline Markup Tool [69, 73, 74], DELT/A [71, 76]. AsbruView, Asbru

Interpreter and DELT/A are freely available for download.

GLIF

Overview: GLIF was introduced in 1998 by the InterMed Collaboratory, a collaboration of

medical informatics departments at Harvard, Columbia and Stanford Universities, with the

support of the National Library of Medicine [17, 18, 81, 88, 89]. Its purpose was to create a

representation model that could be shared among different institutions and software systems,

and to bring together the most useful features of other guideline models while incorporating

health care standards. GLIF specifies an object oriented model for guideline representation and

is essentially a flowchart that represents a temporally ordered sequence of steps [77]. It

encourages a top down process of modeling and supports three levels of abstractions: level 1 is

conceptual, a human readable flowchart of decisions and actions; level 2 is a computable

specification that can be verified for logical consistency and completeness; level 3 is an

implementable specification that has information needed for local adaptation, as well as for

mapping the guideline onto the patient record [18, 82].

Tools: GLIF uses Protégé as an authoring tool [17, 18, 22, 82]; Partners Computerized

Algorithm Processor and Editor (P-CAPE) [85, 90], DSG tool suite [77, 81, 82], GLEE [18, 79,

87, 88, 89], PRESGUID project [80], OCL compliant GELLO engine [16].

Decision Support Systems: Pilot execution engine integrated with MUltimedia Distributed

Record version 2 [83], MEKRES [78, 161], GC3 [84], ASTI [91], EsPeR [158].

GUIDE/NEWGUIDE

Overview: GUIDE was proposed by the University of Pavia in 1998 as a web distributed

framework designed to manage the development and implementation of guidelines, taking

patient and organizational preferences into consideration [94, 96]. The framework initially

consisted of a formal graphical representation language augmented with the possibility of

linking decision analytic models, a front end for accessing the patient database, and an interface

to the organizational model of the practice [94, 96]. The formalized guideline was linked to the

patient record enabling an inference engine to provide real time patient specific advice upon

request of the user. A library of guidelines was also maintained by GUIDE. NewGuide evolved

from GUIDE. It introduced a Virtual Medical Record (vMR), a logging system that allowed the

tracing of health care process details, and a repository for storage of formalized guidelines [93,

94].

Tools: GUIDE consists of an editor [17, 93, 94, 95, 96], inference engine [92, 95] or GET

(Guideline Enactment Tool) [94].

Decision Support Systems: Decision support system for ulcer prevention [97, 98], careflow

(18)

18 |

P a g e

GEM

Overview: GEM, developed in 2000, was derived from previous work with augmented decision

tables done at Yale University School of Medicine [15, 100]. It uses a document centric

approach to define an XML based model that stores and organizes heterogenous information

contained in clinical guidelines. GEM models the names of guideline elements and their

attributes, element contents, and document structure, and defines them into a multi-level

hierarchy, which consists of more than 100 discrete elements in 9 major branches. GEM became

an ASTM standard in 2006. In [100] the authors discuss their efforts in translating GEM II from

XML into OWL, a semantic web ontology language, in order to enable wider integration.

Tools: GEM Cutter [15, 100, 159]. The tool is freely available for download.

Decision Support Systems: Decision support system for smoking cessation [101]

DeGeL (Digital electronic Guideline Library)

Overview: DeGeL was introduced in 2001 at the Ben Gurion University in Beer Sheva, Israel.

The goal was to create and implement a distributed web based modular architecture that enabled

gradual conversion of clinical guidelines from text to a fully structured machine comprehensible

representation, passing through semi structured and semi-formal stages [107]. Asbru was used

for the formal representation, however other languages such as GEM could be used as well. In

[108, 109] there is also mention of the intention to use GLIF ontology. Besides mark up and

run time application tools, DeGeL has offered a variety of accessory tools. These include:

Vayduria, a search, retrieval and visualization tool; an eligibility determination module;

Qualiguide, a retrospective quality assessment tool; IndexiGuide for guideline classification;

and VisiGuide browsing tool. DeGeL offers Idan, an architecture that allows access to any

heterogeneous medical database so as to query raw clinical data and its abstractions. KNAVE

II intelligent visualization and exploration client uses Idan’s computational capabilities to

display and explore the patient's raw data or derived concepts. MEIDA system includes a

vocabulary server and a search engine and enables search and retrieval of standard terms

defined using controlled medical vocabularies. DeGeL.NET is a new version of the digital

library and its different tools. It also uses Asbru ontology, and its main modules include a

knowledge base server, GESHER as the guideline-specification tool, and Spock for runtime

application of clinical guidelines. Other modules include a guideline database that supports the

hybrid multiple ontology representation; a module responsible for guideline-knowledge

creation, reading, updating, and deletion; DeGeLook, a new guideline search engine for

enhanced guidelines retrieval; DeGeLock, an authorization and authentication module for

group-based authorizations; and a web-service API that enables the guideline knowledge-base

server to accept client requests and perform requested transactions.

Tools: Uruz [105, 106, 107, 108], Gesher [102, 103, 104, 110, 112], Spock [102, 103, 106, 108,

109, 113, 114, 115]. Gesher and Spock are available for licensed use.

(19)

19 |

P a g e

Stepper

Overview:

Stepper is a markup tool developed in 2001, that supports step by step formalization

of clinical guidelines. The tool transforms guideline text with increasing level of detail,

rearranges it into an XML knowledge base, and eventually exports it into an operational

representation. Stepper comprises of an embedded XSLT processor, which is responsible for

the non-interactive part of the transformation, while markup and interactive transformations are

carried out via rules expressed in XKBT language. The tool and methodology were created with

the purpose of guiding the formalization process, at the same time reducing the amount of

involvement of the domain expert and minimizing information loss. Stepper is freely available

for download [116, 117, 118].

SAGE

Overview: SAGE (Standards-Based Sharable Active Guideline Environment) project was

started in 2002 as a collaboration between IDX Corporation, Stanford Medical Informatics,

Mayo Clinic (Rochester), the University of Nebraska, Intermountain Health Care, and Apelon

Inc., with the aim of providing a shared guideline model which could be integrated with vendor

systems for decision enhancement, without replacing vendor software functionality. To ensure

interoperability with vendor systems, SAGE employs a common layer of standard information

models (patient data, organizational and workflow models), and terminologies. According to

[124], SAGE model derived elements from PROforma, GLIF, EON, and PRODIGY. SAGE

has been used in tandem with rule and inference engines such as uEngine [7, 121, 126].

Tools: SAGE uses Protégé as its authoring tool [7, 119, 120, 122, 123, 124]; KWIZ [47, 119],

SAGEDesktop [119], SAGE execution engine [119, 123].

Decision Support Systems: GBDSS for headache diagnosis [125].

HELEN

Overview: In 2004, the Department of Medical Informatics at the Heidelberg University

Medical Center in Germany introduced a modular framework called HELEN, that handled both

guideline authoring and execution. Its purpose was to implement clinical guidelines for the

Department of Neonatology [127].

Tools: Comprises of a Protégé based guideline editor, guideline viewer for web based browsing,

engine for executing encoded guidelines [127]. The framework is freely available for download

from the University’s web page.

Semantic Web Technologies and Inference Engines

Overview: Recent research indicates a preference for using semantic web technologies for

guideline representation, and inference engines for their execution. A number of decision

support systems have been developed and are listed below.

Tools: HeD editor (authoring tool) [133], OWL 1 DL, OWL 2 DL and OWL 2 DL + SWRL

(20)

20 |

P a g e

Decision Support Systems: CPG-EX (Protégé and JENA) [128,131]; Decision support system

for nursing clinical practice guidelines (Protégé and JENA) [130]; Decision support system

prototype (Protégé and JESS) [129]; FCM-uUTI DSS (Protégé and EYE) [21, 134]. Protégé, as

well as rule engines JENA and EYE, are free and open source tools.

Other

SIEGFRIED (System for Interactive Electronic Guidelines with Feedback and Resources for

Instructional and Educational Development) was a web based system that was developed to

integrate clinical guidelines into clinical flow by interactively presenting them at the point of

care. It elicited clinical data from the physician and provided appropriate recommendations

based on the underlying clinical guideline. To facilitate the entry of content and logic, the

Guideline Entry Wizard, a tool modeled after the wizard concept in PC based applications, was

developed [135, 157].

HGML Markup Tool was first presented at the 2000 AMIA Annual Symposium by the

Department of Computer Science at Rutgers University in New Jersey. Its aim was to present

a guideline delivery system that allowed a clinician to extract relevant content easily. It enabled

existing guidelines to be manipulated and viewed in different formats, at various levels of detail

based on the needs of the user, while preserving their original published format [136].

Medical Text Markup Tool was an experimental tool developed for the modularized

representation of clinical guidelines. Through text markup, hierarchical modules, which could

be part of a larger module or have other modules as their parts, were formed. Portions of the

guideline were copied and pasted into an XML editor, thereby creating XML files. MTM tool

complemented guideline execution tools such as GUIDE by determining which tasks could be

represented and which portions of text were related to a particular task [137].

Table 2 presents a summary of the toolchains with their main features. For a more detailed

description see Appendix B and C.

(21)

21 |

P a g e

Table 2: Toolchains

Author(s)

System

Function

Institute

Arden Syntax

Lam, 1993 HELP system GBDSS University of Utah, UT, USA Jenders and Barnett, 1993 NEXPERT OBJECT Editor and inference engine Massachusetts General Hospital, MA, USA

Jenders et al., 1994 NEXPERT OBJECT Editor and inference engine Massachusetts General Hospital, MA, USA Gietzelt et al., 2012 Arden2ByteCode

Authoring and Execution University of Braunschweig & Hannover Medical School, Germany

Seitinger et al., 2014 GBDSS for Lyme

borreliosis; ArdenSuite GBDSS; Execution engine

Medical University of Vienna, Medical University Graz, & Medexter Healthcare GmbH, Austria Seitinger et al., 2016 GBDSS for Lyme

borreliosis, Obstetrics; ArdenSuite

GBDSS; Execution engine Medexter Healthcare GmbH; Medical University of _{Vienna; Vienna Hospital Association, Austria}

PROFORMA

Fox et al., 1997 PROforma editor &

enactment engine Authoring and Execution

Imperial Cancer Research Fund & North End Medical Centre, UK

Fox et al., 1998 PROforma editor & _{enactment engine} Authoring and Execution Imperial Cancer Research Fund, UK Bury et al., 2000 Arezzo Authoring and Execution Imperial Cancer Research Fund, UK Bury et al., 2001 PROforma execution engine Execution engine Imperial Cancer Research Fund, UK Peleg et al., 2003 Arezzo Authoring (case study) Cancer Research, UK (PROforma part) Sutton and Fox, 2003 Arezzo, Tallis Authoring and Execution Oxford Brookes University, UK

Sutton et al., 2006 GBDSS using Tallis Authoring and Execution Oxford Brookes University, UK Grando et al., 2012 COGENT Authoring and Execution University of California, San Diego, CA

Isern et al., 2012 HeCaSe2/Tallis Authoring and Execution Rovira i Virgili University; ITAKA Research Group, Spain

Peleg et al., 2014 Tallis (Protégé)

Authoring and Execution (Tallis part)

University of Haifa, Israel; Oxford University, UK; University College London, UK; Whittington Health

NHS Trust, UK; Deontics Ltd., UK; Harvard Medical School, MA, USA; Mayo Clinic College of Medicine, MN, USA; Albano and University la Sapienza, Italy;

University of Leipzig, Leipzig, Germany; Endocrinology Associates, AR, USA; Arcispedale Santa

Maria Nuova, Italy; Odense University Hospital, Denmark

PRESTIGE

Gordon et al., 1997 GAUDI Authoring

Royal Brompton & Harefield NHS Trust, UK; NHS Information Management Centre, UK; Sowerby Unit for

Primary Care Informatics, University of Newcastle, UK Gordon and Veloso, 1999 GAUDI, GLEAM Authoring Royal Brompton & Harefield NHS Trust, UK

Barahona et al., 2000 GBDSS (Diabetes) GBDSS Nova University of Lisbon and UNINOVA, Portugal

EON

Tu and Musen, 1996 T-HELPER system GBDSS Stanford University School of Medicine, CA, USA Tu and Musen, 1999 EON (Protégé based) Authoring Stanford University School of Medicine, CA, USA Tu and Musen, 2000 Dharmapadda Authoring and Execution Stanford University School of Medicine, CA, USA Goldstein et al., 2000 ATHENA GBDSS Stanford University School of Medicine, CA, USA

Tu and Musen, 2000 EON (Protégé based)/

Dharmapadda Authoring and Execution Stanford University School of Medicine, CA, USA Tu and Musen, 2001 Dharmapadda Authoring and Execution Stanford University School of Medicine, CA, USA Peleg et al., 2003 EON (Protégé based) Authoring (case study) Stanford University School of Medicine, CA, USA

PRODIGY

Johnson et al., 1999 PRODIGY Authoring and Execution University of Newcastle upon Tyne, UK; Stanford University School of Medicine, CA, USA Johnson et al., 2000 PRODIGY Authoring and Execution University of Newcastle upon Tyne, UK Johnson et al., 2001 PRODIGY 3 Authoring and Execution University of Newcastle, UK; Stanford University

School of Medicine, CA, US Peleg et al., 2003 PRODIGY Authoring (case study) University of Newcastle, UK (PRODIGY part) Shankar et al., 2004 KWIZ Authoring University of Newcastle, UK; Stanford University

School of Medicine, CA, US

GASTON

De Clercq et al., 1999 CritICIS knowledge base

editor/GASTON Authoring

Eindhoven University of Technology, University of Maastricht, Catharina Hospital, Eindhoven, NL De Clercq, 2000 GASTON Authoring and Execution Eindhoven University of Technology, NL De Clercq et al., 2000 GASTON Authoring and Execution University of Maastricht, NL

De Clercq et al., 2001 GASTON Authoring and Execution University of Maastricht, Eindhoven University of Technology, Catharina Hospital, Eindhoven, NL De Clercq et al., 2001 GASTON Authoring and Execution University of Maastricht, NL Latoszek-Berendsen et al.,

(22)

22 |

P a g e

Latoszek-Berendsen et al.,

2007 GASTON Authoring and Execution

University of Maastricht, MEDECS, Eindhoven, University of Amsterdam, NL Latoszek-Berendsen et al.,

2009 GASTINE Authoring and Execution University of Maastricht, NL

GLARE

Guarnero et al., 1998 CGM Authoring and Execution Laboratorio di Informatica Clinica, Italy. Terenziani et al., 2000 CGM Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2001 CGM Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2002 GLARE Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2003 GLARE Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2004 GLARE Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2004 GLARE Authoring and Execution University of Eastern Piedmont, Italy Molino et al., 2006 GLARE Authoring and Execution University of Eastern Piedmont, Italy Giordano et al., 2006 GLARE Authoring and Execution University of Eastern Piedmont, Italy Bottrighi et al., 2006 GLARE Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2007 GLARE Authoring and Execution University of Eastern Piedmont, Italy Terenziani et al., 2008 GLARE Authoring and Execution University of Eastern Piedmont, Italy Leonardi et al., 2012 GLARE Authoring and Execution University of Eastern Piedmont, Italy Bottrighi et al., 2013 GLARE Authoring and Execution University of Eastern Piedmont, Italy

Asbru

Miksch et al., 1998 AsbruView Authoring Vienna University of Technology, Austria; Stanford _{University, CA, USA} Shahar et al., 1998

Asbru knowledge-acquisition tool; Asbru

Interpreter

Authoring and Execution Stanford University, CA, USA Kosara and Miksch, 2001 AsbruView Authoring Vienna University of Technology, Austria

Kosara et al., 2002 Guideline Markup Tool Mark-up tool Vienna University of Technology, Austria Peleg et al., 2003

AsbruView

GLIF, EON, PRODIGY,

Arezzo, GUIDE

Authoring Vienna University of Technology, Austria (Asbru part) Votruba et al., 2004 Guideline Markup Tool Mark-up tool Vienna University of Technology, Austria Votruba et al., 2004 Guideline Markup Tool Mark-up tool Vienna University of Technology, Austria Votruba et al., 2008 Asbru Interpreter Execution engine Vienna University of Technology, Austria Seyfang et al., 2009 DELT/A Mark-up tool Vienna University of Technology, Medical University of

Vienna, & Danube University Krems, Austria

GLIF

Zielstorff et al., 1998 P-CAPE Authoring and Execution Partners HealthCare System, MA, USA Ohno-Machado et al., 1998 Evaluation Criteria Evaluation Criteria

Brigham and Women's Hospital, MA, USA; Stanford University, CA, USA; Massachusetts General

Hospital, MA, USA; Columbia University, NY, USA; BBN Systems and Technologies, MA, USA;

DSG, MA, USA

Greenes et al., 1999 DSG's tool suite Authoring and Execution DSG Harvard Medical School, MA, USA Boxwala et al., 1999 DSG's tool suite Authoring and Execution DSG Harvard Medical School, MA, USA

Peleg et al., 2001 Evaluation Criteria Evaluation Criteria Stanford University School of Medicine, CA, USA Peleg et al., 2001 DSG's tool suite/Protégé Authoring and Execution Stanford University School of Medicine, CA; DSG

Harvard Medical School, MA, USA Seroussi et al., 2001 ASTI GBDSS Service d'Informatique Medicale, France.

Gillois et al, 2001 EsPeR GBDSS

SPIM, Faculte de Medecine France; Laboratoire SPIEAO, Faculte de Medecine Nancy I, France, LERTIM, Faculte de Medecine, Marseille, France Wang and Shortliffe, 2002 GLEE Execution engine Columbia University, NY, USA

Dufour et al., 2003 PRESGUID project Authoring and Execution LERTIM University of the Mediterranean, France Maviglia et al., 2003 P-CAPE Authoring and Execution Partners Healthcare System, Inc., MA; Harvard Medical

School, MA; Healthvision, Inc., MA, USA Peleg et al., 2003 GLIF (Protégé based) Authoring (case study) DSG Harvard Medical School, MA; Columbia

University, NY, USA Wang et al., 2004 GLEE Execution engine

Columbia University, NY; Stanford University, CA; DSG Harvard Medical School, MA; Eclipsys Corporation, MA, USA; University of Haifa, Israel Boxwala et al., 2004 Protégé/GLEE Authoring and Execution

DSG Harvard Medical School, MA; Stanford University; Columbia University, NY; Eclipsys Corporation, MA, USA; University of Haifa, Israel Kolesa et al., 2005 MUDR2 system Execution engine EuroMISE Center and Center for Biomedical

Informatics, Czech Republic Choi et al., 2007 GLEE Execution engine

Spaulding Rehabilitation Hospital, MA; Columbia University, NY; University of Rochester, NY,

USA

Buchtela et al., 2008 GLIF editor/MEKRES Authoring EuroMISE Center and Center for Biomedical Informatics, Czech Republic

(23)

23 |

P a g e

Buchtela et al., 2008 MEKRES Authoring EuroMISE Center and Center for Biomedical Informatics, Czech Republic Peleg et al., 2008 GLEE Execution engine University of Haifa, Israel

Mei et al., 2011 GELLO engine Execution engine IBM Research, China Liu et al., 2012 GELLO engine Execution engine IBM Research, China

Peleg et al., 2014 Protégé (Tallis) Authoring (GLIF part)

University of Haifa, Israel; Oxford University, UK; University College London, UK; Whittington Health

NHS Trust, UK; Deontics Ltd., UK; Harvard Medical School, MA, USA; Mayo Clinic College of Medicine, MN, USA; Albano and University la Sapienza, Italy;

University of Leipzig, Germany; Endocrinology Associates, AR, USA; Arcispedale Santa Maria Nuova,

Italy; Odense University Hospital, Denmark

GUIDE/NEWGUIDE

Dazzi et al., 1997 GUIDE Authoring and Execution Consorzio di Bioingegneria e Informatica Medica, Pavia, Italy

Quaglini et al., 1998 GUIDE Authoring and Execution

Consorzio di Bioingegneria e Informatica Medica, Pavia; University of Pavia; Istituto Nazionale per lo

Studio e la Cura dei Tumori, Milano, Italy Quaglini et al., 1999 GUIDE Authoring and Execution University of Pavia, Italy Quaglini et al., 2000 GUIDE Authoring and Execution University of Pavia, Italy

Quaglini et al., 2000 GUIDE Authoring and Execution University of Pavia; Consorzio di Bioingegneria e Informatica Medica, Pavia, Italy Peleg et al., 2003 GUIDE Authoring (case study) University of Pavia, Italy (GUIDE part) Ciccarese et al., 2003 NEWGUIDE Authoring and Execution

University of Pavia, Italy; Consorzio di Bioingegneria e Informatica Medica, Pavia, Italy; Ness-ISI Ltd., Beer

Sheva, Israel Ciccarese et al., 2004 NEWGUIDE Authoring and Execution University of Pavia, Italy Ciccarese et al., 2005 GUIDE/NEWGUIDE Authoring and Execution University of Pavia, Italy

GEM

Shiffman et al., 2001 GEM Cutter Mark-up tool Yale School of Medicine, CT, USA Shiffman et al., 2004 GEM Cutter Mark-up tool Yale School of Medicine, CT, USA Shiffman et al., 2004 GEM Cutter Mark-up tool Yale School of Medicine, CT; Yale New Haven

Hospital, CT, USA. Koch, et al., 2010 GEM Cutter Mark-up tool Mayo Clinic, MN, USA

DeGeL

Shahar, 2002 Uruz Mark-up tool Ben-Gurion University of the Negev, Israel Shahar et al., 2003 Uruz Mark-up tool Ben-Gurion University of the Negev, Israel Shahar et al., 2003 Uruz Mark-up tool Ben-Gurion University of the Negev, Israel Shahar et al., 2004 Uruz/Spock Mark-up tool/Execution

Engine Ben-Gurion University of the Negev, Israel Shalom and Shahar, 2005 Gesher Mark-up tool Ben-Gurion University of the Negev, Israel Young and Shahar, 2005 Spock Execution engine Ben-Gurion University of the Negev, Israel Young and Shahar, 2005 Spock Execution engine Ben-Gurion University of the Negev, Israel Shahar, 2006 Uruz/Spock Mark-up/Execution Engine Ben-Gurion University of the Negev, Israel Young et al., 2007 Spock Execution engine

Ben-Gurion University of the Negev, Israel; Wolfson Medical Center, Israel; Stanford University School of Medicine, CA, USA; Palo Alto Health Care System,

CA, USA; GRECC, CA, USA Hatsek et al., 2008 Gesher/Spock Mark-up/Execution Engine Ben-Gurion University of the Negev, Israel Hatsek et al., 2008 Gesher/Spock Mark-up/Execution Engine Ben-Gurion University of the Negev, Israel Hatsek et al., 2010 Gesher Mark-up tool Ben-Gurion University of the Negev, Israel

Topaz et al., 2013 Gesher Mark-up tool

University of Pennsylvania School of Nursing, PA, University of Texas School of Nursing, TX, & University of Minnesota School of Nursing, MN, USA;

Ben Gurion University of the Negev, Israel Shalom et al., 2016 Picard GBDSS Ben Gurion University of the Negev, Israel; Soroka

Medical Center, Israel

SAGE

Campbell et al., 2003 SAGE (Protégé based) Authoring and Execution

University of Nebraska Medical Center, NE; Stanford University Medical Center, CA; IDX Systems Corporation, WA; Intermountain Health Care, UT;

Mayo Medical School, MN; USA Ram et al., 2004 SAGE (Protégé based) Authoring and Execution IDX Systems Corporation, WA, USA Shankar et al., 2004 KWIZ Authoring University of Newcastle, UK; Stanford University _{School of Medicine, CA, US}

Berg et al., 2004 SAGE Desktop Authoring IDX Systems Corporation, WA, USA Tu et al., 2007 SAGE (Protégé based) Authoring and Execution

Stanford University School of Medicine, CA; University of Nebraska Medical Center, NE; GE Healthcare Integrated IT Solutions, WA; Mayo Clinic, MN; Apelon

(24)

24 |

P a g e

Inc., CT; RemedyMD Inc., UT; Hospira, Inc., IL; Kea Analytics, WA; USA

Kim et al., 2008 SAGE (Protégé based) Authoring Kwandong University; InHa University; Seoul National _{University; South Korea} Lee et al., 2010 SAGE + (uEngine +

BRAIN)

Authoring, inference + rule engines

Ajou University; Kwandong University; Inha University; Seoul National University, South Korea Yin et al., 2013 SAGE module GBDSS Zhejiang University, China

Liu et al., 2014 SAGE (Protégé based) Authoring Fourth Military Medical University; China Three Gorges University; China

Kim et al., 2015 SAGE + uBRAIN Authoring, inference + rule engines

Eulji University; Seoul National University; InHa University; Kwandong University: South Korea;

University of Utah, UT, USA

HELEN

Skonetzki et al., 2004 HELEN Authoring and Execution Heidelberg University Medical Center, Germany

SEMANTIC WEB & INFERENCE ENGINES

Abidi et al., 2007 GEM/CPG-EX Authoring and Execution Dalhousie University, Canada Hussain and Abidi, 2008 CPG-EX Authoring and Execution Dalhousie University, Canada Din et al., 2010 Protégé + Jena Authoring and Execution Dalhousie University, Canada Chen et al., 2011 Protégé + Jess Authoring and Execution Taipei Medical University; National Chengchi

University; Taiwan

Papageorgiou et al., 2012 FCM-uUTI DSS GBDSS Technological Educational Institute of Lamia, Greece; Agfa HealthCare NV, Belgium

Papageorgiou, 2012 FCM-uUTI DSS GBDSS Tech. Educational Institute of Lamia, Greece Linan et al., 2015 HeD editor Authoring Arizona State University, AZ; Mayo Clinic, MN; USA Jafarpour et al., 2016 OWL 1 DL, OWL 2 DL,

OWL 2 DL + SWRL Execution engine Dalhousie University, Canada

OTHER

SIEGFRIED

Lobach et al., 1998 SIEGFRIED GBDSS Duke University Medical Center, NC, USA Porcelli and Lobach, 1999 SIEGFRIED GBDSS Wake Forest University School of Medicine, NC, USA;

Duke University Medical Center, NC, USA

HGML

Hagerty et al., 2000 HGML Markup tool Mark-up tool Rutgers University, NJ, USA

STEPPER

Svatek and Ruzicka, 2002 Stepper Mark-up tool Cardio University of Economics, Czech Republic Svatek and Ruzicka, 2003 Stepper Mark-up tool Cardio University of Economics, Czech Republic Ruzicka and Svatek, 2004 Stepper Mark-up tool Cardio University of Economics, Czech Republic

MEDICAL MARKUP TOOL

Kumar et al., 2003 MTM Tool Mark-up tool University of Pavia, Italy

Guideline representation and execution tools: An evaluation study

1 |

P a g e