2006 BRP Focus on the evaluation of factors that influence enterprise application integration (EAI) adoption in the health care sector Informational Enterprise Application Integration (EAI) Healthcare organisation - 3 Khoumbati, K., &amp

(1)

Appendices

Appendix 1 Definitions of the fields (table follows):

• Column for Brief description gives short explanation concerning the articles, especially their objective

• Column for Type of study defines as categorisation of articles used in this study

• Column for Type of ICT defines the type of ICT being used in certain article

• Column for Type of healthcare defines the type of healthcare setting which is the assessment being deployed

• Column for Assessment method defines the method being used by author(s) in the articles to conduct assessment

• Column for Method name refers to name of the assessment method used in the article

• Column for Assessment process guideline defines as a complete‐explained plan or principle set by the author(s) as a standard in order to assist other assessors or evaluators by providing direction or course of action when conducting their own assessment. The guideline should be easily distinguishable from the author(s)’s methodology in conducting their research

• Column for Stakeholders guideline defines as a complete‐explained plan or principle set by the author(s) as a standard that provides and assists other assessors or evaluators in considering who are the

stakeholders and why and how to deal with them in evaluating certain ICT investment

• Column for Notion of value is the author(s)’s general idea, view, or perception of the whole assessment process and criteria which they considered as the most important thing in assessing or evaluating the ICT investment

• Column for Stakeholders refers to people, groups or parties which have interest or in some reason have the stakes for the healthcare entity, therefore, being involved in the assessment process of certain ICT investment

(2)

1 Chau, S., & Turner, P. 2006 BRP Reports on research into the implementation and evaluation of a wireless hand-held clinical care management system

0 0.0 Case/empirical Wireless hand-held clinical

care management system

Aged Care Facility Qualitative

2 Khoumbati, K., Themistocleous, M., &

Irani, Z.

2006 BRP Focus on the evaluation of factors that influence enterprise application integration (EAI) adoption in the health care sector

6 3.0 1.818 Informational Enterprise Application

Integration (EAI)

Healthcare organisation -

3 Khoumbati, K., &

Themistocleous, M.

2006 BRP to evaluate and present the benefits and barriers of the integration approaches by healthcare organisations

0 0.0 0.680 Review Health Information System

(HIS) integration

Healthcare organisation Mixed/multi

4 Kocakulah, M. C., &

Upson, J.

2005 BRP Evaluates the implementation of CPOE by using Value Stream Mapping techniques

0 0.0 Case/empirical Computerised Physician Order

Entry (CPOE)

Hospital Quantitative

5 Leonard, K. J., Lin, J., Dalziel, S., Yap, R., &

Adams, D.

2005 BRP Outlines the development and functionality of PIMS and to evaluate its contribution in term of improved data quality and health outcome

0 0.0 0.597 Case/empirical Panacea Information

Management System (PIMS)

6 Ovretveit, J., Scott, T., Rundall, T. G., Shortell, S. M., & Brommels, M.

2007 MEDLINE Describes &

assesses an implementation of one information technology system (EMR) in one hospital, the perceived impact, the factors thought to help and hinder implementation and the success of the system and compare this with theories of effective IT implementation

Draws on previous research, empirical data from this study is used to develop IT implementation theory

0 0.0 1.444 Case/empirical Electronic Medical Record

(EMR)

Hospital Qualitative

7 Adhikari, N., & Lapinsky, S. E.

2003 MEDLINE Briefly reviews the process of health technology assessment (HTA) and outline some of the methodologic considerations particular to the assessment of information technology

6 1.2 1.054 Informational Clinical Decision Support

Systems (CDSSs)

Intensive Care Unit (ICU) -

8 Amarasingham, R., Diener-West, M., Weiner, M., Lehmann, H., Herbers, J. E., &

Powe, N. R.

2006 MEDLINE Develops and evaluates a survey- based metric that assesses the automation and usability of a hospital's information system

3 1.5 3.745 Case/empirical Clinical Information

Technology (CIT)

Hospital Qualitative

9 Ammenwerth, E., Iller, C., & Mansmann, U.

2003 MEDLINE Presents some background on the theory of triangulation

Discusses what benefits can be obtained from applying triangulation in an evaluation study

24 4.8 1.726 Case/empirical Computer-based nursing

documentation system

University Medical Center Mixed/multi

10 Apkon, M., Mattera, J.

A., Lin, Z., Herrin, J., Bradley, E. H., Carbone, M., Holmboe, E. S., Gross, C. P., Selter, J.

G., Rich, A. S., &

Krumholz, H. M.

2005 MEDLINE Evaluates the utility of a decision- support tool with a patient;level randomized trial conducted at 2 military practices

5 1.7 7.920 Case/empirical Decision-Support Information

Technology (DSIT)

Ambulatory Mixed/multi

(3)

11 Averbukh, A. N., Channin, D. S., &

Homhual, P.

2005 MEDLINE Compares human, contrast-detail perception on different computer display subsystem

6 2.0 1.304 Case/empirical Digital imaging on LCD display Radiology Qualitative

12 Bellazzi, R., Arcelloni, M., Bensa, G., Blankenfeld, H., BruguÃ©s, EulÃ l., Carson, E., Cobelli, C., Cramp, D., D'Annunzio, G., De Cata, P., De Leiva, A., Deutsch, T., Fratino, P., Gazzaruso, C., GarcÃ¬a, A., Gergely, T. s., GÃ³mez, E., Harvey, F., Ferrar

2003 MEDLINE Conducts a randomized controlled study, with clinical, organisational, economic, usability, and users' satisfaction outcomes 2 total cited version

16 3.2 Case/empirical Telemedicine Diabetes Qualitative

13 Blaser, R., Schnabel, M., Biber, C., BÃ¤umlein, M., Heger, O., Beyer, M., Opitz, E., Lenz, R., &

Kuhn, K. A.

2007 MEDLINE Gives experience in integrating and evaluating IT application into the clinical workflow

4 4.0 1.726 Case/empirical General University Medical Center Mixed/multi

14 Bratsas, C., Koutkias, V., Kaimakamis, E., Bamidis, P. D., Pangalos, G. I., &

Maglaveras, N.

2007 MEDLINE Examines how the system corresponds to different descriptions according to the knowledge background of each medical specialist

0 0.0 0.624 Case/empirical Medical Computational

Problems (MCPs)

Biomedical research community

Qualitative

15 Cariello, F. P. 2003 MEDLINE Evaluates service quality and cost of a nationally recognised computerised telephone nurse triage (CTNT) service

7 1.4 Case/empirical Computerized telephone nurse

triage

Paediatric Mixed/multi

16 Chaiken, B. P. 2003 MEDLINE Discusses the idea that organisations should go beyond traditional measures that weigh monetary costs versus quantifiable benefits

2 0.4 Informational Clinical technology General -

17 Choong, M. K., Logeswaran, R., &

Bister, M.

2007 MEDLINE Finds a cost- effective way to digitize X-ray films,

Identifies a suitable compression algorithm and the degree of compression,

Compares among different monitors

0 0.0 1.726 Case/empirical Digitization of medical image Radiology Quantitative

18 Colombet, I., Bura- RiviÃ¨re, A., Chatila, R.

m., Chatellier, G., &

Durieux, P.

2004 MEDLINE Evaluate the effect on the frequency of overanticoagulation of the implementation in the computerised physician order entry system of two types of tool to adjust OAT doses

Assesses any advantages of using the personalised tool rather than the non- personalised tool

1 0.3 1.198 Case/empirical Computerized decision support

system

Oral anticoagulant therapy (OAT)

Quantitative

19 Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., & Lang, T.

2004 MEDLINE Analyses the impact of CBPRS on medical practice, quality of care, and user and patient satisfaction

30 7.5 1.444 Review Computer-based Patient

Record Systems (CBPRS)

General -

20 Dominich, S. n., GÃ³th, J. l., & Kiezer, T. s.

2006 MEDLINE Describes evaluation from three different information retrieval techniques to brain computer tomography (CT)

0 0.0 0.909 Case/empirical Neuroradiological information

retrieval

Neuroradiology Quantitative

(4)

21 Eisenstein, E. L., Ortiz, M., Anstrom, K. J., Crosslin, D. R., &

Lobach, D. F.

2006 MEDLINE Determines whether efficiency information was being properly incorporated into evaluation studies to determine whether the quality of efficiency information included has changed since publication of medical technology economic evaluation guidelines in the mid 1990s

0 0.0 Informational General General -

22 Eisenstein, E. L., Ortiz, M., Anstrom, K. J., Crosslin, D. R., &

Lobach, D. F.

2006 MEDLINE Describes the finding of fact that the volume and variety of economic evaluations had increased.

However, investigators routinely omitted key cost or effectiveness elements in their design, resulting in publications with incomplete, and potentially biased, economic findings

23 Fang, Y.-C., Yang, M.- C., & Hsueh, Y.-S.

2006 MEDLINE Determine the differential cost between film-based radiology and a hospital-wide picture archiving and communication system (PACS) implemented all at once

0 0.0 1.304 Case/empirical Picture Archiving and

Communication System (PACS)

Radiology - government-owned hospital

Quantitative

24 Farley, J. E., Srinivasan, A., Richards, A., Song, X., McEachen, J., & Perl, T. M.

2005 MEDLINE Investigates the accuracy and cost- effectiveness of using handheld personal digital assistants (PDA) and computer- based UTI surveillance in comparison with traditional surveillance of UTIs among medical intensive care unit (MICU) patients

2 0.7 2.489 Case/empirical Handheld computer Urinary tract infection (UTI)

surveillance - hospital

Quantitative

25 Fischer, S., Stewart, T.

E., Mehta, S., Wax, R., &

Lapinsky, S. E.

2003 MEDLINE Systematically reviews to summarize the current literature regarding the use of handheld devices in medicine so it will increase the awareness among physicians about the potential roles for handheld computers in medicine and encourage the further evaluation of their use

166 33.2 3.979 Review Handheld computer General -

26 Fung, C. H. 2006 MEDLINE Develops and

assesses the usefulness of condition-specific computerized templetes in a primary care setting

2 1.0 2.964 Case/empirical Computerized condition-

specific templetes

Academic Veterans Affairs medical center

Mixed/multi

27 Green, C. J., Kazanjian, A. e., & Helmer, D.

2004 MEDLINE Evaluates the quality of the information content on bone-mineral density (BMD) testing posted on consumer health website (CHWS) by comparing with HTA report

12 3.0 1.151 Case/empirical Consumer health websites Bone-mineral density Qualitative

(5)

28 Ignacio de UlÃ-barri, J., GonzÃ¡lez-MadroÃ±o, A., de Villar, N. G. P., GonzÃ¡lez, P., GonzÃ¡lez, B., Mancha, A., RodrÃ-guez, F., &

FernÃ¡ndez, G.

2005 MEDLINE Presents an screening tool that allows an automatic daily assessment of nutritional status of all inpatients that undergo routine analysis

4 1.3 Case/empirical Controllong nutritional status

(CONUT) - automatic screening method

Clinical Epidemiology Unit - university hospital

Quantitative

29 Kaboli, P. J., McClimon, B. J., Hoth, A. B., &

Barnett, M. J.

2004 MEDLINE Determines the accuracy of computerised medical histories

12 3.0 2.060 Case/empirical Computerized medication

histories

Iowa Veterans Affairs Medical Center (VAMC)

Mixed/multi

30 Kaur, K., Forducey, P.

G., & Glueckauf, R. L.

2004 MEDLINE Describes the development of a database that link the delivery of telerehabilitation services, reimbursement, and outcomes evaluation

Compares the time taken to complete identical case histories using the paper-based forms and the computer- based forms

3 0.8 0.816 Case/empirical Telerehabilitation Rehabilitation center -

31 Kern, J. 2006 MEDLINE Identifies criteria used for evaluation of teleconsultation systems in evaluation studies to estimate trend of their changes

3 1.5 1.726 Review Teleconsultation Telemedicine -

32 Kilbridge, P. M., Welebob, E. M., &

Classen, D. C.

2006 MEDLINE Outlines the development of CPOE test (system should alert physicians to at least 50% of common serious prescribing errors) which evaluates the ability of implemented CPOE systems to prevent the occurrence of medication errors that have a high likelihood of leading to adverse drug events (ADEs)

5 2.5 2.382 Case/empirical Computerized Physician Order

Entry (CPOE)

33 Kocsis, O., Costaridou, L., Mandellos, G., Lymberopoulos, D., &

Panayiotakis, G.

2003 MEDLINE Proposes a set of quantitative measurements for compression assessment, based on parameters adopted from medical image quality concept

11 2.2 0.624 Case/empirical Digital compression algorithms

for Picture Archiving and Communication Systems (PACS)

Radiology Quantitative

34 Lee, S., Broderick, T. J., Haynes, J., Bagwell, C., Doarn, C. R., & Merrell, R. C.

2003 MEDLINE Evaluate the usefulness of low- bandwidth telemedicine in remote surgical evaluation

Conclude that low- bandwidth, internet- based telemedicine is a cost-effective technology that can efficiently and effectively prescreen surgical patients in remote areas

8 1.6 1.109 Case/empirical Telemedicine Surgery - hospital -

35 Lising, M., & Kennedy, C.

2005 MEDLINE Reviews a rigorous longitudinal evaluation methodology being employed by a freestanding tertiary care peadiatric hospital as they implement a CCIS

8 2.7 1.042 Case/empirical Critical Care Information

System (CCIS)

Hospital Mixed/multi

(6)

36 Lorence, D. P., &

Greenberg, L.

2006 MEDLINE Creates a benchmark technology assessment of online health search trends and practices, with corresponding evaluation of its applicability within the Federal Health Architecture (FHA) plan for nationwide, interoperable health information infrastructure

3 1.5 2.964 Informational Online health search General Qualitative

37 Magnus, M., Herwehe, J., Proescholdbell, R. J., Lombard, F., Cajina, A., Dastur, Z., Millery, M., &

Sabundayo, B. P.

2007 MEDLINE Provides guidelines to consider prior to adoption and evaluation of IT in HIV care setting

1 1.0 Informational General HIV care setting -

38 Miyahara, S., Tsuji, M., Iizuka, C., Hasegawa, T., & Taoka, F.

2006 MEDLINE Attempts to estimate economic values (in term of willingness to pay [WTP] and willingness to undertake [WTU]) of telemedicine and to extract factors that promote telemedicine through the use of survey data

1 0.5 0.816 Case/empirical Telemedicine General Mixed/multi

39 Murray, M. E., Brennan, P. F., & Moore, S. M.

2003 MEDLINE Provides an approach to performing an economic assessment of innovatives involving technology

Guides the economic analysis of the development of health care program, technologies, and innovations

2 0.4 1.810 Case/empirical Home-based self-care

technology research project

HeartCare Quantitative

40 Nahm, E.-S., Vaydia, V., Ho, D., Scharf, B., &

Seagull, J.

2007 MEDLINE Provides clinicians and frontline informaticians with a practical guide to assess CIS outcomes, focusing on outcome variable, assessment methods, and timing of assessment

0 0.0 1.419 Review Healthcare Information

Systems (HIS) and Clinical Information Systems (CIS)

General -

41 Niiranen, S., Lamminen, H., Niemi, K., Mattila, H.,

& Kalli, S.

2003 MEDLINE Develops a comparative cost evaluation between conventional and new media and near patient testing supported anticoagulant (ac) treatment follow-up in a primary health setting

1 0.2 1.726 Case/empirical Mobile communication & digital television

Anticoagulant treatment - Health centre

Quantitative

42 Olve, N.-G. r., &

Vimarlund, V.

2005 MEDLINE Discuss a model to be used for analysing ICT proposals and their benefits, especially for elderly care

1 0.3 0.551 Informational General Elderly care Qualitative

43 Rahimi, B., & Vimarlund, V.

2007 MEDLINE Reviews published articles in the area evaluations of IT- based systems in order to gain knowledge about methodologies used and findings obtained from the evaluation of IT- based systems applied in healthcare setting

0 0.0 0.581 Review General General -

44 Ramnarayan, P., Winrow, A., Coren, M., Nanduri, V., Buchdahl, R., Jacobs, B., Fisher, H., Taylor, P. M., Wyatt, J. C., & Britto, J.

2006 MEDLINE Aims to measure the clinical impact of DDSS system on diagnostic decision making

2 1.0 Case/empirical Web-based diagnostic

reminder system - Diagnostic Decision Support Systems (DDSS)

Paediatric - hospital Mixed/multi

(7)

45 Reardon, T. 2005 MEDLINE Reviews the state of telemedicine cost research, to examine major issue affecting the yield from this research, and finally to recommend strategies for imrpoving future research

9 3.0 0.816 Review Telemedicine General -

46 Rhodes, K. V., Drum, M., Anliker, E., Frankel, R.

M., Howes, D. S., &

Levinson, W.

2006 MEDLINE Determines the effect of computer screening on health care provider- patient DV communication at 2 socioeconomically diverse emergency departments (EDs)

18 9.0 7.920 Case/empirical A self-administered computer-

based health risk assessment tool

Domestic violence - urban and suburban hospital

Quantitative

47 Roback, K., & Herzog, A. 2003 MEDLINE Presents an inventory of risks and adverse events specific to telehomecare

Proposes guidelines for planning and assessment of IT- based hospital-at- home schemes

8 1.6 Review Telehomecare Hospital-at-home -

48 Shahmoradi, L., Ahmadi, M., & Haghani, H.

2007 MEDLINE Describes a descriptive crosssectional study in order to identify the indicators that healthcare and health industry professionals considered most relevant to the evaluation of HIS in use

0 0.0 Informational Healthcare Information

Systems (HIS)

General -

49 Thompson, D. S., Oberteuffer, R., &

Dorman, T.

2003 MEDLINE Evaluates the reliability of an automated computer-based real-time eligibility screening tool and use a time-motion diary to assess the effect of the system on study coordinator resource consumption compare with manual chart review

4 0.8 1.042 Case/empirical Sepsis alert and diagnostic

system - automated patient screening system

Intensive Care Unit (ICU) - hospital

Mixed/multi

50 Tomuro, K. 2004 MEDLINE Develops and

evaluates a telecare programme for home-dwelling elderly to provide education on oral and general health care

1 0.3 Case/empirical Telecare Home-dwelling elderly Qualitative

51 Trivedi, M. H., Claassen, C. A., Grannemann, B.

D., Kashner, T. M., Carmody, T. J., Daly, E.,

& Kern, J. K.

2007 MEDLINE Examines the effectiveness of a computer-based decision support system for algorithm-based treatment of MDD (CDSS-D) compared to both a traditional paper- and-pencil-based, algorithm-driven system of care (PPA-D) and nonalgorithm- based expert usual care (UC-D)

0 0.0 1.333 Case/empirical Computer-assisted Decision

Support Systems (CDSS)

University Medical Center Mixed/multi

52 Wilson, A., Duszynski, A., Turnbull, D., &

Beilby, J.

2007 MEDLINE Describes an evaluation undertaken to assess the usability and acceptability of CDSS tool

0 0.0 Case/empirical Computer Decision Support

Software (CDSS)

Cardiovascular risk assessment and management - metropolitan general practices

Qualitative

(8)

53 Winterbottom, L. M., Fong, A. M., Benkstein, K. L., Liang, B., Snodgrass, L. S., &

Parks-Huitron, H.

2006 MEDLINE Determines whether a computerized clinical pharmacy approval and follow- up consult process decreased the number of patients without documented treatment benefit while incresing follow-up and documentation of effectiveness

1 0.5 Case/empirical Computerized clinical

pharmacy consult service

Medical center Quantitative

54 Wyatt, J. C., & Wyatt, S.

M.

2003 MEDLINE Discusses the reasons we need to evaluate large scale of HIS, and a range of appropriate methods to carry out evaluations

29 5.8 1.726 Informational Health Information Systems

(HIS)

Hospital -

55 Zheng, K., Padman, R., Johnson, M. P., &

Diamond, H. S.

2005 MEDLINE Assesses medical residents' acceptance and adoption of a clinical reminder system for chronic disease and preventive care management and to use expressed preferences for system attributes and functionality as a basis for system re-engineering.

Gives new insight of system usability

10 3.3 1.726 Case/empirical Clinical Decision Support

Systems (CDSSs)

56 Ammenwerth, E., Brender, J., NykÃ¤nen, P., Prokosch, H.-U., Rigby, M., & Talmon, J.

2004 MEDLINE Identifies and addresses the frequent problems of getting evaluation understood and recognised, to promote transdiciplinary exhange within evaluation research, and to promote European cooperation

48 12.0 1.726 Informational General General -

57 Ammenwerth, E., & de Keizer, N.

2005 MEDLINE Provides an overview to get insight into trends of evaluation research in the last 20 years

58 Ammenwerth, E., GrÃ¤ber, S., Herrmann, G., BÃ¼rkle, T., &

KÃ¶nig, J.

2003 MEDLINE Discusses some of problems during evaluation of information technology in health care, and proposes possible solutions for these problems

59 Bell, D. S., Cretin, S., Marken, R. S., &

Landman, A. B.

2004 MEDLINE Develops a conceptual evaluation framework for anticipating the effects of alternative designs for outpatient e- prescribing systems

43 10.8 3.979 Informational Electronic prescribing (e-

prescribing)

General -

60 Beuscart-ZÃ©phir, M.- C., Anceaux, F. o., Menu, H., Guerlinger, S., Watbled, L., & Evrard, F.

2005 MEDLINE Describes a multi- dimensional assessment method to support the choice and acquisition process for Clinical Information System

3 1.0 1.726 Case/empirical Clinical Information System

(CIS)

Anaesthesiology Mixed/multi

(9)

61 Braaksma, J., Commandeur, A., &

Berghout, E.

2006 - Describes the design of an evaluation method for nonprofit organisations, the special characteristics of (public) nonprofit organisations compared to for- profit organisation, and defines the concept of public value and a framework for the creation of public value

0 0.0 Informational - - -

62 Brender, J. 2004 - Discusses some

lessons from case studies on criteria for success and failure, as well as obstacles and barriers towards the success of health information application (IT- based solution) within a lifecycle perspective of these systems

0 0.0 Informational Health information application General -

63 Buccoliero, L., Calciolari, S., & Marsilio, M.

2008 MEDLINE Proposes an example of e- health project evaluation aiming to develop measures which get close to the notion of benefits to the different stakeholders involved

0 Case/empirical e-health Health care organisation Mixed/multi

64 Chaudhry, B., Wang, J., Wu, S., Maglione, M., Mojica, W., Roth, E., Morton, S. C., &

Shekelle, P. G.

2006 MEDLINE Reviews systematically evidence on the effect of health information technology on quality, efficiency, and costs of health care

102 51.0 14.780 Review Health Information Technology

(HIT)

General -

65 Connell, N. A. D., &

Young, T. P.

2007 BRP Examines how adopting different perspectives of the enterprise might inform both design and evalution criteria of large- scale healthcare IS

66 Currie, L. M. 2005 MEDLINE Examines the strengths and weaknesses of the published evaluation frameworks and enumerates the qualitative research methods in use in each of these frameworks

67 de Keizer, N. F., &

Ammenwerth, E.

2008 MEDLINE Obtains an overview of study designs and study methods used in research evaluating IT in health care, to present a list of quality criteria by which all kinds of reported evaluation studies on IT system in health care can be assessed, and to assess the quality of reported evaluation studies on IT in health care and its development over time (1982-2005)

1 1.726 Review General General -

68 Georgiou, A., Westbrook, J., Braithwaite, J., &

Iedema, R.

2006 MEDLINE Investigates the impact of the electronic ordering system on work processes and relations within a major teaching hospital

2 1.0 Case/empirical Electronic Ordering Systems

(EOS)

Pathology Qualitative

(10)

69 Georgiou, A., Williamson, M., Westbrook, J. I., & Ray, S.

2007 MEDLINE Reviews current evidence of the impact of CPOE on hospital pathology services

8 8.0 1.726 Review Computerised physician order

entry (CPOE) systems

Pathology -

70 Häkkinen, H., Turunen, P., & Spil, T.

2003 - Develops a model for the evaluation on information flow in health care service chains.

Offers the health care organisations a valid and standardised toolkit for evaluationg and enhancing their processes

71 Harison, E., & Berghout, E.

2006 - Applies various qualitative and quantitative methods to measure the influence of information tehcnology on the performance of operating rooms (ORs) and provides a comprehensive set of indicators to evaluate the impact of IT on the quality, efficiency and performance of ORs

0 0.0 Informational Information technology Operating rooms (ORs) Mixed/multi

72 Hyppönen, H., Nykänen, P., Salmivalli, L., Pajukoski, M., Ruotsalainen, P., &

Vuorela, S.

2005 - Describes the evaluation of the national ePrescription system in Finland

3 1.0 Informational ePrescription database General -

73 Kaushal, R., Jha, A. K., Franz, C., Glaser, J., Shetty, K. D., Jaggi, T., Middleton, B., Kuperman, G. J., Khorasani, R., Tanasijevic, M., & Bates, D. W.

2006 SD Assesses the costs and financial benefits of the CPOE system at Brigham and Women's Hospital over ten years

15 7.5 3.979 Case/empirical Computerised physician order

entry (CPOE) systems

74 Kushniruk, A. W., &

Patel, V. L.

2004 MEDLINE Reviews of general area of systems evaluation with the motivation and rationale for methodological approaces underlying usability engineering and cognitive task analysis as applied to health information systems

52 13.0 2.346 Review Clinical Information System

(CIS)

General -

75 Lærum, H. 2004 - develops an

evaluation method adapted to EMR systems in hospital that should be applicabled to a number of sites and EMR systems and allowed for comparisons between sites or systems from various vendors

3 0.8 Case/empirical Electronic medical records

(EMR)

76 Littlejohns, P., Wyatt, J.

C., & Garvican, L.

2003 MEDLINE Describes how the HIS project and its evaluation were set up and examine where the project went wrong. It gives the applicable lessons to the installation of all hospital information systems

74 14.8 9.245 Informational Health Information Systems

(HIS)

Hospital -

(11)

77 May, C., Mort, M., Williams, T., Mair, F., &

Gask, L.

2003 MEDLINE Explores the practical conduct of evaluation and the social organisation of RCTs of new technologies

Presents a simple model of the social and technical contingencies within the evaluation process

29 5.8 2.749 Informational Telehealthcare General -

78 McLeod, T. G., Ebbert, J. O., & Lymp, J. F.

2003 SD Conducts survey on PDA use pattern in medical setting

49 9.8 3.979 Informational Personal digital assistant

(PDA)

Health care clinic Quantitative

79 Neville, D., Gates, K., MacDonald, D., Barron, M., Tucker, S., Cotton, S., Farrell, G., Hoekman, T., Bornstein, S., &

O’Reilly, S.

2004 - Describes a project report that reviews of current approaches to evaluate the impact of health information systems and develops an evaluation framework which address the information needs of key stakeholders and the identified best practices in the evaluation of such initiatives

1 0.3 Informational Electronic Health Record

(EHR)

General -

80 Neville, D., Gates, K., MacDonald, D., Barron, M., Tucker, S., Cotton, S., Farrell, G., Hoekman, T., Bornstein, S., O’Reilly, S., & Keough, M.

2004 - With addition to previous article, the report develops a framework for future evaluation efforts of the initiative which builds on the identified best practices and feedback received from key stakeholders during the process of the review

0 0.0 Informational Electronic Health Record

(EHR)

General -

81 Odhiambo-Otieno, G. W. 2005 MEDLINE Designs evaluation criteria for assessing the viability, sustainability and ultimate contribution of DHMIS in the management of the district health system (DHS) in Kenya

4 1.3 1.726 Informational District health management

information systems (DHMISs)

District health system (DHS) -

82 Oroviogoicoechea, C., Elliott, B., & Watson, R.

2008 MEDLINE Reviews existing nursing research on patient hospitals' information technology (IT) systems in order to explore new approaches for evaluation research on nursing informatics to guide further design and implementation of effective IT systems

0 1.430 Review Nursing information system General -

83 Poissant, L., Pereira, J., Tamblyn, R., &

Kawasumi, Y.

2005 MEDLINE Performs a systematic review to examine the impact of electronic health records (EHRs) on documentation time of physicians and nurses and to identify factors that may explain efficiency differences accros studies

38 12.7 3.979 Review Electronic Health Record

(EHR)

General -

(12)

84 Roderer, N. K. 2004 MEDLINE Presents a framework for describing evaluation measures, discuss definitions

85 Sequist, T. D., Cullen, T., Hays, H., Taualii, M.

M., Simon, S. R., &

Bates, D. W.

2007 MEDLINE Evaluates the implementation of an EHR within the Indian Health Service (IHS)

3 3.0 3.979 Case/empirical Electronic Health Record

(EHR)

Indian Health Service (HIS) health center

Mixed/multi

86 Shaw, V., Jacucci, E., &

Braa, J.

2005 - Provides an insight into the process of an ex-ante or explanatory assessment of information system in hospitas in developing countries

0 0.0 Informational General Hospital Qualitative

87 Sistrom, C. 2005 MEDLINE Explores some of the organisational, cultural, cognitive, and economic forces that interact to influence success of HIT initiative in health care organisation

2 0.7 1.781 Informational General Radiology -

88 Stengel, D., Bauwens, K., Walter, M., KÃ¶pfer, T., & Ekkernkamp, A.

2004 ARP Determines whether the introduction of a handheld computer could improve both quantitaive and qualitative aspects of medical records

25 6.3 Case/empirical Handheld computer-assisted Orthopaedic surgery Mixed/multi

89 Stoop, A. P. 2005 - Aims to contibute

to the discussion about the role of evaluation studies for improving information systems, the use of evaluation methods and the need for evaluation frameworks and guidelines by using a sociotechnical perspective on the design, use and success and failure of ICT application in health care

90 Stoop, A. P., van't Riet, A., & Berg, M.

2004 MEDLINE Reports on two evaluation projects showing that the failure of both patient information system was cused by the fact that the designers had not made a thorough analysis of the advantages and limitiations of IT compared to traditional means for patient education

7 1.8 1.778 Informational Patient information systems Hospital -

91 Van Der Meijden, M. J., Tange, H. J., Troost, J.,

& Hasman, A.

2003 MEDLINE Identifies attributes that were used to assess the success of patient care information systems ant to test the ability of a framework developed by Delone and McLean for management informationsystem to catagorise these attributes correctly

70 14.0 3.979 Review Clinical Information System

(CIS)

General -

(13)

92 VanDenKerkhof, E. G., Goldstein, D. H., Rimmer, M. J., Tod, D.

A., & Lee, H. K.

2004 SD Assesses the utility of personal digital assistants for documentation at the bedside on an acute pain service

7 1.8 Case/empirical Handheld computers Acute pain service Mixed/multi

93 Westbrook, J. I., Braithwaite, J., Iedema, R., & Coiera, E. W.

2004 MEDLINE Addresses the problems of conceptualizing the design features in an evaluation framework and specifying what data will be gathered and how in order to assess the impact of ICT on organisational process and outcomes

7 1.8 Informational General Point of care clinical systems

(PoCCS)

-

94 Williams, T., May, C., Mair, F., Mort, M., &

Gask, L.

2003 BRP Critically contrast the normative expectations of the wider field of HTA with those configured within debates about Telehealthcare Evaluation, and critically review models that provide structures within which the production of evidence about telehealthcare can take place

18 3.6 1.201 Review Telehealthcare -

95 Winkelman, W. J., &

Leonard, K. J.

2004 MEDLINE Identifies characteristics that either enhance or mitigate the influence of medical record structure on patient utilisation of an electronic patient record (EPR) system

17 4.3 3.979 Review Electronic Patient Record

(EPR)

General -

96 Wu, R. C., & Straus, S.

E.

2006 MEDLINE Performs a systematic review of the evidence assessing the effects of handheld electronic medical records on clinical care

3 1.5 Review Handheld electronic medical

records

General -

97 Yusof, M. M., Kuljis, J., Papazafeiropoulou, A., &

Stergioulas, L. K.

2008 SD Introduces the new framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organisational fit, with a case of Fundus Imaging System (FIS)

0 1.726 Informational Health Information Systems

(HIS) - Fundus Imaging System (FIS) as case

General -

98 Yusof, M. M., Papazafeiropoulou, A., Paul, R. J., &

Stergioulas, L. K.

2008 SD Presents an overview of evaluation in health informatics and information systems

1 1.726 Review Health Information Systems

(HIS)

General -

99 Yusof, M. M., Paul, R. J.,

& Stergioulas, L.

2005 MEDLINE Provides a brief description of HIS and their classification

Presents the features of HIS evaluation and a discussion of early evaluation studies using CDSS as an example

2 0.7 Review Clinical Decision Supports

System (CDSS)

General -

(14)

1 Chau, S., & Turner, P. 2006 BRP Socio-technical trial - v End-user perspective Organisational perspective Technology perspective

Benefit Nurse manager

Registered nurses Enrolled care assistants

2 Khoumbati, K., Themistocleous, M., &

Irani, Z.

2006 BRP Fuzzy cognitive mapping - - - - -

3 Khoumbati, K., &

Themistocleous, M.

2006 BRP Shang & Seddon - - Operational

Managerial Strategic IT infrastructure Organisational

Benefits & barriers -

4 Kocakulah, M. C., &

Upson, J.

2005 BRP Value Stream Mapping - - Capital cost

Support cost

Full Time Employee (FTE) cost Adverse Drug Reactions (ADR) cost

Financial -

5 Leonard, K. J., Lin, J., Dalziel, S., Yap, R., &

Adams, D.

2005 BRP Operation research - - Improved data quality

Health outcomes

Benefit -

6 Ovretveit, J., Scott, T., Rundall, T. G., Shortell, S. M., & Brommels, M.

2007 MEDLINE Semi structured interviews and documentation

- - Interview questions:

Practice culture, cultural mediation, leadership, defining moments/critical events/turning points, previous IT implementation, use of HER, chronic disease management, system functionality

- Senior clinicians,

managers - clinics &

division, project team members, doctors and nurses

7 Adhikari, N., & Lapinsky, S. E.

2003 MEDLINE - - - Safety and technical performance

Clinical effectiveness

- -

8 Amarasingham, R., Diener-West, M., Weiner, M., Lehmann, H., Herbers, J. E., &

Powe, N. R.

2006 MEDLINE CIT assessment tool v - Automation: test result, notes &

records, order entry, other processes Usability: effectiveness, ease, support

Benefit Physician

CIO

9 Ammenwerth, E., Iller, C., & Mansmann, U.

2003 MEDLINE Triangulation - - Acceptance and attitudes

Triangulation (as method):

Data triangulation Investigator triangulation Theory triangulation Methods triangulation

Mixed Nurses

10 Apkon, M., Mattera, J.

A., Lin, Z., Herrin, J., Bradley, E. H., Carbone, M., Holmboe, E. S., Gross, C. P., Selter, J.

G., Rich, A. S., &

Krumholz, H. M.

2005 MEDLINE A patient-level randomised controlled trial (RCT)

- - Quality of care: 24 healthcare quality

process measures (opportunities to provide evidence-based care) Patient and provider satisfaction

Mixed Patients

Provider - healthcare

11 Averbukh, A. N., Channin, D. S., &

Homhual, P.

2005 MEDLINE - - - Observer sensitivity

Observer confidence Contrast/noise

Medical - accuracy Radiologist