Assessment of ICT investment in healthcare: an overview
Arviansyah
Faculty of Economics and Business, Business & ICT, University of Groningen Abstract
Background: The use of information and communication technology (ICT) is emerging in healthcare sector; a careful assessment of this technology is recommended and has a high importance for decision makers and users. On the other hand, there is hardly any general model to assess ICT investment in healthcare.
Objective: The research provides an overview of what have been done in the theme of assessment of ICT investment in healthcare and enriches the knowledge of ICT in healthcare. Moreover, the overview discusses on the need of specific assessment framework for healthcare, the appropriate time and prescription in conducting assessment, stakeholders to be involved and assessment criteria and dimensions should be applied.
Method: The research conducts a systematic review using published literature from EBSCO Host Research Database which included two of cross interdisciplinary between business and medical i.e. Business Resource Premier and MEDLINE from 2003 until 2008.
Results and conclusions: First, recent studies already have seen assessment by looking more than
quantitative data but have combined aspects of stakeholders, organisation process changes and technical technology involving qualitative data into multi approaches or mixed method and triangulation method for better and more accurate assessment. Second, assessment of ICT in healthcare should take slightly different approach due to its uniqueness. Therefore, further research should confirm the uniqueness and investigate the different approach. Furthermore, whether the assessment is conducted as formative or summative, assessment timing should achieve certain purpose (such as improving implementation or assessing achievement) and consider various factors (such as type of ICT or learning curve). It is better to see assessment as iterative process and see the technology itself as moving target, which then emphasis the need of multi‐snapshoot over time or life cycle. Third, regardless the type of ICT and healthcare setting, the stakeholders should be included in the assessment process are physicians, nurses, patients, managers of healthcare organisations. The assessment criteria and dimensions which are needed to be included in assessment process are system’s usage, satisfaction, time‐related‐comparing measurement, usability, data or information quality, organisation issue, quality of life (QOL), human issue and technology. Finally, it is needed to have a general framework of assessment as the guidelines that will be combined with specific criteria and stakeholders based on the type of ICT and healthcare setting. This framework should produce a more standard, comprehensive, and evidence‐based result ready to be applied.
1. Introduction
Information and communication technologies (or well known as ICTs) are seen as a magic bullet to solve many problems faced by organisation, whether it is profit or non‐profit oriented entity. The magic of this bullet emerges into many sectors of industry and into our daily work and life. The healthcare sector is seen as non‐profit oriented industry which is usually funded by public funds or private contribution and
government as service to its citizen. One would claim that this technology will lead them into more cost effective and error free medical care, but the arguments are lacking in quantity. In the United States billions
of dollars have been spent on information technology for healthcare with very little benefit but significant adverse effects on patients, physicians, and nurses (Sistrom, 2005).
There are some reasons pointing out on the need of ICT assessment. Farbey, Targett, & Land (1993) mentioned some of these reasons. Few reasons that they mentioned emphasising the need of ICT assessment are:
1. The amount of money invested on ICT investment is huge; such huge money should be allocated into the right project of investment decision.
2. Problem in choosing alterative ICT projects for investing. An appropriate assessment of ICT would give knowledge for better decision.
3. Benchmark for measuring the success of investment. The assessment result would be useful in comparing the success of ICT investment toward other investment decision.
4. For making better and well‐based decision. Better and well‐based decision would be achieved based on the assessment result.
On the other hand, it is known that the attempt to assess or evaluate ICT investment will end up in facing many challenges. Despite of many debates on the value of ICT in improving business performance,
Mahmood & Mann (1993) argued that the aggregate of IT investment was positively and significantly related to each of the organisational performance measures. On the other word, they argued that organisations with more aggregate IT investment had higher performance.
Of course that IT investment was not the single factor influencing organisation performance. Many other factors such as economic, regulatory, competitive and technological environment will also impact
organisational performance (Mahmood et al., 1993). This research believes that ICT brings value to any type of organisation, either profit or non‐profit oriented. Therefore, any type of ICTs invested by organisation should then be assessed in order to show their value and worthiness.
Challenges in assessing ICT investment in healthcare setting rise from the so‐called uniqueness and complexness of healthcare sector as non‐profit institution. Braaksma, Commandeur, & Berghout (2006) argued that nonprofits organisations have specific characteristics compared to for‐profit organisations and there is hardly literature about the evaluation of IT in nonprofits organisation. While the health information system (HIS), as one example of ICT investment, is emerging in healthcare sector, a careful assessment of this technology is recommended and has a high importance for decision makers and users. On the other hand, there is hardly any general model to assess ICT investment in health care.
A great hunt for IT benefit (good phrase taken from Barbara Farbey in 1994) is still continue although searching and structuring these benefits will face many challenges. Proving the business value of ICT for increasing productivity and customer value and decreasing error and complexity is a wide‐opportunity for researchers in developing their knowledge and creativity for the sake of science. Bringing knowledge in healthcare cluster will have great impact to people and community wealth fare. More efficient spending in healthcare sector means lower cost to the society and more funds can be spent to other public beneficially programs.
The research provides an overview of what has been done in the theme of assessment of ICT in healthcare and enriches the knowledge of ICT in healthcare and makes some suggestions on the theme of ICT
investment in healthcare. The research aims at making an overview on the need of specific assessment framework for healthcare, the appropriate time and prescription in conducting assessment, stakeholders to be involved and the assessment criteria and dimensions that should be applied.
2. Theoretical foundation
2.1. Assessment and evaluation of ICT
There are many methods and approaches have emerged in this area of study. Every attempt gives their best shoot to provide comprehensive and accurate assessment, yet easy to use, frameworks. Assessment define as the act of performing evaluation, verification and validation and it referred as the act of measuring quality characteristics (Lærum, 2004). Remenyi, Money, & Bannister (2007) define evaluation as the act of
comparing a process, an artefact, a person, an organisation or any other situation with other comparable entities and/or with a set of standards which the evaluator regards as appropriate to that situation.
Moreover, Berghout (1997) defined evaluation as the activity of comparing and evaluating alternatives together with the act of choice.
More recent definitions are provided from studies in the healthcare research area. Ammenwerth (2004) and other researchers in the Exploratory Workshop on “New Approaches to the Systematic Evaluation of Health Information Systems (HIS‐EVAL) defined evaluation as the act of measuring or exploring properties of a health information system resulting information of decision to be made concerning that system in a specific context. They referred the measurements and the explorations of such properties occur in planning, development, implementation, or operation phase of health information system.
The assessment or evaluation consisted of process and aspects need to be taken and considered. Berghout (1997) referred evaluation process as the steps that are followed during the evaluation. He referred evaluation aspects as particular characteristics of an information system that are considered in evaluation.
These assessment process and aspects are the basis for framework used by academician and practitioners in assessing ICT investment. There are many arguments in this research area concerning which appropriate method and criteria should be used in the assessment process. The basic types of assessments and methods can be divided into formative‐summative type and objectivist‐subjectivist method (Lærum, 2004).
Lærum (2004) described the formative as the type of assessment which is conducted during the life time of a project and intended to provide input for development and design. On the other hand, summative is the assessment type which is conducted at the end of a project and intended to look for system’s achievement.
Furthermore, objectivist deals with more quantitative measurement in its method instead of subjectivist method which deals with more qualitative instrument.
Farbey et al. (1993) argued that methods based on a search process which involve multi‐objective and multi‐
criteria techniques is much better, since it is better in capturing the plurality of interest attending the introduction of new system. Hence, multi‐objective and multi‐criteria techniques will likely to address a wider range of benefit and disadvantage. Based on Mintzberg study in 1983, they proposed a framework of classification system for IT benefit in the areas of strategic, management, operation efficiency and
effectiveness, technostructure or functionality and organisation support.
Hogbin & Thomas (1994) explained Parker and Benson study in 1988 which proposed information economics as concepts and tools which is quantifying benefits and costs for IT project beyond what traditional role of cost‐benefit analysis (CBA). Hogbin et al. (1994) described information economics as a concept dealing with value based on business performance, IT infrastructure and process of decision making that connect into strategic impact on the company. They argued that Parker and Benson’s framework was attractive as they define the criteria, scores and weights in order to produce consistent reapplication and to apply it.
IT or IS department in business sector is responsible for most of ICT matters. Among many arguments drawn in the debatable function of this department, one of the argument by Leyland, Watson, & Kavan (1995) is related to assessment. They suggested viewing IS department fundamental service as the provider of information. Therefore, the effectiveness of an IS unit can be assessed by its capability to provide quality service to its user. They recommended using the SERVQUAL model (to measures service dimensions of tangible, reliability, responsiveness, assurance and empathy) as the instrument to measure IS service quality in order not to fail in measuring the IS effectiveness, to which most ICT investments belong.
Furthermore, Renkema & Berghout (1997) made a comparative review of investment evaluation method for information system at the proposal stage. They categorised their findings into four basic approaches to evaluate information system investment, that is, the financial approach, the multi‐criteria approach, the ratio approach and the portfolio approach. Their observations showed that non‐financial methods are hardly underpinned by theory and many methods focus on the evaluation criteria instead of evaluation process.
They suggested that portfolio methods can help in making non‐financial criteria more easy to use and the differences among the methods can be overcome by combining features of different approaches.
Eventually, Remenyi et al. (2007) made some guidelines for evaluation to be accepted consisting of the following aspects:
1. Evaluation is a management process requires discipline.
2. Evaluation should be comprehensive, involving the opinions of all the major stakeholders.
3. Evaluation is focused on a full range of benefits or outcomes, both tangible and intangible.
4. Evaluation is most effective when it is performed regularly perhaps even continuously.
5. Evaluation encourages full discussion of co‐evolving requirements.
6. Evaluation ensures that the object or process being evaluated continues to be relevant.
They argued that there is no self‐evident methodology for the measurement or management of IT costs and benefits, but still organisations need to be able to evaluate their business process changes and the
accompanying IT investments.
2.2. Assessment of ICT in healthcare context
The effort to improve the utilisation of such high cost investment in ICT is likely to start at assessing its worthiness. One of well known assessment principle is health technology assessment (HTA). Health technology assessment (HTA) is defined as policy researches approach that examines the short‐ and long‐
term social consequences (for examples, economic, ethical, legal) of the application or use of technology
(Oortwijn, Banta, Vondeling, & Bouter, 1999). HTA also conducts a systematic evaluation of properties, effects, or impacts of healthcare technology (Goodman & Ahn, 1999). Throughout the years, HTA has continues to adapt and improve its methods in response to the rapid and great diversity of changing and evolving technologies (software, hardware and infrastructure) into many applications in the field of medical informatics (Goodman et al., 1999).
Friedman & Wyatt (2006) pointed out what Donabedian in 1966 concerned in healthcare innovation that will influence three aspects of healthcare system. These three aspects are the structure of healthcare system, the processes that happen during healthcare activity and the outcome of healthcare for both individual patients and the community. They argued that there will be a great demand for evident‐based technology before the healthcare organisation decides to adopt it, since healthcare is a safety‐critical area. This reason shows another challenge in assessing the ICT investment in healthcare setting. But from another standpoint, it generates opportunities to fulfil such demand.
Yusof, Papazafeiropoulou, Paul, & Stergioulas (2008b) emphasised the importance of a framework with comprehensive and specific measures that incorporates technological, human and organisational issue to facilitate health information system (HIS) evaluation. They came up with framework of HOT‐fit, i.e. human, organisation and technology‐fit factor which is applicably‐proven in evaluating HIS in general. Moreover, Oroviogoicoechea, Elliott, & Watson (2008) in their 10‐year‐review of evaluating information system also urged the need of considering social and organisational aspects in evaluation studies.
The attempt in assessing ICT investment in healthcare setting is influenced by conventional healthcare research cluster. It is the reason that the golden standard for evaluation new technology –as in medicine– is the randomised controlled trial (RCT) (Berg, 2004). Therefore, RCT is well known and widely used in many researches as evaluation model to assess the ICT investment, such as they study of Apkon et al. (2005), Fung (2006) and May, Mort, Williams, Mair, & Gask (2003). Berg (2004) criticised the domination of the RCT model as an evaluation framework, especially in patient care information systems (PCIS).
He argued that isolating information system in RCT framework is dubious, since hardware, software and people (as component of information system) are hardly can be treated as the same as introducing new medical intervention or new medicine. He also argued that RCT is labour intensive and expensive method.
Moreover, he argued that explaining social phenomenon that emerges from the introduction of new ICT in healthcare setting is not always through an RCT or through quantification on every measurement. The approaches that involved qualitative data are needed for measuring intangible benefits and explaining such social phenomenon.
On the other hand, de Keizer & Ammenwerth (2008) showed in their 23‐year‐review on development of quality in healthcare IT evaluation that 70% of the studies used a design other than RCT although the quality of RCT’s publications was significantly higher than the quality of non‐RCT studies. Moreover, they noted that there is a trend toward more multi‐centre studies and more authors frequently coming from various
departments. Moreover, review made by Rahimi & Vimarlund (2007) showed the dominance of economic and organisational aspects in the evaluation of health information system and that there was no standard framework for evaluation effects and outputs of implementation and use of IT in the healthcare setting.
One of recent study in the healthcare research cluster was made by Sorenson, Drummond, & Kanavos (2008). They addressed the concept and controversy concerning health technology assessment in Europe.
There are some findings in their review in their way to identify and address current considerations regarding HTA methodological and process issues related to the prioritisation and financing of modern healthcare. In their executive summary, they argued that:
• In spite of lacking its transparency in prioritizing decision criteria and stakeholder involvement, HTA needs more stakeholders (physicians, health economists, pharmacists and patient group
representatives) involvement in order to improve the HTA processes and the implementation of evidence‐based decisions and related policy
• Assessments should adopt a broader definition of value and benefit by considering patient preferences, quality, equity, efficiency and product acceptability among a wide range of stakeholders. Therefore, assessments should take account of indirect benefits and costs. It is needed to capture other benefits that are important to the patients and to the public.
• The timing of assessments can significantly affect the decision on adopting or not adopting such technology (based on HTA result) and also affect patient access into the technology which is being assessed. Re‐evaluation is the key to the HTA process in order to maintain the accuracy of assessments.
There should be further exploration of applying the principles and methods of economic evaluation to preventive measures. Moreover, they concluded that:
Overall, HTA can play a valuable role in health care decision‐making, but the process must be transparent, timely, relevant, in‐depth and usable. Assessments need to use robust methods and be supplemented by other important criteria. Maximization of HTA will enhance potential decision‐makers’
ability to implement decisions that capture the benefits of new technologies, overcome uncertainties and recognize the value of innovation, all within the constraints of overall health system resources.
The challenge of technology implementation in healthcare sector comes from the slow adoption of this cluster concerning its character of safety‐critical area. Healthcare organisations need better prove and evidence in order to adopt the technology. This phenomenon is happened to all kind of technology, not just medical treatment‐related technology but includes also the information and communication technology (ICT). The effort of proving better evidence is shown into the use of protocol or guideline known as HTA which is originated from the growing concern about the expansive diffusion of costly medical equipment in the 1970s (Sorenson et al., 2008).
Since assessment of such technology has a great role in decision making for healthcare organisation (whether to adopt or not the innovation), many researchers emphasised the need of a better and
comprehensive assessment framework which could be out of ordinary assessment of medical intervention.
Therefore, in order to produce such assessment framework, we need more knowledge of the appropriate stakeholders, appropriate notion of value (including assessment criteria and dimensions) and appropriate time in conducting the process of assessment.
3. Method
The research conducts a systematic review using published literature from EBSCO Host Research Database which included two of cross interdisciplinary between business and medical i.e. Business Resource Premier – known as the major international business and management database – and MEDLINE – known as the
provider of medical information on medicine, nursing and healthcare system database. The search strategy and flow of systematic review is shown in details in Figure 1.
3.1. Search strategy
In order to obtain appropriate articles in the study, it is necessary to have a good search strategy in using the entered keywords for the search engine database. The strategy in entering these keyword searches are: 1.
For proxy of assessment, the keywords use are assessment, evaluation and valuation (entered as:
assessment OR evaluation OR valuation) 2. For proxy of ICT, the keywords use are ICT, information system, information technology, communication technology, informatics and computer (entered as: ICT OR
information system* OR information technology OR communication technology OR informatics OR
computer*) 3. For proxy of investment, the keywords use are investment, cost, benefit, economic and cost‐
benefit analysis (entered as: investment OR cost OR benefit* OR economic* OR cost‐benefit analysis) 4. For proxy of healthcare, the keywords use are healthcare, medical, hospital, health care (entered as: healthcare OR medical OR hospital OR health care). The option to be included in abstract is set for the proxy of
assessment, ICT and healthcare. For the proxy of investment, the option is set to any field. The research also set some limiters, they are: the starting year of articles is set from 2003 (until recent ‐2008), type of articles is set to scholarly (peer reviewed) and articles are set to be taken from academic journal.
The first phase of search produced 291 articles. The full text version was available for 212 articles. Limited source from library subscription made the rest of articles were unavailable for their full text. The second phase started by reviewing abstract and applying inclusion and exclusion criteria. This phase yields 109 articles that continued to the third phase of detail reviewing. There are 65 articles eligible for the study plus an addition of 44 articles resulted from forward and backward search of references and authors (Levy & Ellis, 2006) from the third phase. Finally, the final phase of literature searching ended with 99 articles which are included into the study.
3.2. Inclusion and exclusion criteria
To have appropriate articles use in the research, some restrictions are applied in the phase of detail reviewed. These restrictions establish the scope and framework of the research. Since the theme of the study is the assessment of ICT investment in healthcare, each term should be defined clearly. First, the research refers assessment as what Lærum, Remenyi, Money, & Bannister, Berghout and Ammenwerth et al.
defined. Furthermore, assessment is related to valuation, validation and verification. Valuation refers to the process of determining the current worth of an asset or company. Braaksma et al. (2006) based on
shareholder value network of Rappaport’s study in 1986 referred valuation as components of operation cash flow, discount rate and amount of debt which are driven by some “value drivers” together creating, and that is also showing, shareholder value. These valuation components have a direct cause‐effect from value drivers to shareholder value creation.
Moreover, Miyahara, Tsuji, Iizuka, Hasegawa, & Taoka (2006) used valuation method called contingent valuation method (CVM) which has been developed for the use in the fields of health economics,
environmental economics and experimental economics to analyse survey’s data for evaluating services and projects which are not traded on the market in pecuniary terms. In line with previous term, validation refers to the act of comparing the measured quality characteristics (or a property of an object to be studied) to a
stated goal or frame of reference and verification refers to the of checking well‐defined properties of an object against its specification (Lærum, 2004).
Second, the research defines ICT as the definition of ICT which is derived from the term of IT (term to describe the use of computers and telecommunications) with the emphasis to the fact that computing in the twenty‐first century is highly dependent on data communication (Remenyi, Bannister, Money, 2007). ICT also known as a set of information processing, storage and transmission technologies that have been enabled by the advent of microprocessors in the early 1970s (Cantwell, Gambardella, & Granstrand, 2004).
Third, the term of investment in the research means the transformation of capital into assets for a period of more than one year (Berghout, 1997). Hogbin & Thomas (1994) mention three basic questions to be
answered about an investment. These three questions are (1) whether the investment potentially represent value for money or whether the benefit from investment could be obtain in less cost, (2) whether the investment in line with current goals and (3) whether the investment has been managed well or the investment could be done more effectively. These questions imply that investment should be assessed to meet theirs worthiness, aligned with strategy and obtained them efficiently. Moreover, they characterised an IT investment decision by (1) the function which the proposed systems will provide, (2) the maturity of the IT application and (3) the stage in the project life cycle. The Longman Dictionary of Contemporary English defines investment as the use of money to get a profit or to make a business activity successful, or the money that is used and something that you buy or do because it will be useful later.
Finally, the literatures must be associated to healthcare setting. The American Heritage Dictionary of the English Language defines healthcare as the prevention, treatment, and management of illness and the preservation of mental and physical well‐being through the services offered by the medical and allied health professions. The research has narrowed down the breadth of theme by comprising the preceding definitions.
Therefore, articles which are included must comprise of:
• An assessment or evaluation process. The author(s) then should conduct an assessment or evaluation process with certain evaluation aspects, criteria, or dimensions. In the other word, the author(s) or researcher(s) should describe the steps and the aspects they have taken in assessing the case of ICT investment. Other processes, e.g. implementation without any part mentioning attempt to assess are not included. The assessment could be formative or summative and subjective or objective.
• Appropriate technologies, which are in line with the definition of information and communication technology (ICT). These ICT could be functioning as the chain in delivering value i.e. service or supporting the organisation process. The research excluded computer‐based training, computer‐assisted instruction and any kinds of education matters for healthcare professional. This exclusion is based on the reason that the systems are not part of direct patient care (Ammenwerth & de Keizer, 2005). Another reason is that the healthcare sector, based on previous definition, is highlight on the services offered by the health profession (doctors, physicians, nurses, etc.) to customers i.e. patients. Computer‐based training or computer‐assisted instruction belongs to education sector that has more focuses on delivering value of service from expert/teacher/lecturer (whom could be doctors, physicians, or nurses) to students (as customers).
• The consequences of investment decision. Investment decision consequently transforms capital into assets, i.e. ICT assets, for the purpose of making business activity successful or gaining usefulness of such
assets in the future. The assessment process should be ended by explicitly state the outcome whether the investment is worthy or not. The worthiness could be as tangible, e.g. quantitative measurement (performance, financial, etc.), or intangible, e.g. qualitative measurement (benefit, useful, etc.) or even mix combination of them.
• Assessment and technologies that take place in the healthcare sector.
The research also included the articles that review in a systematic way these aspects. Articles which are in these categories are decided as the core articles of the research, otherwise they will be a secondary source of information.
All the 99 articles were identified of their absolute number of citations and relative number of citations based on the Google Scholar. The research also documented their impact factor on the basis of the Journal Citation Report (JCR). JCR is a research tool that can be used to evaluate and compare journals using citation data provided by The Thomson Reuters Corporation. All the articles are analysed and synthesised according to their type of study, assessment method, assessment criteria and dimensions and stakeholders, followed by table in Appendix 2.
Figure 1 Search strategy for systematic review
4. Results
With careful examination, the research makes categorisation for the articles. These categories are:
• Articles that conduct their own assessment or evaluation process and method with certain evaluation criteria. The author(s) applied the evaluation criteria on certain ICT which is operated in healthcare setting. They mostly concluded the worthiness of such technology based on their notion of value. These kinds of articles are categorised as case or empirical.
• Articles that systematically review any aspects belonging to the assessment of ICT investment in healthcare theme. These articles could be reviewing about the assessment process or method being used, the evaluation criteria mostly used, or discussion concerning any points from previous explanation about definitions and restrictions. These articles are identified by their characteristic in describing their finding. Their description is not focus on proposing idea or innovation. Nevertheless, they reviewed the theme with highly degree of systematic procedures. This kind of articles is categorised as review.
• Articles that discuss and suggest an idea or an innovation related to any points from assessment of ICT investment in healthcare theme, e.g. suggest a framework in assessment process, criteria that should be used in evaluation, or discuss certain ICT related to healthcare. These articles are identified by their characteristic of giving arguments based on not‐so‐systematically literatures review or combine them with examples of case. This kind of articles is categorised as informational.
Descriptive statistics from the study show that 20% of the articles are from 2003, 19% are from 2004, 20%
are from 2005, 22% are from 2006, 13% are from 2007 and 6% are from 2008. There are 7% of eligible articles sourced from Business Resource Premier, 76% sourced from MEDLINE, 5% sourced from
ScienceDirect, and the rest are taken from other sources. 35% of the articles are not listed in the Journal Citation Report (JCR). Thus, they do not have the impact factor measurement. 27% of the articles have zero number of citations. It happened because these are new articles that were published around the end of 2007 or 2008 and were not cited yet.
Furthermore, the descriptive statistics show that 46% of the articles refer to the category of case or empirical study, 30% to an informational study and 24% to a review study (Figure 2). From Appendix 2, we can see that there are various types of ICT and healthcare settings throughout the study. From 99 articles included in the study, there are 80 articles discussing specific types of ICT and 19 articles refer to general types of ICT (Appendix 2). From the category of case or empirical study, most of the articles use combination methods or multi approaches in their assessment process (Figure 3). More researchers use quantitative methods than qualitative studies during the period of published‐articles (Figure 3). There are only 19% and 4% from all the articles that included guidelines for assessment process and stakeholders respectively into their studies.
Figure 2 Type of study from included‐articles
46%
30%
24%
Case/empirical Informational Review
Figure 3 Assessment method from included‐articles
26%
33%
41% Qualitative
Quantittative Mixed/multi
Some notions of values can be drawn from the research. These notions of values are accuracy, benefit, benefit and barrier, cost and benefit, efficiency, financial, functionality, medical (accuracy, effectiveness, and requirement), quantity and utilisation. There are 4 studies discussing the need of a specific assessment
framework for healthcare and there are 15 studies discussing the appropriate time and prescription when conducting assessment. The following sections are the research’s result of stakeholders and assessment criteria and dimensions.
4.1. Result of stakeholders
Many stakeholders were identified throughout the study. The research has categorised them into 17 stakeholders’ categories and summarised their definitions1. Since it is assumed to be already known, definition of stakeholders are hardy could be found in included‐articles. Sorted by their importance (Figure 4), these stakeholders are:
1. Physician refers to a skilled health‐care professional trained and licensed to practice medicine;
specifically: a doctor of medicine or osteopathy. It distinguished from general practitioner as physician is more specialised medical doctor
2. Nurse refers to a person who cares for the sick or infirm; specifically: a licensed health‐care professional who practices independently or is supervised by a physician, surgeon, or dentist and who is skilled in promoting and maintaining health
3. Patient refers to a sick individual especially when waiting or under the care and treatment of a physician or surgeon
4. Healthcare organisation refers to institution or entity providing healthcare service to patients or clients (i.e. management in certain level)
5. Clinician refers to an individual qualified in the clinical practice of medicine, psychiatry, or psychology as it is distinguished from one specializing in laboratory or research techniques or in theory
6. (End) User – undefined refers to what type of personnel the system is intended (Laerum, 2006). It refers to someone or something that uses a particular product, service, etc, rather than the people who make or develop it. It is the ultimate consumer of a finished product. This term is given when there is no definite meaning of who the user is
7. IT/IS department/developer related refers to department or division which gives service to the organisation in term of any IT/IS related infrastructure (i.e. hardware, software, databases and network) or support (people or expertise and procedures)
8. Staff refers to employee or personnel who works mostly administrative or clerical matters. Included in this category are secretary, site personnel and ancillary staff
9. Assistant related position refers to someone who supports the primary medical role (i.e. doctor, nurse, physician, etc.) in their work. Assistants included in this category are care assistant, research assistant and physician assistant
1 Most of definitions are taken from Merriam‐Webster's Medical Dictionary, Merriam‐Webster, Inc. (Dictionary.com.
http://dictionary.reference.com) and Longman Dictionary of Contemporary English Online (http://www.ldoceonline.com/)
10. Client refers to person or party who act on behalf of the patients
11. Head of clinic/laboratory manager refers to someone who is in charge of a clinic or laboratory 12. Clinical researcher/hospital scientist: refers to someone who does a research and specialises on
research in laboratory, clinic or hospital. It is distinguished from clinician who more applies his/her expertise in clinical practice
13. Top management refers to the highest level of management. On this level the managers conduct most of strategy matter instead of technical matter. Remenyi et al. (2007) referred to top
management as a term used to refer to the chief executive and other senior members of the board of directors
14. General Practitioner (GP) /Doctor refers to a physician or veterinarian whose practice is not limited to a specialty
15. Medical student refers to someone who studies medical for the purpose becoming one of healthcare profession (doctor, nurses, etc.)
16. Head of division/manager refers to someone who is in charge of a division or department in healthcare setting
17. Community/external refers to external entity outside the healthcare setting (e.g. social community, citizens)
Figure 4 Stakeholders stacked bar
0 5 10 15 20 25 30 35 40
Community/external Head of division/manager Medical student General Practitioner (GP) /Doctor Top management Clinical researcher/hospital scientist Head of clinic/laboratory manager Client Assistant related position Staff IT/IS department/developer related User ‐ undefined Clinician Healthcare organisation Patient Nurse Physician
Frequency Absolute no. of cited Relative no. of cited Impact factor
Figure 5 Stakeholders stacked bar without frequency
0 2 4 6 8 10 12 14 16 18
Community/external Head of division/manager Medical student General Practitioner (GP) /Doctor Head of clinic/laboratory manager Clinical researcher/hospital scientist Client IT/IS department/developer related Top management Assistant related position Staff Clinician User ‐ undefined Healthcare organisation Patient Physician Nurse
Absolute no. of cited Relative no. of cited Impact factor
Figure 6 Stakeholders difference
0 2 4 6 8 10 12 14 16 18 20
Top management Community/external Client Clinical researcher/hospital scientist Staff Assistant related position Medical student Head of division/manager Head of clinic/laboratory manager User ‐ undefined General Practitioner (GP) /Doctor Healthcare organisation IT/IS department/developer related Clinician Nurse Patient Physician
Difference
Since there are many types of stakeholders are involved in the assessment of ICT investment in the healthcare sector, the research gathers and makes a categorisation of 17 stakeholders emerge from all included‐articles to make it easier to analyse. Figure 4 and Figure 5 show stacked bar of category that compare contribution of number of citation, impact factor and frequency across every category of stakeholders.
For comparison between these two figures, Figure 6 shows bar chart of the difference across every category of stakeholders. All bar charts are sorted from the largest to the smallest. Figure 4 shows the importance of stakeholders on the basis of impact factor, relative and absolute number of citation and frequency. From the graph’s rank order we can see that physician, nurse, patient, healthcare organisation and clinician are the top five most important stakeholders.
On the other hand, Figure 5 shows the importance of stakeholders on the basis of impact factor and relative and absolute number of citation. From the graph’s rank order we can see that nurse, physician, patient, healthcare organisation and user are the top five most important stakeholders. Since these two graphs are the result of a weighted average calculation, therefore, the rank of importance of these 17 stakeholders can be interpreted in Figure 4 and Figure 5.
Moreover, Figure 6 shows the difference between those two analyses of importance. It is the result from subtraction of data in Figure 4 and Figure 5. From this difference analysis (Figure 6), we can see that physician, patient, nurse, clinician and IT/IS department or developer are the top five stakeholders which have large difference between the two analysis of importance. Interestingly, IT/IS department or developer
appears as the stakeholder which has large difference (Figure 6) though it does not emerge as the top five of important stakeholders in both importance analysis (Figure 4 and Figure 5).
Figure 7 Stakeholders stacked bar for high impact journal
0 5 10 15 20 25 30 35 40
Top management Assistant related position Healthcare organisation Patient Nurse Physician
Frequency Absolute no. of cited Relative no. of cited Impact factor
Figure 8 Stakeholder stacked bar without frequency for high impact journal
0 5 10 15 20 25 30 35 40
Top management Assistant related position Healthcare organisation Patient Nurse Physician
Absolute no. of cited Relative no. of cited Impact factor
The research analyses articles which have high impact factor, i.e. top 20 percent from list of articles based on their impact factor. Figure 7 and Figure 8 show stacked bar of category that compare contribution of number of citation, impact factor and frequency across every category of stakeholders limited to articles with high impact factor. Therefore, Figure 7 shows the importance of stakeholder based on impact factor, relative and absolute number of citation and frequency. The stacked bars from Figure 7 and Figure 8 are sorted from the largest to the smallest.
The high‐impact journals only yield 6 stakeholders. From the graph’s rank order, we can see that physician, nurse, patient, healthcare organisation, assistant related position and top management clinician are the most important stakeholders. Moreover, Figure 8 shows the impact factor and number of citation measurement without frequency data. However, it shows the same rank position as shown previously in Figure 7
Figure 9 XY diagrams of impact factor versus absolute number of citation for stakeholders
0 1 1 2 2 3
0 2 4 6 8 10 12
Impact factor
Absolute no. of citation
Assistant related position Client
Clinical researcher/hospital scientist Clinician
Community/external General Practitioner (GP) /Doctor Head of clinic/laboratory manager Head of division/manager Healthcare organisation IT/IS department/developer related Medical student
Nurse Patient Physician Staff Top management User ‐ undefined
Figure 10 XY diagrams of impact factor versus relative number of citation for stakeholders
0 1 1 2 2 3
0 1 1 2 2 3 3 4
Impact factor
Relative no. of citation
Assistant related position Client
Clinical researcher/hospital scientist Clinician
Community/external General Practitioner (GP) /Doctor Head of clinic/laboratory manager Head of division/manager Healthcare organisation IT/IS department/developer related Medical student
Nurse Patient Physician Staff Top management User ‐ undefined
Figure 11 Trend line of impact factor versus number of citation for stakeholders
y = 0.0926x + 0.7642 R² = 0.2316 y = 0.3663x + 0.6478
R² = 0.2856
0 1 1 2 2 3
0 2 4 6 8 10 12
absolute vs impact factor relative versus impact factor Linear (absolute vs impact factor) Linear (relative versus impact factor)
Figure 9 and Figure 10 are XY graph which plot absolute and relative number of citation versus impact factor.
They show position of stakeholders and concordance between number of citation and impact factor
measurement. The more a stakeholder’s position lays away from origin the more important the stakeholder is. Figure 11 plots absolute and relative number of citation versus impact factor together and draws their trend lines. It shows positive linear relationship between two measurements.
4.2. Result of the assessment criteria and dimensions
Many assessment criteria and dimensions were identified throughout the study. Hence, the research has categorised them into category of 28 criteria and dimensions and summarised their definition. Sorted by their importance (Figure 12), these assessment criteria and dimensions are:
1. Usage refers to the amount of users using the system that can be shown by time, frequency, log, etc.
Examples of this criterion, the study of Blaser et al. (2007) made an argumentation that the system is acceptable and usable in the daily routine shown by a high usage rate. Zheng, Padman, Johnson, &
Diamond (2005) argued that the basis of further qualitative assessments were formed by the result of system use in system setting provided by the usage data analysis. Included in this criterion are usage patterns, usage analysis and frequency of use.
2. Satisfaction refers to degree or level of the system or service can fulfil user and non user‘s need or desire. User satisfaction defines as a measure of system performance that meets basic requirements and standards (Nahm, Vaydia, Ho, Scharf, & Seagull, 2007). Another study of Harison et al. (2006) defined healthcare employee satisfaction as the extent to which the operating room (OR) personnel is pleased with its work. In the study of Lising et al. (2005), user satisfaction was measured using user
assessment of potential effects of computers (UAPEC), i.e. instrument with established reliability and validity use to assess the influence the system has on. This instrument contains items that assess concepts such as the degree to which computer use may improve or worsen quality of patient care, job satisfaction and patient privacy. And another instrument to measure user satisfaction in Lising et al. study was user satisfaction survey (USS), i.e. instrument consists of questions on users’
perceptions of potential benefit or harm derived from the system functionality and implementation.
Included in this criterion are patient and provider satisfaction (Apkon et al., 2005), user (physician and nurse) and patient satisfaction (Delpierre et al., 2004), satisfaction of patient with patient care (Kern, 2006)
3. Time related refers to any related approach to measure the length of time used by or in the system when doing certain process or task. Examples of this criterion, length of the consultation, i.e. how much time is spent to make the consultation with the patient (Delpierre et al., 2004), time taken or time to complete i.e. how much time to complete the tasks (VanDenKerkhof, Goldstein, Rimmer, Tod, & Lee, 2004), average time per patient i.e. time to review charts per patient, average time per eligible patient, i.e. time to review charts per eligible patient (Thompson, Oberteuffer, & Dorman, 2003), time spent‐efficiency, i.e. time differences between computer and paper‐based
documentation (Poissant, Pereira, Tamblyn, & Kawasumi, 2005), waiting time (Stoop, 2005), provider time, i.e. outcomes related to time‐consuming process done by healthcare provider (e.g.
time to take image, time to document, nursing and staff time spent, etc.) (Roderer, 2004).
4. Usability refers to the system’s degree of easiness (easy to understand and easy to use) from user point of view. The study of Amarasingham et al. (2006) defined usability as the degree to which the clinical information system is effective, easy to use, and well supported. The sub domains for usability in their study were effectiveness, ease, support. Kushniruk & Patel (2004) defined as the capacity of a system to allow users to carry out their task safely, effectively, efficiently, and
enjoyable. Moreover, the study of Beuscart‐Zephir et al., 2005 (2005) referred usability assessment as way of assessing whether an application is compatible with users’ activity and with the cognitive and collaborative aspects of activity in particular in order to make sure that the application is intuitive enough, and will be easy to learn and to master. Dominich, Goth, & Kiezer (2006) used the Leighton‐Srivastava calculation, usability of system (e.g. access, buttons, navigation, and search possibility), usability of answer (e.g. display, size, overview, readability, navigation) which was measured the ability of the system to return relevant results or correctness, known also as relevance effectiveness. And, Blaser et al. (2007) used user feedback (qualitative survey)to measure user’s responses from the using the system based‐on its acceptability and usability.
5. Data or information quality refers to the degree or level of data, processed data (information) and document’s fitness according to requirement or necessity. The study of documentation‘s quality in comparing paper‐based and computer‐based system by Lising & Kennedy (2005) referred this assessment criteria as an audit tool to address quality of documentation that will be used on pre implementation, post implementation and during follow‐up. Moreover, Leonard, Lin, Dalziel, Yap, &
Adams (2005) argued that some improvement in data or information quality can be shown through better data capture, information processing, or decision support. Finally, the study of Van Der Meijden, Tange, Troost, & Hasman (2003) argued that information quality attributes (e.g., accuracy, completeness, legibility) is concerned to the input and output of the system.
6. Organisation refers to organisational issue or perspective in assessing the benefit of certain technology. The organisational aspects are concerned not only to management issues (patterns of competence and responsibilities) and structural arrangements (establishment and maintenance of work procedures, principles of communication, etc), but also with vision and values (Brender, 2004).
Impacts of ICTs on organisation can be shown as decrease in hospitalisation, improvement in
accessibility of data and achieve effective clinical and administrative management, increase business efficiency. Study of Bellazzi et al. (2003) defined organisational evaluation as impact on
organisational parameters associated to the medical treatment or therapy. Moreover, The study of handheld technology by Chau et al. (2006) referred organisational perspective as the utilisation of handheld computers that offers substantial advantages at the point‐of‐care, care planning and assessment. Some aspects of the health care process, i.e. outcomes resulted from healthcare process such as physicians visit, medication errors, blood pressure measurements (Roderer, 2004), also included in this criterion.
7. Human refers to human issue or perspective in assessing the benefit of certain technology. It refers also to people (user and non user) concerning to behavioural aspects (psychological, social, cultural aspects) (Brender, 2004). Examples of this criterion, Chau & Turner (2006) argued that the ability for mobile technology to perform multiple functions in a handheld device is an attractive proposition from an end‐user perspective. Van Der Meijden et al. (2003) referred human assessment criterion as the impact on individual users measure by the attributes related to changed work practices and varying from a change in frequency of documentation to shifts in responsibilities for certain tasks.
Moreover, Yusof, Kuljis, Papazafeiropoulou, & Stergioulas (2008a) argued that human factor also can be measured by the component of system use and user satisfaction.
8. Technology refers to hardware, software, network and IT infrastructure or any technical matters in implementing ICT in healthcare sector. This criterion included also comprehensive perspective of technological or system assessment which refers as system quality, information quality and service quality. For instance, study of Yusof et al., (2008a) proposed a framework of human, organization and technology‐fit (HOT‐fit) to construct for the capability of being useful in conducting a thorough evaluation study. Included in this criterion are technology perspective in handheld computing, i.e.
the use of handheld technologies that offers some unique advantages over conventional computing devices found in the health care domain (Chau et al., 2006), technical evaluation, i.e. the quality measurement of software and hardware showed by logs and user questionnaire (Bellazzi et al., 2003)
9. Medical outcome – desired or undesired refers to any desired or undesired result caused by medical treatment, therapy and examination. The study of Leonard et al., (2005) defines health outcomes as better care resulted from providing timely and appropriate care to patients as show in deceased patients by days in program by time period, discharged patients by days in program by period, deceased and discharge patients by year, patient days on program (DOP) and occurrence of actual
‘time to assessment’ in days. Medical outcome criterion also shows whether the use of technology contribute to improved health of the patient (Goodman et al., 1999). Examples of undesired outcome are OAT overdose, haemorrhagic accidents (Colombet, Bura‐Riviere, Chatila, Chatellier, &
Durieux, 2004), undesired examination or unsafe diagnostic i.e. the instances in which subjects’
diagnostic workup deviated from a ‘minimum gold standard’ provided by an independent expert
panel of clinicians (Ramnarayan et al., 2006). Included in this criterion are patient outcomes i.e. the assessment of symptom severity, social and work function, and quality of life (Trivedi et al., 2007) and improvement in risk, survival, length of hospitalisation (Delpierre et al., 2004), clinical outcomes i.e. important clinical outcomes for clinicians, include patient clinical status, patient safety, length of stay, and mortality rates (Nahm et al., 2007), changes in patient status i.e. outcomes related to patient after treatment or therapy (e.g. adverse health events, metabolic outcomes, postoperative complications, etc.) (Roderer, 2004).
10. Service quality/quality of care refers to level of excellence on the service or care being delivered or the highest quality of treatment to patients being offered (Harison et al., 2006). Example of this criterion, the study of Beuscart‐Zephir et al. (2005) referred quality management as the
determination that all the required functions ware actually working and meet the users’
expectations. Moreover, Cariello (2003) referred service quality in computerised telephone nurse triage as the perception of reliability, responsiveness, assurance and empathy obtained from callers who used the system on behalf of patient (client).
11. Cost related refers to any type of cost that shown in economic or monetary value. Also, any cost related criterion which is occurred by using or not using the system and cost comparison between new introduction system and older or previous system. Included in this criterion are capital cost, support cost, full time employee (FTE) cost, adverse drug reactions (ADR) cost, surveillance start‐up cost (occurred in implementing automated surveillance system), cost to conduct surveillance (occurred in conducting surveillance), cost of care or direct cost of treatment i.e. the assessment of overall service utilisation cost (Trivedi et al., 2007) and cost associated with service or care being delivered, cost‐efficiency, willingness to pay (WTP) and willingness to undertake (WTU) i.e. estimate measurement of economic values of the system (Miyahara et al., 2006), breakeven point (BEP), and cost savings.
12. Performance refers to degree of the system’s accomplishment in doing tasks or jobs compare to previous system or method. This criterion involved some of technical matter conducting in
laboratory, in simulated clinical situations, or in the clinical setting side by side with standard care in order to assess the components individually, as a whole, and together with other applications (Adhikari & Lapinsky, 2003). The study of Beuscart‐Zephir et al. (2005) referred performance analysis as the evaluation of the exhaustiveness of documentation and evaluation of the quality of
documentation (in such way to make sure that the application will allow at the least as a good medical information management as the previous method) and that the quality of the
documentation will be just as good or improved. Moreover, Adhikari et al. (2003) referred safety and technical performance as the measurement of reliability of computer‐gathered patient data and technical robustness under a variety of operating condition, including performance of hardware, user interface, connections to external device and software.
13. QOL refers to the impact of system on improving patients’ QOL and their degree of satisfaction with the therapy using the system (Bellazzi et al., 2003).
14. Functionality refers to the system’s capability in providing useful features or functions for certain task or job. Bell, Cretin, Marken, & Landman (2004) referred the health benefits of an e‐prescribing system would be a function of its ability to reduce preventable ADEs and increase the appropriate
