Usability problems do not heal by themselves: National survey on physicians’ experiences with EHRs in Finland

(1)

Citation for this paper:

Kaipio, J., Lääveri, T., Hyppönen, H., Vainiomäki, S., Reponen, J., Kushniruk, A.,

Borycki, E. & Vänskä, J. (2017). Usability problems do not heal by themselves:

National survey on physicians’ experiences with EHRs in Finland. International

Journal of Medical Informatics, 97, 266-281.

UVicSPACE: Research & Learning Repository

_____________________________________________________________

Faculty of Human and Social Development

Faculty Publications

_____________________________________________________________

Usability problems do not heal by themselves: National survey on physicians’

experiences with EHRs in Finland

Johanna Kaipio, Tinja Lääveri, Hannele Hyppönen, Suvi Vainiomäki, Jarmo

Reponen, Andre Kushniruk, Elizabeth Borycki, Jukka Vänskä

2017

© 2016 The Authors. Published by Elsevier Ireland Ltd. This is an open access

article under the CC BY-NC-ND license (

http://creativecommons.org/licenses/by-nc-nd/4.0/

).

This article was originally published at:

(2)

ContentslistsavailableatScienceDirect

International

Journal

of

Medical

Informatics

jo u r n al h om e p a g e :w w w . i j m i j o u r n a l . c o m

Usability

problems

do

not

heal

by

themselves:

National

survey

on

physicians’

experiences

with

EHRs

in

Finland

Johanna

Kaipio

a,∗

_,

_Tinja

_Lääveri

b

_,

_Hannele

_Hyppönen

c

_,

_Suvi

_Vainiomäki

d

_,

Jarmo

Reponen

e,f

_,

_Andre

_Kushniruk

g

_,

_Elizabeth

_Borycki

g

_,

_Jukka

_Vänskä

h

a_Aalto_University,_School_of_Science,_Department_of_Computer_Science,_Espoo,_Finland

b_University_of_Helsinki_and_Helsinki_University_Hospital,_{Inﬂammation}_Center,_Clinic_of_Infectious_Diseases,_Helsinki,_Finland c_National_Institute_for_Health_and_Welfare_(THL),_Information_Department,_Helsinki,_Finland

d_University_of_Turku,_Department_of_Clinical_Medicine,_General_Practice,_Turku,_Finland

e_University_of_Oulu,_{Finntelemedicum,}_Research_Unit_of_Medical_Imaging,_Physics_and_Technology,_Oulu,_Finland f_Hospital_of_Raahe,_Department_of_Radiology,_Raahe,_Finland

g_University_of_Victoria,_School_of_Health_Information_Science,_Victoria,_British_Columbia,_Canada h_Finnish_Medical_Association,_Helsinki,_Finland

a

r

t

i

c

l

e

i

n

f

o

Articlehistory:

Received15December2015

Receivedinrevisedform2October2016 Accepted10October2016 Keywords: Usability Userexperience Physician Questionnaire

Electronichealthrecordsystem Healthinformationsystem Nationalsurvey

a

b

s

t

r

a

c

t

Purpose:Surveystudiesofhealthinformationsystemsusetendtofocusonavailabilityoffunctionalities, adoptionandintensityofuse.Usabilitysurveyshavenotbeensystematicallyconductedbyanyhealthcare professionalgroupsonanationalscaleonarepeatedbasis.Thispaperpresentsresultsfromtwo cross-sectionalsurveysofphysicians’experienceswiththeusabilityofcurrentlyusedEHRsystemsinFinland. Theresearchquestionswere:Towhatextenthastheoverallsituationimprovedbetween2010and2014? Whatdifferencesaretherebetweenhealthcaresectors?

Methods:Inthespringof2014,asurveywasconductedinFinlandusingaquestionnairethatmeasures usabilityandrespondents’userexperienceswithelectronichealthrecord(EHR)systems.Thesurveywas targetedtophysicianswhowereactivelydoingclinicalwork.Twenty-fourusability-relatedstatements, thatwereidenticalin2010and2014,wereanalysedfromthesurvey.Therespondentswerealsoasked togiveanoverallratingoftheEHRsystemtheyused.Thestudydatacomprisedresponsesfrom3081 physiciansfromtheyear2014andfrom3223physiciansintheyear2010,whowereusingtheninemost commonlyusedEHRsystembrandsinFinland.

Results:Physicians’assessmentsoftheusabilityoftheirEHRsystemremainascriticalastheywerein 2010.Onascalefrom1(‘fail’)to7(‘excellent’)theaverageofoverallratingsoftheirprincipallyusedEHR systemsvariedfrom3.2to4.4in2014(andin2010from2.5to4.3).Theresultsshowsome improve-mentsinthefollowingEHRfunctionalitiesandcharacteristics:summaryviewofpatient’shealthstatus, preventionoferrorsassociatedwithmedicationordering,patient’smedicationlistaswellassupport forcollaborationandinformationexchangebetweenthephysicianandthenurses.Evenso,supportfor cross-organizationalcollaborationbetweenphysiciansandforphysician-patientcollaborationwerestill consideredinadequate.Satisfactionwithtechnicalfeatureshadnotimprovedinfouryears.Theresults showmarkeddifferencesbetweentheEHRsystembrandsaswellasbetweenhealthcaresectors(private sector,publichospitals,primaryhealthcare).Comparedtoresponsesfromthepublicsector,physicians workingintheprivatesectorweremoresatisﬁedwiththeirEHRsystemswithregardstostatements aboutuserinterfacecharacteristicsandsupportforroutinetasks.Overall,thestudyﬁndingsaresimilar toourpreviousstudyconductedin2010.

Conclusions:SurveysabouttheusabilityofEHRsystemsareneededtomonitortheirdevelopmentat regionalandnationallevels.Toourknowledge,thisstudyisthefirstnationaleHealthobservatory ques-tionnairethatfocusesonusabilityandisusedtomonitorthelong-termdevelopmentofEHRs.The resultsdonotshownotableimprovementsinphysician’sratingsfortheirEHRsbetweentheyears2010 and2014inFinland.Instead,theresultsindicatetheexistenceofseriousproblemsanddeficiencies whichconsiderablyhindertheefficiencyofEHRuseandphysician’sroutinework.Thesurveyresults callforconsiderableamountofdevelopmentworkinordertoachievetheexpectedbenefitsofEHR systemsandtoavoidtechnology-inducederrorswhichmayendangerpatientsafety.Thefindingsof repeatedsurveyscanbeusedtoinformhealthcareproviders,decisionmakersandpoliticiansaboutthe ∗ Correspondingauthor.

E-mailaddress:Johanna.Kaipio@aalto.ﬁ(J.Kaipio). http://dx.doi.org/10.1016/j.ijmedinf.2016.10.010

(3)

J.Kaipioetal./InternationalJournalofMedicalInformatics97(2017)266–281 267 currentstateofEHRusabilityanddifferencesbetweenbrandsaswellasforimprovementsofEHRusability. Thissurveywillberepeatedin2017andthereisaplantoincludeotherhealthcareprofessionalgroups infuturesurveys.

1. Introduction

Healthcare IT (information technology) adoption rates are rapidly increasing along with the expected beneﬁts of system usage.InmostmodernhealthcareorganisationsITplaysan essen-tialroleincaredeliveryandclinicians’dailywork.IntheNordic countriestheavailabilityanduseoflocalElectronicHealthRecord (EHR)functionalitieshasreachedahighlevel(i.e.closeto satura-tion)[1].IntheEUcountriesaccesstobasicEHRsisbynownearly universalamonggeneralpractitioners[2].IntheUSAtheadoption ratesofEHRsystemsinhospitalshaveincreasedfrom15.6%in2010 to75.5%in2014[3].

Theeffects oftheadoptionanduseofEHRsystemshavenot all been positive. Several studies have revealed that usability problems, technology-inducederrors and lack of end-user par-ticipationin EHR development arecontinuing issues that need to be addressed (e.g. [4–11]). Poorly designed user interfaces havebeenrecognizedtoleadtotechnology-inducederrors and thereby may detrimentally affect patient safety [8,12]. Indeed, many technology-associated adverse events in medicine have beenattributedtopoorinterfacedesignratherthanhumanerror alone[13].

Clinicians’acceptance of andattitudes towardsEHRsystems havebeenshowntorelatecloselytosystemusability,forinstance easeofuse,integrationofthesystemsintoclinicians’workﬂows and helpfulnessof thesystemsin thecare ofpatients[14–16]. In addition,poor systemdesign, systemslowdown and system downtimehavebeenconsideredthemostcommonfactorsin inﬂu-encingclinicians’ negativeattitudestowards clinicalITsystems

[15].Usabilityandhumanfactorsapproachesneedtobeintegrated intothedesignandmonitoringofEHRsystemdevelopmentinorder toovercometheprevailingmismatchbetweenclinicalworkandIT systemsandtosupportpracticesthatimprovepatientsafety.Asa result,thereareincreasingattemptstounderstandhowsystems shouldandcouldbeimproved(e.g.[17,18]).

Currently, survey studies of healthcare IT usetend to focus onavailabilityoffunctionalities(e.g.[19])alongwithaspectsof technologyadoptionandacceptance(e.g.[20,21]).TheOECD (Orga-nizationforEconomicCo-operationandDevelopment)hasledan effortto provide member states withreliable data in order to compareinformationandcommunicationtechnology(ICT) avail-abilityandadoptionratesinthehealthcaresector[22].Moreover, pre-implementationandpost-implementationsurveyshavebeen conductedtoinvestigateclinicians’attitudes,satisfactionand reac-tions towards systems (and their new releases) (e.g. [23]). By contrast,usabilityanduserexperiencerelatedquestionnaireshave mainlybeenappliedduringITdevelopmentprocesses.The ques-tionnaireshavebeenusedforlearningaboutinitialuseexperiences or to compare two or more versions of differing systems (e.g.

[14,24–27])ratherthangatheringlong-termdataonexperiences aboutfullyadoptedsystemsafterlongerperiodsofuse.

Intheacademicliteratureonhuman-computerinteraction(HCI) andusabilityengineering(UE)severaldefinitionshavebeen pre-sented for theconcepts of usability and user experience (UX). CommonlyciteddefinitionsforusabilityaregivenbytheISO 9241-11standard[28]andJakobNielsen[29].Thesedefinitionsshare similarusabilitycomponentsincommon–forinstanceefficiency, satisfactionandeffectiveness_−andemphasizetheroleofcontext. Atamoreconcretelevel,usabilityhasbeendescribedasfollows:“A

systemwithgoodusabilityiseasytouseandeffective.Itisintuitive, for-givingofmistakesandallowsonetoperformnecessarytasksquickly, efficientlyandwithaminimumofmentaleffort.Taskswhichcanbe performedbythesoftware(suchasdataretrieval,organisation, sum-mary,cross-checking,calculating,etc.)aredoneinthebackground, improvingaccuracyandfreeinguptheuser’scognitiveresourcesfor othertasks.”[30].Moreover,asusabilityliesintheinteractionofthe userandthesystem[31],qualityofusehasbeendescribedasthe objectofusability.Aqualityofusemodel,described bytheISO 25010standard [32], includesfivecharacteristics:effectiveness, efficiency,satisfaction,freedom fromrisk andcontextcoverage. Thefirstthreeofthesecomponentsarealsopartofwidelyknown usabilitydefinitions[28,29].

Incontrast,UX(userexperience)asaconceptstillremainsvague despitedozensofattemptstodeﬁneit[33–35].Severalofthese deﬁnitionsdescribeUXasapersonalexperienceincludingaspects ofemotions,beliefsandperceptionsthatoccurbefore,duringand aftersystemuse[36–38].Theseaspectscanbealsoseenaspart oftheconceptofusabilityassuggestedbyISO9241-210standard

[36].Usabilityshouldbeunderstoodasacontextualproperty.In theﬁeldofhealthinformaticsthismeansthataspectsofsafetyand preventionofmedicalerrorsaswellascharacteristicsofhealthcare workneedtobetakenintoconsiderationwhendesigningusability studies.Kushniruketal.[8]havestatedthat“theabilityofmethods fromusabilityengineeringtobeabletopredictmedicalerrorsholds considerablepotentialforassessinghealthcareinformationsystems regardingsafetyandensuringthatsuchsystemsdonotinadvertently introducemedicalerrors”.

In ourownstudies[4,39–41] wehave applieddefinitions of usabilityfromtheHCIfieldwhendescribingtheusabilityof clin-icalICTsystemsfromtheviewpointofdifferentend-usergroups withtheaimofincreasingtheunderstandingofcontextualaspects uniqueinclinicalcontexts.Theobjectiveofdesigningsystemsfor usabilitycanbedescribedasenablinguserstoachievegoalsand meettheirneedsinaparticularcontextofuse[28,36].Following fromthis,wehavepresentedadescriptionforusabilityof clini-calICTsystemsfromthephysician’sviewpoint[4]:Theusability ofclinicalICTsystemsreferstotheabilityofthesystemstohavea positiveimpactonpatientcarebysupportingphysiciansin achiev-ingtheirgoalswithapleasantuserexperience.Inordertosupport physiciansintheirdailyclinicalwork,ICTsystemsneedtobe com-patiblewithphysicians’tasks.Atamoreconcretelevel,thisindicates thatsystemsshouldprovidephysicianswithkey(context-matching) functionalities,beefficient(especiallyintermsofrecord-keepingand informationretrieval),andhaveintuitiveuserinterfaces.Inaddition, ICTsystemsshouldsupportinformationexchange,communicationand collaborationinclinicalworkandbeinteroperableandreliable.Since clinicalICTsystemsareusedinnumerousenvironments,theyshould alsoadjusttovarioususerneedsandorganisationalsettings.

ThefocusofthisarticleisonusabilityofEHRsystemsand physi-cians’experiencesinusingthesesystems.Onlyafewstudieshave beenconductedonalargescaleabouttheusefulnessand usabil-ityofEHRsystemsfromtheend-users’viewpointoverthepast fewyears.Nationalsurveysthatincludeusability-relatedquestions havebeenconductedinsomeNordiccountries[1,42–44],butthe focusoftheseworkshasmainlybeenonadoptionandintensityof use.Toourknowledge,besidesourpreviousstudy[4,9,41,45–49], speciﬁcnational usability surveyshavenot beensystematically conductedamonganyhealthcareprofessionalgroups.

(4)

Longitudi-naleHealthobservatorystudiesonusabilitywillbecomeessential whenmovingfromadoptiontothenextlevel:monitoringofuse, usabilityandend-userexperiencesofEHRsystemsastoolsfor sup-portingandrenewingworkprocessesandcareoutcomesaswell astheimpactsofdevelopmentactivitiesandupdatesduring estab-lisheduse.

1.1. Contextofthestudy 1.1.1. HealthcareinFinland

Healthcarein Finlandconsistsof adecentralized,three-level publichealthcaresystemandapartlypubliclyreimbursedprivate sector.Municipalitiesareresponsibleforprimaryhealthcare. More-over,alloftheover300municipalitiesbelongtoone ofthe21 hospitaldistricts,whichprovidepublicsecondarycare,both inpa-tientandoutpatient. Outofthese,ﬁveuniversityhospitalsalso providetertiarycare.Thevastmajorityofseverelyillpatientsare treatedbythepublichealthcaresystem,forinstancethereareno privateintensivecareunitsandallorgantransplantationsare per-formedinthepublicsector.Generalpractitioners in healthcare centresactasgatekeeperstoservicesatsecondaryandtertiary lev-elsofcare,andareferralfromaprimarycareproviderisnecessary innon-urgentcases.However,theprivatesectorcoversasmuchas onethirdofoutpatientvisits[50].Manyoftheseareoccupational healthcarevisits.ManyFinnishpatientschooseprivateprovidersin ordertochoosetheirphysicianinasecondary/tertiarycaresetting ortoavoidlongwaittimesforprimarycarephysicianappointments

[51].

1.1.2. PhysiciansinFinland

In2014,therewere16,350working-age(<65years)physicians living in Finland that were doing clinical work [52,53]. Physi-ciansworkinginhospitalsoftenworkboth inoutpatientclinics andinpatientwards.Thenumberofphysicianshasincreasedby approximately2000since2010.Theproportionofphysiciansin theyoungestandoldestagegroupsaswellasfemalephysicians hasgrownsince2010[54].

1.1.3. Healthinformationsystemsandinformationexchangein Finland

EHRcoveragereached100% inpublichealthcare in 2010.In addition,thevastmajorityofprivatehealthcareprovidersuseEHR systems[42,55].Asingleadministrativeregisterofpatient infor-mationgeneratedbydifferentpublichealthcareproviderswithina hospitaldistrictwasmadepossibleonlybythenewhealthcareact in2011.Thishasincreaseddatasharingbetweenprimaryand sec-ondarycare[42]viajointregistersorregionalhealthinformation systems.

Atthetimeofthesurveyinthespring2014,healthinformation systemswereundergoingaremarkablereformintermsof connec-tivityandnewfunctionalities:thee-prescriptionfunctionalityhad alreadybeenfullyimplementedinallpublichealthcaresettingsand intwooutofthreeprivatesectorEHRsystemsthatareincludedin thisstudy.Allprescriptionsareinonesingledatabasethatcanbe accessedbypharmacies,healthcareprovidersandpatientsalike. Bycontrast,theimplementationofthenationalpatientdata repos-itory(Kanta)hadonlystartedinonesmallhospitaldistrict,but someoftheEHRsystemshadalreadydeployedthefunctionalities neededforintegrationwiththeKantaservices.1

1 _The_National_Archive_of_Health_Information_(Kanta)_is_the_name_of_the_national

datasystemservicesforhealthcareservices,pharmaciesandcitizens.Theservices includetheelectronicprescription,pharmaceuticaldatabase,MyKantapages,and patientdatarepository.TheservicesaredeployedinphasesthroughoutFinland. Moreinformationisavailableat:http://www.kanta.ﬁ/en/kanta-palvelut.

1.2. Aimofthepaper

This paper presents a follow-up of a cross-sectional study of physicians’ experiences with currently used EHR systems in Finland. The ﬁrst national study was conducted in 2010

[4,45,46,49].Thisstudyformedthebaselineforthefollow-upstudy resultstobereportedinthisarticle.

Theaimofthispaperistopresentthe2014surveyresultson Finnishphysicians’experienceswithEHRuseandcomparethose withthe2010resultsbyanalysingdatafromtwoperspectives: a)To what extent hasthe overall situation regarding usability

changed?.

b)What differences are there between public hospitals (both outpatient and inpatient), publicprimary healthcare centres (outpatient)andprivateproviders(mainlyoutpatient)?.

2. Relatedresearch:questionnairestudiesonEHRusability

TheapproachandfocusofsurveysonEHRadoptionand usabil-ityseemtodifferacrosscountries[56].Forexample,Canadaisatan earlierstageinEHRadoptionanddoesnothaveanational question-nairefocusedexclusivelyonquestionsaboutEHRuse.Questions aboutITusagearepartofalargerCanadianphysiciansurvey,but thecurrentsurveydoesnotcontainquestionsaboutphysicians’ assessmentoftheusabilityofEHRs[56].TheNordiccountrieshave setupaneHealthgrouptobenchmarkthedeploymentand use ofhealthITwithinthefiveNordiccountries[1].AlbeittheNordic surveysshareseveralcommonvariablesabouttheusabilityofthe systems,theyarenottied tospecificfunctionalities ortypesof informationcollected[1].Usabilitydatahasbeencollected com-prehensivelywithnationalusabilitysurveysonlyinFinlandandin Iceland[1].InFinland,thefirstusabilitysurveyforphysicianswas conductedin2010[4]andtheIcelandicsurvey,conductedin2014, wasbuiltbasedontheFinnishsurvey[1].

2.1. Nationalusability-focusedquestionnairestudyinFinlandin 2010

Thenational questionnairestudy aimedtostudyphysicians’ experiences ofuse, usability anddevelopmentclinical informa-tion and communication (ICT) systems, particularly EHRs, and therebyprovidegeneralizedpictureabouttheadvantages, prob-lems,andchallengesthatwererelatedtothesesystems.Forthe studyatailoredusability-focusedquestionnairewasdesignedby amultidisciplinary grouporresearchers[4].Usabilityquestions inthequestionnairewerederivedfromtheconceptualizationfor usabilityofclinicalICTsystems(describedin[4]).Thequestions addressedvariousaspectsofclinicalICTsystemusefroma physi-cian’sviewpoint:

-Compatibilitybetweensystemsandphysicians’tasksincluding statementsaboutkeyfunctionalities,efﬁciencyofuse, intuitive-nessofEHRuserinterface

-Systemsupportforinformationexchange,communicationand collaborationinclinicalwork

-Integrationandinteroperabilitybetweenthesystems,aswellas reliabilityandtechnicalfunctionality[4].

Inaddition,theusabilityquestionsreﬂectedvarioususability aspects,forinstance

-Efﬁciencyofconductingroutinetasks

-Physicians’abilitiestoutilizekeyfunctionalitiessuchassummary views

(5)

J.Kaipioetal./InternationalJournalofMedicalInformatics97(2017)266–281 269

-Learnabilitywithrespecttotherequiredamountoftraining -Safetyorerrors(asinlowerrorrates)ofmedicationordering

pro-cessesandseriousadverseeventsforthepatientcausedbyfaulty EHRsystemfunctions.

Mostoftheusabilityquestionswereformulatedasstatements. Thequestionnairealsoincludedasummativequestionaboutthe overallrating(socalledschoolgrade)tothephysician’sprincipally usedEHRsystem.

Altogether3929physiciansactivelyworkinginpatientcarein thepublicandprivatesectorsrespondedtotheweb-based ques-tionnairein2010[4].

Results from the study have been reported in several pub-lications [4,9,41,45–49]. The main usability ﬁndings were that physicians’estimatesabouttheusabilityoftheirclinicalICT sys-temsindicated thattheywerecritical oftheirsystem’s usablity

[4]. The overall grades given to EHR systems varied signiﬁ-cantly: scores for systems used in private sector were higher than for those usedin public hospitalsand healthcare centres. In general, the physicians indicated that the systems lack a properpatientoverview/dashboardandtheyshouldbetter sup-portroutinetasks,decisionmaking,preventionofmedicalerrors, cross-organizationalinformationexchangeandcommunication,as wellascollaborationbetweenphysicians,nursesandpatients[4]. Theresultsalsoshoweddifferencesbetweenresponsesfrompublic hospitals,publichealthcarecentresandprivatesector organisa-tions[4,48].Physiciansworkinginpublichospitalsandinwards weremorecriticalthantheircolleaguesinotherorganisations[48]. Thequestionnairestudyforphysiciansconductedin2010canbe seentohaveanimpactonmonitoringuserexperiencesoneHealth: thesurveytoolandresultshavebeenexploitedwithinFinlandas wellasinternationally.InFinland,thestudyresultspromoted dis-cussionsamong vendors,ministry,healthcare organisationsand researchinstitutesabout howto betterunderstandthecurrent challengesandenhancecollaborativeactionsinimprovingthe sit-uation.ThestudyhasalsoimpactedtheneweHealthstrategyin Finland[57].InFinlandthereisacontinuedcommitmentto mon-itoringusabilityand end-userexperiences.Suchsurveyswillbe continued,andextended fromphysicianstonursingstaff.Many oftheFinnishsurveyvariableshavebeentakenasabenchmark intheNordiceHealthindicatorwork[58].Icelandusedidentical variablestomonitoruserexperienceswithindifferentstakeholder groupsin2014.Futureworkincludesestablishingapermanent sys-temforgathering,analysisandpublicationofresultsofthecommon benchmarkingvariables[59].

2.2. LiteraturereviewonquestionnairestudiesonEHRusability Basedonourpriorquestionnairestudyin2010(describedin Section2.1)andrelatedreviewofliterature[4],theassumption wasthattherearenotmanymonitoringstudiesaboutEHRusein thehealthinformaticsfield.Tocomplementourknowledgeon,we conductedaliteraturesearchtofindrecentlypublishedacademic articlesonusabilitysurveyspublishedafter2009.Ouraimwasto findarticlesreportingwide-scalesurveysonclinicians’experiences withusabilityofEHRsystemswhichwecouldcomparetheresults ofourcross-sectionalstudywith.

Articles weresearched followinga scoping review approach fromPubMedusingthekeywords:“usability”and“questionnaire” combinedwith“EHR”;“EPR”(abbr.ofelectronicpatientrecord); “healthinformation exchange system”; and “national”.In addi-tion;searchterms“longitudinal+study+usability”wereusedto speciﬁcallysearcharticlesreportingusabilityrelatedlongitudinal; cross-sectional;andfollow-upstudies.Theinclusion criteriafor articleswasthefollowing:

-publishedbetween2010and2015,inEnglish

-studiesusabilityofhealthcareITsystemsinuseorrecently imple-mented

-focusesonlargehealthinformationsystemslikeEHRs(notina mobileapplicationorasmallpartofalargerITsystem)

-studiesusabilityfromtheperspectiveofclinicians’experienceon use

-reportsanempiricalstudyincludingdatagathering

-reports astudy,inwhich surveysare usedasanindependent method(e.g.notcomplementingusabilitytesting).

Intotal163citationswerereturned.Thetitlesofallthefound citationswerereviewed.Thereviewwasconductedbytheﬁrst authorofthispaper.Ninearticlesmetthecriteriabasedontheir titleandabstractreview.Twoofthesearticlesreportedresultsfrom ourpriorquestionnairestudyin2010inFinland(Refs.[4]and[49]) andwerenotincludedinfurtherin-depthreview.Thesummary of theremainingsevenarticlesis presented inTable 1.Twoof thearticlesfocusedonclinicalITsystems(includingEHRs)inuse

[64,69],oneonhealthinformationexchange(HIE)system[61]and oneonanorderentrysystem[70],whereasthreestudieswereon theimplementationofEHRsystems[60,63,66].Thestudiesapplied variouskindsofquestionnaires(e.g.Avl[60],IsoMetrics[65])and themesofquestionsrangedfromgraphicallayoutandadequacyof trainingtoperceivedpatientoutcomes.Thenumberofrespondents perstudyvariedfrom32toabout1000.

Based onthereview itappears that long-termresearchand follow-upstudiesofusabilityofEHRsystemsinusearescarce.Only theGermanstudy[64]reportedresults,whichcanbeusedtoget anoverviewofthecurrentsituationofusabilityandcomparea numberofassessedsystems.Thestudycoveredawiderangeof ITsystemsinhospitalsandwastargetedtodifferentusergroups: clinicians,radiologistsandlaboratorypersonnel,aswellas admin-istrative,managementandIT-employees.

EHRimplementationstudies[60,63,66]showedthatclinicians weresatisﬁedwiththeEHRingeneral,butdissatisﬁedwithEHR usability[63].Factorsrelatedtosystemdesignhadstrongeffectson acceptance,evenoneyearafterimplementation[66].Skilled clini-ciansmaybeabletoovercomeusabilityobstaclesbutthisrequires thattheyaregiventheproperresources,educationandtraining

[60]. Afour-yearstudyofITsupportedclinicalpathwaysfound thattheend-users’estimatesofusabilitywererathersatisfactory: onafourpointLikert-scale(1=bad,2=acceptable,3=good,and 4=verygood)theaveragewascloseto2.5.Basedonanalysis,the subjectiveestimatesslightlyimprovedtwotofouryearsafterHIS softwareimplementation,however,nostatisticalsigniﬁcant dif-ferenceswerefoundinthetimecourse[69].InregardtoHIE,the ﬁndingsinastudyinUSAconcerningtheusersatisfactionon inter-facedesignshowencouragingresults[61]:Allthe35itemsofQUIS (QuestionnaireforUserInteractionSatisfaction)had scoresover 50%,whichwereaboveneutral(aQUISscoreof5isneutralanda scoreof>5favourable)whilethemeanscorewas6.5.

3. Methods

The aimof ournationalquestionnaire study wastoexplore Finnishphysicians’experienceswiththeuseoftheircurrent clini-calITsystems,particularlyEHRsystems.Theideawastoutilizethe usability-focusedquestionnairedesignedforthestudyin2010to gatherrepeateddataandtoﬁndouttowhatextenttheoverall sit-uationregardingusabilityhaschangedandwhatdifferencesthere arebetweenhealthcaresectors.

ThebackboneEHRsystemsincludedinthisstudyarebasedon anarchitecturewherecorepatientinformation,narrativepatient recordtextsfrommedicalspecialitiesandadministrativehospital

(6)

Table1

Summaryofarticles.

Authors/Yearof

publication/Referencenumber

HISconcerned Researchmethods Themes/questionsinthe questionnaire

Numberofrespondents Janols,Lind,Göranssonand

Sandblad,2014[60]

Deploymentofthreemodules ofEPRsystemsinSweden

AvalidatedquestionnaireAvl wasoneoftheusedmethods

16questionsonsystem development,usage,utility, competence,stressandhealth, relations

Physicians,nurses,clerks(N notreported)

Gadd,Ho,Cala,Blakemore, Chen,FrisseandJohnson,2011 [61]

HIEsysteminuseinUSA Selecteditemsfromavalidated instrumentQuestionnairefor UserInterfaceSatisfaction (QUIS)[62]

Themes:Overallreactions, screen,terminology andsysteminformation, learning,systemcapabilities, systemfunctionality

165physicians,nursesand others(70%

responserate)

Sockolow,Weiner,Bowles, AbbottandLehmann,2011[63]

RecentlyimplementedEHRin USA

Cliniciansatisfactionsurvey wasoneoftheusedmethods inthestudy

22questionsonimpactof theEHRonclinician satisfactionwithteam communicationandperceived patientoutcomes

37+32physicians(11and17 monthspostimplementation, responseratesof95%and82%)

Bundschuh,Majeed,Bürkle, Kuhn,Sax,Seggewies,Vosseler andRöhrig,2011[64]

ClinicalITsystemsinGerman hospitals

Web-basedquestionnaire, basedonIsoMetricsinventory [65]

37questionsonsuitabilityfor thetask,suitabilityforthe learning,conformitywithuser expectations,effectiveness

1003respondents(including 658cliniciansand73 non-bed-sidemedical personnel)from158hospitals (11%responserate) Carayon,Cartmill,Blosky,

Brown,Hackenberg, Hoonakker,Hundt,Norfolk, WetterneckandWalker,2011 [66]

RecentlyimplementedEHR systeminintensivecareunits (ICU)inUSAregionalmedical centre

Surveybasedonestablished instrumentstomeasure technologyacceptance,EHR usability,andEHRusefulness [62,67,68]

Themes:Overallacceptanceof theEHRtechnology,perceived usability,perceptionsof usefulness,information receivedbytheend-users aboutEHRimplementation, participationin

implementationactivities

121+161nurses(3and12 monthspostimplementation, responseratesof51%and72%)

Schuld,Schäfer,Nickel,Jacob, SchillingandRichter,2011[69]

IT-supportedclinicalpathways inaGermanhospital(HIS softwareincludinge.g.patient datamanagementsystem)

Annualsurvey2006–2009 (interviewsandstandardized questionnaires)toresearch staffsatisfaction

Themes:comprehensibility, usabilityandgraphicallayout (thearticledoesnotdescribe detailsoftheuser questionnaire)

Physiciansandnurses(Nnot reported)

Tan,FloresandTay,2010[70] Orderentrysystemusedin hospitalsinSingapore

Asurveytooldesignedforthe purposesofthisstudy

16questionsonreliability, speedofthesystems,easeof use,adequacyoftraining, impactonproductivity,impact onpatientcare,overall satisfaction

52physiciansandnurses(52% responserate)

Abbreviations:

EHR=Electronichealthrecord. EPR=Electronicpatientrecord. HIE=Healthinformationexchange. HIS=Healthinformationsystem.

informationsysteminformationareavailableindigitalformatvia acommonuserinterface.Thesameuserinterfacegivesaccessto localmedicationdata,nursingdocumentsandadiagnosishistory. DependingontheEHRsystem,radiologyimagesandlaboratory dataarestoredinaseparatedatabase,butusuallytheyareretrieved byexchangingthepatientcontextandseenthroughthemainEHR system.Alsothecomputerizedorderentriesaremanagedinthe mainEHR.Therearesomeelectronicdepartmentalsystems typ-ically in intensive care, operationtheatre or emergency rooms whicharenotintegratedtothemainsystem.Virtuallyall docu-mentationinFinnishhealthcareisnowadayselectronic[42].

Therespondentswereaskedtogivetheirassessmentsin rela-tiontothebackboneEHRsystemtheyhadchosenearlierinthe questionnaire.Thetradenamesofthesesystemsandtheir geo-graphicaldistributionhadremainedthesamebetweenthestudies in2010and2014.Also,thebasicarchitectureanduserinterface concepthadremainedthesame.Inpublichospitalsandin pub-licprimaryhealthcarecentressomefunctionsoftheEHRsystem hadchangedfrom2010:theyincludedtheadditionofamodule toconnectwiththenationalelectronicprescriptiondatabase.Also, thisbasicconnectivitytothenationalhealthinformationexchange (HIE)hascausedmodificationstotheinternalstructureofthe soft-ware:stronguserauthenticationwitha nationalsmartcardwas takenintouseandmanycodesand classificationswereunified. However,fullconnectivitytothenationalpatientrecordarchive

wasstillahead asitwasplannedfor theyears 2014and 2015. AmongprivateprovidersofEHRsystems,thesechangeshadnot yettakenplaceatthetimeofthesurvey.Forsoftwareproviders, thistransitionperiodinconnectivitytothenationalHIEhad prob-ablytakenalotofdevelopmentresourceswhichotherwisecould havebeenusedforuserinterfacedevelopment.

3.1. Questionnaire

The usability-focused questionnaire included18 background questions,38corestatementswithaﬁve-pointLikert-scaleand a questionforthe overallrating oftheEHR-systemin use(see Appendix1inSupplementarymaterial).Inaddition,therewere other groups of questions addressing issues of management, patientsafety,workwell-being,informationsystemsdevelopment, EHRfeaturesthatareworkingwellorareconsideredasthemost importantdevelopmenttargets.Thephysicianswereinstructedto answerquestionsabouttheEHRsystemtheyprimarilyuseinthe contextoftheirdailywork.Outof38corestatements,24usability statementswhichwereidenticalin2010and2014wereselected for analysisin thisstudy. Based onearlierexperiencewiththe 2010responsessomequestionswhererephrasedtoincrease clar-ity.Sincethesequestionsmaynowcarryadifferentmeaning,they havenotbeentakenintoaccountintheanalysis.

(7)

3.2. Datagathering

ThedatawasgatheredfromFebruarytoMarch2014byusinga web-basedquestionnaire.Theindividuallinkstothequestionnaire weresentviae-mailtoallphysicianswhowerecurrentlylivingin Finlandandwhowereundertheageof65.Thee-mailaddressesof thestudypopulationwereobtainedfromtheregisteroftheFinnish MedicalAssociation,whichcoversmorethan90%ofactive physi-cianaddresses.Theresearcherswerenotabletoidentifyindividual respondents.

Theinvitationtothesurveywasemailedtoallworkingage physiciansin Finland.Theactualtargetgroup,physicians doing clinicalwork,washighlightedinthee-mailmessage.Basedonthe FinnishMedicalAssociation’sregister,therewere16350 working-agephysiciansengagedinclinicalworkin2014[52,54]whilethe correspondingﬁgurein2010was14411.Thenumberof respon-dentswas3781(23%)in2014and3929(27%)in2010.Inthe2014 surveysomeoftherespondentsweredifferentfromthoseinthe 2010survey(i.e.respondentswereatleastpartiallydifferent indi-vidualsineachsurvey).

Therewasapossibilitytochoose“other”asamainEHRsystem iftherespondentdidnotconsideranyoftheoneslistedastheir pri-maryEHRsystem.Thereweredozensofsmallerbrandswithless than25respondentseach.Theresponseswereanonymizedafter therespondentshadusedthelinksentbytheFinnishMedical Asso-ciation.Inordertomakethesituationmorecomparablebetween 2010and2014theresearchersdecidedtodiscardthebrandswith theleastnumberofphysicianusersbecausewecouldnottrack thechangesthathave occurredineither theuserorganisations andpractices,usergroupsortheEHRsystemsthemselvesbetween 2010and2014.

3.3. Analysis

Ananalysiswasconductedofuserresponsestothe24 usability-relatedstatementsthatwereidenticalinthe2010and2014survey. Sincetherespondentswerenotidentiﬁedandinordertomakethe resultsmorecomparable,onlytherespondentsthat hadreplied aboutusingtheninemostcommonlyusedEHRsystembrandswere included.Thisresultedintheexclusionof706respondentsfrom 2010and700from2014data(seeFig.1).

The following modiﬁcations to the original data were per-formed:

1)Theoriginalscaleofoverall ratingswasfrom4to10,which followsthetypicalscaleusedatFinnishschools(4meaning‘fail’ and10meaning‘excellent’)andthereforewasfoundintuitive fortheFinnishrespondents.Thescalewaschangedtobefrom1 (‘fail’)to7(‘excellent’)forthepurposesofpresentingtheresults toaninternationalaudience.

2)Theﬁve-pointLikert-scaleanswers‘Fullyagree’and‘Somewhat agree’werecombinedtoformthecategory‘Agree’.Similarly theanswers‘Fullydisagree’and‘Somewhatdisagree’were com-binedtoformthecategory‘Disagree’.

For categoricalvariables thestatistical analyseswerecarried outwithChi-squaretestsorFisher’sexacttest,whenapplicable. Continuousvariableswerecomparedusingaone-wayanalysisof variance.StatisticalsigniﬁcancewasdeﬁnedasP<0.05.The sta-tisticalanalysiswasconductedusingSPSS22software(IBMCorp, Armonk,NY).

4. Results:changesbetween2010and2014

Theresultsaredividedintothreesections:respondents’ demo-graphics2010and2014(Table2),overallratingsforEHRsystems (Table 3), and responses to usability statements presented by healthcare sector: public hospitals (both outpatient and inpa-tient),publicprimaryhealthcarecentres(outpatient)andprivate providers(mainlyoutpatient)(Table4).

4.1. Respondentdemographics

Thedemographicsoftheusersoftheninemostcommonlyused EHRsystembrandsaredescribedinTable2whichshowssimilar demographic features of oursubsetof physicians withthe tar-getpopulation(describedin[4]and [53,54]).Theproportionof femalesamongrespondentshadincreasedbetween2010and2014. Theyoungestagegroupofrespondentshadalsogrowninsizein bothsurveys.Correspondingchangescouldalsobeseeninthe tar-getpopulation. Theproportionofdifferentworkingsectorshad remainedsimilar.

Fig.1. Selectionofdatafortheanalysis.

*EHRbrandsthatwereusedbylessthan25respondentsortherespondentsdidnotdeﬁnethebrand. **r.g.GovernmentsectororUniversityorundeﬁned.

(8)

Table2

Respondents’demographicsinthe2010and2014surveys.

2010 2014 PhysiciansinFinland

2014under65years [52]

p-valueforchange between2010and 2014

Numberofrespondents inthewholesurvey

3929 3781 Numberofrespondents inthisstudy 3223 3081 Numberofphysicians inFinlandunder65 years 18933 20110 n(%ofrespondentsin thestudy) n(%ofrespondentsin thestudy) % Gender <0.001 Female 1840(57) 1909(63) 58 Men 1370(43) 1130(37) 42 Age <0.001 <35years 454(14) 633(21) 21 35–45years 797(25) 669(22) 25 46–55years 1156(36) 841(28) 29 >55years 785(25) 873(29) 25 Healthcaresector 0.232 Publichospital 1807(56) 1667(54) 53 Publichealthcare centre 894(28) 875(28) 26 Privateproviders 522(16) 539(18) 21 Table3

Physician’soverallratingswithscalefrom1(or‘fail’)to7(or‘excellent’)fortheirprincipallyusedEHRsystems(N>30).In2014thephysicianshadtogiveanoverallrating scaleorchoose“Idonotwishtoanswer”.Theproportionsofdifferentbrandsweresimilarbetween2010and2014(p=0,232).

EHRsystem 2010 2014 pfordifference

between2010and 2014 Respondents (total=3223)N(%) Respondents havinggivena score(total=2041) N(%) Meanopinion score(scale1–7) (std.deviation) Respondents (total=3081)N(%) Respondents havinggivena score(total=3057) N(%) Meanopinion score(scale1–7) (std.deviation) Publichospital K 218(7) 148(7) 4.2(1.2) 195(6) 194(6) 4.3(1.2) 0.341 Lh 75(2) 51(2) 2.5(1.3) 47(2) 46(2) 4.2(1.3) <0.001 Mh 462(14) 289(14) 3.7(1.1) 432(14) 428(14) 3.5(1.2) 0.043 Nh 102(3) 60(3) 3.1(1.2) 107(3) 106(3) 3.4(1.7) 0.186 O 950(29) 580(28) 3.1(1.2) 886(29) 881(29) 3.2(1.2) 0.295

Publichealthcarecentre

P 44(1) 31(2) 3.9(1.4) 37(1) 37(1) 4.4(1.0) 0.092 Lc 50(2) 34(2) 3.9(1.5) 71(2) 71(2) 4.0(1.2) 0.952 Mc 439(14) 274(13) 4.1(1.1) 424(14) 419(14) 3.6(1.2) <0.001 Nc 361(11) 231(11) 3.2(1.2) 343(11) 343(11) 3.4(1.1) 0.031 Privateproviders Q 375(12)2 249(12) 4.2(1.0) 379(12) 373(12) 4.3(1.1) 0.324 R 87(3) 60(3) 4.3(1.0) 96(3) 96(3) 4.0(1.3) 0.097 S 60(2) 34(2) 3.5(1.1) 64(2) 63(2) 3.5(1.3) 0.937

4.2. Physicians’overallratingsoftheirEHRsystems

Table3presentstheEHRsystemsusedandtheircontextofuse, aswellasthemeanopinionscoresin2010and2014onascale from1(‘fail’)to7(‘excellent’).Theaverageoftheratingsvaried from3.2to4.4in2014(from2.5to4.3in2010).ThreeoftheEHR systems(systemslabelledL,MandN)areusedbothinpublic hospi-tals(denotedwithasubscript‘h’)andinpublichealthcarecentres (denotedwiththesubscript‘c’).Theratingswereratherlow in 2010,especiallyforthehospitalEHRsystems.Ingeneral,themean opinionscoresgivenin2014accordwithourearlierstudy ﬁnd-ings.However,somechangesarealsoshown:theaveragerating hadimprovedforthesystemsLh(from2.5to4.2)andNh(from

3.1to3.4),whereasthemeanopinionscorehaddecreasedforthe systemM(Mhfrom3.7to3.5andMcfrom4.1to3.6).

4.3. Responsestousabilitystatements:2010comparedto2014

Table4presentsthesummaryoftheﬁndingsforeach24 usabil-itystatement.Thetableshowschangesbetween2010and2014 ‘agree’responsesbyhealthcaresector.Thesimilaranalysisof ‘dis-agree’responsesofthesamedataisavailableasSupplementary material(Appendix2inSupplementarymaterial).

Analysisof‘agree’responsesfrom2010and2014shownotable improvementsinthefollowingEHRfunctionalitiesand character-istics:

-Helpinpreventingerrorsandmistakesassociatedwith medica-tions(allagree(%)from28to41)(statement1)

-Summaryviewonpatient’shealthstatus(allagree(%)from14to 24)(statement2)

(9)

J. Kaipio et al. / International Journal of Medical Informatics 97 (2017) 266–281 273 Table4

Summaryofquestionnaireitems(usabilitystatements)andphysicians’responses:Changesbetween2010and2014forAGREEresponsesbyhealthcaresector.Theboldnumbersrepresentthe”best”responseinthetwoyears thatarecomparedandreﬂectwhetherthesigniﬁcantchangeisanimprovementornot.

Itemno. Statement Publichospitals

AGREE(%of respondentsin workingsector) (N=3474) p-value (between 2010and 2014) Publichealthcare centresAGREE(% ofrespondentsin workingsector) (N=1769) p-value (between 2010and 2014) Privatesector AGREE(%of respondentsin workingsector) (N=1061) p-value (between 2010and 2014)

AllAGREE(%) p-value (between 2010and 2014) Totalnrof responsesto statement 2010 2014 2010 2014 2010 2014 2010 2014

Totalnrofrespondentsinyear andworkingsector

1807 1667 894 875 522 539 3223 3081 6304

1 Informationsystemshelpin

preventingerrorsandmistakes associatedwithmedications.

19 38 <0.001 46 52 0.005 30 34 0.136 28 41 <0.001 6203

2 TheEHRsystemgeneratesa

summaryview(e.g.ona timeline)thathelpstodevelop anoverallpictureofthe patient’shealthstatus.

13 28 <0.001 14 18 0.137 17 17 0.975 14 24 <0.001 3970

3 Informationsystemshelpto

improvethequalityofcare.

30 33 0.022 38 40 0.308 37 42 0.120 33 37 0.003 6215

4 Routinetaskscanbeperformed

inastraightforwardmanner withouttheneedforextra stepsusingthesystem.

28 25 0.092 36 25 <0.001 56 54 0.238 35 30 <0.001 6265

5 Thesystemrespondsquicklyto

inputs.

36 26 <0.001 47 34 <0.001 72 67 0.065 45 36 <0.001 6262

6 Thearrangementofﬁeldsand

functionsislogicalon computerscreen.

36 42 0.001 44 40 0.173 58 63 0.090 42 45 0.008 6252

7 Thesystemkeepsmeclearly

informedaboutwhatitisdoing (forexamplesavingdata).

28 29 0.558 44 34 <0.001 45 47 0.451 35 34 0.164 6253

8 Terminologyonthescreenis

clearandunderstandable(for exampletitlesandlabels).

40 39 0.459 54 45 <0.001 60 59 0.636 47 44 0.012 6254

9 Learningtousetheelectronic healthrecordsystemdoesnot requirealotoftraining.

38 37 0.253 37 24 <0.001 61 50 <0.001 42 36 <0.001 6270

10 Theinformationinthenursing recordisineasilyreadable.

27 26 0.400 35 34 0.431 47 48 0.886 33 32 0.391 5981

11 Thepatient’scurrent

medicationlistispresentedin aclearformat.

(10)

J. Kaipio et al. / International Journal of Medical Informatics 97 (2017) 266–281 Table4(Continued)

Itemno. Statement Publichospitals

AGREE(%of respondentsin workingsector) (N=3474) p-value (between 2010and 2014) Publichealthcare centresAGREE(% ofrespondentsin workingsector) (N=1769) p-value (between 2010and 2014) Privatesector AGREE(%of respondentsin workingsector) (N=1061) p-value (between 2010and 2014)

AllAGREE(%) p-value (between 2010and 2014) Totalnrof responsesto statement 2010 2014 2010 2014 2010 2014 2010 2014 12 Informationonmedications

orderedinotherorganisations iseasilyavailable.

5 9 <0.001 5 12 <0.001 4 9 0.002 5 10 <0.001 6142

13 Obtainingpatientinformation

fromanotherorganisation oftentakestoomuchtime.

76 81 <0.001 75 75 0.991 72 79 0.005 75 79 <0.001 6143

14 Informationsystemshelpto

ensurecontinuityofcare.

37 42 0.002 46 47 0.977 46 54 0.012 41 45 <0.001 6233

15 Informationsystemssupport

collaborationandinformation exchangebetweenphysicians workinginthesame organisation.

67 63 0.312 70 68 0.299 67 68 0.785 65 65 0.673 6258

collaborationandinformation exchangebetweenphysicians workingindifferent organisations.

17 17 0.564 15 19 0.010 5 9 0.016 14 16 0.016 6201

collaborationandinformation exchangebetweenthe physicianandthenurses.

34 45 <0.001 57 60 0.196 48 48 0.992 43 50 <0.001 6203

18 Thesystemmonitorsand

notiﬁeswhentheordersgiven tonurseshavebeencompleted.

14 18 0.010 5 10 0.008 3 7 0.084 11 15 <0.001 3947

collaborationandinformation exchangebetweenthe physicianandthepatients.

12 9 0.004 9 9 0.889 21 20 0.388 12 11 0.024 6106

20 Useoftheinformationsystems

frequentlytakesmyattention awayfromthepatient.

71 76 <0.001 67 71 0.056 50 58 0.008 66 71 <0.001 6246

21 Thesystemisstableintermsof technicalfunctionality(does notcrash,nodowntime).

46 41 0.001 54 46 <0.001 73 72 0.680 53 48 <0.001 6283

22 Information

entered/documented occasionallydisappearsfrom theinformationsystem.

34 28 <0.001 27 25 0.259 27 21 0.015 31 26 <0.001 6225

23 IfIhaveproblemswiththe systemIcaneasilygethelp.

45 45 0.913 52 45 0.005 61 60 0.826 49 47 0.139 6246

24 Faultysystemfunctionhas

causedorhasnearlycauseda seriousadverseeventforthe patient.

(11)

-Theformatofpatient’smedicationlist(allagree(%)from24to 37)(statement11)

-Supportforcollaborationandinformationexchangebetweenthe physicianandthenurses(allagree(%)from43to50)(statement 17).

Incontrast,theanalysisshownotablenegativechangesinthe followingEHRfunctionalitiesandcharacteristics:

-Quickresponsetoinputs(allagree(%)from45to36)(statement 5)

-TheamountoftrainingrequiredtolearntousetheEHR(allagree (%)from42to36)(statement9).

Overall,theportionofrespondentsagreeingwiththepositively formattedusabilitystatementsisratherlow.Thefollowing find-ingscanbepointedoutasexamples.TheEHRsystemsstilllack summaryviewsordashboards(inallsectorsonly17–28%ofthe respondentsagreedwiththestatementabouttheEHRgenerating anappropriatesummaryview),althoughinhospitalEHRsystems thesituationhadimproved(from13%to28%).ITsystems’ability tosupportcollaborationcontinuetoberatedaspoor:Physicians inallsectorsweredissatisfiedwiththesupportforinformation exchangebetweenprofessionalsworkingindifferentorganisations (16%of allagree, statement 16)as wellas betweenthe physi-ciansandpatients(11%ofallagree,statement19).Thisfindings isstronglysupportedbytheanalysisof‘disagree’responses(see Appendix2inSupplementarymaterial),whichshowthatthe por-tionofdisagreeanswershasgrowninallsectorswhencompared toyear2010(inpublichospitalsfrom59%to63%;inpublic health-carecentresfrom60%to65%;inprivatesectorfrom41%to50%). Furthermore,allphysicianscontinuesharingtheirconcernsabout ITsystemusetakingtimeawayfromanddisruptingcaregiving activities.Threeoutoffourphysiciansinpublicsector organisa-tions(76%inhealthcarecentresand71%inhospitals)andmorethan half(58%)intheprivatesectorbelievedthattheuseofthesystems frequentlytakestheirattentionawayfromthepatient(statement 20).

Theanalysisshowsomedifferencesbetweenresponsesfrom differenthealthcaresectors.Comparedtoresponsesfromthe pub-lic sector, physicians working in the privatesector were more satisfiedwiththeirEHRsystemswithregardstosupportfor rou-tinetasks(statement4),responsivenesstoinputs(statement5), intuitivenessoftheEHRuserinterface(statements6–8),andthe requiredamountoftraining(statement9).Incontrast,the assess-mentsbyphysiciansworkinginpublichealthcarecentrestowards theseaspectshadbecomemorecriticalthanin2010.Onlyinthe privatesector,themajorityofphysicians(72%in2014)seemto besatisfiedwiththeirEHR systemswithregardstostability of thesystems(statement21).Publicsectorusersweremoreeven moredissatisfiedthanin2010(physiciansinpublichospitals41% andinpublichealthcarecentres46%in2014).Onthestatement of‘faultyfunctionshavecausedornearlycausedaseriousadverse eventforapatient’(statement24)8%ofphysiciansintheprivate sectoragreedwiththestatementwhereastheportionof physi-ciansworkinginpublichospitalsandinpublichealthcarecentres andagreeing withthestatementremainmuchhigher:42%and 31%.

In general, the analysis of ‘agree’ responses (presented in Table 4) and ‘disagree’ responses (presented in Appendix 2 in Supplementary material) show similar patterns. The compari-sonoftheﬁndings,particularly“best”responsesperhealthcare sector,indicate only somesmall differences which are marked inAppendix 2inSupplementarymaterial. Mostofthese differ-encessupporttheconclusionsmadebasedonanalysisof‘agree’ responses.

5. Discussion

5.1. Themaincontributions

Inthispaper,wereportFinnishphysicians’experienceswiththe usabilityofcurrentlyusedEHRsystems,aswellasonthechanges intheirperceptionsbetween2010and2014.Overall,theresults indicatethesituationhasnotimproved.Physicians’assessments oftheirEHRsystemsstillindicateinadequacies:onascale1–7the averageoftheratingsvariedfrom3.2to4.4.Incontrast,therewere markeddifferencesbetweentheEHRsystembrandswithineachof theusercategories(privatesector,publichospitalsandprimary healthcare).Althoughthesituation hadnot improved consider-ablyingeneral,twointerestingchangesinphysicians’assessments arose:vendorLhwasgiventhemostnegativeassessmentsin2010,

butwasamongthebestin2014,andtheassessmentsofoneof thebiggestvendors(vendorM)haddeterioratedoverafouryear period.Wespeculatethatthereasonsbehindthesechangesarethe following:In2010LhhadreplacedthepreviousEHRsystemonly

threemonthspriortothestudy.Thiswasthebrand’sfirst imple-mentation in a hospital environment. The reason behind more positiveassessmentscouldhavetheiroriginsinsuccessful develop-mentoftheproduct,unsuccessfulinitialimplementationprojects, orboth.Itisunlikelythatinitialphysicianchangeresistancewould havebeentheonlyexplanationforthecriticalassessmentsof2010. Importantly, mostof the users are within one hospital district where healthcare centres andhospitals usethesame EHR sys-temandthebenefitsof sharingdatacouldexplainatleastpart oftheincreaseinusersatisfaction.ForbrandMthesituationis morecomplex.Oneofthereasonscouldbethatintegrationwith nationalhealthandinformationsystemserviceshasrequiredmajor revisionsoftheEHRsystems’functionalities,andthis mayhave disrupted physicianworkflows.In particular, thee-prescription functionality hasevokedmore criticismthan withotherbrands (datanotshown).AtleastfourchangesintheITenvironmentcould explainthelackofimprovementinusers’opinionsabouttherest oftheEHRbrands:Firstly,thenationaleHealthservices(Kanta ser-vices)haverequiredmajorrevisionstoalloftheEHRsystems,and thesecouldhavecausedahaltinallotherdevelopments.Secondly, theimplementationoftheserequirementsmayhavenotimproved theusabilityof thesesystems.Thirdly,theexpectedbenefitsof theKantaservices(standardizationofdataandsharing informa-tion)hadnotbeenrealizedasyetatthetimeofoursurvey;only e-prescriptionhad beenwidelyimplemented.Fourthly,otherIT solutionsthattherespondentsuseintheirdailyliveshave devel-opedduringthepastfouryears;theEHRsystemsseemtolagbehind inthisdevelopment.

Physiciansshouldbeabletostartgettingtheworkdoneinthe wayitissupposedtobedonewithouterrors.Therefore,the com-ponentsofuserinterfacesthatprovideinformationaboutcontrols fortheusershouldbeimmediatelyunderstandable. Theresults showthatlessthanhalfofallrespondentsagreedwiththe state-mentsaboutintuitivenessofEHRuserinterfaces.Theseﬁndings reﬂectexperiencedlearnabilityandsuccessofuserinterfacedesign fromthephysicians’viewpoint.Albeittheend-usersweremore experiencedinusingEHRsystems,theyexperiencedanincreasein thetimerequiredtolearntousethesystems.Ontheotherhand, clinicalworkpracticestendtobecomemoredependentondigital processesovertheyears.Atthesametimelessexternalguiding informationisavailableoutsidetheinformationsystems.

Itisalsoeasytoseethelinkbetweenintuitivenessanderror rateorsafetyofITuse.Thephysicianhasabetterchancetoavoid technology-inducederrorsandmistakeswhenusingtheEHR sys-tem,ifthesystemincludesfunctionalitytopreventerrors,ifthe systemkeepstheuserclearlyinformedaboutwhatitisdoing,and iftheusercaneasilygethelpwhenneeded.Inparticular,the

(12)

ﬁnd-ingthatonethird ofallrespondents,andevenahigherportion (42%)ofrespondentsfrompublichospitals,hadexperiencedthat afaultysystemfunctionhascausedorhasnearlycausedaserious adverseeventforthepatientindicatesthattheuseofthecurrent EHRsystemsmayposeaseriousrisktopatientsafety.

EHRsystemsshouldbeefﬁcienttouse,sothatahighlevelof productivityinahecticandcriticalenvironmentispossible.The surveyresultsindicatethatwiththecurrentEHRsystems physi-ciansarenotabletoconducttheirworkinanefﬁcientway.Thisis particularlyshownbystatementsabouttheavailabilityofkey func-tionalitiesandthestatementaboutperformanceofroutinetasks. Withregardstophysicians’responsestothesestatements,twoout ofthreephysicianssharethisconcern:thesystemslackthose prop-ertiesthatareneededtoperformkey clinicalcaringtaskswith patientsandthesystemsforcethephysicianstoperformadditional tasksoradaptnewinappropriateworkprocesses.

Fromtheusabilityviewpointitcanbearguedthatefficiency ofworkisalsohinderedbecauseofpoorITsupportfor informa-tionexchange,communicationandcollaboration.EHRandrelated ITsystems shouldserve a singlephysician but alsotheirwork withnumerousotherpartiessinceclinicalprocessesare charac-terizedwith a highdegree of communicationand cooperation. Theresultspointoutparticularlythefollowingareasof improve-ment:availabilityofinformationaboutmedicationsprescribedat anotherorganisation,supportforcross-organizational collabora-tionbetween physicians,monitoring of ordersgiven tonurses, and support for physician-patient collaboration.Less than 20% of all respondents agreedwith these four positivelyformatted statements.It appears that after its first years of deployment, thenationale-prescriptiondatabasehasnotas yetameliorated theavailability of medicationinformation across organisations. Thestudy findings provide thebaseline status for the national information exchange in Finland. It will be interesting to see howphysicianexperienceswillevolveinthepost-implementation situationof2017.Basedonresults, ITsystems’supportfor col-laborationandinformationexchangebetweenthephysiciansand patientshadnotimproved;apparentlyfewsolutionshaveemerged tosupportpatient-centred care and patientabilities to partici-pateintheirowncare.Therefore,basedontheresultsofanother nationalsurvey,theusageratesforthepatientportal functionali-tieswerestillverymodest[71],andinformationexchangebetween patientsandphysicianswasonlyavailableinafeworganisations

[42].

Satisfactionwithtechnicalfeatureshasnotimprovedinfour years. We regardthe availabilityof theEHR systems asone of thecentralfeatureswhenthephysiciansassesstheuseofthese systems.However,theproportionofphysiciansagreeingwiththe statementconcerningthedisappearanceofdocumenteddatafrom thesystem(statement22)hadslightlydiminishedinallworking environments.Aswearenotawareofmajortechnicalfailuresin datastorageintheEHRsystemsincludedinthisstudy,we inter-pret“disappearingdata”asanindicatorofthemostsevereusability problems.Theseincludeforexample(a)theuserdoesnotknow wheretofindtheinformationneeded,(b)theuserhasdocumented itinincorrectfieldsormodulesof theEHRsystemor,(c)most importantly,atthefinalstagesofdocumentationprocess,theuser doesnotnoticethathe/sheneedstopressanotherbuttoninorder toactuallysavetheinformation.Userstendtofindwaystomove aroundtheusabilityproblemsovertime[72,73]sowellthatthe developersandtrainersmaynotseethemasusabilityproblems atalland blametheend-usersfornot havingattendedenough trainingsessions.

Theresultsalsoshowsomedifferencesbetweenhealthcare sec-tors.EHRsystemsusedintheprivatesectorappearedmorestable andrespondedmorequicklytoinputsaswellasprovidingbetter supportforphysicians’routinetasks.Thepatientsintheprivate

sectorarelessseverelyillandseemtohavefewercomorbidities; accordinglythelackofdashboardsorpatientoverviewsislesslikely tointerferewithroutinetasks.Theresponsesfromprimary health-careindicatednegativechangesintheseaspectsascomparedtothe year2010.Oneofthereasonsbehindthiscouldbetheincreased documentationneedsfornationalreporting(otheraugmented doc-umentationneedsaremorelocal).Theresultsalsoindicatethat physiciansworkinginthepublicsector(hospitalsandhealthcare centres)wererathersatisﬁedwiththeirITsystemabilitiesto sup-portpreventionoferrorsrelatedtomedication.Healthcarecentre EHRsystembrandshadimplementedmedicationinteractionalerts alreadyby2010whereasthisfunctionalitywasintroducedtothe largestbrandusedinhospitalsonlyafterourprevioussurvey,and wasstilllackingintheprivatesector.

Theanalysisof‘disagree’responsesshowhighlysimilarpatterns withtheanalysisof‘agree’responsesandtherebysupportedthe conclusionsmadebasedonTable4.However,thecomparisonof theﬁndingspointoutsomedifferences(whichareallshownin Appendix2inSupplementarymaterial).Probablythemostnotable differencesarerelatedtothestatementaboutsysteminforming theuseraboutwhatitiscurrentlyperforming(statement7),the statementaboutthearrangementofﬁeldsandfunctionson com-puterscreen(statement6)andthestatementaboutinformation systemsupportforphysician-patientcollaboration(statement19). Inpublichospitalsandinprivatesectortheportionofphysicians disagreeingwiththestatementno.7hasgrown(inhospitalsfrom 50%to54%andinprivatesectorfrom30%to36%)eventhoughthe analysisof‘agree’responsesshowthatthephysiciansinboththese sectorshavegivenmorepositiveestimatesthaninyear2010.On thestatementno.6theanalysisof‘agree’responsessuggestthat theportionofallphysiciansagreeingwiththestatementhasgrown (from39%to42%).Interestingly,theportionofdisagreeinghasalso grownfrom42%to45%.Basedonourexperience,anadditional analysiscanbeparticularlyusefulregardingthosestatementsand resultswhichshowonlylittleornochangeatallbetweenresponses fromdifferentyears.

Oneofthecentralﬁndingsofthisstudyconcurswiththe ﬁnd-ings fromearlier usability studies [60,72]: Time does not heal usabilityproblemseventhoughtimeallowsuserstolearn strate-giesfor overcomingsomeoftheproblems.Kjeldskovetal.[72]

conductedlaboratory-basedusabilitytestingwithanaimto com-paretheusabilityofthesystemasexperiencedbynoviceandexpert users.Theexpertuserswerenotmoreefﬁcientoncomplextasks anda remarkablenumber ofseriousandcriticalproblems with thesystemstillremainedafteroneyearofextensiveuse.Janols

[60]reportssimilarﬁndings:Evenaftertwoyearsofdeployment, thesystemwasnotconsideredtobeassupportiveastheold sys-tem.Inaddition,cliniciansoftenfoundwaystoovercomeproblems withsystemusage;however,thesework-aroundstypically gener-atednewproblems.Inconclusion:poordesignremainspooreven thoughtimeallowspeopletolearnstrategiestoovercominga sys-tem’sspeciﬁcpeculiarities.

Earlierstudieshavealsoshownthatdifferentusergroups(e.g. physiciansand nurses)havedifferentjobrolesand responsibil-ities leadingto differentneeds and expectationsof theclinical systems [70]. Similarly, physicians working in different sectors havedifferentneedsandrequirementsfortheirsystems. Special-izedinformationsystemswithdefinedfunctionalitieshavebeen reported to receive more favourable assessments than clinical informationsystemsingeneral[64].Thefindingcanbeattributed totheimprovedcustomizationofthespecializedsystemsfor spe-cificworkingenvironments[64].Itisnoteworthythatoursurvey didnot cover IT systems used typically in intensivecare units oroperatingtheatres,and,accordingly,moreinternational stud-iesareneededtoreassessthecurrentsituationintheseworking contexts.

(13)

5.2. Relevanceoftheresearch

Fromtheacademicliteraturewefoundonlyafew examples ofusabilitysurveyswhichhavebeenusedtoresearchthe usabil-ityofEHRsystemsinuseovertimeinrealsettings.Approaches to researchingadoption and usability of healthcare IT systems varybetweencountries[56],however,tothebestofour knowl-edgethestudiesconductedinFinlandin2010and2014arethe onlycross-sectionalquestionnairestudiesfocusedonusabilityand aimedtomonitorandfollow-upthedevelopmentofEHRsystems atanationallevel.Reliableacademicsurveysareneededtostudy theusabilityanduserexperiencesofcurrentlyusedEHRsystems andtomonitortheirdevelopmentatregionalandnationallevels. Resultsofimplementationanddeploymentstudiessuggestthat EHRfunctionalityandusabilityimpactcliniciansatisfaction, efﬁ-ciency,andclinicaluseoftheEHR.Therefore,oneofouraimswas toinformdecisionmakers,healthcareorganisationsandpoliticians abouttheusabilityofcurrentlyusedEHRsystemsand improve-mentsthathaveoccurredduringthepastfewyears.Assuggested byBundschuhetal.[64],resultsfromnationalusabilitystudies canbeusedasreferencedatafor evaluationandbenchmarking ofuser-orientedsoftwareengineeringforclinicalIT,whichis rele-vantforthedevelopmentandmarketingofthesesystems,aswell asforclinicalpracticeandcarequality.Furthermore,Carayonetal.

[66]havestatedthat“itisimportantforhealthcareorganisationsto continuetheireffortstooptimizethedesignanduseofEHRafterthe technologyisimplemented,sincethecharacteristicsofEHR technol-ogy,particularlyusabilityandusefulness,haveasigniﬁcantimpacton acceptanceanduseofthetechnology”.Theyalsosuggestthatmore researchusingalong-termdesignisneededtofurtherunderstand howEHR-relatedpredictors oftechnologyacceptance,including usability,maychangeovertime.

5.3. Historyofourusability-focusedquestionnaire

Designingausabilityquestionnairestudyforphysiciansis chal-lenging.Itrequiresin-depthknowledgeaboutusabilityresearch issuesandaboutdomainspeciﬁccharacteristicsofphysicians’ con-textofwork.Comparedtostandardizedusabilityquestionnaires (suchasSUMI [74],SUS[75] orQUIS [62]), thestrengthof our questionnaireisthatitisfocusedandspecializedaround physi-cians’workandtheiruseofEHRsystems.Ashasbeenstatedin ourearlierarticle[4],thereasonsfornotusingthesestandardized usabilityquestionnaireswerethattheyfocusonasinglesystemof software,evaluatetheusabilityofauserinterface,andarecontext anddomainindependent.However,weutilisedthose question-naireswhentheoriginalquestionnairewasdesignedandalsowhen weupdatedtheformbeforedatagatheringin2014.

Questionnaireitemsneedtobecarefullyformulatedsothatthey arecorrectandappropriatefromtherespondent’spointofview.Itis importantforthequestionnairetooltoaddressissuesitissupposed toaddressandtobespecializedforthecontextofitsintendeduse. Forexample,Tanetal.[70]foundthatintheirstudyphysicianshad adifferentconceptofusersatisfactionascomparedtonursesand thedatacollectiontoolwithitsconceptswasmoreeffectivein mea-suringnursingconstructionsthanaphysician’smindset.Therefore, thequestionnaireneedstobecarefullypilot-testedwith poten-tialrespondents.Afterourstudy,wefoundthatsomespecialist groupssuchasradiologistsandlaboratoryphysiciansdidnotﬁnd allstatementsrelevanttotheireverydaywork.OnlyEHRsystem brandsthatwereusedas“foundationsystems”wereavailablefor selection,therefore,physiciansworkinginintensivecareunitsor operatingtheatrescouldnotreplybasedonthespecialityITsystem brandtheyused.MostphysiciansuseseveralotherITsystemsthat storepatientinformation(suchasradiologyinformationsystems, laboratoryinformationsystems)daily.Therespondentswerenot

askedtoassessthedevelopmentofEHRsystemswithregardsto thesituationfouryearsearlier.

The design of our national usability-focused questionnaire startedin2009andhashadseveraliterationssince.Theoriginal version ofit wasdesigned bya multidisciplinary group,whose sevenmemberswereexpertsin theareasofusability research, medicalinformatics,sociologyoftechnology,medicineand medi-calpractices,andoccupationalhealthresearch[4].Thetheoretical background of the questionnaire development workoriginated fromareviewoftheusabilityliterature,particularlyfromwidely knowndefinitionsofusabilityand ananalysisof contextofuse characteristics[4].Beforethefirstdatagatheringintheyear2010 thequestionnaire had two pilottest phases [4]. Afterthat, the questionnairewasmodifiedbasedonourexperiencesfromdata gathering,analysisandacademicdiscussions.Asdescribedin ear-lier,somestatementsweremodified andadded forexample to reflectdevelopmentsatthenationallevel.Thiskindofiteration andupdateneedstotakeplaceinthefutureaswell.Furthermore, itisimportanttopilottestthequestionnaireeverytimebeforeit isusedwithsufficientnumberofpotentialrespondents.Although themodifiedquestionnaireusedin2014wastestedbeforehand, itis likelythatphysiciansfromdifferentworkingenvironments mayunderstandthestatementsdifferently.Ontheotherhand,user experienceis context-relatedand respondentsassessthe state-mentsbasedtheirownexperiences.

Eventhoughourquestionnairehassuchhistory,onecould ques-tionthevalidityandthereliabilityofthemethodandtheresults.To ourbestknowledge,oursurveywithFinnishphysiciansisthefirst nationaleHealthobservatoryquestionnairefocusedonusability andusedtomonitorthelong-termdevelopment.Therefore,wefind themethodandtheresultsarenovel,valuableandofhigh impor-tance.In addition,theresultshavepracticalrelevanceand they havebeenexploitedwithinFinlandaswellasinternationally.Our planistokeepupmonitoringthelevelofusabilityofclinicalIT sys-temsaswellastocontinuethequestionnairedevelopmentwork.At thesametime,wecallforcollaborationwithotherresearchersand communitiesaroundthisworktodevelopvalidatedandreliable academicmonitoringsurveysforhealthcareITfield.

5.4. Strengthsandweaknessesofthestudy

Theuseofquestionnairesisavaluablemethodforgathering subjectiveexperiences–directclinicalresponse–withITusage. Compared with other more qualitative usability methods (e.g. usabilitytesting,expertevaluation,observationsorinterviews)use ofquestionnairesis asuitable techniquefor gathering informa-tionfromalargetargetgroupofend-users.Aweb-basedformat makesiteasytoreachahighnumberofdesiredrespondentsand inquireaboutnumerousIT-userelatedthemes.Forthesereasons, aweb-basedquestionnairewasfoundsuitabletobeusedinour study,whichaimedatresearchingandmonitoringtheoveralllevel ofachievedusabilityofEHRsystemsandimpactsofdevelopment activitiesatanationallevelwithinrecentyears.

The study suffered fromgeneric limitations typical of stud-iesconductedwiththeinternetsurveymethod[76]:TheFinnish MedicalAssociation registerdidnothaveemailaddressesof all physicians. Theinvitationemailsmaynotnecessarily reachthe respondentsbecauseofﬁrewallsettingsorothertechnicalissues. Therefore, we can’t be sure how many physicians actually got theinvitation.Thosethatwerereachedmightnothaveanswered online surveys.Surveyswith closed-endedquestions mayhave alowervalidityratethanotherquestiontypes.Data errorsdue toquestionnon-responsesmayexist.Thenumberofrespondents whochoosetorespondtoasurveyquestionmaybedifferentfrom thosewhochosenottorespond,thuscreatingbias.Surveyquestion answeroptionscouldleadtouncleardatabecausecertainanswer

(14)

optionsmaybeinterpreteddifferentlybyrespondents.For exam-ple,theansweroption“somewhatagree”mayrepresentdifferent thingstodifferentsubjects,and have itsown meaningtoeach individualrespondent.‘Yes’or‘no’answeroptionscanalsobe prob-lematic.Respondentsmayanswer“no”iftheoption“onlyonce” isnotavailable.In addition,issuesof anonymousquestionnaire methodarerelevanttopointout.Inourstudy,weﬁndassuranceof respondentanonymityakeyissue.Wethinkthishasinﬂuencedthe responserateinapositiveway.Multipleresponsesfromasingle respondentwerenotpossiblesincepersonalizedlinksweresendto respondents.Whenusinganonymousquestionnaireitis,however, notpossibletoevaluatethecausality.

Whenestimatedaschangesinpercentagesofphysicians agree-ingordisagreeingwithastatement,ourresultsdonotshownotable improvementsregardingtheusabilityofEHRsystems.However, asthenumber of respondentswas high,the statisticalmethod used(ChiSquaretest)mayoverestimatethedifferencesbetween thefindingsof2010and2014.Whenconsideringtherelevance ofthedifferencesbetweenthefindings,itisrecommendedtopay attentiontoboththep-valuesandthesignificanceofthechange expressedaspercentagevalues.

Itisunlikelythatthechangeinrespondentdemographicswould explainthelackofimprovementsintheopinionsofphysicians:The proportionofwomenhadincreasedamongbothrespondentsand physiciansinFinlandbetween2010and2014.Womenhadgiven higheropinionscoresthanmeninbothsurveys[45,53]. Respon-dentswereyoungerin2014thanin2010,buttheopinionsscores oftheyoungestagegrouphadslightlyrisenbetween2010and2014 (datanotshown).Theproportionsofworkingsectorsanddifferent EHRsystemshadremainedsimilar.Since therespondentswere notidentiﬁed,someoftherespondents,atleasttheonesthathad ﬁnishedtheirstudiesorretiredafter2010,werenotthesamein 2010and2014.However,itisunlikelythatphysicianswhofeelless positivelyabouttheirEHRsystemswouldhaverespondedtothis survey,butnotthepreviousone.Ofthelargestusergroups,only onehospitaldistricthadchangeditsEHRsystembrandbetween 2010and2014,manyoftherespondentsinthissurveyarelikely tohavefouryearsmoreexperienceintheuseofthesystemsthan inthepreviousone.

ThefindingsonthecurrentstateofusabilityofEHRandrelated clinicalITsystemshave novelvalue,sincetheappliedresearch approachwasnottypicalofhealthinformationstudies.The liter-aturereviewofrelatedstudiessuggeststhatausability-focused nationalquestionnairewithnearly4000respondentscanbe con-sideredexceptional ascomparedtoothersimilarstudiesinthe field.However,ourscopingreviewhadsomelimitationsaswell. Thereviewwasfocusedontheacademicliteraturepublishedin PubMedbetween2010and2015.Weareawarethatsome stud-iesonthetopic“questionnairestudiesonEHRusability”havebeen reportedbefore,forinstancetheacademicstudiesbyChristensen etal.[77] andEdwardsetal.[78] aswellassome comprehen-sivereviews on EHR systems in the USA market (e.g. surveys by AmericanEHR [79–81]). Thesenon-academic surveysabout usersatisfactionwithEHRsintheUSAhavebeenconductedfor someyears.Conclusionsbasedonthesestudies,however,seem tobesomewhatcontradictoryandmanifold:HIMMS13reported EHRsatisfactiondiminishing[82],whereasrecentlypublished sur-veyreportbyAmericanEHRPartnersindicatea growingoverall satisfactionamongphysicianswiththeirEHRsystems[81]. Inter-estingly, the report also points out how usability ratings vary betweenspecifictasks(e.g.refillingaprescriptionwasratedaseasy whereasimportationofa patient’smedicationlistdifficult) and betweenspecialities(primarycarephysiciansgivingmorepositive evaluationsthanotherspecialists)[81].Whatismore,theuseof othersearchterms(e.g.cross-sectionalorusersatisfaction)could haveresultedinmorerelatedarticles.However,forthekeyterms

weselected“usability”sinceourgeneralﬁndingisthattheterms satisfaction,usability,userexperience,usefulnessand meaning-fuluseareusedinhealth informaticsliteraturewithsomewhat contradictorymeanings.

5.5. Furtherresearch

Thisarticleistheﬁrstinternationalpublicationwhichreports resultsfromournationalsurveydatagatheredin2014.Resultsfrom the2010surveywerereportedinseveralarticles[4,9,47–49]. Like-wise,infuturestudieswewillanalysethequestionnairedatafrom otherperspectives,forinstancehealthinformation exchange.In addition,ouraimistoresearchforinterveningvariablestoﬁndfor exampleifmoreexperiencedITusersmaybecomemorecriticalof theEHRsystemstheyuseforwork.

Ouraimistocontinuethemonitoringofdevelopmentof health-careITsystemsinFinlandasseriesofcross-sectionalstudieson physicians’experiencesofEHRsystemuseandusability.Thisalso meansthatthesurveyquestionnaireneedstobeupdatedtoreﬂect thechangesintheﬁeld (e.g.regionalandnationalregulations). UpdatestoeHealthstrategyandpolicygoals(e.g.patient empow-ermentviapatientportalfunctionalities)callforupdatesofthe surveyinstrument.Thenextnationaldatagatheringwith physi-ciansisplannedtotakeplacein2017.Alotofexpectationsare associatedwiththeforthcomingmoregeneralHIEanduser experi-encesmightshowdifferentresultsafterKanta(thenationalarchive ofhealthinformationinFinland)installationsareinfullservice. Thestudyresultscanbeusedtoinformhealthcareproviders, deci-sionmakersandpoliticiansaboutthecurrentstateofEHRusability anddifferencesbetweenbrandsaswellasimprovementsofEHR usabilityatanationallevel.

Our current survey did not cover other health professional groupssuchasnurses.AstheEHRisamultidisciplinaryplatform thatisexpectedtosupportteamwork,otherprofessionalsmightbe includedinfuturequestionnaires(e.g.physiotherapistsandvarious specialityprofessionalsuseacommonEHR). Medicalsecretaries performimportanttasksininformationlogisticsandscheduling. However,thesurveyquestionnaireneedstobemodiﬁedforuse withtheotherprofessions,sincetheworktasksandresponsibilities ofotherprofessionsdifferfromthoseofphysicians.

Ournationalquestionnairestudyhasgainedinterestinseveral countries,includingCanadaandNordiccountries.Thereis ongo-ingresearchactivity betweenFinlandand Canadatodevelop a generalizedusability-focusedquestionnaireforvariousgroupsof healthcareprofessionals, includingnurses,based ontheFinnish nationalquestionnaireforphysicians.Itwouldbeinterestingtobe abletocomparetheresultsfromnationalusabilitystudiesacross morecountriesandmonitorthedevelopmentofhealthcareIT sys-temsataninternationallevel.Fromaconceptualperspective,our studysuggeststhatmoreresearchisneededtounderstandthe rela-tionshipbetweenconceptsofusability,technology-inducederror andpatientsafety.

6. Conclusion

Thehealthcarefieldis continuouslychanging.Political, orga-nizational and technological changes as well as increasing digitalizationhaveeffectsonhealthcareITsystemdevelopment andimplementation.ResearchonexperiencedusabilitywithEHR systemsisessentialtofindouthowthesechangesappearin clin-icians’ daily work in clinical environments. Based on our best knowledge,thereported cross-sectionalsurvey withphysicians inFinlandisthefirstnationaleHealthobservatoryquestionnaire focusedonusabilityandusedtomonitorlong-termdevelopment inthisarea.