Determinants of evidence use in public health policy making: Results from a study across six EU countries

Tilburg University

Determinants of evidence use in public health policy making

van de Goor, L.A.M.; Hämäläinen, R.M.; Syed, A.; Juel Lau, C.; Sandu, P.; Spitters, H.;

Eklund Karlsson, L.; Dulf, D.; Valente, A.; Castellani, T.; Aro, A.R.

Published in:

Health Policy

DOI:

10.1016/j.healthpol.2017.01.003

Publication date:

2017

Document Version

Peer reviewed version

Link to publication in Tilburg University Research Portal

Citation for published version (APA):

van de Goor, L. A. M., Hämäläinen, R. M., Syed, A., Juel Lau, C., Sandu, P., Spitters, H., Eklund Karlsson, L.,

Dulf, D., Valente, A., Castellani, T., & Aro, A. R. (2017). Determinants of evidence use in public health policy

making: Results from a study across six EU countries. Health Policy, 121(3), 273–281.

https://doi.org/10.1016/j.healthpol.2017.01.003

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal

Take down policy

Contents lists available atScienceDirect

Health

Policy

j o u r n a l h o m e p a g e :w w w . e l s e v i e r . c o m / l o c a t e / h e a l t h p o l

Determinants

of

evidence

use

in

public

health

policy

making:

Results

from

a

study

across

six

EU

countries

Ien

van

de

Goor

_,

_Riitta-Maija

_{Hämäläinen}

_,

_Ahmed

_Syed

_,

Cathrine

Juel

Lau

_,

_Petru

_Sandu

_,

_Hilde

_Spitters

_,

_Leena

_Eklund

_Karlsson

_,

Diana

Dulf

_,

_Adriana

_Valente

_,

_Tommaso

_Castellani

_,

_Arja

_R.

_Aro

_,

_on

_behalf

_of

the

REPOPA

consortium

a_{Tranzo,TilburgSchoolofSocialandBehavioralSciences,TilburgUniversity,Tilburg,TheNetherlands} b_{Welfare:EqualityandInclusion,NationalInstituteforHealthandWelfare,Helsinki,Finland} c_{SpecialisedServices,NHSEngland,London,UK}

d_{PreventionandHealthPromotion,ResearchCentreforPreventionandHealth,CapitalRegionofDenmark,Glostrup,Denmark} e_{CenterforHealthPolicyandPublicHealth,DepartmentofPublicHealth,Babes-BolyaiUniversity,Cluj-Napoca,Romania} f_{UnitforHealthPromotion,InstituteofPublicHealth,UniversityofSouthernDenmark,Esbjerg,Denmark}

g_{InstituteofResearchesonPopulationandSocialPolicies,NationalResearchCouncil,Rome,Italy}

a

r

t

i

c

l

e

i

n

f

o

Articlehistory: Received19July2016

Receivedinrevisedform9January2017 Accepted10January2017

Keywords:

Policydevelopmentprocess Evidence-informedpolicy Publichealthpolicy Barriersandfacilitators Semi-structuredinterviews Individualandsocialfactors Policycontext

Structuralcollaborationbetween researchersandpolicymakers

a

b

s

t

r

a

c

t

Theknowledge-practicegapinpublichealthiswidelyknown.Theimportanceofusing differenttypesofevidenceforthedevelopmentofeffectivehealthpromotionhasalso beenemphasized.

Nevertheless,inpractice,interventiondecisionsareoftenbasedonperceivedshort-term opportunities,lackingthemosteffectiveapproaches,thuslimitingtheimpactofhealth promotionstrategies.Thisarticlefocusesonfacilitatorsandbarriersintheuseofevidence indevelopinghealthenhancingphysicalactivitypolicies.

Datawascollectedin2012byinterviewing86keystakeholdersfromsixEUcountries (FI,DK,UK,NL,IT,RO)usingacommontopicguide.Contentanalysisandconceptmapping wasusedtoconstructamapoffacilitatorsandbarriers.

Barriersandfacilitatorsexperiencedbymoststakeholdersandpolicycontextineach countryareanalysed.Alackoflocallyusefulandconcreteevidence,evidenceoncosts,and alackofjointunderstandingwerespecifichindrances.Alsousers’characteristicsandthe rolemediaplaywereidentifiedasfactorsofinfluence.

Attentionforindividualandsocialfactorswithinthepolicycontextmightprovidethekey toenhancemoresustainableevidenceuse.Developingandevaluatingtailoredapproaches impactingonnetworking,personalrelationships,collaborationandevidencecoproduction isrecommended.

©2017TheAuthor(s).PublishedbyElsevierIrelandLtd.Thisisanopenaccessarticle undertheCCBY-NC-NDlicense(http://creativecommons.org/licenses/by-nc-nd/4.0/).

∗ Correspondingauthorat:TilburgUniversity,TilburgSchoolofSocialandBehavioralSciences,DepartmentTranzo,P.O.Box90153,5000LETilburg, TheNetherlands.

E-mailaddresses: L.vandegoor@uvt.nl (I. vande Goor), Riitta-maija.hamalainen@thl.fi(R.-M. Hämäläinen), Ahmed.sy3d@gmail.com (A. Syed),

Cathrine.juel.lau@regionh.dk(C.JuelLau),Petru.sandu@publichealth.ro(P.Sandu),H.p.e.m.spitters@uvt.nl(H.Spitters),Leklund@health.sdu.dk(L.Eklund Karlsson),Diana.dulf@publichealth.ro(D.Dulf),Adriana.valente@cnr.it(A.Valente),T.castellani@irpps.it(T.Castellani),Araro@health.sdu.dk(A.R.Aro).

1 _{TheauthorswouldliketothankallmembersoftheREPOPAconsortium(http://www.repopa.eu/content/consortium),especiallythosewhohave}

contributedtodoinginterviewsandprocessingdata.

http://dx.doi.org/10.1016/j.healthpol.2017.01.003

2 I.vandeGooretal./HealthPolicyxxx(2017)xxx–xxx

1. Introduction

Itiswellacknowledgedthattheuseofrobustevidence toinformpublichealthpolicyislikelytoensurethegreatest andmostequitablepopulationhealthgains[1,2]. Increas-ingfocusonevidence-informedpublichealthinwhichnext todifferentsourcesofresearchevidencecontextual fac-tors alsoplay a substantialrole in thedecision making process[3]andhasnumerousdirectandindirectbenefits. Amongtheseareaccesstomoreandhigherquality infor-mationonwhatworks,ahigherlikelihoodofsuccessful programsandpoliciesbeingimplemented,greater work-forceproductivity,andmoreefficientuseofresources[4]. Nevertheless,inpractice,interventiondecisionsareoften basedonperceivedshort-termopportunities,lacking sys-tematicplanningandreviewofthebestevidenceregarding effectiveapproaches[4]thusresultinginslowuptakeof researchevidenceinpractice.Ithasbeenestimatedthatit takesanaverageof17yearsfor14%ofresearchto trans-lateintopractice[5].Morerecentresultsshowthatevenin clinicalpracticewhichissupposedtobemoreevidence ori-ented,theuptakeofevidencehasnotchangedsubstantially sincethen,indicatingthatthegapbetweenevidenceand practicehasnotdiminishedsubstantially[6].Generallyin publichealthpolicymakingtheuseofresearchevidenceis lessthananticipatedwhenconsideringtheextensive avail-abilityofresearchevidence.Whileresearchevidenceon effectivehealthenhancingphysicalactivity(HEPA)policies andinterventionsisavailable,itappearsnottobe opti-mallyusedtoinformhealth relatedpolicydevelopment

[7–11].Amultitudeoffactorsthatimpede(orfacilitate) evidence-informedpolicymakingexistsresultinginbelow optimalhealth outcomeswhen implemented.Literature showsthatspecificcontextsandtraditions,political prior-ities,individualbeliefsandpreferences,socialvalues,and availableresourcesallplay amajorrole [12,13].Among thesefactorsthreemaincategoriescanbedistinguished. Firstly,easy accessto relevantand useful research[14]

alsoentailingtimelyaccesstogoodqualityandrelevant researchevidence[15].Secondly,frequentopportunities to interactwith researchers [7] including collaboration and networking with policymakers[15]. Thirdly, work-inginresearchreceptiveorganizations[16,17]facilitates evidence-informedpolicymaking.

Theupdatedsystematicreviewonbarriersand facili-tatorsofevidenceuseinpolicymakingbyOliveretal.in 2014[15]concludedthatoverthepast10yearsthesehave basicallyremainedthesameandthatitisdifficulttofind newperspectives.Somerecent researchhoweverpoints inthedirectionofpersonalrelationshipsandpolicy mak-ers’networksaswellasdifferencesincontextualfactors tobeofutmostimportanceinrelationtoimprovingthe uptakeofevidenceinpolicy.Policymakersappeartohave aneedforandalsouseamuchwiderrangeof informa-tionsourcesthanresearchevidenceandtheyaccessmost ofthesethroughpersonalcontacts[18].Inaddition,policy makers’relationshipswithinnetworksandcharacteristics oftheorganizationalcontextsuchasthemuchneglected roleofmanagersinpolicydecisionmakingappearstobe ofgreat influencein evidenceuse[18–22]. Alsostudies withempiricaldataoninteractionsbetweenstakeholders

inpolicymakingreportthattheuseofevidenceinthe pol-icyprocesswasdifficulttotraceorthattheprocessitself appearstoberatherclosed[23–27].

Furthermore the literature shows that policymakers with respect to use of evidence need to pay attention tolargerentities andmulti-dimensional factorssuchas communities,municipalities,resources,politicsandother factorsascomparedtoforinstanceclinicians[21,28].This makesuseofevidencebypolicymakersmuchmore com-plicatedandmaybethereasonthattheextentofevidence usebythemislowerincomparisontoclinicianswhofocus ononespecificissueonly,i.e.thephysicalconditionofthe individualpatient.

Despiteseveraldecadesofworkonevidenceinformed policy,thegoalstoimproveevidenceuptakeandpromote greateruseofevidencewithinpolicymakingarestill elu-sive.Recentliteraturewarrantsmoreresearchonevidence usebypolicymakersthroughinteractionandpersonal con-tacts, relationshipswithinnetworksand thecomplexity andvariedcontextofpolicymaking.

In 2011 the European Commission (EC) funded the ResearchintoPolicytoenhancePhysicalActivity(REPOPA) project.Oneofitsaimswastostudytheextenttowhich EUmemberstatesuseresearchevidenceandotherkinds ofevidenceinHEPApoliciesandwhatpromotesorhinders theuptakeofresearchevidenceinthepolicy-making pro-cessofHEPApolicies[25].Thegeneralaimoftheproject wastofacilitate theintegrationof researchevidenceto stimulatemoreevidence-informedphysicalactivity poli-cies.Theaim,design, methodsandpreliminarybaseline resultsof theoverallREPOPA—(www.repopa.eu)project aredescribedbyAroetal.[25].Preliminaryresultsshow thatsupportiveinstitutionalresources,accessto applica-ble context-relevantresearchevidence,mediaattention, goodpersonalrelationshipsandnetworks,jointlanguage and collaborationbetweenresearchers and policy mak-erswerefoundtofacilitatetheuseofresearchevidence. Barriersidentifiedwererelatedtonon-supportive institu-tionalmanagement,lackofeasyaccesstobestavailable evidence,limitedcontactsbetweenadministrative person-nel,expertsandresearchers[25].

Theaimofthisarticleistofurtherexplorebarriersand facilitators intheuseofresearchand otherevidencein developingHEPApoliciesfromsixEUcountriesusing semi-structuredinterviewsconductedwithkeystakeholdersas partoftheREPOPAproject.Morespecificallyitfocuseson aspectsthat(local,regionalornational)policymakersand otherstakeholdersin differentEuropeancontexts expe-rience asmost influencingin theuptake ofevidence in real-lifepolicymakingprocesses.

2. Methodsanddesign

wereundertakenusingastandardtopicguideacrossall sixparticipatingcountries.Theinterviewsfocusedon per-ceptionsandexperiencesonhowresearchandothertypes of evidence, comprising a broad variety of information sourceswereusedbypolicymakers[18,29]andwhat fac-torsfacilitatedorhinderedtheiruseinthepolicymaking process. The interviews wereconducted by researchers from the respective countries in theirnative language. Eachinterview tookabout1.5-hwastape-recorded and transcribed(tape-recordingwasonlydonewhen partic-ipants provided consent), otherwisehand-written notes weremade[26].

Bypurposefulsamplingatotalof86stakeholderswho were directly involved in the policy making process of the selected policies and who could inform about the useof researchorotherevidence inthis specificpolicy makingprocesswereidentifiedandincluded.The stake-holdersincluded(national,regionalorlocal)policymakers, researchers,publicsectorofficersorotherinfluential stake-holders.Allintervieweeswerecontactedbyemailorphone by the research team in the country with background informationontheprojectandconsentformsinthelocal language [26]. On the (rare) occasion of an identified stakeholderwasnotavailableforinterview,adeputywas contacted.Theinterviewswerespecificallydesignedto col-lectin-depthinformation onstakeholdersexperiencein termsoffacilitatorsandbarriersintheuptakeofevidence intheeverydayworldofpolicymaking.

Thedatacollectedwereanalysedbyeachcountryteam usingacommonguidelineforqualitativecontent analy-sis(analysiswerecarriedoutmanuallyorusingsoftware packages forqualitative researchsuchas NVivo, MaxQ-data).Codingwasdonebytworesearchersindependently fromeachotherinthecountryteams[26].Eachcountry producedareportpresentingitsfindingswhichwere inte-gratedintoasingle(internal)projectreportinEnglish.

Fromthecountryreports,listsoffacilitatorsand barri-ersidentifiedfortheuseofresearchevidenceweregrouped andcategorizedusinganonlineconceptmaptool[30].The conceptmaptoolhelpedfindassociationsandpulltogether similaritemsandseparatedifferingitemsmentionedinall theinterviews.Byusingtheconceptmaptoolthe quali-tativedatafrominterviewsformedapatternoffacilitators andbarriersfortheuseofresearchevidenceinHEPApolicy making.Sincethefocuswasnotonfindingdifferencesin viewsofdifferentstakeholdergroups,overallviewswere takentogetherinthecategorizationandnoweightingof datawascarriedout.

REPOPA developed an Ethics Road Map and Ethics GuidanceDocumenttocoordinatevaryingnationalethics clearanceproceduresinpartnercountries.Ethicsclearance wasdonein each countryaccordingtocountry-specific regulations and procedures (for details see Ref. [31]: Edwardset al.);however, irrespectiveofeach country’s requirements,informedconsentwasobtainedfromall par-ticipants.Ethical Committees for each countryincluded wereasfollows:EthicsCommitteeoftheRegionofSouth DenmarkandtheNationalDataprotectionAgency(DK); CentralCommitteeonResearchInvolvingHumanSubjects (NL);ResearchEthicalCommitteeoftheNationalHealth andWelfare Institute(FI);EthicsCommitteeofthe

Uni-versity of Babes-Bolyai(RO); NationalResearch Council Research Ethics and Bioethics AdvisoryCommittee(IT); HealthandSocialSciencesReviewBoardofUniversityof Ottawa(CA);EthicsapprovalbytheResearchCouncilsUK (UK).

The ethical clearance papers of all countries were approvedbytheECbeforestartoftheproject.TheEChad oversightoftheethicsoftheoverallproject.

3. Results

Fig. 1 presents an overview of the facilitators and barriers for the use of evidence in the policy develop-mentprocesshighlightedbytheinterviewees.Although facilitatorsandbarriersweretosomeextentinitself com-plementary,theresultspresentedhereshowthosefactors thatintervieweesrecognizedasbeingmainlyfacilitatingor mainlyhinderingintheirrealworldpolicymaking expe-rience. Factors found to facilitate or hinder the use of evidencecanbecategorizedintothreemaindomains: • Domain1-organizations,systemsandinfrastructure; • Domain2-accessandavailabilityofrelevantevidence; • Domain 3-networking and collaboration between

researchersandpolicymakers.

In additionfactors that appeared specificwithin the countryorpolicycontextaredescribed.

3.1. Domain1:organizations,systemsandinfrastructure Within the domain of organizations, systems and infrastructurethefollowingfacilitatingfactorsweremost frequentlymentioned. Supportof administration,which includedtheorganizations’structure,resources,systems, staffandtheirskills(Fig.1upperleft),positiveattitudes frommanagers,settingclearcutcriteriaforhowthe pol-icyprocessshouldevolve,closemonitoringoftheprocess andtrainingofpersonnelintheuseofevidenceinpolicy making.

“Myexperiencefromlocallevelisthatthemanagementis concernedwithahighlevelofprofessionalism...hence thefactthatwehavethe(professional)capacityaswell asamanagementconcernedwithevidencebasedpolicy making(Localpolicymaker,Denmark).”

Inadditionsomeintervieweesmentionedtheroleof media.Evidencefromtraditional mediasourcessuchas televisionandnewspaperarticles,butalsorelativelynew sourcessuchassocialmedia(blogs,internetandtwitter) werefelttohaveasubstantialimpactthroughframingthe problemanditspotentialsolutionsbyprovidingexposure fordecisionmakers.

“Yes,Ihavetheideathatit[priorities]alsocomesfrom thecitizens,sometimesbecauseofwhatismentionedin thenewspapersandsometimesitisalsowhattheminister ofstatehasexperiencedhim-orherself(National policy-maker,Netherlands)”

mak-4 I.vandeGooretal./HealthPolicyxxx(2017)xxx–xxx

Fig.1.OverviewoffacilitatorsandbarriersfortheuseofevidenceinHEPApolicymakingasmentionedbyinterviewees.

ing (Fig. 1 upper right), lack of resources (Fig.1 lower right), financial and time constraints, and the fact that researchevidencehardlyeverisdirectlyapplicablein pol-icywithinthetightpoliticaltimeframe.Inaddition,lackof staffcompetencesandfactorsrelatedto‘relevantresearch achievable’werementioned(Fig.1middleleft).A regu-larlyupdated,well-functioningresearchinfrastructurewas mentionedasabasicprecondition.

3.2. Domain2:accesstoandavailabilityofrelevant evidence

Intheseconddomainoftenmentionedhinderingfactors wereinapplicabilityofevidencebecauseofits complex-ity,extensiveness orbeingtoo theoretical(Fig.1upper right).Alsothelackofdirectrelevancewasmentioned fre-quentlybecauseoftenpolicymakersfacesituationswhere researchevidencedoesnotcompriseinformationon

(eco-nomic)impactandcostsonconcretepolicymeasures,or onthespecificlocalproblemorpolicycontext.

“Evaluating nationalpolicyin asystematic way leaves roomforimprovementsotospeak...OfcoursetheCourt ofAuditorsprioritizesthiscontinuously.Theyputthislack ofevaluativeinformationonnationalpolicyexpenditures ontheagendacontinuously...(Nationalpolicyadvisor, Netherlands).”

“If we should work evidence based we should not be doing anything as there is no evidence we can trans-late...directlytomunicipalcontext.(Localpolicymaker, Denmark).”

preferences[forguidingourwork]...Theefficacy indica-torsfromtheNationalProgramcannotbeevaluateddue tolackofdata.(Localpolicyadvisor,Romania).”

Someintervieweesstressedtheimportanceofthe avail-abilityoftoolsandmethodologiesinordertogettimely, concreteandpracticallyapplicableevidence,namely tai-loredresearchevidence.However,fundsespeciallyatlocal levelappearedlimited.

“Icannolongerpayforascientificadvice.Itisnotonly afundingproblem,butalsoanormativeproblem:thelaw doesnotallowmetoaskforadvice.(Localpolicymaker, Italy).”

3.3. Domain3:networkingandcollaborationbetween researchersandpolicymakers

Thethirddomainisaboutfactorsrelatedtonetworking andcollaborationbetweenresearchersandpolicymakers, whichwereoftenmentionedaspotentialfacilitatorsfor evidenceuse(Fig.1lowerleft).Morespecifically,creating crosssectorcooperation,seminarsandconferenceswhere researchersandpolicymakerscouldmeetinorderto facil-itatebuildingajointlanguageandreferenceframework.

“..thatonecanparticipateinahalf-orfull-dayconference andyes,listentoresultsandconclusionsbeingpresented whileIalsogettheopportunitytodiscussmyown expe-rience,that’showitbecomesaplatformyoucanindeed standupon(Localpolicymaker,Denmark).”

“Academicsdonotacceptnottobeprotagonists.Viceversa, policymakersbelievetheworldisonlywhattheysee.A neutralplacewouldweakenthesenseofsupremacyofthe twoparts.Theremustexistaneutralplaceinwhicheach onefeelslessself-referentialandacceptpeerdiscussion (Localpolicymaker,Italy).”

Similarly,directandfrequentface-to-faceinteractions between researchers and policy makers and working together as early as possible in policy-related research projects were mentioned as facilitators for improving mutualunderstanding.

“Iftherewouldbeclearandapplicableresearchstudies, thatwouldallowdecisionmakerstounderstandtheways usingtheresultscouldbebeneficial,thendecision mak-erswouldbeopen[tousethesestudies].Otherwise,ifthe studiesareencryptedinascientificlanguage;theywillonly remainonpaper(Nationalpolicymaker,Romania).” “Webasedourpolicyonfacts,notopinions.Weused inter-nationalresearchresults,butalsodatafromtheresearch projectsonthenationallevelandinformationproducedby researchinstitutionsinvariousresearchprojects.In addi-tion,webenefitedfromindividualresearchersandtheir knowledgeonspecificresearchresultsinFinlandor else-where(Nationalpolicymaker,Finland).”

Intervieweesalsomentionedpersonalliaisonsor ‘link-ingpins’betweenknowledgeinstitutesandpolicymaking organizationsasfacilitators.Workingcloselytogetherin variouswaysfacilitatesbalancingtheuptakeofevidence

stemmingfromresearchandothertypesofevidence,like ‘best practices’ and ‘common sense’or ‘what works by intuition’typeofknowledge.Thisalsorelatestohowthe politicalrelevanceof(research)evidenceisvalued(Fig.1

upperright).Inthecasethatpoliticalinterests(becauseof pursuitofvoters,politicalvalues)andscientificevidencedo notcomplementeachother,itmayleadtocompromisesto theextenttowhich(research)evidenceuptakeoccurs.

“Idon’tknowifIreallywanttosaythis,buttherewasn’t anyevidencetosaythatwecoulddoit.Itwasactually anaspiration andanabsolutebeliefbythebidteamin particularabout’Wecan’(Nationalpolicymaker,UK).” Thisseemsespeciallyrelevantbecausepersonal char-acteristics,interestsandvaluesofallstakeholders(policy makers,researchersandpoliticians)werementioned by someasimportantastowhetherindividualsactuallyshow ‘a willtocooperate’andare opentodiscusstheuseof evidenceinpolicymakingatall.

“Intheverybeginningitwasverydifficulttogetinvolved stakeholderstocometogethertoprepareajointpaperon nutritionandphysicalactivity.Thefearwasthatthe sig-nificance,theamountofresourcesandentityofphysical activityasitsownissuewouldbedecreasedifnutrition isinthesamepolicye.g.atthesamesandboxsharingthe sandcakeswithphysicalactivity’(Nationalpolicymaker, Finland).”

3.4. Countryandcontextspecificfactors

Theanalysisalsorevealedsomefindingsastohow barri-ersandfacilitatorsappearedspecifictothecountrycontext. Politicalrelevanceoftheissueatstakeandmedia sensitiv-ityismentionedinUK,DenmarkandDutchcasesasbeing quiteinfluential.Mediacaninfluencepoliticalprioritizing andagendasetting.Italsohelpscreateresearchfriendly trendsbymakingpolicydecisionsmoretransparenttothe public.

TheexampleofthesportspolicylinkedtotheOlympic GamesinLondon2012illustratedthatregardlessofample research evidence, a contextual factor such as political willinlargepartdeterminedwhetherandhowthis evi-dencewasused.Theconclusionfromtheintervieweesin thisrespectwas:politicsdecidedwhetherornota cer-tainprogramorpolicyislaunched,independentfromwhat researchevidencesuggested[32,33].Politicianssimplyfelt thattheyhadtotry(becausetheyhadahunchthatitmight work)andthatLondon2012OlympicGamescouldinspire agenerationtotakeupsport.Thisideahadalreadybeen accepteddespitethefactthatresearchevidencefromother countrieswherethisalreadyhadbeentriedbefore[26,27]

clearlyshowedalackofsuccessinmotivatingthegeneral publicbyanOlympicGamespolicyinitiative.

6 I.vandeGooretal./HealthPolicyxxx(2017)xxx–xxx

time,resourcesandcompetencieswithholdpolicyofficers fromevaluating and implementing all this information. Whenevidencewasuseditmostlystemmedfromdirect (personal)relationshipsbetweenresearchersand policy-makers,thatbythewayinFinlandwerepresentquiteoften. TheDutchandDanishcasesweresimilartotheextent thatHEPApoliciesaremainlylocallyinitiatedand imple-mented.TheDanishcasesfocusedonthelocalandregional level only (no national HEPA policy) while the Dutch casehadageneralnationalumbrellapolicy,howeverthe DutchHEPApolicymakingwascarriedoutat municipal-ity level. Both cases showed that locally relevant and directlyusableevidenceappearedtobewhatstakeholders werelackingmostly.Despiteawell-functioningnational researchinfrastructureinbothcountries,municipality pol-icyofficersappearedtolackpracticalinformationonfor examplecost-effectivenessandconcreteinterventionsor policymeasuresfittingtheactuallocalcontextand prob-lems. In the Netherlands the distance (created by law) betweennationallevel policyambitions andthe imple-mentationoflocalpolicywasmentionedtohinderuptake of evidence. Trust based (personal) relations between researchers,policyadvisorsandpolicymakersatlocallevel werementionedinbothcasesasfacilitatingtheuseof evi-dence.

Thelackofconcretenessandrelevanceofevidenceto thelocalcontextwasmentioned byItalian interviewees asanimportantbarrierforuseofevidenceinpolicy.The Italian case also showed incomprehensibilityand diffi-cultytounderstandevidenceasakeybarrier.Translation and synthesis of the evidence and organizing meetings betweenresearchers,policymakersandother stakehold-erssothat(personal)relationshipsandnetworkscouldbe established,wereseenasmainfacilitators.

Finallythe Romaniancases showedhow a country’s nationalandlocaladministrativecontextisofspecific influ-enceontheuptakeofevidenceinpolicy.Itshowedthatin Romaniaaconjunctionofempirical,experiencegrounded evidenceandavailableresourcesandopportunitiesatlocal level were leading in HEPA policymaking, as research evidenceappearednotavailableornotofinterestto poli-cymakersasitwasjudgedtobepolicyirrelevant.Romania differedfromtheotherfivecountriesin thatthere was littleinformation availableonhowthepolicywasbeing developed,whichstakeholderswereinvolvedandtowhat extent.Intervieweeswerealsodifficulttoreachorwere notwillingtoparticipateandtheliteracylevelregarding evidence-informedpolicy makingand related terminol-ogyoftheintervieweeswasratherlow.Thestakeholders thatparticipatedmentionedthatingeneral,research evi-dencewaslackingandthattheevidencethatwasavailable wasmostlynotrelevanttothespecificpolicycontextand thereforenot very usefulfor implementation. However, oneintervieweementionedthattheresearchevidencethat wasavailablesuchthatfromPhDtheseslackedvisibility orpoliticalinterestandwasthereforenotused.In addi-tionpolicymakersnot beingusedtoincludingevidence intheirworkandthelackoffinancialresourcesatlocal levelwerementionedasspecificbarriers.Tocompensate thislackofconcrete(research)evidence,otherinformation suchasdirect(field)observations(e.g.fieldvisits),

expe-riencesfrompreviouslyimplementedprogramsandlocal levelpractitioners’feedbackwasused.

4. Discussion

This article presents findings from semi-structured interviews with 86 stakeholders involved in the policy developmentof21HEPApoliciesfromnational,regionalor locallevelsinsixEUcountries.Factorshinderingor facili-tatinguseofevidenceinreal-lifepolicymakingprocesses wereexploredandanalysed.Barriersandfacilitators men-tionedmostfrequentlywerecategorizedintothreegeneral domains:organizations,systemsandinfrastructure;access andavailabilityofrelevantevidence;andnetworkingand collaborationbetweenresearchersandpolicymakers. Fur-therspecificsofhowbarriersandfacilitatorsappearedto interactwithcountryorpolicycontextsweredescribed. Thisstudyconfirmsthatmostofwhatisalreadyknown from recent literature on key hindrances in uptake of evidenceinpolicymakinginmostlyEnglishspeaking coun-tries,alsoholdstrueforotherEuropeancountries.Besides this finding which is based ona substantial number of interviewsin6Europeancountries,wealsofound contex-tualdifferences inthecountrycases.Firstly,somecases showed that the attitude of media towards underpin-ningpolicywithevidence(asdoneintelevisiondebates, newspapers, social media) as wellas the political con-textinwhichpolicymakingtakesplaceinfluencesevidence uptakeinpolicy.Policydecisionmakers(e.g.politicians) mayfeelthatthemedia’sattitudehasalargeimpacton votersandthereforeactaccordingly.Reviewsinthe liter-atureonbarriersandfacilitatorsintheuseofevidenceto develophealthpolicymainlyfocusonfactorsinfluencing researchers,practitionersandpolicyofficersasmain stake-holders.Mediaappearnottoplayasubstantialroleamong thesefactors[34,15,35,36,10].Ortonetal.[10]domention aspectssuchaspoliticalviabilityanddegreeof commu-nity support as non-evidentiary factors of influence on policymaking.Mediaengagementcanofcoursestrongly affecttheseaspects.Thefactthatwefoundstakeholders explicitlymentionthatmediahasanimpactondecision makers’opinionsontheuseofevidenceaddsnewinsight tothisknowledge.Inaddition,therapiddevelopmentof socialmediamakesitsinfluenceontheuptakeofevidence hardtofindinrecentliteraturewhichmakesitevenmore relevant.Thesefindingsindicatethatthetacticalor polit-icalmodelofevidenceutilizationinpublichealthpolicy resemblesrealitymorecloselythantheproblem-solving orknowledge-drivenmodel[16,37,38,39].

of‘individualfactors’isalsoknownintheresearch liter-aturetoplay acrucialroleintheknowledgeconversion process [41,16,10]. Our results also seem to be consis-tentwiththe‘interactionexplanation’accordingtowhich ‘knowledgeutilizationdependsondisorderlyinteractions betweenresearchersandusers’andinteractionsacrossthe interfacebetweenpolicymakers andresearchers which areimportanttothetransferofevidencetopolicy[42,34]. Decisionmakers’e.g.politicians’personalbeliefsand per-ceptionsaswellasculturalcircumstancesandtraditions in appreciatingevidence ormore systemoriented limi-tationswillalwaysinteractwiththemereavailabilityor transferringofresearchevidence[39,43].

Inaddition,structuralpartnershipswithall stakehold-ers generatingevidence emerging fromactive two way partnershipsbetweenresearchers,policydecisionmakers, fundersand otherstakeholders and therefore contextu-allyembeddedfromtheoutset,inwhichevidenceismade synthesizedortranslatedinto‘easiertounderstandand applicableevidence’mayofferpotentialbenefit[49,50].

Thirdly,ourstudyshowedthatactualuseofevidence requires both optimal policy relevance, which in some countriesneedstobelocallyembedded.Amorepersonal andtrustbasedinteractionbetweenresearchersand(local) policymakersisalsorequired.Havingevidenceathandon whenandwhereit fitsthephaseandcontextof policy-making,inparticularevidenceoncostsandconsequences ofconcretepolicymeasuresandinterventions,wasshown to support useof evidence in policy making. Although a lot of valuable work is beingdonein this respect by nationalandinternationalinstitutions[56,57], policymak-ersstillappearedtoexperiencealackofevidenceespecially oncosteffectivenessandimplementationofinterventions to the local or regional context. This is also acknowl-edgedin literaturewhere timelyaccesstogood quality andrelevantresearchevidenceandinteractionsacrossthe interfaceofresearchandpolicy-makingarereportedtobe themostimportantfactorsininfluencinguseofevidence

[15,34,12,39].

4.1. Strengthsandlimitations

InthisstudyHEPApoliciesfromdifferentlevelsinsix EUmemberstateswereincluded.Thiscanbeconsidered bothastrengthandalimitation.Astrengthsincealarge varietyofcountriesandstakeholdersinvolvedincreased thepossibilitiesforgatheringandexploringthewhole con-tinuumofpotentialbarriersandfacilitatorsthatplayarole throughoutdifferentcontextsandondifferentlevelsof pol-icymaking.Selectingpoliciesfromdifferentpolicycontexts offeredapossibilitytolearnfromdifferentpolicymaking systemsacrossEUmemberstates.Althoughthedatastem from2012,findingsarerelevant aspolicyprocessesare long(4–10years)anddependantonelectionsand admin-istrativeunderpinningsthatarestillinforce.Ourfindings thereforecanpotentiallyinformnextpolicymakingcycles. Inaddition,havingdifferentstakeholdergroupsinvolved, withresearchersbeingaminority,canbeseenasastrength

[15].Thisstudyontheonehandoffersavastamountofrich datainitsvarietyandrepresentation.Itspansacross dif-ferentpoliticalandsocialcontextsthatwarrantsanalytic

generalization[38,44].Whileontheotherhand,itleaves littleroomforstrictcomparison,whichcanbeseenasa limitation.Thisisillustratedbytheparticipationof Roma-niaforinstance.Romaniaisahighlycentralizedcountry comparedtootherEUmemberstatesthatparticipatedin thisresearchproject[39].Thereforethedevelopmentof HEPApoliciesinRomaniamainlytakesplaceatnational levelandwithinasinglesector(i.e.thesportsector)[45]. Thisnationalpoliticalandadministrativecontextmainly resulted in less articulated information on barriersand facilitatorsin crosssectorpolicymakingin Romania.In general,thebarriersandfacilitatorsmentionedin Roma-niareflectsimilaraspectssuchaslackofrelevantevidence forpolicy,althoughfocusingmoreonimplementationof programsthanonHEPApolicydevelopmentprocesses.A furtherstrengthisthattheintervieweeswerechosenfor beingactuallyinvolved intheprocessofdevelopmentof selectedHEPApolicies.Thismadetheirresponsesreallife basedratherthantheoreticalpolicyprocess.Thisis con-sideredamajorstrengthbecauseourfindingsaddtothe existingliteraturewithnewinformation onthebarriers andfacilitatorsthatstakeholdersexperienceinthe every-daypolicymakingprocesses.

4.2. Recommendations

For all the barriersand facilitators identified in this study,recommendationscanbeformulated.Withrespect toorganizations,systemsandinfrastructure,structural fac-torswithinthepolicymakers’organizationcreateastrong potentialforacceleratingresearchevidenceuptake. How-ever,toenhanceevidenceuptakemoreattentionneedsto bepaidtosocialandpersonalfactors.Theroleofmanagers

[18],relationshipsandnetworksinwhichresearchersand decisionmakersact[20,19]aswellasimprovedskillsof policymakerscanmakeacleardifferenceintheextent ofevidenceuptakeinthepolicyprocess[28,10,58].Also bothresearchersandpolicymakersshouldbemoreaware ofhowmediaimpactresearchutilizationbyinfluencing politiciansandhowtomakebetteruseofitsimpact. ‘Fram-ing’ofa problemisoneof theways policymakersand politicianstrytoreduceambiguity anduncertainty ofa specificproblem. Researcherscan supportpolicymakers withprovidingevidencetosupportthisprocessof fram-ing[59].Inaddition,moretimelyinteraction,collaboration andpromotingtrustbasedinteractionsacrosstheinterface betweenpolicymakersandresearcherscouldfacilitateuse ofresearchevidencebybringingabouta jointlanguage, moremutualunderstandingofeachother’snorms,values andeverydayliferealities[15,34,46].Andfinallythe con-textualrelevanceisamainaspectofhowtheapplicability ofevidenceis valued.Therefore, researchersneedtobe abletoconvincepolicymakers‘thatthisworksinthis pol-icycontext’withtheevidencetheyprovide.Theyneedto doitsuchthatattitudesandideologiesofdecisionmakers, theirskillsandpersonalvaluesareknownandtakeninto account;otherwiseresearchevidencewillnotbetakeninto consideration.

collab-8 I.vandeGooretal./HealthPolicyxxx(2017)xxx–xxx

orationandunderstandingbetweendifferentstakeholders suchasstewardship-based intervention [47] and policy games[48]. One shouldbeaware of theimportance to keepoffresearchers’independenceandacademicfreedom

[51–53,11].Inthatrespect,theroleof‘pivotal’peoplewho caninspireandtriggerthepolicymakingprocessandan enhancedfunctioningofknowledgebrokersmaybe stud-iedfurther[54].Knowledgebrokerswereshowntohavea positiveeffectinorganizationsthatarenotveryevidence oriented[55].Furthermore,itseemsequallyimportantthat researchers,policymakers,politiciansandmedia-officers learnabouteachother’srealitiesandthedifferent obsta-clestheyarefacing. Thiswillhelp increaseevidence to beexperienced as‘relevant’ amongdifferent stakehold-ersand consequentlywillenhanceevidencebeingused inpolicy,thusultimatelyresultinginmoreeffective pub-lic health policy. Our study results therefore urge for interventionsthat supportmore structuralandpersonal interactionbetweenallstakeholdergroupsthroughoutthe entirepolicymakingprocess ingenerating, interpreting anddiscussingtheevidenceneededtooptimallyinform publichealthpolicyindifferentcontexts.

Conflictofinterest

Authorsdeclarednoconflictofinterest.

Acknowledgements

TheResearchintoPolicyintoPhysicalActivity(REPOPA) has received funding by the European Union Seventh FrameworkProgram (FP7/2007-2013); grant agreement no.281532.Thisdocumentreflectsonlytheauthors’views andneithertheEuropeanCommissionnoranypersonon itsbehalfis liablefor anyusethat maybemadeof the informationcontainedherein.

References

[1]MacintyreS. Evidence based policy making: impact onhealth inequalitiesstill needs to be assessed.British Medical Journal 2003;326:5–6.

[2]RoseG.TheStrategyofPreventiveMedicine.OxfordUniversityPress; 1992.

[3]OxmanAD,LavisJN,LewisS,FretheimA.Supporttoolsfor evidence-informedhealthpolicymaking:Whatisevidenceinformedpolicy making?HealthResearchPolicyandSystems2009;7(Suppl.1):S1.

[4]Brownson RC, FieldingJE, Maylahn CM.Evidence-based public health:afundamentalconceptforpublichealthpractice.Annual ReviewofPublicHealth2009;30(April):175–201.

[5]BalasEA,BorenSA,HicksLL,ChonkoAM,StephensonK.Effectof linkingpracticedatato publishedevidence:a randomized con-trolledtrialofclinicaldirectreports.MedicalCare1998;36(January (1)):79–87.

[6]HucksonS,DaviesJ.Closingevidencetopracticegapsinemergency care:the Australianexperience.AcademicEmergency Medicine 2007;14(November(11)):1058–63.

[7]LomasJ.Connectingresearchandpolicy.CanadianJournalofPolicy Research2000;1(1):140–4.

[8]McCaughey D, Bruning NS. Debate Rationality versus real-ity: the challenges of evidence-based decision making for health policy makers. Implementation Science 2010;5(May),

http://dx.doi.org/10.1186/1748-5908-5-39.

[9]LiveraniM,HawkinsB,ParkhurstJO.Politicalandinstitutional influ-encesontheuseofevidenceinpublichealthpolicy.Asystematic review.PLoSOne2013;8(October(10)):e77404.

[10]Orton L, Lloyd-Williams F, Taylor-Robinson D, O’Flaherty M, CapewellS.Theuseofresearchevidenceinpublichealth deci-sionmakingprocesses:systematicreview.PLoSOne2011;6(July (7)):e21704,http://dx.doi.org/10.1371/journal.pone.0021704. [11]ShineKT,BartleyB.Whoseevidencebase?Thedynamiceffectsof

ownership, receptivity and values on collaborative evidence-informed policy making. Evidence & Policy: A Journal of Research, Debate and Practice 2011;7(November (4)):511–30,

http://dx.doi.org/10.1332/174426411x603489.

[12]InnværS,VistG,TrommaldM,OxmanA.Healthpolicy-makers’ per-ceptionsoftheiruseofevidence:asystematicreview.Journalof HealthServicesResearch&Policy2002;7(October(4)):239–44.

[13]LarsenM,GulisG, PedersenKM.Useofevidenceinlocal pub-lichealthworkinDenmark.InternationalJournalofPublicHealth 2012;57(June(3)):477–83.

[14]Lavis JN, Oxman AD, Lewin S, Fretheim A. SUPPORT Tools for evidence-informed health Policymaking (STP). Health Research Policy and Systems 2009;7(December (Suppl. 1)):I1,

http://dx.doi.org/10.1186/1478-4505-7-S1-S3.

[15]OliverK,InnvarS,LorencT,WoodmanJ,ThomasJ.Asystematic reviewofbarrierstoandfacilitatorsoftheuseofevidenceby pol-icymakers.BMCHealthServicesResearch2014;14(January(1)):1,

http://dx.doi.org/10.1186/1472-6963-14-2.

[16]BowenS,ZwiAB.Pathwaystoevidence-informedpolicyand prac-tice:aframeworkforaction.PLoSMedicine2005;2(May(7)):e166.,

http://dx.doi.org/10.1371/journal.pmed.0020166.

[17]MooreG,RedmanS,HainesM,ToddA.Whatworkstoincrease theuseofresearchinpopulationhealthpolicyandprogrammes:a review.Evidence&Policy:AJournalofResearch,DebateandPractice 2011;7(August(3)):277–305.

[18]OliverKA,deVochtF.Defining‘evidence’inpublichealth:asurveyof policymakers’usesandpreferences.TheEuropeanJournalofPublic Health2015;(July):ckv082.

[19]OliverK,deVochtF,MoneyA,EverettM.Whorunspublichealth?A mixed-methodsstudycombiningqualitativeandnetworkanalyses. JournalofPublicHealth2013;(April):fdt039.

[20]ShearerJC,DionM,LavisJN.Exchangingandusingresearchevidence inhealthpolicynetworks:astatisticalnetworkanalysis. Implemen-tationScience2014;9(October(1)):1.

[21]ArmstrongR,WatersE,DobbinsM,AndersonL,MooreL,Petticrew M,ClarkR,PettmanTL,BurnsC,MoodieM,ConningR.Knowledge translationstrategiestoimprovetheuseofevidenceinpublichealth decisionmakinginlocalgovernment:interventiondesignand imple-mentationplan.ImplementationScience2013;8(October(1)):1.

[22]Oliver KA, de VochtF, MoneyA, Everett M.Identifying public health policymakers’ sourcesof information: comparingsurvey and network analyses. The European Journal of Public Health 2015;(July):ckv083.

[23]DaugbjergSB,KahlmeierS,RacioppiF,Martin-DienerE,MartinB, OjaP,etal.PromotionofphysicalactivityintheEuropeanregion: contentanalysisof27nationalpolicydocuments.JournalofPhysical Activity&Health2009;6(November(6)):805.

[24]JansenMW,DeLeeuwE,HoeijmakersM,DeVriesNK.Workingatthe nexusbetweenpublichealthpolicy,practiceandresearch.Dynamics ofknowledgesharingintheNetherlands.HealthResearchPolicyand Systems2012;10(October):33.

[25]AroAR,BertramM,HämäläinenRM,vandeGoorI,SkovgaardT, ValenteA,etal.Integratingresearchevidenceandphysical activ-itypolicymaking—REPOPAproject.HealthPromotionInternational 2015;(February):dav002,http://dx.doi.org/10.1093/heapro/dav002. [26]Hämäläinen RM, Aro AR, van de Goor I, Lau CJ, Jakobsen MW, Chereches RM, et al. Exploring the use of research evidence in health-enhancing physical activity policies. Health Research Policy and Systems 2015;13(October (1)):43,

http://dx.doi.org/10.1186/s12961-015-0047-2.

[27]EklundKarlssonL,JakobsenM,HeibergW,AroAR.Involvementof externalstakeholdersinlocalhealthpolicymakingprocess:acase studyfromOdenseMunicipalityDenmark.Evidence&Policy2016,

http://dx.doi.org/10.1332/174426416x14609162710134. [28]AroAR,SmithJ,DekkerJ.Contextualevidenceinclinicalmedicine

and health promotion. The European Journal of Public Health 2008;18(December(6)):548–9.

[29]FlitcroftK,GillespieJ,SalkeldG,CarterS,TrevenaL.Gettingevidence intopolicy:theneedfordeliberativestrategies?SocialScience& Medicine2011;72(April(7)):1039–46.

for Human and Machine Cognition. 2008 January 22. avail-able at: http://cmap.ihmc.us/Publications/ResearchPapers/ TheoryUnderlyingConceptMaps.

[31]EdwardsN,ViehbeckS,HämäläinenR-M,RusD,SkovgaardT,vande GoorI,etal.Challengesofethicalclearanceininternationalhealth policyandsocialsciencesresearch:experiencesand recommen-dationsfromamulti-countryresearchprogramme.PublicHealth Reviews2012;34(January(1)):1.

[32]Mahtani KR, ProtheroeJ, Slight SP, Demarzo MM,Blakeman T, BartonCA,etal.CantheLondon2012Olympics‘inspirea gen-eration’todomorephysicalorsportingactivities?Anoverview of systematic reviews. BMJ Open 2013;3(January (1)):e002058,

http://dx.doi.org/10.1136/bmjopen-2012.

[33]McCartneyG,ThomasS,ThomsonH,ScottJ,HamiltonV,HanlonP, etal.Thehealthandsocioeconomicimpactsofmajormulti-sport events:systematicreview(1978–2008).BMJ2010;340(May):c2369,

http://dx.doi.org/10.1136/bmj.c2369.

[34]Hanney SR,Gonzalez-BlockMA,BuxtonMJ,KoganM.The util-isationofhealthresearch inpolicy-making:concepts,examples andmethodsofassessment.HealthResearchPolicyandSystems 2003;1(January(1)):2.

[35]WallaceJ,NwosuB,ClarkeM.Barrierstotheuptakeofevidencefrom systematicreviewsandmeta-analyses:asystematicreviewof deci-sionmakers’perceptions.BMJOpen2012;2(January(1)):e001220,

http://dx.doi.org/10.1136/bmjopen-2012-0012202.

[36]LavisJ,Davies H,OxmanA,DenisJL,Golden-BiddleK,FerlieE. Towardssystematicreviewsthatinformhealthcaremanagement andpolicy-making.JournalofHealthServicesResearch&Policy 2005;10(July(Suppl.1)):35–48.

[37]NutbeamD.Howdoesevidenceinfluencepublichealthpolicy? Tack-ling healthinequalitiesinEngland.HealthPromotionJournalof Australia2003;14(December(3)):154–8.

[38]WeissCH.Themanymeaningsofresearchutilization.Public Admin-istrationReview1979;39(September(5)):426–31.

[39]PetticrewM,WhiteheadM,MacintyreSJ,GrahamH,EganM. Evi-denceforpublichealthpolicyoninequalities:1:therealityaccording topolicymakers.JournalofEpidemiologyandCommunityHealth 2004;58(October(10)):811–6.

[40]ElliottH,PopayJ.Howarepolicymakersusingevidence?Modelsof researchutilisationandlocalNHSpolicymaking.Journalof Epidemi-ologyandCommunityHealth2000;54(June(6)):461–8.

[41]GubbinsC,CorriganS,GaravanTN,O’ConnorC,LeahyD,LongD, etal.Evaluatingatacitknowledgesharinginitiative:acasestudy. EuropeanJournalofTrainingandDevelopment2012;36(November (8)):827–47.

[42]LandryR,LamariM,AmaraN.Theextentanddeterminantsofthe utilizationofuniversityresearchingovernmentagencies.Public AdministrationReview2003;63(March(2)):192–205.

[43]BestA,HolmesB.Systemsthinking,knowledgeandaction:towards bettermodelsandmethods.Evidence&Policy:AJournalofResearch, DebateandPractice2010;6(May(2)):145–59.

[44]PolitDF,BeckCT.Generalizationinquantitativeandqualitative research:mythsand strategies.InternationalJournalofNursing Studies2010;47(November(11)):1451–8.

[45]The Romanian Administrative System—French Inspiration and National Adaptation,

www.publicresearch.ro/library/files/sistemadministrativ dragos dincaen.pdf.

[46]GlasgowRE, Emmons KM.Howcan weincreasetranslation of researchintopractice?Typesofevidenceneeded.AnnualReview ofPublicHealth2007;28(April):413–33.

[47]The Nuffield Council on Bioethics [Internet]. Nuffield Bioethics. 2016 [cited 21 Dec 2015]. Available from:

http://www.nuffieldbioethics.org.

[48]GeurtsJL,DukeRD,VermeulenPA.Policygamingforstrategyand change.LongRangePlanning2007;40(December(6)):535–58.

[49]GreenhalghT,WieringaS.Isittimetodropthe‘knowledge trans-lation’metaphor?Acriticalliteraturereview.JournaloftheRoyal SocietyofMedicine2011;104(December(12)):501–9.

[50]ArmstrongR,WatersE,RobertsH,OliverS,PopayJ.Theroleand theoreticalevolutionofknowledgetranslationandexchangein pub-lichealth.JournalofPublicHealth2006;28(December(4)):384–9,

http://dx.doi.org/10.1093/pubmed/fdl072.

[51]FuntowiczS,RavetzJ.Science forthepost-normalage.Futures 1993;25(7):735–55.

[52]DahlRA.Ademocraticdilemma:systemeffectivenessversuscitizen participation.PoliticalScienceQuarterly1994;109(April(1)):23–34.

[53]Sebba J.Getting Research intoPolicy: theRole ofThink Tanks and Other Mediators. Impact of Social Sciences Blog; 2011

http://eprints.lse.ac.uk/35740.

[54]ChoiBC,PangT,LinV,PuskaP,ShermanG,GoddardM,etal.Can scientistsandpolicymakersworktogether?JournalofEpidemiology andCommunityHealth2005;59(August(8)):632–7.

[55]LaRoccaR,YostJ,DobbinsM,CiliskaD,ButtM.Theeffectivenessof knowledgetranslationstrategiesusedinpublichealth:asystematic review.BMCPublicHealth2012;12(September(1)):1.

[56]NationalInstituteforHealthandClinicalExcellence.Supporting InvestmentinPublicHealth:ReviewofMethodsforAssessingCost Effectiveness,CostImpactandReturnonInvestment.Proofof Con-ceptReport.NHS;2011.

[57]McDaidD,SassiF,MerkurS.PromisingHealth,PreventingDisease. TheEconomicCase.WHO/OECD,EuropeanObservatoryonHealth SystemsandPolicies.McGrawHillEducation.OpenUniversityPress; 2015.

[58]YostJ,DobbinsM,TraynorR,DeCorbyK,WorkentineS,GrecoL.Tools tosupportevidence-informedpublichealthdecisionmaking.BMC PublicHealth2014;14(July(1)):1.