Enabling the human in the loop: Linked data and knowledge in industrial cyber-physical systems

(1)

University of Groningen

Enabling the human in the loop

Emmanouilidis, Christos; Pistofidis, Petros; Bertoncelj, Luka; Katsouros, Vassilis; Fournaris,

Apostolos; Koulamas, Christos; Ruiz-Carcel, Cristobal

Published in:

Annual Reviews in Control

DOI:

10.1016/j.arcontrol.2019.03.004

IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from

it. Please check the document version below.

Document Version

Publisher's PDF, also known as Version of record

Publication date:

2019

Link to publication in University of Groningen/UMCG research database

Citation for published version (APA):

Emmanouilidis, C., Pistofidis, P., Bertoncelj, L., Katsouros, V., Fournaris, A., Koulamas, C., & Ruiz-Carcel,

C. (2019). Enabling the human in the loop: Linked data and knowledge in industrial cyber-physical systems.

Annual Reviews in Control, 47, 249-265. https://doi.org/10.1016/j.arcontrol.2019.03.004

Copyright

Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).

Take-down policy

If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum.

(2)

ContentslistsavailableatScienceDirect

Annual

Reviews

in

Control

journalhomepage:www.elsevier.com/locate/arcontrol

Enabling

the

human

in

the

loop:

Linked

data

and

knowledge

in

industrial

cyber-physical

systems

Christos

Emmanouilidis

a,∗

_,

_Petros

_Pistoﬁdis

b

_,

_Luka

_Bertoncelj

a

_,

_Vassilis

_Katsouros

b

_,

Apostolos

Fournaris

c

_,

_Christos

_Koulamas

c

_,

_Cristobal

_Ruiz-Carcel

a

a Cranﬁeld University, UK

b Institute for Language and Speech Processing, “Athena” Research Center, Greece c Industrial Systems Institute, “Athena” Research Center, Greece

a

r

t

i

c

l

e

i

n

f

o

Article history: Received 21 July 2018 Revised 18 January 2019 Accepted 11 March 2019 Available online 19 March 2019

Keywords:

Cyber-physical systems Internet of things

Context information management Product lifecycle management Asset lifecycle management Maintenance

Human in the loop

a

b

s

t

r

a

c

t

IndustrialCyber-PhysicalSystemshavebenefittedsubstantiallyfromtheintroductionofarangeof tech-nologyenablers.Theseincludeweb-basedandsemanticcomputing,ubiquitoussensing,internetofthings (IoT)withmulti-connectivity,advancedcomputingarchitecturesanddigitalplatforms,coupledwithedge orcloudsidedatamanagementandanalytics,andhavecontributedtoshapingupenhancedornewdata valuechainsinmanufacturing.Whilepartsofsuchdataflowsareincreasinglyautomated,thereisnowa greaterdemandformoreeffectivelyintegrating,ratherthaneliminating,humancognitivecapabilitiesin theloopofproductionrelatedprocesses.HumanintegrationinCyber-Physicalenvironmentscanalready bedigitallysupportedinvariousways.However,incorporatinghumanskillsandtangibleknowledge re-quiresapproachesandtechnologicalsolutionsthatfacilitatetheengagementofpersonnelwithin techni-calsystemsinwaysthattakeadvantageoramplifytheircognitivecapabilitiestoachievemoreeffective sociotechnicalsystems.Afteranalysingrelatedresearch,thispaperintroducesanovelviewpointfor en-ablinghumanintheloopengagementlinkedtocognitivecapabilitiesandhighlightingtheroleofcontext informationmanagementinindustrialsystems.Furthermore,itpresentsexamplesoftechnologyenablers forplacingthehumanintheloopatselectedapplicationcasesrelevanttoproductionenvironments.Such placementbenefitsfromthejointmanagementoflinkedmaintenancedataandknowledge,expandsthe powerofmachinelearningforassetawarenesswithembeddedeventdetection,andfacilitatesIoT-driven analyticsforproductlifecyclemanagement.

1. Introduction

While industrialcyber-physicalsystems (Colombo,Karnouskos, Shi, Yin, & Kaynak, 2016) bring together the physical anddigital worldsinmanufacturing,thehumanintegrationinproduction en-vironments has onlyrecently beganreceiving increased attention (Nunes,Zhang,&Silva,2015).Termssuchas“Operator4.0” are em-ployedtodenotethevisionofhumanempowermentwithIndustry 4.0technologies(Romeroetal.,2016).Withinsuchavision,the co-existence ofhuman andengineeringactorsisviewedthroughthe prismofthenatureoftheirinteractioninvariousformsof physi-calanddigitalaugmentationofhumanactivity.Manyconceptsand numerouspracticalimplementationexamplesofsupportedhuman

∗ _{Corresponding author.}

E-mail addresses: christosem@cranfield.ac.uk , christosem@ieee.org (C. Em- manouilidis), pistofid@ceti-athena.innovation.gr (P. Pistofidis), vsk@ilsp.gr (V. Katsouros), fournaris@isi.gr (A. Fournaris), koulamas@isi.gr (C. Koulamas),

cruizcarcel@cranﬁeld.ac.uk (C. Ruiz-Carcel).

action in industrial environments are reported in the literature. However,theactualcognitive contributionofhumanactivities to-wardsthe operationoftechnicalsystems,althoughacknowledged tobeimportantinsociotechnicalsystems,itremains lesswell ex-plored.

Arguably, the effectiveness of industrial Human In the Loop (HIL)CyberPhysical Systems(CPS)islinked totheability to cap-tureandactuponthecontextofsuch interactioninanenterprise system(ElKadiri etal.,2016;Nunesetal., 2015).Theaimofthis paper is to introduce an approach for enabling industrial HIL in CPS,asacontributortosuccessfulintegrationofsociotechnical sys-tems.StartingformanoutlineofresearcheffortsrelatedtoHILin CPSwithanapplicationfocusonproductandassetlifecycle man-agement,the paper outlines key emerging HIL-CPS concepts and relevantcognitivecapabilities,highlightstheroleofcontext infor-mation management, and offers examples of placing the HIL at selected relevantapplication cases. Data andknowledge ﬂows in suchactivitiesneedtobetakenintoaccountandmethodsforthe jointmanagementoflinkeddataandknowledgeareintroducedas

https://doi.org/10.1016/j.arcontrol.2019.03.004

(3)

250 C. Emmanouilidis, P. Pistoﬁdis and L. Bertoncelj et al. / Annual Reviews in Control 47 (2019) 249–265

akeymechanismforestablishingsharedcontext,whichinturnis a key ingredient forsuccessful HIL-CPS engagement inindustrial environments.Theaimisnot toseektoreplacethehumanfactor buttoempower itwithmoreeffectiveintegrationofhuman cog-nitivecapabilitieswithintechnicalsystems.

The rest of the paper is structured as follows. Section 2 dis-cusses related work on data ﬂows in Product Lifecycle Manage-ment (PLM), HIL in CPS, as well as Linked Data, Knowledge, ContextManagement andVisualAnalytics. Section3introduces a linkagebetweenHIL-CPSandhumancognitivecapabilitiesandthe conceptofblendingsociotechnicalsystemswithadvanced capabil-ities for interaction, supported by visual analytics and a context managementarchitecturethatincludesthehandlingofdata, meta-datainformation,andknowledge,together withappropriate busi-ness logic. Section 4 presents the key concepts implemented to demonstrateHILonapplicationcasesrelevanttoproductandasset lifecyclemanagement.The ﬁrstone employsHILforLinkingData and Knowledge Management and for enhancing machine learn-ing capabilities and integrates operating stage technical systems (condition monitoring) with design stage knowledge (e.g. Failure Modes,Effects,andCriticalityAnalysis– FMECA).Thesecondcase engagesHILwithvisualanalyticstocommunicatecondition mon-itoring outcomes in visually relevant ways for augmenting hu-mancapabilitieswhenperforming analytics-drivendecisiontasks.

Section5concludeswithanoutlineofthemaincontributionsand pointersforfurtherresearch.

2. Relatedwork

While much emphasis has been placed on increasing levels of automationin production systems, the case forstrengthening the role of HIL is growing stronger, asemerging technology en-ablersempower human operators tobecome moreeffectively in-tegratedinproduction activities (Romero etal., 2016). Such inte-grationmakes humanactorsandtheircognitive capabilitiesmore engagedwithdata,knowledge,anddecisionprocesschainsin pro-ductionenvironmentsandPLMactivities.Thereforeresearchneeds to consider the relevance of data ﬂows when integrating HIL in PLM, the technology enablers that support more effective inte-gration ofhuman cognitive capabilitiesinCPS, the role oflinked data and knowledge in supporting context information manage-mentandestablishing shared context tofacilitate theintegration ofHILin technical systems,including theempowering impact of visualanalyticsasaninteractive approachtoHILindecision mak-ing.Thesearediscussednext.

2.1.DataﬂowsinPLM

The early vision of closed loop product lifecycle management involvedcreatinginformationloopsbetweendifferentproduct life-cyclephases,namelyBeginningofLife(BoL),MiddleofLife(MoL), and End of Life (EoL) activities (Kiritsis, Bufardi, & Xirouchakis, 2003).Consequentresearchfocusedonfacilitatinginformation ex-changes between the different lifecycle activities (Jun, Kiritsis, & Xirouchakis,2007). Physical dataaspects of such exchangeswere handledwithradiofrequencyidentiﬁcation(RFID),introducingthe conceptofproduct embeddedinformation devices(PEID)(Kiritsis etal.,2008).Thishassupportedtheintroductionofsmartproducts or assets (Brintrup et al., 2011; McFarlane, Sarma, Chirn, Wong, & Ashton, 2002; Meyer etal., 2009), a key enabler for the joint handlingofoperations,maintenance,andlogistics(VanBelleetal., 2011) butalsoof monitoringand control functions(Meyer etal., 2009). However, such joint handling required an upgrade in the level of data exchanges well beyond basic product data ID ex-changes. This upgrade could be served by further advancements

IoT technologies and collaborative digital engineering systems (Kiritsis,2011).

Among the key challenges in IoT-enabled product and asset lifecycle management is the integration of data, informationand knowledgefromdisparateandheterogeneoussources.Thisrenders theconventional approachtointegratingdatathrough acommon enterprisedatawarehouseincreasinglyproblematicinmodernbig data enterprise environments (Vathy-Fogarassy & Hugyák, 2017). Instead,theemergingdatamanagement patternisthatof retriev-ing relevantdatafromdisparate sources andseekingtointegrate thematthepointofendconsumption.However,theoften hetero-geneous natureofthedatacreates furtherchallengestosuch ap-proaches andhasled toresearch efforts toestablishsemantic in-teroperability of connectedproducts, supported by developments that looked into how semantic (Cassina et al., 2008) and ontol-ogybasedmodelling(MatsokisandKiritsis,2010)canenablesuch productlifecycledataexchanges.Thisthreadofresearchhasledto standardized,andsemanticallyenhancedproductlifecycledata ex-changes(Framlingetal.,2014;Kubleretal.,2015).IoTconnectivity nonetheless givesrise toa multi-layered view ofdata exchanges, whichrequiresamappingoftheIoTinformationprocessinglayers withproductdatamodelling,fromthephysicaltotheapplication layer(Framling,Kubler,&Buda,2014).Productdatainteroperability thereforeisrelevant acrosstheIoT stack andinvolvesboth lower tierdata,suchassensormeasurements,butalsohigherlevel infor-mation(Yooetal., 2016),inordertoenableproductdata integra-tioninproximitytothepointofdataconsumption.

Among theprimeinterests inclosed loopPLMis feedingMoL informationbacktoBoLactivities,so astoenableabetter under-standingof how specific design choicesmightperform in opera-tionanddriveproduct designenhancementsaccordingly.The na-ture of the required data acquisition, transmission, management, andprocessingvariesfromcasetocaseandmaycreatesignificant challenges.Forexample,rawsenseddatatransmissionmayrequire considerablebandwidth,whileradiofrequency(RF)operationmay beconstrainedbythenatureofthesurroundingenvironment.The business value of embedded intelligence for smart services was recognizedevenbeforethedawnofIoT(Kaplanetal.,2005). Tech-nological solutions involvedsensor networking protocols,such as those linked toIEEE 802.15.4,andhavebeen adoptedin wireless sensor network applications for asset monitoring (Willig, 2008). Embeddedprocessingoflocallyacquiredmeasurementsonsensor nodes enabled assetintelligence beyond identificationand track-ing(Liyanage,Lee,Emmanouilidis,&Ni,2009).Suchprocessing en-ablesassetsandproductstoofferahigherlevelofself-awareness (Katsouros, Koulamas, Fournaris, & Emmanouilidis, 2015), consis-tentwithanagent-basedviewofintelligentcyber-physicalentities (Leitãoetal.,2016).Suchcyber-physicalmonitoredassetsfeaturea basiccycleofperception,analysis,decision,and(re)action.

Coupling internetworking connectivity with local, distributed andcloud computing, together withsemanticallyenriching prod-uct information, hasbeen recognized as key contributor towards connectedandintelligent products inenterprise systems(Kiritsis, 2011).Different terms,suchasproduct avatars(Wuest,Hribernik, & Thoben, 2015), shadows (Vermesan & Friess, 2016), or digi-tal twins (Vermesan etal., 2011) have all been employed to de-scribe the cyber version of a physical asset, acting as a smart agent (Leitão, Member, Ma, & Vrba, 2013), or intelligent prod-uct in a cyber-physical world of interconnected physical entities (Leitãoetal.,2016).Accompanyingphysicalassetswiththeircyber counterparts(Wuestetal.,2015)enhancetheeﬃciencyofmodern enterpriseinformationsystems,(ElKadiri etal., 2016). Communi-catingproduct lifecycletothe applicationlayerofenterprise sys-tems is best achieved with application layer relevant means, for example with visual product representations. For example, MoL relevantproductdatacanbe superimposedtoBoLproduct views,

(4)

such as a 3D product CAD model (Emmanouilidis, Beroncelj, Bevilacqua, Tedeschi,& Ruiz-Carcel, 2018). Thus, product dataare madeavailabletousersasavisualproduct designrepresentation, aidingtheunderstandingofmaintenancerelatedconcepts,suchas the occurrenceoffailure modes. Such an approach can be useful when dealing withreal data streams from products asa natural visualanalyticsextensiontoclosedloopPLM.Whatismore,it en-ablestheintegrationofHILclosingPLMinformationloopsvia en-richeddata andknowledge(Wuest etal.,2015). Thenext section looksintomoredetailonhowthisintegrationisbeingpursuedin industrialCPS.

2.2. Humanintheloopandcyberphysicalsystems

Most research effortson data exchangesin PLMhave focused onautomated dataexchanges, whilehuman-contributeddata and knowledgehasreceivedfarlessattention.However,HILintegration requires furtherdevelopment ofmethods andtechnological solu-tions asit isnowbeingrecognisedto beamajor enablerforCPS (Nunesetal.,2015).Recently,humanintegrationinindustrial envi-ronmentsisreceivingincreasingattention,withtermssuchas “Op-erator4.0” usedtodenotethevisionofhumanempowermentwith Industry4.0technologies(Romeroetal.,2016).However,whilethe concept of‘people-centric’IoT hasbeenhighlighted inarangeof applicationdomainsanditscognitivecontributioninempowering interaction betweenhumanandnon-humanactors isemphasized in recentstudies (Feng,Setoodeh,& Haykin, 2017), the nature of such interaction in industrialenvironments isstill primarily con-ﬁned to intelligent operator support. Indeed,the actual cognitive contribution ofhumanactivities towards theoperation of techni-cal systems, although acknowledged to be of importance, is still lesswellexplored.

When consideringhumanintegrationinCPS, signiﬁcantadded valuemayariseasaresultoftheinteractionbetweenhumanand non-human actors. Recent research hasproposed the integration of HILof criticalsystems and processes,withthe role of techni-cal systems being toprevent cognitiveoverload forhuman oper-ators, rather than replacing orreplicating their function. The ap-proachcombinedasupervisoryloopforsituationalawarenesswith a dedicated machine learning approach (Gross et al., 2017). HIL also enhances decisionmaking capabilities. Speciﬁcally, a human operator receiving a range of automated decision recommenda-tions, needs to identify an appropriate recommended action and applyittophysicalassets.ThiscanresultinIoT-drivenintelligence, wheretheiterativenatureoftheHILinteraction,aidedby natural interaction interfaces (e.g. natural language-based, visual analyt-ics),aswellvia semanticallyenriched abstractionofdatathrough knowledge,greatlyenhancehumandecisioncapabilities(Maetal., 2017), butalsomachinelearningtasksindiagnostics(Subramania &Khare,2011).

Further scenarios ofHILormore broadlyHumanin theMesh involvement in industrial environments have been proposed, in-cluding interaction with ERP, MES, SCADA, simulation, analytics, data management, as well as lower level shop ﬂoor activities delivering ﬂexibility in CPS-enabled manufacturing environments (Fantini et al., 2016). Identifying appropriate ways to make such interaction more effective andbetter integrate humanwith non-human actors is still open to research but methods andtools to alignthisintegrationwithhumancognitivecapabilitiesisa natu-ral promising path.The next section discussesoptions to achieve suchintegration.

2.3. Linkeddata,knowledge,context,andvisualanalytics

The eﬃciency of human cognitive activities and therefore of HILinCPS cruciallydependsonthehuman actorhavinga sound

understandingofthecontextofthetargetedproblemorsituation. Inconnected factories,humanactors, aswell asIoT-enabled pro-ductionequipmentandenvironmentscreateproductionand prod-uct– relatedstreams ofdata.Inorderto efficientlymanagesuch dataproduced by multipleproduction sitesandstakeholders, so-lutionsfor scalabledata processingare needed. Context Informa-tionManagementhasemergedasakeyconceptinmanagingsuch complexityinIoT-enabled environments (Perera,Zaslavsky, Chris-ten,&Georgakopoulos,2014).Themainprincipleisthatinorderto enableefficientaggregationandprocessingofdatafromdisparate sources, only contextually relevant data need be made available atthepoint ofdata orservicesconsumption. While domain spe-cificcontext greatlyvariesdependingonthetargetedapplication, higherlevelcontextcanbecategorizedtofallundercertainbroad categories,such asasset, user,business, environment andsystem context.

Inindustrial CPS, context informationmanagement can deter-mine the situational circumstances of decisions (El Kadiri et al., 2016). Inproductandassetlifecyclemanagement,high-level con-textcanbeclassifiedaccordingtotheaforementionedbroad cate-gorieswithdomain-specificsemantics,asillustratedinFig.1.Each contextcategory comprises parameters whichcan be acquiredor computedandtheirsemantic interpretationwouldimpacton the waya specific situation needs to be assessed. Forexample, rele-vantinformationandservices dependon theassetunder consid-eration. Therein,the asset contextmay be determined by consid-eringthe status of the modelled assetin the assethierarchy, its functionwithintheproductionsystem,historicaldataaboutits op-eration, includingprognostics andhealth management (PHM), as well asreliability – relatedmaintenance knowledge,such as Fail-ureModes, Effects,andCriticalityAnalysis (FMECA),orFaultTree Analysis(FTA).Toresolvethecontextofan eventorinteractionin asociotechnicalsystem,othertypesofcontextmustbetakeninto account(Fig.1).Thedetailedanddomainspecificmodelingofthe broadcontextcategoriesmaybe furtherdiversifieddependingon theexactnatureoftheapplication.

Contextdeterminationdependsonlinkingdataandknowledge andthisisinlinewiththesemanticwebparadigmoflinkeddata andproduct knowledge (Pistofidis, Emmanouilidis, Papadopoulos, &Botsaris,2016).Therealvalueofcontextunderstandinglieswith thequalityofdataandknowledgeuponwhichanalysis,decisions, andactionsareexercised.Forexample,(Fig.2)rawdatacanbeof littlevalue ifthere isa lack ofunderstanding about their prove-nanceandunderlyingcontext.Thefigureillustratesthatannotated data(information)canbemorevaluableifadequatelyanalysed to obtaininsightsabouttheunderlyinggeneratingprocesses,leading toknowledge-enhanceddata,whicharemorelikelytodriveaction recommendations.Asoundunderstandingofthedatacontextand overall situation awareness may produce additional insights and leadtomoreinformeddecisionsorchanges(e.g.supplierselection inFig.2).The enhancedvalue ofproductdata acrosssucha data value chain justifies the viewpointthat data is to be considered a value addingassetitself (Kubler etal., 2015). Context informa-tionmanagementisasemanticallyscaledextensionofinformation fusionforIoT (Snidaro,García,& Llinas,2015),enablinglinks be-tweenhumanandnon– humanactors.Therefore,suchlinks facil-itatemoreeffective informationflows,better interfacing between actors,andallowforamoreefficientintegrationofHILin produc-tionandspecificallyPLM.

A Visual Analytics environment can be a signiﬁcant enabler of such context-driven interaction. From the early years of ex-ploratory dataanalysis (Tuckey, 1962) (Tuckey, 1977) all the way tocurrentbig dataanalytics (Idreos etal., 2015), thequality and value of data-driven decision making depends among other on datapre-processinginitiatedbyhumanexperts.Whereascomputer data analysis in the past had a very limited set of options for

(5)

Fig. 1. Context categories in product and asset lifecycle management.

Fig. 2. Context and data value.

userinteractionwithdata,currentvisualanalyticsgreatlyupgrade interaction capabilities, steering expert judgement through visu-allypresented aspects of data characteristics (Thomas and Cook, 2005). Visually enhanced data presentation makes it easier for userstocomprehendthedatacontextcomparedtoreviewingraw data(Endertetal.,2014).Situationalawareness andcontext man-agement also play a key role and can be combined with Visual Analytics, for example, aiming at combining automated process-ing through condition monitoring with human contributed ob-servations, as a means for context-based information fusion for diagnostics(Emmanouilidis et al., 2016). Furthermore, situational awareness is invaluable in resolving context ambiguity in han-dlingindustrialalertandalarmmanagement,whichifunresolved canoverwhelmhumanoperators withunmanageablenumbers of alerts(daSilva,Pereira,&Gotz,2016).

The overviewofrelatedresearchhighlights thatthereismuch tobegainedthroughtheintegrationhumanandnon-humanactors insociotechnicalsystemsbutforsuchintegrationtobecomemore effective,further researchis neededtoalign suchactors notonly bymeansoftechnologyintegrationbutalsobyappropriatedesigns

forseamlessdata,information,knowledge,anddecisionﬂows.The nextsectionintroducestheconceptofplacingsuchdesignswithin theviewpointofcognitivecapabilitiesinsociotechnicalsystems. 3. Cognitivecapabilitiesinsociotechnicalsystems

Evenwiththeintegrationof arangeofIndustry 4.0 technolo-gies,productionenvironmentsarestillfarbelowthelevelof intel-ligencenormallyassociatedwithhumanactors.Whilenon-human actors exhibit some level of intelligent function, the active pres-enceof a human actor enablesmore powerful cognitive capabil-ities to be expressed in production activities. Such activities can be considered to drawparallelswith thecapabilitiesof cognitive architectures(Langley,Laird,&Rogers,2009).Whilesuch capabili-tieshavebeenstudiedregardinghumancognitiveabilitiesand ar-tiﬁcialcognitivesystems,thepotentialforjointsociotechnical sys-tems,whichbothamplifyhumancognitiveabilitiesandexpandthe capabilitiesoftechnicalsystems,haveonlyrecentlygained atten-tion(Aricaetal.,2018).Activitieswhichareclosertobecoming in-dustrial practiceinvolveasubset ofcognitive capabilitiesandare

(6)

Fig. 3. HIL-CPS cognitive capabilities.

linked tosensing,stateinference,actiongenerationandexecution inaformofHILintelligenceinCPSsystems(Nunesetal.,2015).

This paperargues that theintegration ofsuch cognitive capa-bilities can also be exploitedinindustrial informationprocessing cycles.Fig. 3illustrates the conceptof integratingcognitive abili-tiesinsociotechnicalsystemstodeliveraddedvalueoutcomesfor industrialenvironments.

Whileinthefigureoutcomesarehighlightedforshopfloor, op-erations, and management, HIL integration has broader potential inbringingtherangeofhumancognitivecapabilitiesinthewhole production and product lifecycle management activities, joining thepowerofCPSentities withproduction,operational,and infor-mation technologies. Forsuch a potential to be realised, a range ofissuesinintegratingindividual humancapabilitieswithin tech-nical systems, still need to be resolved. Forexample, the lackof sufficiently flexible interaction interfaces,despite many advances attheperceptionandrecognitionlevel,isstillasignificanthurdle. Recent studies have shown the dominance of intuitive cognition over proper reasoning and decision making, a finding that calls forfurtherresearch inhuman-automationinteractioninindustrial settings (Patterson, 2017). To this end, recent efforts focused on modellinghumanactivitieswithinCPSsystems(Fantini,Pinzone,& Taisch,2018),identifyingspecificchallengeson:(a)understanding andcontrollingtheinteractionbetweenworkersandCPS entities; (b)howtocapturetheaddedvalueofhuman-contributedactivity; and(c)howtotake intoaccountandmatchaspecific situational contextwithskillsandcharacteristicsofworkers.Context Informa-tion Management is a key concept aimed atsituation awareness andis thereforeappropriate foraddressing theabove highlighted challenges.

CPSaswellasproductandassetlifecyclemanagementactivities increasinglygenerateaverysigniﬁcantamountofdata.While au-tomated dataanalyticsexpectationsare high,there arestillmany situations wherein placing the HIL of analytics is highly beneﬁ-cial.Enhancinghuman cognitivecapabilitiesinanalysing datavia relevant softwaretoolsis linkedto theconcept of“the humanis theloop” (Endertetal.,2014),indicating thevaluablerole of

hu-mananalystsinintegratingtheircognitiveabilitieswhen interact-ing withvisual analyticsenvironments. Aconceptual view ofHIL intheVisualAnalyticsprocessisintroducedinFig.4.

Inatypicalvisualanalyticsscenario,dataaremanaged, aggre-gated,andretrievedthrough adequatedatamanagementbusiness logic. Data visualisation options are offered through an analytics wizard, offering different data visualization and exploration op-tions and retrieving the mostrelevant ones. The user can inter-act with the analytics andvisualization tools to direct data pro-cessing,enrichdata,process themtoobtainprioritisationranking, andincaseofcriticalunexpected eventsbeingdetected,to issue relevantalerts, orotherwise routinely present summarised infor-mation with relevant dashboards. This process augments part of the cognitive processing cycle presented earlier, including recog-nitionandcategorisation, interaction,perceptionandsituation as-sessment,monitoringandprediction,aswellasdecisionmaking.

Havingintroduced thekey concepts ofhuman cognitive capa-bilitiesinsociotechnicalsystems,highlightingtheirroleinHIL-CPS andHumanintheVisualAnalyticsloop,thenextsectionpresents theimplementationofsuchconceptsonselectedapplicationcases. 4. Human-in-the-loopinproductionenvironments

4.1. IntroducingapplicationcasesforHILinproductionenvironments TheroleofHILishighlightedintwoapplicationcases,both rel-evant to production environments. Theyemploy condition moni-toring andthey involvesome commondesign elements, butina manner to serve different application needs. In that sense, they featurediversecontextandimplementationchoicesandthey aim to enhance joint sociotechnical capabilities in distinct ways.The ﬁrst case actively involves HIL for Linking Data and Knowledge Management and for enhancing machine learning capabilities in order to integrate operating stage technical systems (condition monitoring) withdesign stage knowledge (e.g.Failure Modes, Ef-fects, andCriticalityAnalysis – FMECA).The second caseengages HIL with visual analytics to communicate condition monitoring

(7)

Fig. 4. Human in the visual analytics loop.

outcomesin visually relevant waysforaugmenting human capa-bilitieswhen performing analytics-drivendecisiontasks. Thefirst caseillustrates the amplification of machine learningand condi-tionmonitoringcapabilitiesviaHIL,whilethesecondoneisacase whereinhuman capabilitieswhen interacting with technical sys-temsare themselves enhanced. Even though the directionofthe amplificationbetween technical andhuman capabilitiesis differ-ent,theultimateresultinbothexamples isa jointsociotechnical systemcapabilitiesenhancement.

Inbothcasesthetechnicalsystememploysvibrationcondition monitoring to associate extracted vibration signal parameters to machine or component condition. The ﬁrst case additionally employs machine learning to learn this association. The ability ofmachine learning to associate new patterns to conditions not yetcovered by historical dataor tobias learningon the basis of human expert knowledge is aided by human interaction. While blind data-driven machine learning might eventually learn such additional associations, this natural form of human intervention makesthelearningprocessmorefocused.The extractedvibration features in both cases were as in Katsouros et al. (2015) and comprised the signal RMS, skewness, kurtosis, shape factor, crest factor, peak value and impulse factor. A stream of data acquiredinthiswayisatimestampedsequenceofvectorsxi∈R7,

xi=[rmsi, ski,ki, sfi, cfi, pi, ifi]T withi=1,..., K,whereinthe

vector parameters correspond to the extractedfeatures fromthe measurement signal describedabove, and K denotes the number ofsamplesinadatastream.

4.2. Linkeddata,knowledge,andmachinelearning

Thedataandknowledgeprocessingchaininmaintenance prac-tice seeks to upgrade the added value of collecteddata to drive more effective evidencedriven decisionmaking. Event detection, diagnostics, prognostics, and decision support for maintenance have received much attention, but one of the areas where fur-therresearch isneededrelatestothe waysuch solutionscan en-ableandindeedbeneﬁtfromHILinteraction.Inthisexample,HIL concepts are employed to fuse automated data processing with human contributed knowledge in maintenance decision support (Pistoﬁdis et al., 2016). Evidence driven decision making in this wayisnotsolelydata-drivenorbasedonprerecorded knowledge butisenriched withmechanisms thatbring together humanand non-human actors in a waythat linksdata, knowledge and ma-chinelearning(Fig.5).

The application environment is that of an industrial produc-tion facility for lifts and an actual in-service operating installa-tion.Thisisarepresentativecaseofahydraulicliftthathasavery widebaseofresidentialandoﬃceinstallations.Preventive mainte-nanceisperformedonamonthlybasistodetectfailureeventsand driverecommendedmaintenanceactions.Vibrationsensorsare po-sitionedintwoareasofinterestinaliftinstallation,namely,atthe bearingofthelift’sdrivemotorandattherollersofthelift’scabin. Theexperimentsfocusonanalysingthesignalsobtainedfromthe cabinrollingwheels toinfer assetconditionsoasto drive main-tenance actions recommendations. While automated approaches

(8)

Fig. 5. Architecture for Linked data, knowledge, and machine learning.

maysucceedwheredataaresuﬃcientlyrepresentativeofthe un-derlying knowledge, the proposed approach mitigatesrisks when thisisnot thecase. The approachisdemonstrated throughan e-maintenanceplatform,comprisingcomponentsformachine learn-ing, embeddedeventdetectionanddiagnostics, aswell aslinked maintenanceknowledgemanagementandactionrecommendation (Pistoﬁdisetal.,2012).

The objective of this case study is to show how integrating automated processing and human-contributed observations can be fusedwithestablishedreliabilitymaintenance constructs,such as FMECA, to expand the capabilities of either human actors or technical monitoring systems operating in isolation via a joint sociotechnical solution (Fig. 6). Furthermore, it extends machine learningcapabilitiesbyallowing HILto contributetothe learning process,enablingittobedrivennotonlybydatabutalsoby inte-gratinghumanexpertknowledge.

The design approach to HIL in this case is two-fold. First it aims at facilitatingmaintenance knowledge management, includ-ing sharing, enrichment, validation and extension of knowledge. Second,itaimsatamonitoringabstractionwhichcanbeemployed with machinelearning andHILto deliver customisable condition monitoringimplementations abletolearn notonly fromdata but alsofromHILinvolvement.Thisispresentedinmoredetailinthe nextsections.

4.2.1. HILforknowledgeenrichmentinmaintenancewithin production

Akeyaddressedchallengeistodesignametadatamanagement system(MMS)thatbindsthesemanticsofsensoreventswith ref-erencediagnosticsforfailure modes.The linkingprocess includes human experts in the loop, utilising and progressively enriching

thebackboneofanFMECAstudywithinputfromboththe techni-calsystemaswellasmaintenancepersonnel.Originatingfromthe smart sensors that populatethe edge IoT tier of thearchitecture (Fig.7),sensoreventsrepresentanasynchronousﬂowof informa-tionthat signalseventsof interest,which maycorrespondto the identiﬁcationofknownandnovelstates.

These events are handled by services that examine whether failureeventsarecurrentlylinkedwiththetriggeredstates. Appro-priateinputisthenproducedthroughtheuseofsemantic annota-tionsthatcharacterisefailureeventsaseither‘confirmed’or‘False Alarms’.Humanexpertsarealertedtointervenewhennovelevents oreventslinked tofailure modesaredetected. Bothprocesses,ie automated and human-triggered data tagging, produce semantic dataannotations, ie metadata.Overall, the produced timelines of semanticannotationsiscollaborativelydrivenjointlybysmart sen-sorsandhumanexperts(Fig.5),butthelatteroccuratadifferent time scale compared to automated alerts. Risk quantification for monitored machinery andinfrastructure is pursued through Risk Priority Number (RPN) evaluation, as in a typical FMECA study (RPN =Severity XOccurrence X Detection)buttheestimation is nowinfluencedbythejointactionofthetechnicalandhuman ac-torsofasociotechnicalsystem.

Themethodologyencourageshumaninvolvementtoprovide di-rect feedback and eventvalidation. This is achieved by simplify-inghumanfeedback,empoweringstaff to participatein knowledge ﬂowswithminimal,intuitiveandnaturalinterfaces.Suchaprocess ismorelikelyto securestaff participation,offering amore famil-iarinputpatternthathasbeenforyearsnowdrivingtheanalytics ofenterprisesocial networks(tags,‘likes’,andshortmessages).It isapatternthat seekstoaggregate alarge volumeofconcise in-puts intoa knowledge buildingprocess that invests incollecting

(9)

Fig. 6. HIL and cognitive capabilities in machine learning for decision making.

(10)

swiftevaluationsoverthevalidityandqualityofgiveninformation. In this approach,it is less importantto provide a higher quality startingknowledge(initialversion ofanFMECA study),andmore importantto providethecontext, thesemantics, andthetoolsto create a virtual collaboration infrastructure to assess its validity, acceptance and connection to practice. It is exactly this collabo-ration, driven by shortswiftevaluations (maintenancetags), that allowsthehumantoactivelycontributeinaknowledgevalidation loop. The FMECA studyis the reference knowledge baseline. But it is not static and its enrichment, evolution, and validation can be triggered asa product ofconsensus andfusion from a larger group oftechnical staff andexperts.Tosecurethe qualityof this input and furthermore manage the complexityof multiple simi-larcontributions,userrolesandvotesareemployed.Onlyspeciﬁc roles can facilitate tags andvotes, including expertsthat partici-pate intheFMECA reviewteam andmaintenance engineerswith strictly deﬁnedfunctiondomain.Everycontributionisloggedand when reviewed (FEMCA review)should be backed withevidence (conditionmonitoringhistory).ThisFMECAupdatecomprises cor-rections,proposedextensions,andbuildstheenrichedevidenceto backthemup.Itisnotautomatic butistriggeredbythejoint ac-tionofthetechnicalandhumanactors.

The design approach in this application case delivers a so-ciotechnical tool that provides accessto services that offer: (i) a shop-floor contextualisedFMECA study;(ii) a streamlined review collection process; (iii) and a filtered view of human and non-human actors’annotations.Current IoT andcloud – oriented pat-terns and technologies (micro-services, Node.JS, MongoDB) were employedinthetooldesignandimplementation,buttwospecific design aspects are highlighted next, as they define the two key perspectivesofhowHILfunctionsinthisapplicationcaseare:(a) supervisingandtrackingtheenrichmentandversioningof knowl-edgeand(b)encouragingpersonneltocollaborateandshare expe-rience andcriticalthinkingover workingpracticeexperience and establishedknowledge.

4.2.1.1. Data provenance. A timeline ofevents can reveal patterns thatimpactonmaintenancediagnosticsandriskassessment.Each entryinamaintenancesystemistime-stampedandallactionsare logged. Data Provenance refers to the ability to trace and verify data creation,and, in ourcase, failure evaluationsandimportant sensor events,in the formof maintenance metadata. Provenance ofsuchassessmentscanidentifypatternsthatdepictasset reliabil-ityandmayofferhintsforriskanalysis.Therefore,addinga meta-data layer on top of sensorand reliability data further enhances provenance by collecting the evidence of a validation loop. Pop-ulating thislayer and driving this loop, human observations and machine generated events,produce metadata,addingbackground knowledgeandevidencetosupportanFMECAreview.

4.2.1.2. Contextsharing. Socialnetworksorsocial-networkinspired features are increasingly included inenterprise communities and collaborative environments. Professionals are becoming familiar withsuch featuresinstandard socialnetworking contextandcan benefit fromtheir inclusion inenterprise environments, allowing themtooffertheirinputorannotatetheinputofothers,ata real-timemannerandwithmanysharingoptions.Morespecifically, in-formation and features such as‘voters’ and‘votes’ stimulate vir-tual interaction and conduct a social contextualisation of shared content.Insteadoflongformsforreporting,theproposed method-ology employs minimal input via customisable maintenance tags withvotingoptions. Thisis nosubstitute forFMECArevision de-cisions, butallows theaccumulationofevidenceandcapturingof observationsandknowledge frompersonnel.Ifnot enough meta-data are clustered around a specific failure event, a single ob-servation or sensorevent isless likelyto trigger the appropriate

Fig. 8. A causal semantic graph of events.

re-evaluationduringtheFMECA review process.Votes are an ex-tracontextsharingfeaturethatenablespersonneltoactively con-tribute to a crowdsourcing and sharing of observations, evalua-tions,andsensorevents.

Comprehendingthe role ofmetadataandannotations asunits of risk-oriented maintenance knowledge can be more effective withatoolthatmanagesthemontopofawidelyaccepted knowl-edgebackbone,namelytheFMECAstudy.Theswiftcapturingand sharing of such maintenance knowledge by personnel, together with data and events produced by the technical system, leads to an incrementally enriched version of FMECA. This isa princi-plesimilartocrowdsourcingintelligenceinrecommendersystems, wherebytheusers’collaborativecontributionisexploited. Leverag-ingupononeofthemostsigniﬁcantenterpriseassets,namelythe humanfactor, is the key to facilitatingmore effectiveknowledge ﬂowswithintheenterprise.

With increasing adoption, Linked Data have introduced for-malisation frameworks and technologies that can efficiently in-stantiateknowledgerepresentations.Suchframeworkscanemploy annotationfortaggingimportantcontent.Supportingtechnologies providemetadata contextualisation using widely established data constructs.Theamalgamationofrelevantknowledge,data,and an-notations define the Failure Context asthe confluence of factors contributing to the occurrence of a failure. In other words, the FailureContextholdsthecombinedknowledgerelevanttothe oc-currence of a failure mode and the assets-specific time-relevant feedback of maintenance practice. Building upon the established semantics ofan FMECA study,this designprovides the means to formalisingandinstantiatingthiscontext.Adoptingstructuresand components from the established MIMOSA schema,1 _the _FMECA

model is customised to empower the creation of an event map (Fig.8).

Thismap isa semantic graphwhereFailure Events act asthe corenodes. Implemented asa distinct set of semantics, Mainte-nanceTagsareusedtoannotatecoreorsupportingnodesandthus reportwhy,howandwhena failureeventoccurs.The initial ver-sionofthisgraphiscreatedastheproductoftheveryﬁrstFMECA studycompleted by the appropriate team ofexperts. As dictated bycommonpracticeinreliabilityengineeringandriskanalysis,an FMECA study is followed by scheduled reviews and evaluations. The sociotechnical systemapproach introduced in this paper en-ablesenhancedFMECAreviewandevolution,throughfeedinginto

(11)

itcollectiveevidencefromthetechnicalandhumansystemactors. Informationandknowledge aboutfailure modes become increas-ingly important,when human input (from maintenance practice) andtechnicaltriggers (fromthemachine learning– enabled con-dition monitoring system) repeatedly validate known and reveal newconnections (causalitylinks)betweenthemandother events andfailuremodes.Thecollectedinputisadistinctsecondlayerof knowledge,aboveanddirectlylinkedtoFMECA(metadata).FMECA reviewscontinuetobe themilestonesintheenrichmentprocess, butthey can now beneﬁtor even be triggered through this sec-ondlayer, anddriveinsights forthe correctionsandextension of thenext FMECA version, which remains the responsibility ofthe FMECAteam.

The introduced model utilises MIMOSAasa startingpoint for drawing a subset of its coresemantics. Specifically, in MIMOSA, failure events are denoted with the entity Hypothetical Event. In thenewmodel, theHypothetical Event profilehas beenextended withattributesthatrecordsemanticsofoccurrenceand detectabil-ity.Theseholda rankedevaluationforthe events’frequencyand detectionprobability.Alongwiththeinheritedpropertyforevent’s severity,thesescaledattributescandriveaRPN-basedevaluation. Furthermore,actingasthebuildingblockofagrowingeventmap, theintroducedevententitycapitalizesondistinctandwell-defined recursive attributes that link events with cause and effect asso-ciations.A “malfunction” and a “failuremode” are both types of theevententity.Failuremodeshavepopulatedcauses,effectsand solutionsthat areassociatedwiththem.Amalfunctionis primar-ilya simpleeventlinked toa failuremode.Thisisa processthat graduallybuildsaknowledgeinfrastructureforanAssetFaultTree. Traversingsucheventlinksmayfacilitatearootcauseanalysisand provideinsightsforriskassessment.

Whenever a user is prompt for an assessment, it is practical to provide a starting reference point, FMECA knowledge in this case. In the present design the referencing dynamics of ‘ mainte-nancetags’areemployedtocreatemetadatathatbindusers’ feed-back with FMECA knowledge. Instead of the simple string tags, commonlyused inthe context of semanticweb, the present de-sign employed class-types of review assessments, which acquire added value when coupled with FMECA content. Each tag has a straightforwarduseandannotationpurposethat isdefinedbyits tagtemplate.The defaultset oftagtemplates isconfigurableand extendable. Onlyspecific roles(e.g.roles withFMECAreview au-thorization/‘facilitator’)cancreate,modifyandadjustthetypeand purposeoftag templates. The abilityto extendandmap the en-richmentprocessoftenresidesintheskillsetofexpertsthatclearly understandthescope,depthandpurposeoftheannotated knowl-edge. An experienced engineer can be trained to translate new maintenance goals and policies into meaningful tags that create newactionablesemanticlinks.

A tag instance is the modeling entity for maintenance meta-data.Everyannotationactioncreatesataginstance.Eachinstance constitutesatimestampedunitofmaintenanceknowledge.Tag in-stancesaresharedandcanbesearchedorﬁlteredbyusers.Their knowledgecanbefurtherenrichedwithtagvotes(declaring agree-ment)andtagmini-forms(additionalfeedback).Thesearetermed asMaintenancemicro-Knowledge(Fig.9).

The enriched version of FMECAis essentially thecontent that is dynamically tagged and identified as contextually relevant to howrealeventsmanifestedandoccurred,andwhyspecific main-tenance solutions or diagnostic interpretations were reached. In this sociotechnical system design, maintenance tags can be pro-ducedbybothhumanandnon-humanactors.Incorporatingthese twoinformation flows withan FMECAstudyby translatingthem intobriefandaccuratereviewannotationsconstitutesasimple en-richment process that formulatesa growing pool ofmaintenance metadatathat isnativelyorganised andcollaborativelyevaluated.

Fig. 9. Maintenance micro - knowledge.

Thisfusedinformationpoolcontainingenrichedfailuremode pro-ﬁles and timelines of maintenance tags, can be consumed and analysedformismatches,correctionsandadditionstobe inserted inthenextversionofFMECA,orhelpdocumentandsupport crit-icalriskassessmentsandmaintenanceplans.

The metadata are instantiated and stored in a document-oriented database (Fig. 5), allowing the creation of structured, semi-structured, unstructuredandpolymorphic data.Its abilityto handle andquery massivevolumes ofnew andrapidly changing datatypesmeetsthedesigndecisiontoallowthecreationof cus-tom tags andencourage the collection of more and better orga-nizedhumaninput.Furthermore,theimplementationofthe back-endlogic isconsistent withmicro-servicespatterns,breaking the application logic into smaller modules, enabling better ﬂexibility and laying the groundwork forcloud compatibility. This applica-tioncaseimplementstheconceptofcapturingthetacitknowledge relevant tomaintenance practiceand risk assessment,while also acting and planning upon maintenance events that can support maintenance intelligence.Itcapturesandtransformsmaintenance expertiseintoknowledgefragmentsthatinstantlylinkbackground reliabilityknowledgetoeverydaypractice,andcrowdsourcesdata, informationandknowledgefromhumanandtechnicalactors. 4.2.2. HILinmachinelearningloopforassetself-awareness

Machinelearningistypically consideredasan automated pro-cessdrivingdecisionmaking.However, data-drivenlearningis of-tenineﬃcientincomplextaskswithpoorlyrepresentativedata. In-corporatingHILinmachinelearningisrarelyconsidered although it can make a realimpact on real world applications, such asin production environments. The design of the HIL solution in the presentapplication caseconsiders a machine learning infrastruc-turedistributedamongtheedgeandthewebservicelevel(Fig.5). The web service levelinvolves operations neededfor learningto model the associations between signal features and asset condi-tions and the management of all relevant operations to manage

(12)

Fig. 10. Incremental learning of process states.

Fig. 11. Association of failure modes with Gaussian states.

suchprocessesandcommunicateresults.Theseoperationsemploy machine learning, inclusive off ahuman feedback loop forstates detectedasnovelorforvalidation(Fig.10).Theedge implementa-tionincludessensordataacquisition,featureextraction,aswellas noveltydetectionandclassiﬁcationofsenseddatatoasset condi-tionsandisdescribedinmoredetailinthenextsection.

Numerous approacheshave beendeveloped tomodel changes in system states, identify current states,and predict future ones fordifferentprognosticspurposes(Leeatal.,2014).The implemen-tation in the present research adopted the parametric family of Gaussian MixtureModels(GMM)formodellingthefailure modes. GMMs belong to the parametric family of multivariate Gaussian densityfunction, whichcan be combinedwithBayesian statistics tomakestatisticalinferencesregardingthefailuremodes.

GMMscan beutilised toapproximateanarbitrarydistribution within anarbitraryaccuracy.FortheGaussiandensityfunction,it suﬃces to calculate from the feature data points the mean vec-tor andthe covariancematrix.In orderto keep the implementa-tion simple, the choice madeis to relate each failure mode to a numberofGaussiancomponents/statesthatresultfromthe train-ingdatacollectedduringtheoperationofthesystem.Fig.11shows a user interface from the implementation example wherea user can effect the above association. This view displays the associa-tionofafailuremodewithasetofmodelstatesfromspeciﬁc

sen-Fig. 12. Expert conﬁrmation of sensor driven annotation.

sors.Thisassociation isperformedoff line andfailure eventscan beproﬁled byexpertsandlinked withstates.Sensoreventsfrom theedge node,can be processedandfailure eventslinked to the triggeredstatesaretaggedandconﬁrmed.Suchsensoreventsand humaninputproducemetadataandcanserveasexamplepatterns formachinelearning.Ratherthanrelyingonblinddata-drivenonly learning, this effectictely constitutesan incremental, HIL-enabled learningprocess.

For the calculation of the covariance matrix the Minimum Covariance Determinant (MCD) estimator is employed, which is amongtherobustestimatorsofadataset’scovariance(Rousseeuw &Leroy,1987).Intheexperimentalsettingthefeaturevectorsare setsofstatisticaltimeseriesparameterscalculatedonshifted win-dowsover the sensorialdata stream (Katsouros etal., 2015). The bootstrapofthesystemisbasedonamodelofthenormal opera-tionmode ofeachasset, whichistrainedfromfeaturesequences that have been collected from the sensor level. If there is prior knowledge of sensorial features’ association with failure modes then such associations can also be included in the initialisation ofthe system. The parameters ofthe Gaussian models are com-municateddown tothe sensorlevel. Theembeddedalgorithm at theedgenodecalculatesthefeaturesequencesandtheirdegreeof classiﬁcationtoeachoftheknownstates/modes.Featuresthat can-notbeclassiﬁedtoanyoftheknownstatesaremarkedasnovel.

For states which are defined to be determined at the sensor node, theembeddedclassifier assigns readingsto statesfor non-noveldata.Thisisdoneby calculatinganoveralldegreeof classi-fication,applyingaBayesianapproachusingtheindependence as-sumption forthe feature data points and weighting the product oftheprobabilities withtheaprioriprobability ofeachstate.For morecomplexstatesthatrequiredatafrommultiplesensornodes, theclassificationofthefeaturesequencesiscommunicatedtothe relevantwebservice, whichassignsreadingsto classesina simi-larmanner,butformultiplefeaturesetsfromdifferentnodes.The featuresequencesthat belongtostatesoffailure modeswill trig-geralerts,drawing attention forhuman intervention. Inthe case where the eventis relatedto an existing failure mode, the web servicesusemaintenance tags toreport it andthe humanexpert mayverifytheeventorraisedoubtsaboutit,potentiallyasafalse alarmnotrelatedtothefailuremode(Fig.12).

Maintenanceevents oralertscanbe issuedby technical (non-human)orhuman actors(Fig.5).Alerts relatedtodetected novel eventscorrespond to events that are not classiﬁed in anyof the knownstates,ienormalconditionsorfailuremodesforwhich rep-resentativedataareavailable.Such noveldataarestoredinorder to be examined at a later stage by a human expert. These may

(13)

representa newstate that isextendingone ofthe known condi-tions.Alternatively,they canberelatedto anewfailuremode. In thelattercasethehumanexperthastoupdatetheknowledgebase withthe new failure mode and relate thismode to the relevant sampledata,soasGaussianmodelscanbebuiltonthebasisofthe relatedfeaturesequences.Thisisanexampleofmachinelearning whereinhumaninputisasigniﬁcant partofthelearningprocess, playingarolethatisveryhardtobereplicatedbyautomated ma-chinelearning. Such HILinthemachine learningloop can there-forehavea positive impacton enhancing datavalue chains from theedge to the cloud. The next section presents an edge imple-mentation that beneﬁts from and makes use of HIL in machine learning.

4.2.3. HILforIoT-driveneventdetection

The edge implementation for IoT-driven event detection is directlybeneﬁttingfromtheHILofthe machinelearningprocess describedin the previous section, as the human interaction en-ables the association ofsensed data andfeatures to states.State models are then downloaded to the embedded device to drive edge analytics for event detection and diagnostics. The parts of the machine learning for novelty detection and diagnostics that have been assigned to execute at the edge of the system, close to the monitored resources, were implemented using a range of embeddeddevices of different capabilities, interconnected under a wireless sensor network of heterogeneous IoT technologies. HIL contributed knowledge, such as the association of failure modes and machine states is effectively driving the embedded nodeoperation.Theoverallimplementationwasintegratedwithin an e-maintenance platform (Pistoﬁdis, Koulamas, Karampatzakis, Papathanassiou,&Centre,2012),whilethealgorithmic implemen-tation described in the previous section (Katsouros et al., 2015) can triggerproduction process adaptation driven by IoT-captured failureevents (Alexakos,Anagnostopoulos,Fournaris, Kalogeras,& Koulamas,2017).

Theimplementationoverdiversehardwareplatformsand tech-nologies is indicative of the wide range abstraction possibilities, makingtheapproachappropriateforservingverydiverse require-ments.The versatilityof theapproach isalsoevident via the ap-plicationlayerconnectivity,whichisofferedthroughRESTfulWeb Services,2 _implemented _using _JSON3 _over _HTTP4 _or _CBOR5 _over

CoAP.6_These_services_allow_accessing_the_sensor_nodes_directly_or

throughtheappropriate gateways,dependingonthe sensornode capabilities.Allsensornodesexecutesampling,storage,feature ex-traction, classiﬁcation and novelty detection services, with their operationexternallycontrolledthroughawell-structuredsetof re-sourceURIs,accessed through HTTPorCoAP GET/POST/PUTREST operations.Keyresourcesinclude:

• /info: This is a set of implementation dependent non con-ﬁgurable parameters (basic buffer size, maximum number of buffers, the maximumnumberof slidingwindows forthe ac-quired signal processing, supported sampling ratesand maxi-mumnumberofsupportedstatedescriptions).

• /conﬁg: The novelty detection engine conﬁgurableparameters (sampling rate, sliding windows size andstep, samplescaling factor,monitoringperiod).

• /stateset: A set of conﬁgurable parameters, ie state descrip-tionsandthresholdsoverwhichthedistancesofthecalculated

2_{https://en.wikipedia.org/wiki/Representational _ state _ transfer}_. 3_{https://en.wikipedia.org/wiki/JSON}_.

4 _{https://en.wikipedia.org/wiki/Hypertext _ Transfer _ Protocol}_. 5 _{https://en.wikipedia.org/wiki/CBOR}_.

6 _{https://en.wikipedia.org/wiki/Constrained _ Application _ Protocol}_.

feature matricesare compared in orderto generate a novelty event; these can be ‘learned’ following the previous section’ processanddownloadedtoedgedevice.

• /event:This is the observable resource of the last event trig-geredby the node, modelledthrough a structure that is sent totheuppersoftwarelayers,periodically,on-demandorwhen analertistriggeredbythenoveltydetectionmechanism.It en-capsulatesthewholefeature-extraction,noveltydetection,and classiﬁcationprocesschain.Thenoveltydetection implementa-tioncaninstantiateandselectivelyreturneitheronlyaBoolean resultofa novelstate, that is acalculation window with fea-tures distance higher than the conﬁgured threshold from a knownstate;or,additionally,allfeaturevaluesforthewindow andthedistancesfromallknownstates,quantifying dissimilar-ity.

According to the application requirements, there are different measurement classes in terms of sensing elements and relevant qualities.Low-endspecificationcorrespondtosimplescalarvalues fromtemperaturesensorswithlimitedrequirementsforsampling bandwidthandprocessing.Thesecanbesufficientlysupported by resource constraintembeddeddevicesand low bandwidth proto-cols for IoT networking. At the high-end there are high quality, multiaxialvibrationmonitoringrequirementsthatmayposehigher power and processing resources requirements, so as to manage morecomplexdatastreamsandinterfacingtoindustrialgradeIEPE sensors. In between, there canbe mid-range nodesthat can still handleseries andvector measurements butwithlower sampling ratesandaccuracyneeds.Thesenodesmaybesupportedby mid-range hardware regarding analogue and digital data acquisition, processingpower, flash andRAM space fordataacquisition, pro-cessing,storage andradio communicationtransmission.The edge design abstractsand supportsall the above requirements specifi-cationsandabroadrangeofheterogeneoussensornodeswas im-plemented,coveringthewholecapabilitiesspectrum.

Speciﬁcally,atthelowestend, theembeddeddetectionengine was realized with off-the-self hardware components and widely usedIoToperatingsystemssuchastheTelosB/TinyOSplatform,the NXPJennicplatform overitsownAPI andovertheContiki OS,as well asthePrismaSense developmentkitplatform andAPI,using ZigBee, 6LoWPAN andraw IEEE802.15.4 protocol stacks. In these cases,theRESTresourceswereaccessedthroughcustomgateways translatingJSON/HTTPrequests intobinarycommandstransferred overthe aforementionedwireless protocols,asdevice capabilities couldnotsupportcomplexapplicationlayerprotocols.Atthe mid-rangelevel,aspecialresourceconstraintembeddednodewas cre-atedbasedonatwo processorboardsystemstructure,separating theapplicationfromthecommunicationprocessor,coupledwitha numberofexchangeablesensorinterfaceboards.

Thesampling, signal processinganddetectioncomponents ex-ecute on a Freescale FRDMK64F embedded board, while a full IoTnetworkingstack,basedonanIEEE-802.15.4wirelessinterface andthe IPv6, RPL7 _CoAP_and_CBOR _components,_is _implemented

onthe CC2538based Openmoteboard(Fig. 13).These nodescan be accessed either directly through CoAP or indirectly though the HTTP/CoAP gateway,according to the resource URL structure in the conﬁguration database, separating the access (“http://” or “coap://”), node FQDN8 _and _resource _path. _Finally, _the _high _end

vector samplingrequirements andthe supportofindustrialgrade IEPEvibrationsensors havebeencovered byasensornode based on the DT9837B USB acquisition system from Data Translation, controlled by an embedded PC or tablet device which provides

7 _{https://tools.ietf.org/html/rfc6550}_.

(14)

Fig. 13. Mid-range Embedded vibration monitoring with event detection capabili- ties.

Fig. 14. High-end embedded vibration monitoring node with user interface.

the developed interfaces to the IIoT infrastructure,directly using HTTP/JSON(Fig.14).

ThisIoTedgenode implementationenablesamonitoringasset toexhibitself-awarenessfeatures andissueeventalertswhenthe processedsensor readingsareidentiﬁed eitherasnoveloras be-longingtofailuremodestates.Itispartofamonitoring infrastruc-turethat includes HILinthe deﬁnitionorvalidation ofthe asso-ciationofreferencereadingswithassetconditions.ThisHIL inter-action leads to an incremental machine learning approach, grad-ually building a more complete learned model over time. While partofthisapplicationexamplefocusedonHILinmachine learn-ing for assetself-awareness andmonitoring, thenext application caseutilisesmonitoringinadifferentcontext,linkingmaintenance processesinaPLMcontextwithvisualanalytics.

4.3. IoTenabledvisualanalyticsforlinkedmaintenanceandPLM PLMtoolsalreadyoffercontext-adaptedproductviewsenabling ausertoview productrepresentations,data,andinformation rel-evant todifferentlifecyclephase activities.However,user interac-tioninthedataanalysisloop couldfurtherbeneﬁtfromdue con-sideration of human cognitive capabilities to ensure a close in-tegration of HIL in PLM activities. Visual analytics embed visual semantics in data representations, thereby employing simple but cognitively powerfulmeans tobetterengagethehumancognitive capabilities.InPLMactivities,auserdoesnotsimplyneedtoshare

productlifecycledata,butwouldbeneﬁtfromdoingsoviavisually enrichedproductviews,whichisthefocusoftheapplicationcase presentedinthissection.

4.3.1. HILforIoTdrivenvisualanalytics

Consideringthat themostuser-friendlyproductrepresentation is a 3D product model, thekey idea isto employ such a design – stage product representation together with MoL product infor-mation,relatedtoproductconditionmonitoring.Bysuperimposing MoLrelevantproductinformationtoBoLproductviews,suchasa 3D product CADmodel, linked maintenance data andknowledge (Pistoﬁdisetal.,2016)becomevisualfeatures ofa productdesign representation,facilitatingauser’sunderstandingofMoLconcepts, such as the occurrence of failure modes, within a design view-point.Therefore,this3Dvisualizationbecomesanaturalextension ofstandardanalyticsformonitoringdata,includinggraphsof sen-sor readingsand signal features, such as time domain and spec-tralfeatures.Thisconceptofblendeddigitalproductvisual analyt-icswasappliedtodesignalaboratorybaseddemonstratorfor IoT-drivenvisualanalytics.Thedemonstratorwasdevelopedona me-chanical transmissionrig,comprisinga lower shaft withfour 42-toothgears,drivenbyamotor,andanuppershaftwithonelarger 62-toothgear,whichis drivenby theﬁrstshaft throughmeshing the uppershaft gear withany ofthe lower shaft gears (Fig. 15). Loadingconditions canbe adjusted withabrake, attachedto the uppershaft,while therotational speedis controlled byadjusting themotor speed.The lower shaft gears are initially identicalbut defectsareintroducedtogears1–3,whilekeepingonegearin nor-malcondition forreference. The defects are intended to emulate pitting,growingfromsmallerscaleongear1toalevelconsistent withextensivespalling,causingtoothpiecestofallapart(Fig.16).

The aim was to produce an instantiation of the concept of linked knowledge in maintenance and PLM, with knowledge su-perpositionto product views. Therig wasretroﬁttedwitha sim-ple and inexpensive IoT monitoring arrangement anda software demonstratorwasdevelopedtooffervisualanalyticsfeatures.The aimis tohighlight some ofthe possibilitiesforthe ampliﬁcation of cognitive abilities, which can be pursued by integrating this type of visual analyticswith more conventional condition moni-toringand PLMactivities, assummarisedin Fig.17.Forexample, interactionandcommunicationcapabilitiesareofferedthrough vi-sual interfaces.Monitoring outcomes are communicated both via a 3D asset representation aswell as standard signal graphs.The

(15)

Fig. 16. Defect introduction.

understandingofthesituationbytheuserisaidednotonlybythe conventionalvisual representationsof monitored signalsbutalso bysuperimposingtheoutcomeofadetectionmechanismtoa3D productrepresentationandbyhighlightingindifferentcolourparts thatare consideredto be developingafault conditionand there-foreshouldbesubjecttofurtherattention.Thismakesinteraction andfocusofattentionmoreintuitivefortheuser.

4.3.2. DemonstratorofIoT-drivenvisualanalytics

A low cost IoT – enabled monitoring solution, implementing dataacquisitionandbasicdiagnostics,wasintroducedforthis ap-plicationcase.Ratherthandevelopingathoroughengineered solu-tion,thedemonstrationobjectivefocusedoninstantiatingthebasic dataprocess chainfortheblendeddigitalproductvisual analytics concept.Thisprocesschaincomprises

– datagenerationprocess,viaaprototypedataacquisition. – adataprocessingstage,whereinacquireddataareconvertedto

monitoringparameters.

– abasicdiagnosticstage,whereinacquiredparametersare trans-latedintoassetconditions.

Fig. 18. Experimental setup arrangement on the gearbox test rig.

– blendedvisual analytics, jointlyhandling MoLdata (e.g. diag-nostics)withBoL(3Dproductmodel)productviews.

The datageneration process wasimplementedthrough an Ar-duino UNO board and two MPU 6050 accelerometers to capture gearboxvibration(Fig.18).Whilethisisnotasufficientsetupfor an industriallyrelevant solution,it isadequate fordemonstrating theproposedconceptandwasselectedforthispurpose.Thedata processing stage wasimplemented on a Raspberry Pi 3 Model B boardonPython,employingtheSciPylibrary.Thisincludedsignal averaging and extraction of standard statistical parameters from theaccelerationsignal asmentioned inSection4.1,forminga se-quence of measurement vectors. A Fast Fourier Transform (FFT) representationof thevibration signalis alsocalculated onboard, afteradequatefilteringandwindowing.Thefocusinthisexample isnotspecificallyontheconditionmonitoringfunctionalitybuton incorporatingthediagnosticoutputs inanenvironmentoffering a visual analytics view ofthe product. Any other monitoring setup canbeincorporatedinstead.

(16)

Fig. 19. Visual analytics example from the demo application.

Fig. 20. Visual communication of measurement locations and diagnosed failure modes.

Reference data acquisition experiments were performed with different gear coupling setups, starting with gear without de-fects to obtain reference data from normal operating condition. Data were acquiredwitheach oneofthe other lowershaft gears coupled withthe uppershaft gear,to obtain representative sam-ples fromagradual faultprogression.Thedifference between ref-erencesamplesfromnormalandprogressingfaultconditionswere employedtosetsimplethresholdlevelsforeachoneofthe vibra-tionparameters todistinguishbetweendifferentconditions.More advancedsignalprocessingandpatternrecognitiontechniquescan beemployedinstead.However,thefocusinthisexampleison of-fering a visual analytics view of the product, based on the data processingchainandnottheexactsignalandpatternanalysis.

The visualisation application was developed in the Processing environment (processing.org),an Open SourceDevelopment Envi-ronment for Interactive Visualisation. The application presents a range ofoptions for interactive visualisation. The application can producereportsandvisualanalyticsgraphsfortherawsignal,the measured parameters andthe FFT ofthe rawvibration signal, as wellasmotortemperature(Fig.19).

The comparison of threshold values estimatedfrom reference data and parameters extracted from subsequent observations is passedto the visualization layer of the application. This offers a 3D model of the test rig highlighting visual features by colours, conveying contextual meaning. Forexample, sensor locationsare marked in blue colour. The diagnosis outcome is communicated bysuperimposingfaultconditionsfeaturesonthe3DCADproduct representation,whereinmechanicalcomponentsarehighlightedin red toindicate faultycondition. Such visual features can be seen inan example screencaptured fromthe visualisationapplication (Fig.20).

Typicalmonitoring systemsalready conveymeasurement data andfaults tousers.However, blendingvisualfeatures in3D prod-uct representations offers an additional HIL option, further aid-ing a user to interact with product relevant data in a way rele-vanttonon-monitoringcontexts,suchaswhenreviewinghistorical dataandFMECAknowledge(Pistoﬁdisetal.,2016).Inthis applica-tioncasea userinteractingwiththeapplicationisable toaccess maintenancelinkedknowledgewhichisnaturallymoreactionable, asit is shared in a contextually relevant way. As an example, a