berckmansPP

(1)

Online monitoring van

levende organismen

Prof. Dr. Ir. Daniel Berckmans

M3-BIORES

K.U.Leuven

Lessen voor de 21e eeuw - March 8 2010

Van intensive care tot wielrennen en

formule 1

(2)

Head: Daniel Berckmans Prof J. Aerts Ir K. Van Loon Ir G. De Bruyne Ir T. Leroy Ir J. Lefever Ir. S. A. Haredasht Ir. F. Borgonovo Dr V. Exadaktylos Ir A. Bulckaert Ir M. Silva Ir A. Aydin Dr. S. Ferrari Secretary W. Meulemans Technical Assistance Ing J. Lemaire L. Happaerts Private Companies BIORICS N.V. Ir F. Jansen Ir J. Bellinckx Ing L. Wollants Ing M. Milutinovic A. Goeman Dr C. Bahr Prof E. Vranken Ir O. Cangar Ir A. Pluk Ir A. Poursaberi Ir E. Bites Romanini Ir. Y Ir S. Eren Ozcan Ir N. Alban Ir S. De Boodt Ir A. Youssef Ir. X

(3)

Overview

• What is a bioresponse?

• Methodology

• A living organism is a CITD – system

• Modelling bioresponses

• Examples / Results

- Monitoring Bioresponses

- Controlling Bioresponses

• Applications in sports

(4)

(5)

Bioresponses

Micro-environment

Laboratory for Agricultural Buildings Research, K.U.Leuven, Belgium

(6)

Block diagram of a system

System

Process

Disturbance variables (Environment)

Input variables

(t)

Output variables

(t)

(Environment)

(t)

bio

system

bioresponses

(7)

Active monitoring/control of a

biological response??

(8)

DESIRED DIRECTION

PREDICTION

2

Direction Position and balance

on the board PREDICTION-BASED CONTROLLER

Methodology

1

FEEDBACK MEASURE MEASURE

3

(9)

(10)

PREDICTION

2

MEASURE

DESIRED DIRECTION

Modern Process Control e.g. Automatic Pilot

Direction Process MANAGE

1

FEEDBACK MEASURE Steering gear

3

(11)

DYNAMIC BIORESPONSE MICRO-ENVIRONMENT

Process

1

FEEDBACK PREDICTION MODEL

2

DESIRED PROCESS OUTPUT

MONITOR/ REGELAAR

3

MEASURE MEASURE

1991: Modern control theory applicable to

living organisms?

(12)

A living organism is a

CITD system

(13)

A living organism:

Complex

(14)

A living organism:

Complex

(15)

Complex Individual Hea rt bea t (bpm) Time (s)

A living organism:

(16)

(17)

Complex Individual Time-Varying

A living organism:

TIME (HOURS) 0 1 2 3 4 5 HEA T PR OD UCTION (W /KG) 11 12 13 14 15 16 17 MEASURED MODELLED (1ST ORDER) MODELLED (2ND ORDER)

5 days old

TIME (HOURS) 0 1 2 3 4 5 HE A T P RODU CT ION ( W /K G) 7 8 9 10 11 12 13 MEASURED MODELLED (1ST ORDER)

30 days old

Example: Heat production of broiler chickens

1 0 1 1 2 0 1 2

( )

b

b z

y k

a

a z

0 1 0 1

( )

b

y k

a

a z

(18)

Complex Individual Time-Varying Dynamic

A living organism:

1. Measure

2. Model

3. Manage

In an on-line way

Living organism =

CITD

- system

Complex Individual Time-Varying Dynamic

(19)

(20)

Output variables

Swimming

activity

Light intensity

PH

Water temperature

Oxygen concentration

Input variables

Disturbances

Age, Cu-concentration

Example: Daphnia-monitor

*

(21)

Daphnia TIME (SECONDS) 0 20 40 60 80 100 VE RT ICAL PO SIT IO N (M M ) 0 10 20 30 40 50 60 70 MEASURED MODELLED (1ST ORDER) MODELLED (2ND ORDER) 1 2 4 5 6 7 8 9 10 3 10

(22)

(23)

Example: monitoring behaviour of laying hens*

Battery cages Furnished cages

(24)

Quad unit

Digital recorder

Experiments:

– 18 animals, 1 hour of recording each

– 4 Camera‟s on top of the compartments + digital video recorder

– Audio-visual scoring of behaviour by ethologist as reference

(25)

(26)

• Features:

– Position

– Orientation

– Shape

• Experiments:

– 4 Camera’s

– 18 hours of real-time video

– Audio-visual scoring of

behaviour by ethologist as

reference method

(27)

1 2 3 4 5 6 7 8 1.8 1.9 2 2.1 2.2 2.3 1 2 3 4 5 6 7 8 30 32 34 36 38 1 2 3 4 5 6 7 8 14.5 15 15.5 16 16.5 17 1 2 3 4 5 6 7 8 time Video input Image processing Posture parameters p₁ p_m p₂ …

(28)

1 2 3 4 5 6 7 8 1.8 1.9 2 2.1 2.2 2.3 1 2 3 4 5 6 7 8 30 32 34 36 38 1 2 3 4 5 6 7 8 14.5 15 15.5 16 16.5 17 1 2 3 4 5 6 7 8 Video input Image processing Posture parameters time p₁ p_m p₂ …

(29)

(30)

(31)

Step 2: dynamic modelling

Fitting a mathematical model to the posture

parameters in each time window

For example: modelling scratching behaviour

time p₁ „scratching‟

posture parameters

p1,…,pm

p

₁

[k] = d

₁

* u[k] - d

₁

* p

₁

[k-1] -

… - d

₁

* p

₁

[k-n+1]

dynamic parameters:

d1,…, dn

(32)

1 2 3 4 5 6 7 8 1.8 1.9 2 2.1 2.2 2.3 1 2 3 4 5 6 7 8 30 32 34 36 38 1 2 3 4 5 6 7 8 14.5 15 15.5 16 16.5 17 1 2 3 4 5 6 7 8 Posture parameters Dynamic modelling Dynamic parameters 1 2 3 4 5 6 7 8 -1.3 -1.2 -1.1-1 -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 1 2 3 4 5 6 7 8 -0.4 -0.2 0 0.2 0.4 0.6 0.8 time p₁ p_m p₂ … d_n d₁ …

(33)

1 2 3 4 5 6 7 8 1.8 1.9 2 2.1 2.2 2.3 1 2 3 4 5 6 7 8 30 32 34 36 38 1 2 3 4 5 6 7 8 14.5 15 15.5 16 16.5 17 1 2 3 4 5 6 7 8 Posture parameters Dynamic modelling Dynamic parameters 1 2 3 4 5 6 7 8 -1.3 -1.2 -1.1-1 -0.9 -0.8 -0.7 -0.6 -0.5 -0.4 1 2 3 4 5 6 7 8 -0.4 -0.2 0 0.2 0.4 0.6 0.8 time p₁ p_m p₂ … d_n d₁ … „scratching‟

(34)

Classification

Compare dynamic parameters to pre-learned

bounding box

– Result: 1 if behaviour occurs, 0 else

d_l d_k

(35)

10 hours of real-time video data

Today, standing, walking, scratching behaviour, producing

an egg can be classified in an on-line way

(36)

Example: model-based computer vision

**of laboratory mice(*)**

• 135 Swiss mice, 10 min. recording each

• Open field test setup:

(*) Collaboration with Laboratory of Biological Psychology (Prof. R. D’Hooge), Leuven 26 cm

53 cm 34.5 cm

(37)

Walk model fit to real video images

total model output

param. 1

param.

values

(38)

(39)

Model fit to walk patterns

• Reference mouse:

• Drugged mouse (injected with pentobarbital):

(40)

Calving monitor

(41)

Example: Calving monitor for cows*

 X and Y coordinates of the

centre point

 Orientation (degrees)  Body width/length ratio

 Hip length (m)

 Back area (m2₎

 Walking trajectory

 Distance walked (m) (*) In collaboration with TEAGASC (Ireland)

(42)

(43)

Example: lameness detection

*

(44)

Sleepiness monitoring

*

(45)

MODEL 2 1 MODEL-BASED PREDICTIVE MONITOR - Heart rate - Biorhythm - Heat balance - Driving performance Driver sleepiness Signs of sleepiness MEASURE

Driver sleepiness detection & prediction

based on continuous measurement of bio-responses from the driver’s body

(46)

On-line Pig Sound Analysis

*

(47)

Pig Cough Sounds*

Example Healthy cough sound Example Sick cough sound

RESPIRATORY PATHOLOGIES IN PIG FARMS Mortality, Production Use of antibiotics Am plitu de (dB ) F req ue nc y (Hz ) Time (s) A C 77777777 B Time (s) 0 0,2 0,2 0,4 0,4 1000 1500 0 500 Am plitu de (dB ) F req ue nc y (Hz ) Time (s) A C 77777777 B A C 7777777777777777 B Time (s) 0 0,2 0,2 0,4 0,4 1000 1500 0 500

(48)

Pig Cough Localization

Cough hazards Using the difference in time arrival between

several microphones in a stable, the location of the cough sounds was determined

Silva et al. Computers and Electronics in

(49)

Climate controller V(t) T Q(t) Antibiotics sound Therapeutic decision infection Sound analysis micro

Main future application: Reducing the

use of Antibiotics

(50)

• Chronic Obstructive

Pulmonary Disease (COPD)

• More than 680 000 patients

(40+) in Belgium

Cough is more than just a sound

(51)

Monitoring the health status of individual critically ill patients on the basis of on-line modelling approaches

Example : Intensive Care

Project together with UZ Leuven (Prof. G. Van den Berghe) and Computer Sciences (KULeuven, Prof. M. Bruynooghe)

Treatment

_Health

(52)

Patient 728 (WBC) Time (days) 0 5 10 15 20 25 30 35 40 a-parameter -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5

Survivor (intensive insuline treatment)

Trained on 141 patients, validated on 58 patients

Stationarity criterion: a > -1

Patient 1276 (WBC) Time (days) 0 5 10 15 20 25 30 35 a-parameter -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5

(53)

> 16h 8 -16h 0 – 8h

10/12 patients correct

Moment of extubation

(54)

(55)

DYNAMIC BIORESPONSE MICRO-ENVIRONMENT

Process

1

FEEDBACK PREDICTION MODEL

2

DESIRED PROCESS OUTPUT

MONITOR/ REGELAAR

3

MEASURE MEASURE

1991: Modern control theory applicable to

living organisms?

(56)

Example : Control of the crawl trajectory of

larvae of Calliphora vicina

(57)

t t y(t)

u(t)

Response of crawl direction to variations in ligth

Light intensity

crawl trajectory

(58)

-45.00 0.00 45.00 90.00 135.00 180.00 225.00 270.00 315.00 360.00 0 3 6 9 12 15 18 21 24 27 30 33 36 Tim e C ra w l d ir e c ti o n ( °) 270.00 180.00 12.55 315.00 270.00 9.72 45.00 90.00 13.59 90.00 180.00 15.77 225.00 90.00 12.77 135.00 270.00 9.60 0.00 0.00 15.60 180.00 0.00 15.61

Responses of crawl direction to steps in light

direction

(59)

t t y(t)

u(t)

Response of crawl direction to variations in ligth

Light intensity

crawl trajectory

MEASURE MEASURE a₁a₂ … a_n b₀ b₁ … b_m A(z-1_{) = 1+a} 1z-1+ a2 z-2+ … + anz-n B(z-1_{) = b} 0 + b1z-1+ … + bmz-m

)

(

)

(

)

(

)

(

)

(

₁ 1

k

d

k

u

z

A

z

B

k

(60)

MODEL

Light intensity

crawl trajectory

FEEDBACK

MODEL-BASED CONTROLLER

Active control of crawl direction of larvae

(61)

Active control of 2D crawl direction of 6 individual

larvae

(62)

X

Y Z

(63)

(64)

(65)

Running speed

_{Heart rate}

Example : optimisation of physical

training of horses

v

(66)

Heart rate (bpm)

Sensors on the horse

Speed (km/h)

GPS

(Garmin Forerunner 205)

Heart rate monitor (Polar S610i)

(67)

Experimental design

(68)

Methods and Results

Overview of the experiments

Pre-experimental

work

25 Accuracy of the

equipment

4 Step experiments

45 5 horses x 3 riders

x 3 repetitions

Controlling

experiments

6 2 horses x 1 riders

x 3 repetitions

TOTAL

80

(69)

Velocity and Heart rate - Kyrielle (Bert) 01-01 0 5 10 15 20 25 30 0:00:00 0:05:00 0:10:00 0:15:00 0:20:00 0:25:00 time (u:mm:ss) v e lo c it y ( k m /u ) 0 20 40 60 80 100 120 140 h e a rt r a te ( b p m ) Velocity Heart rate

(70)

Input: Speed Output: Heartrate

2

Model Model-based controller Objective ‘On-line’ Measurement

1

Actuator

3

(71)

Procedure of Model-Predictive heart rate

control for horses (MPC)

HR en speed PC with MPC controller

Auditory feedback from MPC controller

(72)

40 50 60 70 80 90 100 110 120 0:00:00 0:10:00 0:20:00 0:30:00 0:40:00 0:50:00 1:00:00 time(h:mm:ss) h e a rt r a te ( b p m )

Controlled heart rate Target heart rate

(73)

Time-constant of HR over time 0 10 20 30 40 50 60 70

Week 0 Week 1 Week 2 Week 3 Week 4

Time (date) T C ( s e c ) TC

Optimisation of physical training of race

horses: Heart rate recovery time

(74)

Training exercises

Performance

Body

What happens today in sports

Process

Reading &

Measuring

Trainers

experience

prediction

feedback

(75)

(76)

Physical training _{Physical performance}

Process: physical monitoring

CITD

(77)

Model

Process

Heart Rate

Physical activity

Exercise type 1

Exercise type 2

……

Physical monitoring

Input

Output

(78)

(79)

(80)

The player

Polar heart rate belt Hosand HR module Inmotio position antenna Telemetry connection Abatec server

Base stations Inmotio/Abatec _transponder

Wire connection 3D Accelerometer

(81)

Inmotio/Abatec Transponder Hosand HR module Antenna 1 Antenna 2

(82)

(83)

Method

• Player arrives at training

• Player is measured in Milan Lab

– Measurements of Physical condition

• Real-Time monitoring on training

field with BioRICS system

• Player in the Mind Room

(84)

Milanlab:

Dry test Physical/Mental condition

Allenamento:

Variable 1 Variable 2 Variable n . . . Activity

(Inmotio) Heart Rate

Mathematical Model Physical/Mental condition Variable 1 Variable 2 Variable n . . . ??? ??? ???

Objective:

Milan Lab references during training

- Test installations in Milan Lab

(85)

Total of 255 experiments over 3 years

In 19 training sessions recorded for 9 Primavera players

(4 players/training = 76 recordings )

Algorithm to monitor

“Physical condition”

(86)

DYNA reference 3 4 5 6 7 8 9 20070731 20070906 20070911 20070913 20070914 20070920 20070921 20071122 20071123 20071126 20071129 20071130 20071203 20071204 DYNA reference

(87)

3 4 5 6 7 8 9 20070731 20070906 20070911 20070913 20070914 20070920 20070921 20071122 20071123 20071126 20071129 20071130 20071203 20071204

DYNA reference Algo Physical

(88)

“Physical algorithm” for Milan Lab

physical reference:

(89)

Results: numeric

• 50 training sessions with Milan Lab physical reference

• 5 Primavera players

• Algorithm correctness: exact Milan Lab physical reference

-> 40/50 training sessions or

80% correct

• Algorithm correctness: error > 1 point on Milan Lab physical

reference score:

-> 48/50 training sessions or

96% correct

on [-1 1]

interval of Milan Lab physical reference

(90)

Physical training Physical performance P erf o rman ce

Total performance = Mental performance + Physical performance

Mental performance Mental training

Process: mental monitoring

CITD

(91)

Heart Rate

Physical

Activity

Reference

mental status

??? Time

On-line mental

monitor

Mental algorithm

Mental monitoring

(92)

MINDROOM

(93)

Method

• Player arrives at training

• Player is measured in Milan Lab

– Measurements of Physical condition

• Real-Time monitoring on training

field with BioRICS system

• Player in the Mind Room

(94)

Milanlab:

Dry test Physical/Mental condition

Allenamento:

Variable 1 Variable 2 Variable n . . . Activity

(Inmotio) Heart Rate

Mathematical Model Physical/Mental condition Variable 1 Variable 2 Variable n . . . ??? ??? ???

Objective:

Milan Lab references during training

- Test installations in Milan Lab

(95)

Mindroom data (1133 and 1156) -1.5 -1 -0.5 0 0.5 1 1.5 2 7 /0 3 /2 0 0 6 3 1 /0 3 /2 0 0 6 3 /4 /2 0 0 6 7 /4 /2 0 0 6 1 3 /4 /2 0 0 6 1 9 /4 /2 0 0 6 2 4 /4 /2 0 0 6 2 7 /4 /2 0 0 6 1 /5 /2 0 0 6 2 /5 /2 0 0 6 3 /5 /2 0 0 6 4 /5 /2 0 0 6 1 0 /5 /2 0 0 6 2 7 /0 3 /2 0 0 6 3 1 /0 3 /2 0 0 6 7 /4 /2 0 0 6 1 1 /4 /2 0 0 6 Date (ddmmyy) Me n ta l s ta tu s [p o s , n e g o r n e u tr a l] MINDROOM 1156 1133

(96)

Milan Lab Mindroom vs. Mental algorithm

Mental algo vs Mindroom

-1.5 -1 -0.5 0 0.5 1 1.5 2 7 /0 3 /2 0 0 6 3 1 /0 3 /2 0 0 6 3 /4 /2 0 0 6 7 /4 /2 0 0 6 1 3 /4 /2 0 0 6 1 9 /4 /2 0 0 6 2 4 /4 /2 0 0 6 2 7 /4 /2 0 0 6 1 /5 /2 0 0 6 2 /5 /2 0 0 6 3 /5 /2 0 0 6 4 /5 /2 0 0 6 1 0 /5 /2 0 0 6 2 7 /0 3 /2 0 0 6 3 1 /0 3 /2 0 0 6 7 /4 /2 0 0 6 1 1 /4 /2 0 0 6 Date (ddmmyy) Me n ta l s ta tu s [p o s , n e g o r n e u tr a l] MINDROOM MENTAL ALGO 1156 1133

(97)

Results: numeric

• 30 training sessions with mental score (MR + Questionnaire)

• 5 Primavera players

• Algorithm correctness:

(98)

• Objective: Quantification of mental status (fear) of a

horse in a non-invasive and continuous way during

physical exercise

• Experimental design

5’ 5’ 1’ 4’ 1’

(99)

Input

5’ 5’ 1’ 4’ 1’

walking trotting walking

Mathematical Model Physical activity (ActiGraph GTM1) Measured Heart rate (Polar RS 800) Output

• Modelling

(100)

5’ 5’ 1’ 4’ 1’

(101)

Input

5’ 5’ 1’ 4’ 1’

Mathematical Model Modelled Heart rate (HR_physical) Physical activity (ActiGraph GTM1) Measured Heart rate (Polar RS 800) Model error (HR_mental) Output

• Prediction

(102)

5’ 5’ 1’ 4’ 1’

(103)

• Results:

a) Blue = input signal

Green = part of input signal

used for modelling

b) Blue = measured heart rate Black = modelled heart rate c) Black = model error

Red = presence of stressor

Legend:

33/37 :

mathematical model describing the relationship

between Physical Activity and Heart Rate Response

could be built (R

2

avg

=0.93).

33/33 :

presence of the stressor could be automatically

detected

(104)

• Remarkable result:

Jansen et al. 2009. The veterinary

(105)

Heart Rate

T

_body

Video

images

Example: development of a real-time monitor

& controller for status F1-driver

Practice\Qualification\Race

G-forces

Breaking

Steering

Car tuning

Weather

conditions

Track

conditions

… Mathematical model Algorithm Physical Algorithm Mental

(106)

Karting: experiments

(Jeffrey van Hooydonk)

(107)

Sensors

• EMG

• Heart rate (ECG)

• Acceleration

• VO2max

-Longitudinal

-Transverse

(108)

3D accelerometer & Unipro

700 750 800 850 900 950 1000 1050 1100 1150 1200 -1 -0.5 0 0.5 1 Time [s] A c c e le ra ti o n [ g ]

Yannick de Brabander (Horensbergdam 23/02/2008; lap 2)

700 750 800 850 900 950 1000 1050 1100 1150 1200 50 60 70 80 90 100 110 120 Time [s] S p e e d [ k m h ] longitudinal transverse 1 2 3 4 5 6 7 8 9 10 1112 Right turns Left turns

(109)

Jeffrey – experiment

0 100 200 300 400 500 600 700 -0.5 0 0.5 T ra n v e rs e A c c [ g ] Time [s] Jeffrey - 23/2/208 - Horensbergdam 0 100 200 300 400 500 600 700 40 60 80 100 120 S p e e d [ k m h ] Time [s] 0 100 200 300 400 500 600 700 160 170 180 190 H e a rt R a te [ b p m ] Time [s]

(110)

Applications

Bicycle helmets Race horses Chickens Mussels Professional Cyclists Intensive Care ESA (2006) Daphnia Tubifex (1991) Larvae Sleep monitor •13 products (1981) •13 patents •>200 journal publications

(111)

Thank you for your attention…

• For more information you can always check

our website:

• http://www.m3-biores.be

(112)

(113)

Light intensity step experiment

150 0 µmol PAR m

-2

_s

-1 Time [minutes] 0 10 20 30 40 50 60 Light Int ensity 0 50 100 150 Time [minutes] 0 10 20 30 40 50 60 Light Intensi ty 0 50 100 150 liup exp lidown exp

Choosing a model structure based on

biological knowledge

Example: net CO

₂

assimilation response of plants to

(114)

Plot

Time [minutes] 0 10 20 30 40 50 N et CO 2 ass imilati on ra te [ µmol C O 2 m -2 s -1 ] -2 0 2 4 6 Measured Modelled 1st order Modelled 2nd order

(115)

TF

₁

TF

2 u x

x = TF

₁

.TF

₂

.u

TF

₁

.TF

₂

TF

₁

TF

₂ u + _x +

x = (TF

₁

+TF

₂

).u

TF

₁

+TF

₂

Parallel

TF

₁

TF

₂ u + x -1 1 2

TF

x=

.u

1+TF TF

1 1 2

TF

1+TF TF

u x

Feedback

Serial

Step 2: Decomposition into first order models

(116)

3 rd

order

(117)