Design, development and programming of a low-cost real-time measurement platform for hand pose estimation

(1)

estimation

real-time measurement platform for hand pose

Design, development and programming of a low-cost

Academic year 2019-2020

Master of Science in Electromechanical Engineering

Master's dissertation submitted in order to obtain the academic degree of

Counsellor: Dr. ir. Tom Lefebvre

Supervisor: Prof. dr. ir. Guillaume Crevecoeur

Student number: 01506940

(2)

(3)

Acknowledgement

Not only during the time I worked on this masters’ dissertation, but also during my whole student time in Ghent I received help form different people. I want to thank my friends from home, who I may know a little bit longer, but also the friends that I got to know over the last five years. I want to thank them for having fun together, for cooking and eating together, for travelling, for doing sports, etc. but also for studying together, for helping me understand everything and for motivating me when needed.

Furthermore I want to thank two other persons in particular. The first one is my grand-mother who supports me all the time and who is probably lighting a lot candles these days. The second person I want to thank is Stijn. I want to thank him for being with me this whole time, for taking breaks in between all the school work and for supporting me. I am also grateful that I have the opportunity to study. This may be something that is normal for me and for most of my friends, but it is definitely not something everyone has the chance to. This includes financial support from home but also the educational support of the university. More specifically the effort the professors and assistants take in order to learn me and all the other students something more. Therefore I want to thank my counsellor for guiding me in the right direction and for giving me more insight in this thesis.

(4)

The author gives permission to make this master dissertation available for consultation and to copy parts of this master dissertation for personal use.

In all cases of other use, the copyright terms have to be respected, in particular with regard to the obligation to state explicitly the source when quoting results from this master dissertation.

Margje Ruyssinck, June 2020

(5)

Preamble

This master’s dissertation was written while the Covid-19 outbreak in Belgium was active. Due to this people were forced to go into lockdown and universities were closed. This brought some difficulties and limitations for the elaboration of this work. These will be summed up below.

- Just before the Covid-19 measures started, two bend sensors were not well prepared yet: the clincher connectors were not closed in a good way. This needed to be done again to make sure the sensors would conduct. New clincher connectors needed to be ordered but since we were prohibited to enter the buildings it was not possible anymore. As a consequence the two bend sensors are still waiting at the lab to be prepared for use and haven’t made it to my home. Therefore only 8 sensors are used and no sensors will be placed on the thumb.

- Not all electronic components were available at home. Capacitors and inductors were not available in order to make an analog filter. Also a new multiplexer was needed but ordering only one would be a waste of money. Therefore I did a call online in a group with people from my village and luckily one of the persons in that group could help me.

- When I was in Ghent I had some trouble getting good data from the bend sensors. From previous test I knew it should work, so something must have gone wrong. In order to track down whether the problem was about the sensors, the circuit board, with the multiplexer or the wiring, the use of a multimeter was needed. I don’t have one myself so I needed to wait until my mother came to Ghent for work so she could drop it off.

- When I started to gather data I was already in Ghent and due to the whole situ-ation it was not allowed to meet different people and therefore only data from my roommate and myself are available.

- While doing some tests with the glove, one of the wires came loose. It needed to be soldered again. To solve this problem my roommate asked his parents to drop off his soldering station but again this took some days to get the equipment.

As is clear from previous points, the biggest problem was lack of materials and the waiting time to get them.

(6)

programming of a low-cost real-time

measurement platform for hand pose

estimation

Master’s Dissertation submitted to obtain the academic degree of MASTER OF SCIENCE IN ELECTROMECHANICAL ENGINEERING:

CONTROL ENGINEERING AND AUTOMATION Margje Ruyssinck

Academic year 2019-2020

Supervisor: Prof. dr. ir. Guillaume Crevecoeur Counsellor: Dr. ir. Tom Lefebvre

Summary — Through the years, the research field of human-computer interaction has become very important. Being able to estimate hand poses accurately and in real-time gives the opportunity to develop many applications such as in the medical sector, learning tools, computer controlling, entertainment, etc.

In this work, we present the design and development of a cheap and mobile glove which can capture information of the hand in real-time. It starts with the design of the glove and ends with an elaborated application: automatically translating sign language. The design, based on a mathematical model of the hand, needs a measure for orientation and for the relative finger angles. To get this information multiple bend sensors and an IMU are attached to the glove. The model is used to get a visualisation of the moving hand. After data collection and processing, a wide range of classification techniques are compared w.r.t. their predictive performance to predict the hand gestures, based on a total of 12 different signs. Four different classification methods (Random forest, SVM, LDA and na¨ıve Bayes) and a combined method are used, this makes it possible to compare the accuracy of each method and to select the best one.

Keywords— wearable glove, real-time gesture recognition, real-time visualisation, bend sensors, internal measurement unit (IMU)

(7)

Design, development and programming of a

low-cost real-time measurement platform for hand

pose estimation

Margje Ruyssinck

Supervisor: Prof. dr. ir. G. Crevecoer Counsellor: Dr. ir. T. Lefebvre Abstract—Through the years, the research field of human-computer

in-teraction has become very important. Being able to estimate hand poses accurately and in real-time gives the opportunity to develop many appli-cations such as in the medical sector, learning tools, computer controlling, entertainment, etc.

In this work, we present the design and development of a cheap and mobile glove which can capture information of the hand in real-time. It starts with the design of the glove and ends with an elaborated application: automatically translating sign language. The design, based on a mathemat-ical model of the hand, needs a measure for orientation and for the relative finger angles. To get this information multiple bend sensors and an IMU are connected to the glove. The model is used to get a visualisation of the moving hand. After data collection and processing, a wide range of clas-sification techniques are compared w.r.t. their predictive performance to predict the hand gestures, based on a total of 12 different signs. Four dif-ferent classification methods (Random forest, SVM, LDA and na¨ıve Bayes) and a combined method are used, this makes it possible to compare the accuracy of each method and to select the best one.

Keywords—wearable glove, real-time gesture recognition, real-time visu-alisation, bend sensors, internal measurement unit (IMU)

I. INTRODUCTION

In our daily life, almost everyone is able to work with one’s hands. We can do many different tasks with them and we use them all of the time. Moving our hands is one of the most easy things to do and no one ever needs to think about how one is doing it exactly. Therefore, using our hands as a direct interface between ourselves and the computer may eliminate the need for a keyboard or touchpad. This can be very intuitive for people and therefore human-computer interaction is a widely spread re-search field which has many application. There are applications in the entertainment industry: interactive TV remote controlling [1], (virtual reality) gaming [2], interacting with the computer [2], controlling a drone, etc. There are also educational appli-cations such as learning to drive a car in a virtual environment [3], hand gesture recognition [4] and many more. The medical sector also benefits also from these novelties: touchless interac-tion with a screen while preserving sterility [5], robot nurses that detect hand gestures in order to assist the doctors [5], improve post-stroke rehabilation [6] and many other helpful applications exists.

However, movements like picking up something, holding a pen, turning a page, waving to somebody, pushing a button, rais-ing one’s frais-inger and other common activities suddenly become very complex to detect by a computer. The computer needs ex-act information on the orientation and the relative position of the fingers in order to know what the hand is doing. All this

information then needs to be gathered and processed before it is useful. While this is a difficult task for the engineer, it will open up a window of new possibilities to make life easier for the user. A. Goal and outline

This abstract describes the process of the development of a real-time state estimation platform that captures complex hand manipulations, together with a method to translate sign lan-guage. First some related work is discussed in section I-B whereafter the proposed system is presented. In section II, the kinematics of the hand are explained. Then the needed sen-sors and design is discussed in section III, after which a section about data processing is given (section IV). Different classifica-tion methods are summed up briefly in secclassifica-tion V and in the end the application is described.

B. Related work

An approach that is already widely used to recognise hand gestures are vision, i.e. camera image, based techniques [4, 5]. This technique uses one or multiple cameras which are able to recognise the hands and distinguish them from other body parts. To do this, different methods are possible. A first option is to use markers, this can be done by wearing a glove with a spe-cific color or with some light points attached to it such that it can easily be detected by the camera. There are also mark-ersless methods such as the shape and color based techniques. These two methods have the disadvantage that they are sensitive to different light intensities in their environment. Yet another technique uses motion: only the hand is moving while the rest of the body is at standstill. The camera only needs to detect the moving parts. This brings a big limitation on possible applica-tions and is not user friendly. A fifth method is to use special depth cameras, these are able to detect the hands since they are assumed to be closest to the camera. The images obtained from these cameras are often coarse-grain and noisy. Furthermore, they can only give accurate results if there is sufficient texture in the picture.

There are clearly some disadvantages connected with the vision-based approach. In case the hand is marked or when a depth camera is used, one camera suffices but often multiple cameras are needed to obtain a good 3D view and to prevent the

(8)

algorithm to match the different 2D images in order to get a 3D view is necessary. This adds an extra computational cost, next to the one needed to recognise the hand. The cameras also need a set-up which means some extra materials are needed for the montage and enough space is required. Furthermore the system is static and sensitive to environmental changes such as acciden-tally walking in front of the camera, lightning and a cluttered environment. This method has clearly a lot of disadvantages and therefore there is need for more a user friendly, cheap and mobile alternative.

C. Proposed system

In order to eliminate the disadvantages named above, the pro-posed system consist of a number of sensors that are all inte-grated in a wearable glove. In this way, only information from the hand itself is captured. This implies that no high computa-tional cost is needed to recognise the hand or to match multiple images into a 3D image. It also implies that the design is less dependent on the environment. The system fulfills the mobil-ity question since the sensors on the hand capture all the data needed. If the interface system between the glove and the com-puter can work wirelessly, the solution is completely mobile. An other important factor is the cost. While cameras are asso-ciated with a high cost, this system, consisting of only a glove and some sensors is supposed to have a low cost. A sketch of the proposed system is given in figure 1.

Fig. 1: Sketch

II. KINEMATICS OF THE HAND

In this section we take a closer look at the anatomy of the hand in order to understand its working principles. Then a

sim-1_{Self occlusion occurs if a part of an object overlaps itself such that another}

part is hidden from the camera and no information is available about that part.

position appear to be very important. At the end of this section, a visualisation of the hand is shown.

A. Anatomy

In literature, the number of DOF of the hand differs depen-dent on the assumptions made [7, 8]. Taking the assumptions from Elkoura et al. [7], we count 4 DOF in each finger: 3 for extension/flexion2_{and 1 for abduction/adduction}3_{. 5 DOF are}

counted for the thumb: 3 for extension/flexion and 2 for abduc-tion/adduction and 6 DOF for the wrist: 3 for the rotation around the three axes (yaw, pitch and roll) and 3 for the translation in the three directions. This sums up to 27 DOF. This is a high num-ber and therefore it is recommended to reduce this numnum-ber when designing the mathematical model. Concerning the fingers, two degrees of freedom per finger are taken into account: the angle at the first and second knuckle. The movement of the third pha-lange and the abduction/adduction of the fingers is omitted. For the thumb also 2 DOF would be taken into account4_{. Also here}

the adduction and one extention movement are not taken into ac-count. Another 3 DOF are added for the orientation of the hand. This makes a kinematic model of 11 DOF.

B. Kinematic model

To make the mathematical model, information about the ori-entation and the relative position of the fingers is needed.

Fig. 2: Joints indicated on each of the fingers

2_{Extension/flextion refers to a movement that increases/decreases the angle}

between two body parts. [9]

3_{Abduction/adduction is the movement away/towards from the midline of the}

hand. [9]

4_{Just before the Covid-19 measures started, two bend sensors were not well}

prepared yet: the clincher connectors were not closed in a good way. This needed to be done again and new clincher connectors were ordered but since we were prohibited to enter the buildings it was not possible anymore. As a consequence only 8 sensors are used and therefore no sensors are placed on the thumb.

(9)

B.1 Orientation

The orientation can be represented in two different ways: Eu-ler angles or quaternions. EuEu-ler angles can be described by 3 successive rotations about linearly independent axes. Due to the fact that the rotations, represented by rotation matrices are not commutative (R(z, ψ) · R(y, φ) · R(x, θ) 6= R(z, ψ) · R(x, θ) · R(y, φ)6= ...) and due to the fact that they are limited due gim-bal lock5_{, quaternions are preferred to represent the rotation. A}

quaternion vector consists of four elements. The first element is a real element and is related to the angle at which is rotated (θ), the last three elements are complex and are related to the rotation axis (vx, vy, vz). This corresponds to an axis-angle

rep-resentation of a rotation.

Since it is easier to use rotation matrices in calculations, Ro-drigues’ formula (Eq. (1)) is used. This formula makes it possi-ble to convert an axis-angle representation into a rotation matrix (Eq. 2).

R(θ, v) = I + sin(θ)ˆv + (1_{− cos(θ))ˆv}2 ₍₁₎

With I the identity matrix, θ the rotation angle, v the vector around which is rotated and ˆv is the cross product matrix of vector v, meaning v × a = ˆv· a. The rotated points are obtained by premultiplying the coordinate of the point with the rotation matrix: Pr= R· P . R =  1− 2 · (q 2 3+ q42) 2· (q2q3− q1q4) 2· (q1q3+ q2q4) 2· (q1q4+ q2q3) 1− 2 · (q22+ q24) 2· (q3q4− q1q2) 2_{· (q}2q4− q1q3) 2· (q1q2) + q3q4) 1− 2 · (q22+ q32)   (2) B.2 Relative finger angles

The position of the different joints of the finger is influenced by extension and flexion. Since this means an increasing or de-creasing angle between the finger parts, some measure for the different angles is needed. We name them α1, α2, β1, β2, γ1,

γ2, δ1, δ2, 1and 2(from left to right).

Starting from some initial points (green in figure 2) and know-ing the angles, the relative position of the other points can be calculated. The positions of point A2to point D3are calculated

in the following way.

- x-direction : stays the same (fingers rotate around the x-axis) - y-direction : P2y= P1y+ lp1· cos(angle1) P3y = P2y+ lp2· cos(angle1+ angle2) - z-direction : P2z= P1z− lp1· sin(angle1) P3z= P2z− lp2· sin(angle1+ angle2)

With P a point from A to D, lp1and lp2the length of the first

and second part of finger p respectively and angle the corre-sponding angle. In the simplified model, it is assumed that the thumb only rotates around the z-axis and therefore the points E2and E3are not calculated the same way the fingers are

cal-culated. The principle is the same but the exact calculations are different.

5_{This is a phenomen where one degree of freedom get lost in a}

three-dimensional system. It happens when one of the axes is in parallel with an other axis and degenerates the system to a two-dimensional system.

- x-direction : E2x= E1x− le1· sin(1) E3x= E2x− le2· sin(1+ 2) - y-direction : E2y= E1y+ le1· cos(1) E3y = E2y+ le2· cos(1+ 2)

- z-direction : stays the same (thumb rotates around the z-axis) C. Visualisation

In figure 3, the visualisation of the hand is shown. A demo of the real-time visualisation can be found by clicking on the following link: https://www.youtube.com/watch?v= Jw8-r2WbGfA.

Fig. 3: Visualisation of the hand III. MEASUREMENT PLATFORM

In this section we will detail the measurement platform. As discussed in the previous section, the study of the hand kinemat-ics revealed that in order to capture basic mobility of the hand, we should measure its absolute orientation and the relative po-sition of the fingers. Therefore multiple sensors are attached to the glove. The orientation can be obtained by using an Internal Measurement Unit (IMU). This is a system that uses the grav-ity, the magnetic field and the acceleration to give an absolute orientation. The IMU is placed on the palm of the hand.

In order to know the angles at the different finger joints, bend sensors are used. Two bend sensors (one on the first and one on the second knuckle) are placed on each finger. The bend sensors have a variable resistance which changes when they are bent. This resistance is used to calculate the corresponding angle.

To collect the data from these two types of sensors, the Ar-duino Uno is used to interface with the computer. A simplified diagram of the connections between the hardware is presented in figure 4. It also shows the use of a multiplexer which is placed between the bend sensors and the Arduino. The multiplexer is used to reduce the amount of input pins needed on the Arduino. An other important aspect for the design is the interchange-ability of the glove. Different people have a different hand size and therefore multiple sizes of gloves are desired. Removable sensors may be the solution, in this way it is also possible to 3

(10)

Fig. 4: Simplified diagram

IV. LOW-LEVEL MEASUREMENT PROCESSING

A. Programming software

The data obtained from the sensors needs to be processed in order to have useful data. Matlab reads the sensor output from the Arduino IDE, this is because this environment makes it pos-sible to get the quaternion values directly from the IMU while this is not possible in Matlab.

B. Filtering

In the case of the IMU, the raw data is already filtered since it contains an internal filter algorithm, based on the Madgwick fusion filter algorithm [10]. In the case of the bend sensors, the data contains clearly some noise that needs to be filtered out. Therefore an Infinite Impulse Response (IIR) low pass filter is used, namely the Butterworth filter. An IIR filter is a recur-sive filter and therefore it needs less computations to achieve the same result as an Finite Impulse Respose (FIR) filter. This makes the filter faster and more suitable to apply in real-time. C. Calibration

In order to do useful measurements, the sensors need to be calibrated first. To calibrate the three internal sensors (ac-celerometer, gyroscope, magnetometer) of the IMU, an internal calibration system is implemented. The calibration is achieved by moving the hand in different directions for a while. The calibration-function gives a value between 0 and 3 for the three sensors and for the overall system with 3 corresponding to a fully calibrated sensor (or system).

Since the size of the hands of different persons is not the same, a calibration for the bend sensors is needed too. The calibra-tion gets the minimum and maximum resistance value for each sensor. These values are needed to do a linear interpolation in order to get the corresponding angle of the finger (Eq. (3)): 0◦

corresponds to the minimum resistance value Rstraightand 90◦

corresponds to the maximum resistance Rbend.

θ = 90◦_{− 90}◦_· Rbend− Rf lex

Rbend− Rstraight (3)

four different classification methods are discussed. A first step is the selection of the features, used to train the models. These features have a very big impact on the model: bad features can make bad predictions and overfitting is possible in the case when too many features are chosen. The selection is done manu-ally (logical thinking and looking at the data) and automaticmanu-ally (minimizing the misclassification rate).

A. Support Vector Machine (SVM)

The main purpose in Support Vector Machine (SVM) is to find the optimal hyperplane that divides the data points in two classes. Dependent how far the data is from the plane, we can be more sure the model makes a correct prediction. To solve a multi-class SVM, a trained Error Correcting Output Codes (ECOC) model is used. This model reduces the classication to a set of binary classication problems. A one-versus-one cod-ing design is used, which determines which classes the binary learners (SVM) train on.

B. Random Forest

The random forest method is an ensemble machine learning algorithm, this means that it combines a group of weak learners (decision trees) to form a strong learner (Random Forest). A tree starts with a root node where a first decision is made, based on a selected feature and divides the whole dataset in two. In the following node the next division is made and so on until you reach the leaf node, which represents the class you’re in. All the trees do a prediction but the final prediction of the forest is obtained by majority voting.

C. Na¨ıve Bayes

The na¨ıve Bayes method is a probabilistic classication method which uses the Bayes theorem (Eq. 4).

P (y_{|x) =} P (x|y) · P (y)

P (x) (4)

The purpose is to classify a certain class y, given the featuresx withx = (x1, x2, x3, ..., xn) and n the number of features used

in the model. These features are assumed to be independent and to have an equal contribution to the outcome. Using the prop-erty of independence (P (x1, x2) = P (x1)· P (x2)) and the fact

that constant terms do not contribute when comparing magni-tudes with each other, the class with the maximum probability is chosen: y = argmaxyP (y) n Y i=1 P (xi|y) (5)

D. Linear Discriminant Analyse

The Linear Discriminant Analyse (LDA) classification relies on the Bayes theorem (Eq. (4)). In this method no assumption of independence of the features is needed but instead an other assumption is made: P (x|y) is assumed to be the normal distri-4

(11)

bution: P (x|y) = √1 2πσe 1 2  X− µ σ   2 (6) With µ the mean value and σ the standard deviation of a class. An other assumption made is that the different classes share the same variance (σ2_{) for x (otherwise a Quadratic Discriminant}

Analyse (QDA) is obtained). When the normal distribution is plugged into the Bayes theorem, the logarithm of the right part of the equation is taken and the constant terms are erased. We get the discriminant function (which is linear in x):

δ(x) = µix σ2 − 1 2 µ2 i σ2+ log(P (y)) (7)

The discriminant function tells us how likely it is that the data is from each class: x is classified to the class that yields to the largest discriminant function.

E. Combined method

Since the predictions of the four different classes above are not always correlated to each other, a combined method is de-signed. This method looks at the predictions made in each model and uses majority voting to select the most occuring pre-diction.

VI. APPLICATION

After the data is collected and processed, it is time to zoom in on the application: the recognition of 12 different signs. The knowledge gained in section V is now applied. First, the fea-tures are selected automatically and for each model separately. Then, the four different models are made, together with a com-bined method. The data, collected from two different persons6

is divided in training data (80%) and test data (20%). In this way, a model is made with training data of hand 1 or hand 2 and is tested with the test data of the same hand. A better evaluation method is when models are trained with the full data of hand 1 or hand 2 and evaluated with the full data of the other hand (table I). This gives an accuracy that will be more realistic to the ones when new persons try the application.

RF SVM LDA Bayes Combined a 99.2766 97.6852 97.2222 97.8009 98.2639 b 85.8681 85.3646 93.1597 81.3021 96.9097 c 99.7396 98.2639 96.5278 96.0069 97.4826 d 47.4711 41.3947 86.8287 67.2743 74.8322 TABLE I: Accuracy (%) of the prediction for test data hand 1 (a,d) or hand 2 (b,c) through models made with training data of hand 1 (a,b) or hand 2 (c,d)

When looking at this table, it is remarkable that evaluation d has much lower accuracy than the others. This can be explained because there is less data of hand 2 available, compared to hand 1. Due to the fact that in c only 80% of the data of hand 2 is used

6_{Due to the Covid-19 measures it was not possible to collect data from many}

different people

as training data, these models are overfitted on data of hand 2, and high accuracy values are obtained.

Although each model gives high accuracy values, only the LDA classication model is used in the application. Combining the four methods gives a better accuracy but it takes more time and this needs to be avoided in a real-time application. There-fore the LDA method ensures a fast and accurate prediction of the hand gestures.

A demo of the real-time classification can be found on youtube: https://www.youtube.com/watch?v= 14MPM7yW6H0

VII. CONCLUSION

At this point we can say that a real-time estimation platform has been created successfully. A wearable glove is made and the data from the different sensors on it can be captured in real-time. A high accuracy is obtained and the system is not prone to the environment it is used in. Self-occlusion cannot happen and a good segmentation is not needed. A visualisation of the real-time hand movements is realized and 12 different hand ges-tures can be recognised by making use of the LDA classification method. Further improvement is still possible. The glove can be optimised with an expansion of the sensors and a more com-fortable design can be made. The possibilities of this glove are immense: once the glove is ready, many applications can be worked out. Going from implementing the glove as a mouse to interact with your computer, to a glove which is used in medical applications to ensure sterility of the screens.

Thus it is possible to have a relatively accurate real-time po-sition acquipo-sition of the hand, without the use of an expensive design. Due to the relatively cheap elements this system con-sists of: the glove itself, 10 bend sensors, clincher connectors, the IMU, an Arduino, a multiplexer and some other small mate-rials like wires, a resistor, velcro, circuit board, etc., this design costs about 160 euros all together (table II).

Amount Parts Cost Total cost 1 Arduino Uno 20.00 20.00 1 BNO055 36.24 36.24 10 Bend sensors 7.95 79.50 10 Clincher connectors 1.12 11.20 1 8-channel multiplexer 0.401 0.401 1 A pair of gloves 2.49 2.49 1 Velcro 5.49 5.49 Total 155.321

TABLE II: Cost of the glove REFERENCES

[1] G. Ren and E. ONeill, “Freehand gestural text entry for interactive tv,” in Proceedings of the 11th European Conference on Interactive TV and Video, EuroITV 13, (New York, NY, USA), p. 121130, Association for Computing Machinery, 2013.

[2] P. Kumar, J. Verma, and S. Prasad, “Hand data glove: a wearable real-time device for human-computer interaction,” International Journal of Ad-vanced Science and Technology, vol. 43, 2012.

[3] Deyou Xu, “A neural network approach for hand gesture recognition in virtual reality driving training system of spg,” in 18th International Con-ference on Pattern Recognition (ICPR’06), vol. 3, pp. 519–522, 2006.

(12)

[5] J. P. Wachs, M. K¨olsch, H. Stern, and Y. Edan, “Vision-based hand-gesture applications,” Commun. ACM, vol. 54, p. 6071, Feb. 2011.

[6] W. Li, C. Hsieh, L. Lin, and W. Chu, “Hand gesture recognition for post-stroke rehabilitation using leap motion,” in 2017 International Conference on Applied System Innovation (ICASI), pp. 386–388, 2017.

[7] G. ElKoura and K. Singh, “Handrix: animating the human hand,” in Pro-ceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Com-puter animation, pp. 110–119, Eurographics Association, 2003. [8] T. L. Baldi, S. Scheggi, L. Meli, M. Mohammadi, and D. Prattichizzo,

“Gesto: A glove for enhanced sensing and touching based on inertial and magnetic sensors for hand tracking and cutaneous feedback,” IEEE Trans-actions on Human-Machine Systems, vol. 47, no. 6, pp. 1066–1076, 2017. [9] O. Jones, “Anatomical terms of movement.”

https://teachmeanatomy.info/the-basics/ anatomical-terminology/terms-of-movement/, 2019. [10] S. Madgwick, “An efficient orientation filter for inertial and

iner-tial/magnetic sensor arrays,” Report x-io and University of Bristol (UK), vol. 25, pp. 113–118, 2010.

(13)

List of figures

2.1 DOF in the hand. Adapted from [1] . . . 6

2.2 Included DOF for fingers . . . 7

2.3 Axis-angle representation. Adapted from [2] . . . 8

2.4 Euler angles [3] . . . 9

2.5 Joints indicated on each of the fingers . . . 12

2.6 Side view of the simplified finger . . . 13

2.7 Top view of the simplified thumb . . . 13

2.8 Demo of the real-time visualisation of the hand . . . 14

2.9 Points on the hand . . . 16

2.10 Different visualisation options . . . 18

3.1 Position of the bend sensors . . . 19

3.2 Simplified diagram . . . 20

3.3 Sketch of the design of the glove . . . 20

3.4 Orientation of the IMU with respect to the hand . . . 21

3.5 Orientations IMU [4] . . . 22

3.6 Bend sensor . . . 22

3.7 Linearity between the digital number and the voltage . . . 23

3.8 Voltage divider . . . 24

3.9 Arduino Uno . . . 26

3.10 Multiplexer pinout MC14051B [5] . . . 26

3.11 Truth table multiplexer MC14051B [5] . . . 27

3.12 Working principle multiplexer . . . 27

3.13 Position and use of the velcro . . . 28

4.1 Low pass filter [6] . . . 32

4.2 Effect of a signal that passes through a filter [6] . . . 33

4.3 Effect of the order on the sharpness of the filter . . . 34

4.4 Comparison FIR and IIR filter with the same order . . . 34

4.5 Illustration of the sensor fusion process [7] . . . 36

4.6 Signal during calibration of the bend sensor, the effect of the filter is visible 39 5.1 Step 1: making the model . . . 41

5.2 Step 2: prediction . . . 41

5.3 Example of data of two classes (blue and purple), plotted against the features of column 1 and column 2 . . . 43

5.4 5-fold cross-validation [8] . . . 44

5.5 Hyperplane [9] . . . 45 xv

(16)

5.8 An example of a decision tree obtained from Matlab . . . 48

5.9 Normal distribution . . . 50

6.1 Sign language alphabet [11] . . . 52

6.2 Accuracy of the prediction for test data of hand 1 through models hand 1 with different percentages of the data . . . 57

6.4 Misclassification rate of the three different models . . . 60

6.5 Misclassification rate LDA . . . 61

6.6 Misclassification rate na¨ıve Bayes . . . 62

6.7 Misclassification rate random forest . . . 63

6.10 Accuracy of the prediction for the combined classification methods . . . . 69

6.11 Demo of the real-time classification of the hand gestures . . . 70

6.12 Screen . . . 70

A.1 Data at standstill . . . 77

A.2 Frequency spectrum at standstill . . . 78

A.3 Random movement . . . 79

A.4 Original and filtered data . . . 80

B.1 Confusion matrix RF . . . 81

B.2 Confusion matrix SVM . . . 82

B.3 Confusion matrix LDA . . . 82

B.4 Confusion matrix Bayes . . . 83

B.5 Confusion matrix Combination . . . 83

B.9 Confusion matrix Bayes . . . 86

D.1 Feature selection LDA . . . 92

D.2 Feature selection Naive Bayes . . . 93

D.3 Feature selection RF . . . 93

(17)

E.2 Confusion matrix SVM . . . 95

E.3 Confusion matrix LDA . . . 95

E.4 Confusion matrix Bayes . . . 96

E.5 Confusion matrix Combination . . . 96

E.6 Confusion matrix RF . . . 97

E.10 Confusion matrix Combination . . . 99

E.11 Confusion matrix RF . . . 100

(18)

1.1 Advantages and disadvantages of different hand motion capturing methods 3

2.1 Initial coordinates for visualisation 1 and 2 . . . 14

2.2 Lengths of the different finger parts . . . 15

2.3 Initial coordinates for visualisation 3 . . . 16

2.4 Time needed for each hand model to visualise 100 samples . . . 17

4.1 Frequency at which the values from the BNO055 are obtained in Arduino IDE . . . 30

4.2 Frequency at which the values from BNO055 are read out in Matlab from the Arduino IDE . . . 30

4.3 Mean values of minimum and maximum resistance for different sensors and for different persons . . . 39

6.1 Confusing signs . . . 53

6.2 Representation columns, features and their corresponding sensor parameter 54 6.3 Accuracy of the prediction for test data (20% of the data) hand 1 through models hand 1 (trained on 80% of the data) . . . 55

6.4 Accuracy of the prediction for test data of hand 2 through models hand 1 (trained on 80% of the data) . . . 56

6.5 Accuracy of the prediction for test data of hand 1 through models hand 2 (trained on full data) . . . 57

6.6 Time it takes to make a model in function of the amount of training data 59 6.7 Accuracy of the prediction for test data of hand 2 through models hand 1 with different percentages (_{≤ 1%) of the data . . . .} 59

6.8 Selected features 1 . . . 64

6.9 Accuracy of the prediction for test data hand 2 through models hand 1 with the features chosen like in table 6.8 . . . 64

6.10 Selected features 2 . . . 64

6.11 Accuracy of the prediction for test data hand 2 through models hand 1 with the features chosen like in table 6.10 . . . 64

6.12 Accuracy of the prediction for test data (20% of the data) hand 1 through models hand 1 (trained on 80% of the data) . . . 65

6.13 Accuracy of the prediction for test data of hand 2 through models hand 1 (trained on 80% of the data) . . . 65

6.14 Accuracy of the prediction for test data of hand 1 through models hand 2 (trained on full data) . . . 66 6.15 Time it takes to make a model in function of the amount of training data 69

(19)

7.1 Cost of the glove . . . 73 C.1 Accuracy of the prediction for test data of hand 1 through models hand 1

with different percentages of the data . . . 89 C.2 Accuracy of the prediction for test data of hand 2 through models hand 1

with different percentages of the data . . . 90 C.3 Accuracy of the prediction for test data of hand 1 through models hand 1

with different percentages of data . . . 90 C.4 Accuracy of the prediction for test data of hand 2 through models hand 1

(20)

ADC Analog to Digital Converter. AI Artificial Intelligence.

DOF Degrees of Freedom.

ECOC Error Correcting Output Codes.

EEPROM Electrically Erasable Programmable Read-Only Memory. FIR Finite Impulse Respose.

IDE Integrated Development Environment. IIR Infinite Impulse Response.

IMU Internal Measurement Unit. LDA Linear Discriminant Analyse.

MARG Magnetic, Angular Rate and Gravity. PCB Printed Circuit Board.

QDA Quadratic Discriminant Analyse. RF Random Forest.

SCL Serial Clock Line. SDA Serial Data Line.

SVM Support Vector Machine.

(21)

Chapter 1 Introduction

1.1 Motivation

In our daily life, almost everyone is able to work with one’s hands. We can do many different tasks with them and we use them all of the time. Moving our hands is one of the most easy things to do and no one ever needs to think about how one is doing it exactly. Therefore, using our hands as a direct interface between ourselves and the com-puter may eliminate the need for a keyboard or touchpad. This can be very intuitive for people and therefore human-computer interaction is a widely spread research field which has many application. There are applications in the entertainment industry: interactive TV remote controlling [12], (virtual reality) gaming [13], interacting with the computer [13], controlling a drone, etc. There are also educational applications such as learning to drive a car in a virtual environment [14], hand gesture recognition [15] and many more. The medical sector also benefits from these novelties: touchless interaction with a screen while preserving sterility [16], robot nurses that detect hand gestures in order to assist the doctors [16], improve post-stroke rehabilation [17] and many other helpful applications. It is also clear that dependent on the application, different specifications and accuracy are required and thus a higher cost can be allowed.

However, the implementation of the computer-hand interface has a downside. Actively knowing at which angle the finger bends or knowing the exact position in space is none of our concern, but it is for the computer. This intuitive way of working with our hands needs to be translated to the computer and this is not something that can be worked out in one day. Movements like picking up something, holding a pen, turning a page, waving to somebody, pushing a button, raising one’s finger and other common activities suddenly become very complex to detect by a computer. The computer needs exact information on the orientation and the relative position of the fingers in order to know what the hand is doing. All this information then needs to be collected and processed before it is useful. While this is a difficult task for the engineer, it will open up a window of new possibilities to make life easier for the user.

(22)

1.2 Related work

A method that is already widely used to recognise hand gestures are vision, i.e. camera image, based techniques [15, 16]. This technique uses one or multiple cameras. Cam-eras need to be able to recognise the hands and distinguish them from other body parts, therefore different methods are possible. A first option is to use markers, this can be done by wearing a glove with a specific color or with some light points attached to it such that it can easily be detected by the camera. There are also markerless methods, for example shape and color based techniques. The first technique uses of the fact that different ob-jects can be recognised based on their specific shape. In this case it is important that the object can be clearly segmented from the background. The color based technique looks at skin colored objects in order to recognise the hands but there can be some confusion with similar colored objects. These two methods are sensitive to different light intensities in their environment. Another technique uses motion: only the hand is moving while the rest of the body is at standstill and then the camera only needs to detect the moving parts. This brings a big limitation on the possible applications and is not user friendly. A fifth method is to use special depth cameras, these are able to detect the hands since they are assumed to be closest to the camera. The images obtained from these cameras are often coarse-grain and noisy. Furthermore they can only give accurate results if there is sufficient texture in the picture. Therefore this method is often combined with other techniques.

Next to the already mentioned drawbacks, there are more disadvantages related to the vision-based method. In case the hand is marked or when a depth camera is used, one camera suffices but often multiple cameras are needed to get a good 3D view and to pre-vent the system from self occlusion. Self occlusion occurs if a part of an object overlaps itself such that another part is hidden from the camera and no information is available about that part. A consequence of using multiple cameras is that the system becomes even more expensive and an algorithm to match the different 2D images in order to get a 3D view is necessary. This adds an extra computational cost, next to the one needed to recognise the hand. The cameras also need a set-up which means some extra ma-terials are needed for the montage and enough space is required. Due to the fact that the cameras static and the hand needs to be in direct line-of-sight, this implies that this method is not mobile. Furthermore the system is sensitive to changes in the environment such as accidentally walking in front of the camera, lightning and a cluttered environment.

One great advantage of the vision-based method is that it can be expanded without mak-ing a big extra cost. There is no limitation to capture only the motions of a specific body part. Movements of other body parts or full body motions can be obtained with (almost) the same hardware. In case of unmarked recognition no extra cost is needed. But some extra effort to work out algorithms to detect the desired objects is required.

However this method has clearly a lot of disadvantages and therefore there is need for a more user friendly, cheap and mobile alternative.

(23)

1.3. PROPOSED SYSTEM 3

1.3 Proposed system

Taking the drawbacks of the vision-based system into account, the proposed system consist of sensors which are attached to the glove itself. In this way, only information from the hand itself is captured. This implies that no high computational cost is needed to recognise the hand or to match multiple images into one 3D image. It also implies less dependence on the environment: people walking around you or different light intensities are no longer a problem. The system fulfills the mobility question since the sensors on the hand capture all the data needed so no external set-up is required. In this way, people can use the design independent on the place they are. Furthermore, the system only needs a computer to process the data and an interface to send the data to the computer. If the interface can be done wirelessly, the system is completely mobile. An other important point for the design is the interchangeability of the glove. Different people have different sizes of hands and therefore multiple sizes of gloves are desired. Removable sensors may be the solution, in this way different persons can wear gloves of different sizes without the need of buying new sensors. Moreover if one of the sensors is damaged, there is no need to replace the whole glove but instead the damaged sensor can be taken off and replaced. An other important factor is the high cost of the cameras, this is for many persons a holdback and therefore an affordable method is desirable. This system, consisting of only a glove and some sensors, is supposed to have a low cost. But some disadvantages are associated with this method too, the glove does not fit every hand perfectly: it may feel unnatural and irritate the skin when hands get sweaty inside the glove. Another drawback is that it is not possible to expand this design to other body parts without extra costs. When e.g. information about the elbow is wanted, a whole new design is needed.

Table 1.1: Advantages and disadvantages of different hand motion capturing methods

Techniques Vision based Wearable glove

Advantages -Recognition of other body -Low cost parts without extra cost

-No glove needed -No problem with segmentation -No self occlusion

-Not dependent on environment -Mobile

Disadvantages -Self occlusion -Unnatural feeling

-Expensive -Completely new design needed

for other body parts -Sensitive to environment

-Fixed to a test room -High computational cost -Line-of-sight needed

1.4 Research question

The main goal of this thesis is to develop a real-time state estimation platform to capture complex hand manipulations. This consists of two main parts. The first part is about the

(24)

design of the glove: the hardware needs to be chosen in order to obtain enough information to make a model of the hand. The different components need to be assembled into one wearable glove. The second part is about the development of the measurement platform. When data from the different sensors is gathered, it needs to be processed in order to gain useful data. Once the data is processed, a real-time visualisation of the hand can be made and it is possible to develop many different applications.

1.5 Thesis outline

To describe the route I followed to accomplish this task, the thesis is subdivided into six chapters, next to this introduction chapter (chapter 1).

Chapter 2 is about the kinematics of the human hand. Some basic information related to the working of the hand is explained first. Then a simplified mathematical model is proposed, taking into account the suggested aspects needed to realize the model. At the end of this chapter a visualisation of the model is shown.

In chapter 3, the components which are required to get the desired functionalities, ex-plained in the first chapter, are discussed. Each component is exex-plained and a way to assemble them all together is proposed.

After the data is collected from these components, it is important to make sure it is useful. Therefore chapter 4 discusses how the data is processed on a low-level scale. This chapter is subdivided into three parts. The first one is about the software which is used, the second part is about the filtering of the data and the last part is about the calibration of the sensors.

After the data is processed on a low-level, it can be processed on a high-level scale too. Chapter 5 gives information on how data can be classified. Four different models are discussed after the section about the feature selection is presented.

Next, in chapter 6, a simplified application about sign language is described. It offers a method to translate automatically the different hand gestures. The knowledge acquired in chapter 5 will be very important in this chapter since the different hand gestures are classified.

(25)

Chapter 2 Kinematics of the hand

In this chapter we take a closer look at the anatomy of the hand in order to understand its working principles. Then a simplification of the real hand is made such that a kinematic model can be designed. In this model two principles associated with the motion of the hand appear to be very important. At the end of this chapter, there is a section about the visualisation of the model.

2.1 Anatomy of the hand

In figure 2.1 the skeleton of the hand is depicted. The different finger joints are clearly visible and they are indicated with a number which corresponds to the number of Degrees of Freedom in that joint. In literature no consistent value for the total number of DOF for the hand is found, this number depends on the assumptions made. In [18] the following assumptions are made:

1. The thumb is independent of the other fingers.

2. Adduction/abduction1 _{of the finger joints are independent.} 3. Motion frequency does not affect joint interdependence. 4. Both hands have the same interdependence model.

5. The posture of the wrist and the rest of the arm do not affect the underlying interde-pendence structure.

Taking these assumptions into account, 27 DOF are counted (4_{· 4 + 5 + 6):}

- 16 DOF: 4 in each finger: 3 for extension/flexion2 and 1 for abduction and adduc-tion.

- 5 DOF for the thumb: 3 for extension/flexion and 2 for abduction/adduction. - 6 DOF for the wrist: 3 for the rotation around the three axes and 3 for the

trans-lation in the three directions.

1_{Abduction/adduction is the movement away/towards from the midline of the hand. [19]}

2_{Extension/flextion refers to a movement that increases/decreases the angle between two body parts.}

[19]

(26)

As mentioned above, other values for the number of DOF are possible [20], dependent on the assumptions made but this number is almost always above 20.

Figure 2.1: DOF in the hand. Adapted from [1]

Due to this high number of DOF, a reduction is preferred when designing a kinematic model. Concerning the fingers, two degrees of freedom per finger can give already a lot of information about the movements. Therefore one DOF for the first knuckle and one for the second knuckle is taken into account as indicated in figure 2.2. In this way the the movement of the third phalange and the abduction/adduction of the fingers is neglected. For the thumb also 2 DOF would be3 taken into account. Also here the adduction and one extention movement are omitted. Another 3 DOF are added for the orientation of the hand. This makes a kinematic model of 11 DOF.

Human limitations When a model of the hand is made, it is important to consider some of the natural limitations. For example the angle of the blue knuckles (figure 2.2) cannot be negative and its maximum values is around 135° while the maximum value for the orange knuckles is around 90°. Furthermore the frequency at which we can move our hands to about 6-12Hz at its maximum.[21]

(27)

2.2. KINEMATIC MODEL 7

Figure 2.2: Included DOF for fingers

2.2 Kinematic model

In this section two important aspects to obtain the reduced kinematic 11 DOF model are discussed. This is about the orientation of the hand and the relative position of the fingers.

2.2.1 Orientation

An orientation of the hand is obtained when rotating in three dimensions. Rotations around the vertical, transverse and longitudinal axis are called yaw, pitch and roll re-spectively. Mathematically these rotations can be described by means of matrices or by the axis-angle representation.

An orientation matrix has the general form of (2.1) with R the rotation matrix and T the translation matrix: _"

R T 0 1 #

(2.1) The rotation matrix is created by multiplying the three main rotation matrices.

Rotation around x-axis: pitch or roll (x-coordinate stays the same)

Rx(θx) =    1 0 0 0 cos(θx) − sin(θx) 0 sin(θx) cos(θx)    (2.2)

(28)

Rotation around y-axis: pitch or roll (y-coordinate stays the same) Ry(θy) =    cos(θy) 0 sin(θy) 0 1 0 − sin(θy) 0 cos(θy)    (2.3)

Rotation around z-axis: yaw (z-coordinate stays the same)

Rz(θz) =    cos(θz) − sin(θz) 0 sin(θz) cos(θz) 0 0 0 1    (2.4)

By multiplying a point with the rotation matrix, the position of the rotated point is obtained.

The axis-angle method does not need these three orthogonal axes. It uses only one axis (vx, vy, vz) to rotate the point around with an angle θ.

Figure 2.3: Axis-angle representation. Adapted from [2]

This method makes it is easier to visualise the rotation directly and have an idea of where the rotated point would be. However, mathematically it is easier to work with matrices. Therefore it is good to know how the this representation can be converted into a rotation matrix.

An algorithm which is named the Rodrigues’ rotation formula is used to rotate a vector in space, given an axis and an angle. This theory can be used to convert the axis-angle parameters (vx, vy, vz, θ) into a rotation matrix and vice versa. Rodrigues’ rotation formula, given in Eq. (2.5) gives the conversion between the axis-angle representation and a rotation matrix.

(29)

2.2. KINEMATIC MODEL 9 I is the identity matrix, θ is the rotation angle, v is the vector around which is rotated and ˆv is the cross product matrix of vector v meaning v_{× a = ˆv · a}

v_{× a =} i j k vx vy vz a1 a2 a3 =    vya3− vza2 vza1− v1x3 vxa2− v2y1    =    0 _−vz vy vz 0 −vx −vy vx 0    ·    a1 a2 a3    = ˆv · a

When acquiring data from sensors, a rotation matrix is usually not the output. What is obtained are typically some values in the form of Euler angles or quaternions. These are discussed in more detail below.

Euler angles

The key idea of Euler angles is that any rotation can be described by 3 successive rotations about linearly independent axes. Doing this, there are 12 different possibilities. The first 6 possibilities are called proper Euler angles and the last 6 Tait-Bryan angles: z-x-z, x-y-x, y-z-y, z-y-z, x-z-x, y-x-y and x-y-z, y-z-x, z-x-y, x-z-y, z-y-x, y-x-z. In the case of z-y-x the rotation matrix becomes a multiplication of the rotation matrices around the z-axis, the y-axis and the x-axis respectively. These correspond with roll, pitch and yaw. The rotation matrix is composed as follows:

R = Rot(z, ψ)_{· Rot(y, θ) · Rot(x, φ)} (2.6) Figure 2.4 is a visualisation of Eq. (2.6).

Figure 2.4: Euler angles [3]

After the first rotation is done, the second rotation around the y-axis is now around a new y-axis (Y’). This indicates that the order of the matrices is of importance. This causes a problem for the uniqueness of the solution.

(30)

Although if the angles are small enough, only a small error occurs each step. There are some ways to overcome this problem:

- Chose one specific combination and make sure the angles stay small each step - Calculate all the different options and take the mean value

- Calculate an other combination every step - Use quaternions instead of Euler angles

It is immediately clear that the first three solutions are not optimal: the first solution prohibits fast rotations in order to hold the angle small, the second solution requires longer calculation time and this needs to be avoided in real-time applications and the third possibility will cause more errors.

A second disadvantage of Euler angles is the fact that they are limited due to gimbal lock. This is a phenomen where one degree of freedom get lost in a three-dimensional system. It happens when one of the axes is in parallel with another axis and degenerates the system to a two-dimensional system. Since also for this problem the solution could be to use a quaternion representation, this is explained in the following paragraph.

Quaternions

As mentioned above, the use of Euler angles is not always ideal. An alternative parame-terization is established by the formulation of quaternions. A Quaternion vector consists of four elements. The first element is a real element, related to the angle at which is rotated (θ). The last three elements are complex and are related to the rotation axis (vx, vy, vz).

q = q(1) + q(2)_{· i + q(3) · j + q(4) · k} (2.8) A property of a quaternion is that it has a unit length:

||q|| = q(1)2

+ q(2)2+ q(3)2+ q(4)2 = 1 (2.9) If the orientation of the new frame relative to the original frame (x, y, z) is wanted, the quaternion can be represented as in equation 2.10, in the reverse case the conjugated

values are used. _

    q(1) q(2) q(3) q(4)     =      cos(θ/2) vxsin(θ/2) vysin(θ/2) vzsin(θ/2)      (2.10)

From this formula, the angle and the vector around which is rotated are obtained:

(31)

2.2. KINEMATIC MODEL 11    vx vy vz    =         q(2) sin(θ/2) q(3) sin(θ/2) q(4) sin(θ/2)         (2.12)

This leads to an axis-angle representation. Using the Rodrigues’ rotation formula (2.5) with θ as in Eq. (2.11) and v as in Eq. (2.12) the corresponding rotation matrix is obtained. R =    1_{− 2 · (q(3)}2_{+ q(4)}2₎ ₂_{· (q(2)q(3) − q(1)q(4)) 2 · (q(1)q(3) + q(2)q(4))} 2_{· (q(1)q(4) + q(2)q(3))} 1_{− 2 · (q(2)}2_{+ q(4)}2₎ ₂_{· (q(3)q(4) − q(1)q(2))} 2_{· (q(2)q(4) − q(1)q(3)) 2 · (q(1)q(2) + q(3)q(4))} 1_{− 2 · (q(2)}2+ q(3)2)    (2.13) Due to the property in Eq. (2.9), the rotation matrix is sometimes defined in literature as follows: R =    −1 + 2 · (q(1)2 _{+ q(2)}2₎ ₂_{· (q(2)q(3) − q(1)q(4)) 2 · (q(1)q(3) + q(2)q(4))} 2_{· (q(1)q(4) + q(2)q(3)) −1 + 2 · (q(1)}2_{+ q(3)}2₎ ₂_{· (q(3)q(4) − q(1)q(2))} 2_{· (q(2)q(4) − q(1)q(3)) 2 · (q(1)q(2) + q(3)q(4)) −1 + 2 · (q(1)}2_{+ q(4)}2₎    (2.14) To implement the orientation into the model, the different points P on the hand (figure 2.5) need to be multiplied with the rotation matrix R (2.13).

Pr = R· P (2.15)

2.2.2 Relative finger angles

The position of the different joints on the finger is determined by the extension and flex-ion. Since this means an increasing or decreasing angle between the finger parts, some measure for the different angles is be needed. We name them α1, α2, β1, β2, γ1, γ2, δ1, δ2, 1 and 2 (from the left to the right). The orange circles in figure 2.2 match with the angles1 and the blue ones with the angles2.

Starting from some initial points (green in figure 2.5) and knowing the angles, the relative position of the other points can be calculated.

(32)

Figure 2.5: Joints indicated on each of the fingers

The positions of point A2 to point D3 are calculated below. Figure 2.6 helps to visualise Eq. (2.17)-(2.19): a side view of a simplified finger is given.

- x-direction : stays the same (fingers rotate around the x-axis) - y-direction : P2y = P1y+ lp1· cos(angle1) (2.16) P3y= P2y+ lp2· cos(angle1+ angle2) (2.17) - z-direction : P2z = P1z− lp1· sin(angle1) (2.18) P3z = P2z − lp2· sin(angle1+ angle2) (2.19) With P a point from A to D, lp1 and lp2 the length of the first and second part of finger p respectively and angle the corresponding angle.

In the simplified model it is assumed that the thumb only rotates around the z-axis and therefore the points E2 and E3 are not calculated the same way the fingers were calculated. The principle is the same but the calculations are different. Figure 2.7 helps to visualise these equations: a side view of the simplified thumb is given.

- x-direction :

E2x= E1x− le1· sin(1) (2.20) E3x= E2x− le2· sin(1+ 2) (2.21)

(33)

2.2. KINEMATIC MODEL 13 - y-direction :

E2y = E1y+ le1· cos(1) (2.22) E3y = E2y+ le2· cos(1+ 2) (2.23)

- z-direction : stays the same (thumb rotates around the z-axis)

Figure 2.6: Side view of the simplified finger

(34)

2.3 Visualisation

Now that the kinematics of the hand are explained, a visualisation of the hand can be created. Three different hand visualisations are made, beginning from a realistic hand which has a thickness in the z-direction but which is very slow to a flat but faster visualisation. These are described one by one after which a table is shown which compares the speed of the different models (table 2.4). This is done by plotting the hand during 100 samples in order to obtain a smooth moving visualisation, while the time is measured. At last, a figure of the three visualisation models is shown. For a demo of the real-time visualisation have a look at following video: https://www.youtube.com/watch?v=Jw8-r2WbGfA.

Figure 2.8: Demo of the real-time visualisation of the hand

Visualisation 1 To begin with, the values of the coordinates of the palm (green points in figure 2.5) are defined (getBeginP oints()).

Table 2.1: Initial coordinates for visualisation 1 and 2

Points x y z A1 -4 1 0 B1 -2 3.5 0 C1 0 4 0 D1 2 3.5 0 E1 4.5 0 0 F 2 -8 0 G -3 -8 0

Next, the length of the different finger parts is initialised (getLengths()). Part 2 and 3 form the length of lp2 (figure 2.6) since there is no third angle to define the position of the third phalange separately.

(35)

2.3. VISUALISATION 15 Table 2.2: Lengths of the different finger parts

finger part 1 part 2 part 3

A 3.5 2.5 1.7

B 4.5 3 1.7

C 5 3.5 1.7

D 4.5 3 1.7

E 4 3

As soon as the angles are known, the other points from figure 2.5 are calculated mak-ing use of Eq. (2.17)-(2.23) (calculateOtherP oints()). When the quaternion values are known too, the coordinates of the rotated points can be calculated, making use of Eq. (2.15) (calculateRotatedP oints()) and the plot can begin.

The knuckles are plotted as spheres with a certain radius, dependent on where they are situated on the finger. The parts between the knuckles are plotted as ellipsoids. This is done by first calculating the midpoint of each finger part such that this point can be used as the midpoint of the ellipsoid. The ellipsoid is plotted by making use of the parametric representation and the surf()-function in Matlab:

X = t1+ b· sin(U) · sin(V )

Y = t2+ a· cos(α) · cos(U) − b · sin(α) · cos(V ) · sin(U) Z = t3+ a· sin(α) · cos(U) + b · cos(α) · cos(V ) · sin(U)

With t the midpoint, a the major axis and b the minor axis of the ellipsoid. U and V are vectors that have multiple points between 0 and 2π. In order to plot a 3D palm, some ”new points” are defined. These are equal to the begin points _{±0.8 (in this way} the palm will have a thickness of 1.6). These points are connected in groups of four to form a plane. Doing this in such a way all the planes are connected with each other, the palm is created. To do this, the patch()-function is used. Following these steps, a hand like in figure 2.10a is created.

Plotting these ellipsoids involves the calculation of many points of the surface of the ellipsoid. The number of these points depends on the size of vector U and V and since these vectors consists of many elements, a lot of waypoints are used to plot the surface. As can be seen in table 2.4, this method of visualising the hand is not that fast.

One way to reduce the time to plot is by using low level plot functions: for example use surface() instead of surf(). Another way is to not use axes labels and titles because these functions take a lot of extra time. After applying these solutions, hand 1 still takes too much time to use in a real-time plot and the plot is less clear since no title or names for the axes are being used. Therefore a second visualisation (figure 2.10b) is made.

Visualisation 2 A simplification of the first visualisation is made by using the exact same principle but with the only difference the representation of the finger parts. The

(36)

different links of the fingers are not made by ellipsoids but by a line between 2 points via plot3(), given a certain thickness. Although this plot is faster than the previous one, it is again too slow to use in a real-time application (table 2.4).

Visualisation 3 In both previous solutions the plot consisted of many different plots: the knuckles (19 different spheres), the finger parts (14 ellipsoids or lines) and a palm, made from different planes. Taking this in mind, it is logical that it takes some time to plot since each plot carries some delay. A solution for this would be to plot the hand by using much less parts.

Observing previous visualisations, another problem is remarked: every time an iteration is done, a new plot is created. This takes time. Therefore, it is recommended to initialise the plot before the iteration and adjust the values in the figure each iteration, instead of replotting everything. In this way, the title and the axes can be declared beforehand such that they are fixed. To adjust a plot, the following Matlab function is used:

set(H, Name, Value)

Due to the fact that in hand 1 and hand 2 many different plots are used, there is no way to apply this function in a simple manner to the previous visualisations. Therefore it is suggested that the hand can be plotted with only a few plot-functions. Looking at the way the palm is plotted, the idea rises of plotting the whole hand.

Because the fingers consists of only three points on a line, no such planes can be made to represent a finger. Therefore some width is created by defining new initial points (figure 2.9 and table 2.3). The dots on the left side of the finger correspond to a and the dots on the right side correspond to b.

Figure 2.9: Points on the hand

Points x y z A1a -4.3 1 0 A1b -3.7 1.1 0 B1a -2.4 3.5 0 B1b -1.6 3.6 0 C1a -0.4 4 0 C1b 0.4 3.9 0 D1a 1.6 3.5 0 D1b 2.4 3.4 0 E1a 4.1 0 0 E1b 4.9 -0.2 0 F 2 -8 0 G -3 -8 0

Table 2.3: Initial coordi-nates for visualisation 3

(37)

2.3. VISUALISATION 17 The visualisation is now reduced to only two plot-functions: one for the fingers (Pf ) and one for the palm (Ph):

Pf = patch(’Faces’,ff,’Vertices’,vf,’FaceAlpha’,0.8,’FaceColor’,’yellow’); Ph = patch(’Faces’,fh,’Vertices’,vh,’FaceAlpha’,0.8,’FaceColor’,’yellow’);

With ff and fh the vertices to connect and vf and vh the coordinates of the different vertices. This more simple way of visualisation reduces the plotting time already a lot. When now making use of the set()-function at each time new data comes in, this visual-isation method can work practically real-time.

Comparing the different visualisation methods The time needed to visualise a set of 100 samples for each model is given in the table below. The test is done 15 times and on the last row a mean value is given. It is very clear that each visualisation is faster than the previous one but only the last one can work in real-time.

Table 2.4: Time needed for each hand model to visualise 100 samples Visualisation 1 Visualisation 2 Visualisation 3

30.648560s 21.217669s 0.566770s 29.723115s 20.583778s 0.629672s 30.514270s 22.792250s 0.589957s 29.802155s 20.714529s 0.573352s 30.272225s 21.669021s 0.631801s 27.071309s 17.679667s 0.571384s 28.207066s 18.360135s 0.583892s 27.255395s 22.414494s 0.609964s 26.991313s 19.299084s 0.549383s 27.904284s 18.846411s 0.436246s 28.293138s 20.926876s 0.515690s 29.841667s 18.790585s 0.468686s 29.654703s 19.004493s 0.482052s 28.745213s 18.778337s 0.474349s 36.090837s 19.002373s 0.514891s 29.401018s 20.007976s 0.512533s

(38)

(a) Visualisation 1

(b) Visualisation 2

(c) Visualisation 3

(39)

Chapter 3 Measurement platform

As already mentioned in chapter 2, there are two important aspects to gain information about the hand: the orientation of the palm and the relative position of the fingers. The orientation can be obtained by using an Internal Measurement Unit (IMU) (section 3.1). This is a system that uses the gravity, the magnetic field and the acceleration to give an absolute orientation. The IMU is placed on the palm of the hand. The other aspect, the relative position of the fingers, can be calculated when the values of the angles at the finger joints are known. In order to obtain these angles, bend sensors (section 3.2) will be used. Due to the fact that bend sensors have a variable resistance which changes when they are bent, they are used to convert this resistance value into the corresponding angle. Two bend sensors are placed on each finger. The first one captures the angle of the first knuckle (orange in figure 3.1) and the second one of the second knuckle (blue).

Figure 3.1: Position of the bend sensors

To collect data from these two types of sensors, an Arduino is used to interface with the computer (section 3.3). A simplified diagram of the connections between the hardware is presented in figure 3.2. It shows the use of a multiplexer (section 3.4), which is placed between the bend sensors and the Arduino. This component is used to reduce the amount

(40)

of input pins needed on the Arduino. In figure 3.3, a first sketch of the design of the glove is represented. All the hardware needed, together with a way to mount them on the glove, will be discussed in this chapter.

Figure 3.2: Simplified diagram