Sitting is the new smoking: online complex human activity recognition with smartphones and wearables

Hele tekst

(1)SITTING IS THE NEW SMOKING ONLINE COMPLEX HUMAN ACTIVITY RECOGNITION WITH SMARTPHONES AND WEARABLES. Muhammad Shoaib.

(2)

(3) SITTING IS THE NEW SMOKING ONLINE COMPLEX HUMAN ACTIVITY RECOGNITION WITH SMARTPHONES AND WEARABLES. Muhammad Shoaib.

(4) Graduation committee: Chairman: Promoter: Referee: Members: Professor dr. Professor dr. Professor dr. Professor dr.. Professor dr. ir. P. M. G Apers Professor dr. ing. P. J. M Havinga Assistant Professor Ir. J. Scholten. ir. M. R. van Steen ir. H. J. Hermens ir. W. Kraaij A. T. van Halteren. University of Twente University of Twente Leiden University VU Amsterdam/Philips. This research is supported by the SWELL project within the Dutch National Program COMMIT. CTIT Ph.D.-thesis Series No. 17-436 Centre for Telematics and Information Technology University of Twente P.O. Box 217, NL – 7500 AE Enschede ISSN 1381-3617 ISBN 978-94-028-0653-3. Publisher: Ipskamp Printing Cover design: Muhammad Shoaib c Muhammad Shoaib Copyright .

(5) SITTING IS THE NEW SMOKING ONLINE COMPLEX HUMAN ACTIVITY RECOGNITION WITH SMARTPHONES AND WEARABLES. DISSERTATION. to obtain the degree of doctor at the University of Twente, on the authority of the rector magnificus, Prof. dr. T.T.M Palstra, on account of the decision of the graduation committee, to be publicly defended on Wednesday, May 31, 2017 at 14:45. by. Muhammad Shoaib born on 1st January 1986 in Mardan, Khyber Pakhtunkhwa, Pakistan.

(6) This dissertation is approved by: Professor dr. ing. P. J. M Havinga (promoter).

(7) Acknowledgments. It has been an interesting journey for the last four and half years with some ups and downs. It was a great experience at the end. Many people supported me during this journey in different ways and I would like to thank them for that. First, I would like to pay my deepest gratitude to my promoter, Prof. Paul Havinga and daily supervisor, Hans Scholten for giving me the opportunity to conduct my research under their supervision. Their feedback and guidance played a very important role in the completion of this work. Most importantly, they provided me with the freedom to choose my own research direction, and also encouraged me to work with others which resulted in many important collaborative publications. It allowed me to gain two important skills: to work independently as well as in collaboration with others. Among these collaborators, I am very thankful to Dr. Ozlem Durmaz Incel who was like a mentor to me during these last four years. She has been very helpful with her continuous feedback and guidance. I would like to thank Stephan Bosch for his feedback and critical discussions about this work. I would like to extend my thanks to my other co-authors for their efforts and help: Dr. Nirvana Meratnia, Sumeyye Konak, Stephen Geerlings, and Fulya Turan. Without data, this work was not possible. For this, I am extremely grateful to all the participants who took time from their busy schedules and performed various activities for my data collection experiments. One again thanks to Hans Scholten who translated the abstract of this thesis into the Dutch language. I would like to thank Prof. Hermie Hermens, Prof. Wessel Kraaij, Prof. Art van Halteren, and Prof. Maarten van Steen for being part of my graduation committee and for their time to review this thesis. This journey started a while ago when I got Huygens Scholarship for my research-oriented master program at the Vrije University Amsterdam. For this, I am extremely thankful to Prof. Maarten van Steen who recommended me for this scholarship program to pursue my master studies. Since then it has been a wonderful journey for me in many ways. For this, I would also like to thank my friend Suhail Yousaf who guided me at that time about possible higher studies.

(8) vi in Netherlands. It was a pleasure working alongside my (ex) colleagues from Pervasive Systems group and SWELL project (Vignesh R. K. Ramachandran, Eyuel Debebe Ayele, Okan Turkes, Kyle Zhang, Viet Duc Le, Jan-Pieter Meijers, Jacob Kamminga, Fatjon Seraj, Kallol Das, and Wei Wang). I am also thankful to our wonderful support staff (Nicole Baveld, Thelma Nordholt, Marlous Weghorst) for all their help. I would like to thank all my friends for their support and encouragement, both here in Enschede and the rest of the world. The list is long but I would like to say special thanks to Tauseef Ali and Yawar Abbas who made my stay in Enschede very comfortable and enjoyable. Finally, I would like to thank my family, especially my parents, for their love and support in their own ways without even really understanding the concept of a Ph.D. At the end, I would like to pay special thanks to my girlfriend Swantje for being there in my life, for her love, and for her continuous encouragement and support all this time. It has been a challenging but a rewarding journey. The journey is what bring us happiness, not the destination (Way of the Peaceful Warrior)..

(9) Abstract. Human activity recognition plays an important role in the development of various applications about fitness tracking, health monitoring, context-aware feedback and self-management of a smartphone or a wearable device. For example, users can monitor their activities in real-time as well as its history over longer periods of time. Health professionals can monitor the daily routine (activities) of patients and any deviations from such routine. It can be used for giving feedback at the right time, such as the device can disable incoming calls while a user is jogging or is in a meeting. In terms of self-management, the device can dynamically turn on and off various sensors, data features, and classifiers depending on the current activity to save resources. For example, a smartphone or a smartwatch can turn off gyroscope and use accelerometer only if a user is sitting or disable WiFi when the user is jogging, thereby saving battery life. Many people already use smartphones in their daily life and there has been an increase in the use of smart wearables, such as smartwatches in recent years. These devices are equipped with different sensors such as accelerometer, gyroscope, and magnetometer which can provide information that can be used to recognize various human activities. Therefore, we mainly use these devices for human activity recognition in our research. A significant amount of work has been done in human activity recognition by different researchers. However, most of the work focuses on simple physical activities. Simple activities are periodic in nature and can be easily recognized; for example, walking, jogging, biking, writing, typing, sitting and standing. Complex activities may involve hand gestures which are not periodic; for example, eating, drinking coffee, smoking and giving a talk. Most of the existing work has been performed offline where data is collected from a smartphone or a wearable device, but the activity recognition process is carried out offline on a desktop machine at a later stage. In the online method, this process is performed in real-time on the device. In this context, we investigate the recognition of both simple and complex activities using different sensors from smartphones and.

(10) viii wearables. We do so both in offline as well as in online mode. To this end, we address the following questions: How to recognize various human activities using different sensors from wearable devices and smartphones both offline and online (on the device)? What are the tradeoffs between different data features, sampling rates, segmentation window sizes, sensor positions on human body, sensors, and classification methods and its consequences on resources consumption and recognition performance of various human activities? To address these questions, we started with data collection experiments where we collected multiple datasets for various human activities over time. We use these datasets for investigating different aspects of human activity recognition using smartphones and wearables. We describe the main contributions of this thesis as follows: •. We investigate the recognition of both simple and complex human activities using various machine learning algorithms. Based on this analysis, we provide recommendations on how and when to use certain sensors, classifiers and body positions for the recognition of a specific activity.. •. We propose to use a hierarchical lazy classification approach for the recognition of complex activities involving hand gestures such as smoking and other similar activities. It uses neighboring information among the data segments on top of a classifier in correcting misclassified segments. We show the performance improvements of this algorithm compared to a single layer classification approach for smoking, eating and drinking activities.. •. We developed an online activity recognition framework for smartphones and smartwatches. Based on this framework, we implement a prototype application for these devices which can recognize various human activities in real-time. As an example use case, we use the smartphone for recognizing seven physical activities, whereas the smartwatch is used for smoking recognition. We also implement a smoking session detection algorithm using a hierarchical approach. We investigate the resource consumption (CPU, memory, and power) of our online activity recognition system on a mobile phone and a smartwatch with respect to different aspects. We also tested this system for three weeks for recognizing various activities in real-time and observed recognition results are encouraging. Based on our offline and online analysis, we propose a context-aware activity recognition (AR) algorithm that can adapt different aspects of the AR process in real-time to save resources..

(11) Samenvatting. Herkenning van menselijke activiteiten speelt een belangrijke rol bij het ontwikkelen van “smartphone” applicaties voor het continu meten van conditie en gezondheid, of het geven van feedback aan de gebruiker ervan. Het geeft zowel de gebruiker alsook, bijvoorbeeld, een dokter of fysiotherapeut de mogelijkheid de conditie over korte en lange termijn te meten. Gezondheidsprofessionals kunnen zo afwijkingen van de dagelijkse routine ontdekken en dit met hun patiënt bespreken. In het dagelijks leven kan het herkennen van activiteiten gebruikt worden om op het juiste moment feedback te geven, of om te voorkomen dat een gebruiker op het verkeerde moment wordt gestoord door automatisch de ringtone uit te zetten tijdens een vergadering. Ook kan het gebruikt worden om de smartphone zuiniger met energie om te laten gaan door op dat moment overbodige sensoren uit te schakelen. Bijvoorbeeld, als de gebruiker zit hoeven gyroscoop en versnellingsmeter niet beide gebruikt te worden, een ervan is voldoende. Smartphones en “wearables” (kleine draagbare apparaten) zoals een “smartwatch” worden al veelvuldig gebruikt. Ze zijn voorzien van een scala aan sensoren, zoals versnellingsmeter, gyroscoop en magnetometer en zijn geschikt voor het meten van diverse menselijke activiteiten. Het onderzoek beschreven in dit proefschrift richt zich derhalve hoofdzakelijk op het gebruik van dergelijke apparaten. Er is door verschillende onderzoeksgroepen al een aanzienlijke hoeveelheid onderzoek gedaan naar herkenning van activiteiten. In de meeste gevallen gaat het echter om het herkennen van simpele activiteiten, zoals wandelen, rennen, fietsen, schrijven, zitten of staan. Deze activiteiten zijn repeterend en zijn gemakkelijk te herkennen. Complexe activiteiten echter zijn niet-repeterend en omvatten vaak handgebaren. Voorbeelden zijn eten, drinken van koffie, roken of het geven van een presentatie. In de meeste bestaande onderzoeken wordt de data eerst verzameld, terwijl de verwerking ervan later “offline” plaatsvindt op een pc. In ons onderzoek is de data zowel offline als “online” verwerkt, voor simpele en complexe activiteiten. Online betekent hier dat de verwerking van de data niet achteraf, maar direct “real-time” gebeurt op de smartphone of de smartwatch zelf. In dit proefschrift worden de volgende.

(12) x onderzoeksvragen geadresseerd: Hoe kunnen menselijke activiteiten worden herkend, zowel offline als online, gebruikmakend van de verschillende sensoren in smartphones en smartwatches. Wat zijn de trade-offs tussen verschillende data functies, bemonsteringsfrequenties, bemonsteringsperiodes, posities van sensoren op het lichaam en classificatiemethodes? En wat zijn de consequenties ervan voor het energieverbruik en de mate van activiteitsherkenning? Om deze vragen te beantwoorden is dit onderzoek begonnen met experimenten waarbij data over langere periodes is verzameld van verschillende menselijke activiteiten. De aldus verkregen datasets verkregen met smartphones en wearables zijn gebruikt om de verschillende aspecten van activiteiten te onderzoeken. De bijdragen van dit onderzoek is in de volgende punten samengevat: •. Gebruikmakend van verschillende “machine learning” algoritmes zijn zowel simpele als complexe activiteiten geanalyseerd. Dit heeft geleid tot aanbevelingen voor gebruik van sensoren (welke sensoren, positie en oriëntatie van sensoren) en classificatiemethodes voor het herkennen van bepaalde activiteiten.. •. Voor de herkenning van complexe activiteiten waar handbewegingen een rol spelen, zoals roken en dergelijke, wordt het gebruik van een hiërarchische “lazy” classificatie aanbevolen. Om fouten in de classificatie van een data segment te corrigeren wordt aanvullend informatie van aangrenzende segmenten gebruikt. We tonen aan dat deze aanpak een verbetering is ten opzichte van een enkele classificatie voor activiteiten als roken, eten en drinken.. •. Er is een raamwerk ontwikkeld voor online activiteitsherkenning op smartphones en smartwatches. Met behulp van dit raamwerk is een prototype app gemaakt waarmee in real-time activiteiten kunnen worden herkend. Met deze app kunnen zeven verschillen activiteiten met een smartphone worden herkend. Een implementatie op een smartwatch herkent wanneer de gebruiker rookt. Speciaal voor het roken is een hiërarchisch detectiealgoritme ontwikkeld. De app is gebruikt voor het meten van verschillende parameters op zowel de smartphone als de smartwatch, zoals CPU belasting, geheugengebruik en energieconsumptie. De app is gedurende enkele weken getest en de resultaten van het herkennen van activiteiten zijn bemoedigend. Gebaseerd op de analyse van de tests wordt een nieuw herkenningsalgoritme voorgesteld dat zich aanpast aan de situatie en activiteit en daardoor efficiënt met CPU, geheugen en energieconsumptie omgaat..

(13) Contents. 1. 2. 3. 4. Introduction 1.1 Introduction . . . . . . . . . . . . . . . 1.2 Human Activity Recognition Process . 1.3 Research Objective and Questions . . 1.4 Thesis Contributions . . . . . . . . . . 1.5 Datasets for Performance Evaluations 1.6 Thesis Organization . . . . . . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. . . . . . .. 1 1 4 6 7 10 11. Introduction to Human Activity Recognition 2.1 Introduction . . . . . . . . . . . . . . . . . 2.2 Related Work . . . . . . . . . . . . . . . . 2.3 Online Activity Recognition . . . . . . . . 2.4 Discussion . . . . . . . . . . . . . . . . . . 2.5 Conclusions . . . . . . . . . . . . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 13 14 16 16 34 38. Simple Physical Activity Recognition with Smartphones 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . 3.3 Data Collection . . . . . . . . . . . . . . . . . . . . . . 3.4 Preprocessing Data . . . . . . . . . . . . . . . . . . . . 3.5 Evaluation Approach . . . . . . . . . . . . . . . . . . . 3.6 Performance Analysis and Discussion . . . . . . . . . 3.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 41 42 44 46 47 50 53 68. Complex Human Activity Recognition with Smartphones 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . 4.3 Data Collection and Experimental Setup . . . . . . . . 4.4 Results and Discussion . . . . . . . . . . . . . . . . . . 4.5 Conclusions . . . . . . . . . . . . . . . . . . . . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 71 72 74 76 80 96. . . . . . ..

(14) xii. CONTENTS. 5. A Hierarchical Lazy Classification Approach for Activity Recognition 97 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 5.3 Data Collection and Preprocessing . . . . . . . . . . . . . . . . . 102 5.4 A Hierarchical Lazy Classification Algorithm . . . . . . . . . . . 104 5.5 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . 106 5.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110. 6. Online Activity Recognition Using Smartphones and Smartwatches 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Framework for Online Activity Recognition . . . . . . . . . . . . 6.3 Training Models for Online Classification . . . . . . . . . . . . . 6.4 Resource Consumption Analysis . . . . . . . . . . . . . . . . . . 6.5 Performance Evaluation . . . . . . . . . . . . . . . . . . . . . . . 6.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 7. Conclusions and Future Work 143 7.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . 143 7.2 Lessons Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 7.3 Future Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147. 113 114 115 119 121 128 141. A Appendix A 149 A.1 Classification results of various classifiers . . . . . . . . . . . . . 149 B Appendix B 156 B.1 Classification Results (KNN and Decision Tree) . . . . . . . . . . 156 C Appendix C 163 C.1 Online Human Activity Recognition . . . . . . . . . . . . . . . . 163 Bibliography. 166. About the author. 182.

(15) CHAPTER 1. Introduction. 1.1. Introduction. Physical activities play a very important role in people’s physical and mental well-being. The lack of such activities can negatively affect their well-being in general [1]. According to the World Health Organization, physical inactivity alongside smoking, poor nutrition, and harmful use of alcohol are the main reasons for premature deaths [2]. Therefore, there is a need for a proper health and lifestyle coaching that detects a user’s activity and his or her situation in real time and provides him or her with the right motivational feedback at the right time. The first step towards achieving such a goal is to recognize various human activities in our daily life. The recognition of various human activities can enable different applications about fitness tracking, health monitoring, context-aware feedback and self-management of a smartphone or a wearable device [3, 4]. For example, health professionals can monitor the daily routine (activities) of patients and any deviations from such routines. People can track their daily activities and get an insight into their daily lifestyle [4]. Various high-level habits (good or bad) can be detected by recognizing low-level physical activities or triggers [3]. Some examples of such habits or behaviors are drinking too much coffee, lack of physical activity, smoking, and missing meals or not taking meals on time. The recognition of hand gestures involved in smoking, drinking, and eating can lead to the detection of these high-level behaviors or habits over a longer period of time. The recognition of such activities can be used in giving context-aware feedback [3]. For example, a user should not be interrupted for a feedback message while typing, writing or giving a talk, but can be interrupted while having a cup of coffee or smoking. Note that this example is for the explanation purpose because users may see the same activity differently in terms of.

(16) 2. 1 Introduction. importance as for as the interruption is concerned. For example, interruption, while watching TV or sitting idle may be fine with one person but annoying for another person. Such coaching mechanisms, if implemented practically, can help in improving people’s well-being [5, 3], thereby helping in reducing the overall cost for governments, employers and insurance companies caused by people’s ill health and unhealthy lifestyle [5, 3]. The context-aware feedback is not limited to the e-health applications and can be used in other scenarios as well. For example, if a smartphone or a wearable device knows what its user is doing at a specific moment, it can act accordingly in terms of notifications and feedback for various applications running on these devices. Many other such use cases and examples for human activity recognition are discussed in [4]. In terms of self-management, a smart device can dynamically use various sensors, data features and classifiers depending on the current activity to save resources. For example, a smartphone can turn off gyroscope and use accelerometer only if a user is sitting or disable WiFi when the user is jogging, thereby saving battery life. Human activities can be recognized using various sensors, mainly in two ways [6]: 1) by ambient sensors, and 2) by wearable sensors. Ambient sensors can be installed in the user’s environment such as in a home or office. Its main drawback is its static nature and therefore cannot be used outside these fixed places when the user is mobile. On the other hand, wearable sensors can be used in all situations. However, they can be interpreted as being obtrusive. In this thesis, we focus on smartphones and smartwatches which can be considered as wearable sensors. We focus on these two devices because they are less obtrusive and a lot of people already use them in their daily life. Once an application is built using these devices for human activity recognition, it can also be used in combination with ambient sensors at home and office for more contextual information. For this purpose, smartphones have been extensively studied for recognizing different physical activities in recent years [7, 8, 9]. They are equipped with different sensors (accelerometer, gyroscope, magnetometer, GPS) which can be used in developing applications based on human activity recognition. The use and capability of smartphones for motivating people to be physically active are discussed in detail in [10, 11, 4, 3]. Recently, smartwatches are also being used for the same purpose because they are also equipped with various sensors such as accelerometer and gyroscope. However, unlike smartphones, activity recognition using smartwatch sensors is relatively new [12, 13, 14, 15, 16]. Most of the research on human activity recognition using mobile phones and smartwatches is done offline in machine learning tools [17, 18, 19, 20, 21, 22, 12, 13, 14, 15, 16]..

(17) 1.1 Introduction. 3. Some of the examples of such tools are WEKA [19], Scikit-learn [23], and MATLAB. Most of the work on human activity recognition using mobile phones was done offline because mobile phones were initially considered as resource-limited devices [24]. For example, they did not possess enough battery resources (lower mAh) to run activity recognition systems for an extended period. Moreover, it is a challenging task to implement and evaluate different recognition systems on these devices. However, in recent years, mobile phones have become capable of running such recognition systems, so there has been a shift towards online activity recognition. For example, we have shown in Section 2.3.4 that for various mobile phones, the battery capacities have increased from 950 mAh in 2008 to 1500 mAh in 2013. Moreover, machine learning tools such as WEKA provides a way to train machine learning algorithms offline and then port it to the mobile devices, thereby making the implementation of such systems relatively easy. Therefore, researchers are moving towards online activity recognition (on the devices). There are a number of studies where activity recognition has been implemented on mobile phones for real-time processing. In some of these studies, the aim is to show that online recognizers can work on mobile phones considering the available resources, while in other studies, the aim is to develop an application where the activities of the users can be tracked, such as a mobile diary or a fitness tracker [4]. Unlike smartphones, most of the work using smartwatch sensors is still being done offline [12, 13, 14, 15, 16]. In this thesis, we initially start with data collection experiments, offline analysis of these datasets, and then we move towards online human activity recognition on smartphones and smartwatches. In existing work, the main initial focus was on the recognition of simple physical activities using an accelerometer. In some studies, the accelerometer had been combined with other sensors, but without evaluating the impact of those sensors individually. This is termed as blind fusion. We investigated the impact of various aspects on the recognition of both simple and complex human activities such as the orientation of the sensors, body positions, activity types, sensors’ individual use and their combinations, personal and generic classifiers, data features, sampling rates, window sizes etc. Simple activities are periodic in nature and can be easily recognized; for example, walking, jogging, biking, writing, typing, sitting and standing. Complex activities may involve various hand gestures which are not periodic in nature; for example, eating, drinking coffee, smoking and giving a talk. Based on our analysis of the recognition of these activities, we proposed various recommendations on how and when to combine various sensors depending on the body position, activity type, and other aspects. We also proposed.

(18) 4. 1 Introduction. a hierarchical lazy classification approach for the recognition of the complex activities. We also developed a framework for online activity recognition (AR) where various aspects of AR process can be evaluated in similar experimental setup. Based on this framework, we implemented an online activity recognition system for Android smartphones and smartwatches. We discuss all the relevant related work associated with each contribution in the respective chapters of this thesis as described in Section 1.4.. 1.2. Human Activity Recognition Process. Activity recognition systems can be categorized into the following main steps [25] and are shown in Figure 1.1:. 1. raw data 3. 2 1 ra. w. d. ata. feature extraction. storage. for offline training. training: calculation of model parameters. sensing. preprocessing. samples. trained classifier. features. feature extraction features. sensors 2. raw data. sensing. preprocessing. samples. sensors. 3. classification. 4a. activity classes 4b. Figure 1.1: Activity recognition steps. (1). (2). Sensing: In this step, data is collected from various sensors at a certain sampling rate. The choice of a specific sampling rate depends on the use case and sensor type. For an accelerometer, various sampling rates are used between a range of 5 to 100 samples per second in different activity recognition studies [7]. Preprocessing: Subsequently, the collected data are processed in various ways. For example, noise is removed. Then, a windowing or segmenta-.

(19) 1.2 Human Activity Recognition Process. (3). (4-a). (4-b). 5. tion scheme is applied to it. Various segmentation sizes can be chosen depending on different factors such as use case, response time, and activity type. Feature extraction: Various data features are extracted from the segmented raw data which can be used to discriminate different activities. These features can be of different types such as time-domain and frequencydomain. Some of the common time-domain features are mean, standard deviation, variance, etc. Training: As shown in Figure 1.1, training is a preparation step to obtain the model parameters by learning from already labeled data for later use in classification (supervised). As shown in Figure 1.2, training can either be offline on a desktop machine or online on the phone or a wearable device itself. Classification: In this final stage, the trained classifiers are used to classify new examples of data into different activities. This step can be done either offline in a machine learning tool, such as WEKA [26], or online on the mobile phone itself as shown in Figure 1.2.. Training. Classification. Preprocessing and Feature Extraction. Training (Offline). Preprocessing and Feature Extraction (offline). Classification. Training. Preprocessing and Feature Extraction Sensing. Sensing. Figure 1.2: Offline and online activity recognition process. Mobile phones are being used in two ways for online activity recognition:.

(20) 6. 1 Introduction. client-server approach and local approach. In the client-server approach, the sensing part is done on the mobile phone, which acts as a client. The collected data are then sent to a server or a cloud for further real-time processing, such as preprocessing and classification. The preprocessing step can partially reside on the mobile phone, too. However, the main classification step is performed on a server. This approach is adapted in order to run the computationally-expensive steps on a server, because of the limited resources of a mobile device or a wearable. For example, in [27], raw data are sent to a server for classification. This approach requires an Internet connection at all times for sending sensor data for further processing to a server or a cloud. On the other hand, in a local approach, activity recognition steps are done locally on the mobile phone in real time. These steps include data collection, preprocessing and classification. In this approach, information about classified activities and raw data can also be sent to a server for further analysis. However, the main three steps are performed locally. The training can still be done on a desktop machine beforehand or on the phone locally. This approach is shown in Figure 1.2. In this thesis, we mainly focus on the local approach because it does not require the Internet all the time, and it can be considered more privacy friendly as the data stays on the user’s device. We also want to investigate the potential of smartphones and wearables (smartwatches) for running activity recognition process locally. Further details on the activity recognition process are presented in Chapter 2.. 1.3. Research Objective and Questions. The recognition of various human activities depends on different factors such as classification methods, sensor types, sensors’ position on the human body, extracted features, segmentation window for the feature extraction, and sampling rates. For example, certain activities can be recognized better on a specific body position or with a specific sensor compared to others. Smoking or eating are the types of activities which cannot be recognized using motion sensors in pocket position, however, they can be recognized using these sensors at the wrist position or its combination with the pocket position. Similarly, certain activities can be recognized better by a specific sensor or a combination of different sensors. For example, jogging can be better recognized using an accelerometer whereas using stairs activity can be better recognized with the combination of accelerometer and gyroscope. To this end, the goal of this thesis is to investigate the recognition of various human activities (both simple and complex) using different sensors, embedded in wearable devices and smartphones. In order to.

(21) 1.4 Thesis Contributions. 7. achieve this goal, we address the following main research question: How to recognize various human activities using sensors from smartphones and wearables such as a smartwatch? We divide this main research question in the following three sub-questions: •. RQ1: How to use data from different wearable sensors (mainly motion) for reliable human activity recognition (both offline and online)?. •. RQ2: What are the tradeoffs between different data features, segmentation window sizes, sensor positions on the human body, sensors, and classification methods and its consequences on the recognition performance of various human activities?. •. RQ3: How do various sensors, classifiers, sampling rates, and window sizes affect the resource consumption of the smartphones and wearable devices such as a smartwatch?. We address these three questions in all chapters. RQ1 and RQ2 are addressed in the context of simple physical activity recognition in Chapter 3 and in the context of complex activity recognition in Chapter 4. These questions are also addressed in Chapter 5 in the context of hierarchical classification approach for recognizing activities involving hand gestures such as smoking, eating, and drinking. RQ1, RQ2, and RQ3 are addressed in Chapter 6 in the context of online human activity recognition on mobile phones and smartwatches where we discuss two use cases: physical activity recognition and smoking recognition.. 1.4. Thesis Contributions. To address our research questions, we first performed data collection experiments where many participants were asked to perform various human activities while they were carrying smartphones and smartwatches. We used these datasets in an offline analysis using machine learning algorithms for human activity recognition. We performed our offline analysis in two tools: WEKA [26] and Scikit-learn [23]. We investigated both simple human activities such as walking, jogging, sitting, standing, biking and complex activities such as smoking, drinking, eating. In the final stage, we developed a framework for online activity recognition using smartphones and smartwatches. Based on this framework, we developed a prototype system in Android for real-time activity recognition for smartphones and smartwatches. In the form of this thesis, we contributed to the existing work in human activity recognition in different ways. We describe these contributions below:.

(22) 8. 1 Introduction. •. Contribution 1 (Chapter 2): We conducted an extensive literature study on existing work in human activity recognition, both offline and online. We discussed the gaps in existing work which needs to be worked on and also recommended guidelines for future research in activity recognition. This work has been published in the following two papers:. •. –. Shoaib, M. and Bosch, S. and Durmaz Incel, O. and Scholten, J. and Havinga, P.J.M. (2015) A Survey of Online Activity Recognition Using Mobile Phones. Sensors (Switserland), 15 (1). pp. 2059-2085.. –. Shoaib, M. and Bosch, S. and Durmaz Incel, O. and Scholten, J. and Havinga, P.J.M. (2015) Defining a Roadmap Towards Comparative Research in Online Activity Recognition on Mobile Phones. In Proceedings of the 5th International Conference on Pervasive and Embedded Computing and Communication Systems, PECCS 2015, 11-13 Feb 2015, Angers, France . pp. 154-159. IEEE.. Contribution 2 (Chapter 3): We provide recommendations on how and when to use certain sensors, data features, classifiers, and body positions for the recognition of a specific activity. For this purpose, we investigated the recognition of seven simple physical using various machine learning algorithms. We evaluated the impact of various sensors (individually and in combination with each other), classifiers, five body positions, data features, and sampling rates on the recognition performance of these activities. We collected two datasets with 14 participants which are publicly available [28]. We developed an Android application for data collection which is also available for public use [28]. This contribution has been published in the following publications: –. Shoaib, M. (2013) Human Activity Recognition Using Heterogeneous Sensors. In Adjunct Publication of the 2013 ACM Conference on Ubiquitous Computing, UbiComp’13 Adjunct, 8-12 Sept 2013, Zurich, Switzerland. ACM.. –. Shoaib, M. and Scholten, J. and Havinga, P.J.M. (2013) Towards physical activity recognition using smartphone sensors. In: 10th IEEE International Conference on Ubiquitous Intelligence and Computing, UIC 2013, 18-20 Dec 2013, Vietri sul Mare, Italy. pp. 80-87. IEEE.. –. Shoaib, M. and Bosch, S. and Durmaz Incel, O. and Scholten, J. and Havinga, P.J.M. (2014) Fusion of Smartphone Motion Sensors for Physical Activity.

(23) 1.4 Thesis Contributions. 9. Recognition. Sensors, 14 (6). pp. 10146-10176. •. •. Contribution 3 (Chapter 4): We investigated the recognition of simple and complex activities involving hand gestures and proposed to use motion information from both wrist and pocket position for their better recognition. We also investigated the impact of varying segmentation window size on the recognition of simple and complex activities because they are affected in different way with an increasing window size. We also evaluated the effect of synchronization delay between the smartwatch and smartphone on the recognition performance of various human activities. We also collected two datasets for these studies which are made public [28]. These results are published in the following publications: –. Shoaib, M. and Bosch, S. and Scholten, J. and Havinga, P.J.M. and Durmaz Incel, O. (2015) Towards detection of bad habits by fusing smartphone and smartwatch sensors. In Proceedings of the IEEE International Conference on Pervasive Computing and Communication Workshops, PerCom Workshops 2015 , 23-27 Mar 2015, St. Louis, MO, USA. 591 -596. IEEE.. –. Shoaib, M. and Bosch, S. and Durmaz Incel, O. and Scholten, J. and Havinga, P.J.M. (2016) Complex Human Activity Recognition Using Smartphone and Wrist-Worn Motion Sensors. Sensors, 16 (4). 426:1-426:24.. –. Konak, Sumeyye and Turan, Fulya and Shoaib, M. and Durmaz Incel, O. (2016) Feature Engineering for Activity Recognition from Wrist-Worn Motion Sensors. In: Proceedings of the 6th International Conference on Pervasive and Embedded Computing and Communication Systems, PECCS 2016, 25-27 Jul 2016, Lisbon, Portugal. IEEE.. Contribution 4 (Chapter 5): We proposed to use a hierarchical lazy classification approach for the recognition complex activities involving hand gestures such as smoking and other similar activities. It uses neighboring information among the data segments on top of a classifier in correcting misclassified segments. We showed the performance of this algorithm compared to a single layer classification approach for smoking, eating and drinking activities. For this purpose, we collected a dataset for smoking, eating, drinking, and other activities. To the best of our knowledge, it is the largest smoking dataset using smartwatch sensors. Eleven participants took part in the experiments over a period of three months. These results have been published in the following publication:.

(24) 10. 1 Introduction –. •. 1.5. Shoaib, Muhammad, Hans Scholten, Paul JM Havinga, and Ozlem Durmaz Incel. "A hierarchical lazy smoking detection algorithm using smartwatch sensors." In 18th IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) 2016, pp. 1-6. IEEE.. Contribution 5 (Chapter 6): We developed an online activity recognition framework for mobile devices and smartwatches. Based on this framework, we implemented a prototype android application for these devices which can recognize various human activities in real-time. For a use case, we used the smartphone application for recognizing seven physical activities whereas the smartwatch application for smoking recognition. We also developed a smoking session detection algorithm using a hierarchical approach. We investigated the resource consumption (CPU, memory, and power) of our online activity recognition system on a mobile phone and a smartwatch with respect to various aspects. We also tested this system for three weeks for recognizing various activities in real-time and reported these results. Based on our observations, we recommend how and when to use different sensors and classifiers for online activity recognition on mobile phones and smartwatches. Based on our offline and online analysis, we propose a context-aware activity recognition (AR) algorithm that can adapt various aspects of the AR process in real-time to save resources.. Datasets for Performance Evaluations. We collected six datasets to evaluate the impact of various aspects on the recognition performance of different simple and complex human activities. For this purpose, we developed two Android applications for data collection. One was specifically for android mobile phones where one can collect various sensor information at a user-provided sampling rate. The second one was for both Android phones and android smartwatches which can collect sensor data from both these devices at the same time. Both these applications were made publicly available for other researchers to use for future research and also to reproduce our work [28]. We present a brief summary of these datasets here in Table 1.1. These datasets will be discussed in the relevant chapters of this thesis as shown in Table 1.1. Two of our datasets are mainly focused on physical activities such as walking, sitting, standing, jogging, biking and using stairs (Chapter 3). The other two focus on an extended set of activities (thirteen activities) which include writing,.

(25) 1.6 Thesis Organization. 11. Table 1.1: Description of Datasets used in this Thesis. Simple_activities_small Simple_activities_extended Complex_activities_small Complex_activities_extended. Number of participants 4 10 10 10. Number of activities 7 7 13 13. Dataset_Smoking. 11. 4. Dataset_Smoking_online. 11. 2 (smoking vs. others). Dataset. 1. Sensors1 A, G, M, GPS A, G, M, LA, GPS A, G, M, LA, GPS A, G, M, LA, GPS A, G, M, LA, GPS, Barometer, Blood-pressure A, G, M, LA, GPS, Barometer, Blood-pressure. Relevant Chapters 3 3 4 4. Results Published in [20] [21] [29] [30]. 5. [31]. 6. NA. A: accelerometer, G: gyroscope, M: magnetometer, LA: linear acceleration sensor, NA: not available. typing, drinking coffee or tea, giving a talk, smoking, and eating (Chapter 4). One dataset was collected mainly for smoking, however, it also includes other similar activities such as eating, and drinking (Chapter 5). For Chapter 6, we collected additional data for the real-time smoking recognition system and it contains many daily activities such as biking, cooking, washing dishes, drinking, sitting and laying in different postures, and working on a computer, etc.. 1.6. Thesis Organization. We present the organization of this thesis in Figure 1.3. In Chapter 2, we present an introduction to human activity recognition process and a detailed review of the existing work on human activity recognition. We propose various recommendations for future research in this area. In Chapter 3, we analyze the role of various smartphone sensors in the context of simply physical activity recognition. In Chapter 4, we move towards the recognition of relatively complex activities such as giving a talk, eating, smoking, and drinking coffee. In Chapter 5, we propose the use of a hierarchical lazy classification approach for the recognition of complex activities. In Chapter 6, we present our framework for online activity recognition and the prototype implementation of our real-time activity recognition system based on this framework. We present the tradeoff of various aspects of human activity recognition process and its consequences on the resource consumption and recognition performance for recognizing different human activities. In Chapter 7, we present our conclusions and future research directions..

(26) 12. 1 Introduction Introduction. 1. Introduction to human activity recognition. 4. 3 Offline activity recognition. Simple physical activity recognition with smartphones. Online activity recognition. 2. 5 A hierarchical classification approach for activity recognition. Complex human activity recognition with smartphones. Online activity recognition with smartphones and wearables (smartwatches). Conlcusions and Future work. Figure 1.3: Thesis Organization. 7. 6.

(27) CHAPTER 2. Introduction to Human Activity Recognition. Physical activity recognition using embedded sensors has enabled many contextaware applications. Initially, one or more dedicated wearable sensors were used for such applications. However, recently, many researchers started using mobile phones for this purpose, since these ubiquitous devices are equipped with various sensors, ranging from accelerometers to magnetic field sensors. In most of the current studies, sensor data collected for activity recognition are analyzed offline using machine learning tools. There is now a trend towards implementing activity recognition systems on these devices in an online manner, since modern mobile phones have become more powerful in terms of available resources, such as CPU, memory, and battery. In this chapter, we present the human activity recognition process and review the studies done so far that implement activity recognition systems on mobile phones which use only their onboard sensors. We discuss various aspects of these studies, and their limitations if any. We present various recommendations for future research in activity recognition.. This chapter is mainly based on: •. Shoaib, M., Bosch, S., Durmaz Incel, O., Scholten, J., Havinga, P.J.M. (2015) A Survey of Online Activity Recognition Using Mobile Phones. Sensors (Switserland), 15 (1). pp. 2059-2085.. •. Shoaib, M., Bosch, S., Durmaz Incel, O., Scholten, J., Havinga, P.J.M. (2015) Defining a Roadmap Towards Comparative Research in Online Activity Recognition on Mobile Phones. In Proceedings of the 5th International Conference on Pervasive and Embedded Computing and Communication Systems, PECCS 2015, 11-13 Feb 2015, Angers, France . pp. 154-159. IEEE..

(28) 14. 2.1. 2 Introduction to Human Activity Recognition. Introduction. Human activity recognition has enabled novel applications in different areas, such as healthcare, security and entertainment [32, 6]. Initially, dedicated wearable motion sensors were used to recognize different physical activities [6, 33, 32, 34, 35]. However, there has been a shift towards mobile phones in recent years, because of the availability of various sensors in these devices. Examples of such sensors are GPS, accelerometer, gyroscope, microphone and magnetometer. Most of the research on human activity recognition using mobile phones is done offline in machine learning tools, such as WEKA [17, 18, 19, 20, 21, 22]. Mobile phones were initially considered as resource-limited devices [24]. For example, they did not possess enough battery resources (lower mAh) to run activity recognition systems for an extended period. Moreover, it is a challenging task to implement and evaluate different recognition systems on these devices. However, in recent years, mobile phones have become capable of running such recognition systems, so there has been a shift towards online activity recognition. For example, we have shown in Section 2.3.4 that for various mobile phones, the battery capacities have increased from 950 mAh in 2008 to 1500 mAh in 2013. There are a number of studies where activity recognition has been implemented on mobile phones for real-time processing. In some of these studies, the aim is to show that online recognizers can work on mobile phones considering the available resources, while in other studies, the aim is to develop an application where the activities of the users can be tracked, such as a mobile diary or a fitness tracker [4]. There is a number of survey publications that have reviewed the work done so far in this area [9, 32, 36, 37, 38]. Though these surveys have partially covered online activity recognition, their focus is mainly on studies with offline analysis. They are generic studies covering different aspects of context-aware applications on mobile phones and wearable sensors. In this chapter, we mainly focus on the online activity recognition using solely the mobile phone sensors. By online activity recognition, we mean that the data collection, preprocessing and classification steps are done locally on the mobile phone. In some cases, the online activity recognition is done on a remote server or in a cloud, we do not consider such studies, as discussed in Section 2.3. Because we report all studies done so far on online activity recognition using mobile phones only, we believe that this study will help researchers in the future work in this area of research. It is important to note that “online activity recognition on smartphones” should not be confused with “online machine learning models”. “Online machine.

(29) 2.1 Introduction. 15. learning models” are able to adapt themselves according to new data points, unlike offline or batch learning models [39]. The details of online vs. offline learning models can be found in [39], but this is not the focus of our chapter. We use the “online” term in a different way, for the practical implementation of activity recognition systems on mobile phones. These implemented systems can be using either an online or a batch learning model. In this chapter, we focus on the work in which such systems have been implemented on mobile phones. We are interested in systems that can recognize different physical activities. For comparing these studies, we use different criteria, such as classification methods, experimental setups, position and orientation independence, real-time feedback, assistive feedback, evaluation methods, dynamic and adaptive sensor selection, adaptive sampling and resource consumption analysis. Because we are only interested in studies that focus on online activity recognition using mobile phones, we used the following criteria for the selection of the studies reviewed in this chapter: •. •. •. They implement the activity recognition system fully on mobile phones, such that sensing, preprocessing and classification are all done locally on these devices. They use only mobile phone sensors, where motion sensors are used as the main sensors in the recognition process. For example, we did not include [40] in our review, because it uses an external motion sensor for physical activity recognition in combination with a mobile phone accelerometer. However, we consider studies that use other onboard sensors as additional sensors, such as a microphone, gyroscope, GPS, and pressure sensor. They are able to recognize different physical activities. We do not include studies on fall detection and posture detection in this work.. There are also some studies [41, 42, 43, 44, 45, 46, 40] that have reported that they have implemented online activity recognition, but we could not find evaluation proof or details for such a claim in the respective papers. Therefore, we did not include these studies in our review. The rest of the chapter is organized as follows. In Section 2.2, we briefly describe the related work. A comparison of all reported studies on online activity recognition is described in Section 2.3. In Section 2.4, we discuss possible improvements to the current research and future directions. Finally, we conclude this chapter in Section 2.5..

(30) 16. 2.2. 2 Introduction to Human Activity Recognition. Related Work. Human activity recognition using wearable sensors is a very broad research subject. Earlier work by Lara et al. [32], Akker et al. [37] and Preece et al. [47] provides an outline of relevant research and applicable techniques. These surveys include all wearable solutions. In contrast, our survey focuses on human activity recognition solutions using a specific wearable sensor platform: the smartphone. The smartphone is quickly gaining popularity as a wearable sensor platform. It is being applied for many applications, including health monitoring, monitoring road and traffic conditions, commerce, environmental monitoring and recognizing human behavior [48]. Earlier work by Incel et al. [9] surveys activity recognition research using smartphones. However, the most research described therein still involves offline processing of the data collected on the smartphone. In contrast, our survey focuses entirely on research that resulted in a practical, online and self-contained implementation on a smartphone.. 2.3. Online Activity Recognition. Activity recognition systems consist of mainly four steps: sensing, preprocessing, feature extraction and either training or classification [25]. These steps are shown in Figure 2.1. We briefly describe them as follows (we discuss these steps in detail in the following sections): (1). (2). (3). Sensing: In this step, sensor data are collected at a specific sampling rate. This sampling rate may vary for different sensors such as accelerometer and GPS. The choice of this value depends on the use case, application requirements, and sensor type. For the accelerometer, the sampling rate of 100 to 5 samples per second has been used in various studies. Preprocessing: Subsequently, the collected data are processed in various ways. For example, noise is removed. Then, a windowing or segmentation scheme is applied to it. Feature extraction: Various data features are extracted from the segmented raw data to differentiate between activities. These features can belong to different categories such as time-domain and frequency-domain. Some of the most common time-domain features are mean, standard deviation, minimum, and maximum. The choice of a specific data feature is also design decision and may depend on many factors such as its computational complexity, and activity types..

(31) 2.3 Online Activity Recognition. (4b). Training: We use supervised classification in this thesis. In such classification approach, a classification algorithm needs to be trained first from already labeled data or examples. As shown in Figure 2.1, training is a preparation step to obtain the model parameters for later use in classification. As shown in Figure 2.2, training can either be offline on a desktop machine or online on the phone itself. If training is performed offline, raw data from example activities is first collected and stored. At a later time, these data are used for obtaining the model parameters, as shown in Figure 2.1. If training is performed online, the raw data are not stored for later use, but instead directly processed for training. The training step is performed infrequently compared to classification or prediction, and the resulting model parameters are stored for future use in the actual online activity recognition. Classification: In this final stage, the trained classifiers are used to classify new data instances into different activities. This step can be done either offline in a machine learning tool, such as WEKA, or online on the mobile phone itself. In this chapter, we are mainly focusing on online classification.. sensors. preprocessing. samples. raw data. sensing 1. feature extraction features. sensors 2. raw data 3. 2. sensing 1 ata. ra. w. d. feature extraction. storage. for offline training. training: calculation of model parameters. preprocessing. samples. trained classifier. features. (4a). 17. 3. classification. 4a. activity classes 4b. Figure 2.1: Activity recognition steps.. Mobile phones are being used in two ways for online activity recognition. These two approaches are client-server and local approach. In client-server.

(32) 18. 2 Introduction to Human Activity Recognition. Figure 2.2: Local approach for activity recognition on mobile phones.. approach, the sensing part is done on the mobile phone, which acts as a client. Then, the collected data are sent to a server or a cloud for further real-time processing, such as preprocessing and classification. The preprocessing step can partially reside on the mobile phone, too. However, the main classification step is performed on a server. This approach is adapted in order to run the computationally-expensive steps on a server, because of the limited resources in a mobile device. For example, in [27], raw data are sent to a server for classification. There are other studies that have done the same [49]. This approach requires an Internet connection at all times for sending sensor data for further processing to a server or a cloud. In local approach, activity recognition steps are done locally on the mobile phone in real time. These steps include data collection, preprocessing and classification. In this approach, information about classified activities and raw data can also be sent to a server for further analysis. However, the main three steps are performed locally. The training can still be done on a desktop machine beforehand or on the phone locally. This approach is shown in Figure 2.2. We only consider studies that have followed the local approach, as most.

(33) 2.3 Online Activity Recognition. 19. of the studies follow this approach. Our goal is to see the potential of mobile phones in running activity recognition systems locally. It is important to note that there are studies that do more than simple activity recognition, such as in [11]. However, we only mention those parts of these studies that fit the scope of this chapter. We discuss these studies in the following aspects: • • • • • • • • • • • • •. Implemented Classifiers on the Mobile Phones Online vs. Offline Training for Classification Methods Platforms, Phones, and Sensors used in Online Activity Recognition Resource Consumption Analysis Real-time Assistive Feedback Validation of Online Activity Recognition Orientation-Independent Activity Recognition Position-Independent Activity Recognition Fixed and Adaptive Sampling Dynamic and Adaptive Sensor Selection Performance Evaluation Recognized Activities Data Features used for Classification. 2.3.1. Implemented Classifiers on Mobile Phones. Classification is an important step in the activity recognition process. There are various classifiers that have been implemented on mobile phones in the last few years. The most commonly used classifiers are decision tree, support vector machine (SVM), K-nearest neighbor (KNN) and naive Bayes. Some of the other implemented classifiers are decision table, rule-based classifier, fuzzy classification, quadratic discriminant analysis (QDA) and neural networks. In some studies, two classifiers are combined in different ways, thereby creating multi-layer or hierarchical classification. For example, decision tree and dynamic hidden Markov model (DHMM) are used in combination in [50]. Studies that show that these classifiers can run on mobile phones are shown in Table 2.1. A mapping of various classification methods with respect to the studies where they were implemented is given in Table 2.1. For specific implementation details about these classifiers, readers are referred to the relevant studies, as shown in Table 2.1..

(34) 20. 2 Introduction to Human Activity Recognition. Table 2.1: Implemented classifiers on mobile phones for online activity recognition. Implemented Classifiers. Relevant Studies. Decision Tree SVM KNN Naive Bayes. [51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] [62, 63, 64, 65, 66, 67] (Clustered KNN [68]) [69, 70, 71] [72, 73, 68, 11, 55] (Decision tree, dynamic hidden Markov model (DHMM) [50]) (SVM, K-medoids clustering [74]) (Decision tree, probabilistic neutral network (PNN) [75]) [76] [77] [70] [55] [78]. Multi-layer Classifiers Probabilistic Neural Networks Rule-based Classifier Quadratic Discriminant Analysis Decision Table Fuzzy Classification. 2.3.2. Total Relevant Studies 11 6 5 4 3 1 1 1 1 1. Online vs. Offline Training for Classification Methods. For classifying test data into various pre-defined classes in supervised classification, the classifiers need to be trained first using training data [25]. In the context of mobile phones, this training can be done in two ways: online and offline. • •. Online: the classifiers are trained on the mobile phones in real time. Offline: the classifiers are trained beforehand, usually on a desktop machine.. We found that most studies have used the offline method. One of the reasons for doing so was because the training process is computationally expensive. Unlike training, it is easy to implement only the classification part on the mobile phone and it is computationally cheaper. Only six out of all 30 studies were using online training where classifiers can be trained on mobile phones in real time. We outline all of the studies where an offline and online method has been used in Table 2.2. It is important to note that all these studies implement real-time classification. In addition to these studies, Google also now provides a real-time activity recognition API [79]..

(35) 2.3 Online Activity Recognition. 21. Table 2.2: Training process on mobile phones (online vs. offline). Training Process (Online vs. Offline) Offline Online. Relevant Studies [51, 78, 72, 69, 77, 76, 64, 65, 11, 52, 53, 54, 55, 56, 50, 59, 58, 57, 70, 71, 74, 60, 61, 75] [62, 63, 73, 68, 66, 67]. Total Relevant Studies 24 6. Table 2.3: Mobile phone platforms used for online activity recognition. Mobile Phone Platforms Android Symbian iOS Debian Linux. 2.3.3. Relevant Studies [62, 51, 72, 69, 63, 73, 77, 64, 76, 65, 68, 11, 52, 53, 55, 66, 59, 58, 57, 70, 74, 61, 75] [54, 56, 50, 59, 57, 70, 60, 61] [54, 67, 71] [78]. Total Relevant Studies 23 8 3 1. Platforms, Phones and Sensors Used in Online Activity Recognition. Activity recognition systems are implemented on different mobile phone platforms. The most commonly-used among these platforms is Android. However, some studies were conducted using iOS and Symbian. We found only one study with an implementation on the Debian Linux platform using the OpenMoko Neo Freerunner mobile device. All of these studies range from 2008 to 2014. Initially, activity recognition research mainly used the Symbian platform. However, in recent years, Android took over, and most of the studies are now using Android. These studies use different types of mobile phones for their implementations. For example, Nokia N95 is mainly used with the Symbian operating system, various android phones with Android and iPhones with iOS. A detailed description of these phones is given in Table 2.6. The relationship between different studies and their used platforms is given in Table 2.3. These studies use different types of motion sensors in the leading role in the activity recognition process. The accelerometer was the dominant sensor in all of these studies. It was used in all studies, in most cases individually, and in a few cases, in combination with other sensors, such as microphone, gyroscope, magnetometer, GPS and pressure sensor. In some cases, these sensors are fused at a raw level, whereas in other cases, at a higher level, depending on the application objective. There are 23 out of 30 studies that used the accelerometer alone. The details about the relationship between the sensors used and their relevant studies are given in Table 2.4. In this table, A stands.

(36) 22. 2 Introduction to Human Activity Recognition. for an accelerometer, G for a gyroscope, LA for a linear acceleration, Mic for a microphone, PS for a pressure sensor, and M for a magnetometer. Table 2.4: Sensors used for online activity recognition. A, accelerometer; G, gyroscope; M, magnetometer. Mobile Phone Sensors. Relevant Studies. A A, M A, Mic A, GPS A, G, M A, PS, Mic A, G, M, Gravity Sensor, LA, Orientation Sensor. 2.3.4. [62, 51, 78, 69, 72, 63, 73, 77, 68, 76, 11, 52, 53, 54, 56, 58, 70, 57, 67, 71, 74, 60, 61] [75] [66] [50, 59] [65] [64] [55]. Resource Consumption Analysis. Resource consumption analysis, such as the analysis of battery, CPU, and memory usages, is an important aspect of online activity recognition. This is one of the factors in shifting from an offline to an online approach. Such analysis is performed to validate if online recognition systems can be run in real-world settings. However, we found that most of the reported studies are missing this analysis, except a few studies so far, as shown in Table 2.5. Table 2.5: Studies with resource consumption analysis. Resources (Performance Metric). Relevant Studies. CPU (percentage) Memory (MBs). [78, 68, 11, 54, 56, 50, 57, 70] [68, 11, 55, 56, 50, 74] [62, 77, 11, 52, 53, 54, 56, 50, 70, 74, 60, 61]. Battery (hours or watt-hours per hour). Total Relevant Studies 8 6 11. As shown in Table 2.5, CPU, memory and battery usage are reported for resource consumption analysis. For the battery consumption, two types of measurements are made. In one case, the amount of time a battery lasted was reported while running online activity recognition systems, as shown in Table 2.7; while in other cases, the power usage was reported in watt-hours per hour for these systems. Though many studies simply report the number.

(37) A1, A5, A8, A9, A11. A1, A5, A8, A9, A11 A1, A4, A5, A12 A1, A5, A8, A10 A1, A2, A3, A5, A6 Different physical activities A1, A2, A3, A4, A5, A9 A1, A2, A3, A5. A1, A3, A5. [54]. [50] [63] [11] [72] [73] [77] [68]. [52]. 1. A1, A5, A6, A7, A12, A17, Idle. [69]. A1, A1 (power), A4, A8 A1 (slow), A2, A3 (relax, normal), A7, A13, A14 A1, A2, A3, A6, A7, A16 A1, A5, A6, A7, A8, A9, A10 A1, A2, A5, A6, A7, A12. A1, A5, A6, A7, A8. A1(slow, normal, rush), A2, A3, A5. A1, A2, A3, A5, A9, A10, A17. A1, A2, A3, A5, A12, A16 A1, A4, A6, A7, A9. A1, A4, A6, A7, A8, A9, A13, A14. A1, A5, A1/A5 on treadmill, A6, A7, A9, A10, A11, A12, A13, A14, A15, idle (A2/A3), watching TV. [67] [61] [62] [51] [76]. [65]. [55]. [57]. [71] [74]. [75]. [64]. Fundamental frequency, average acceleration, max and min amplitude (based on accelerometer magnitude) mean, VAR, mean crossing rate, spectrum peak, sub-band energy, sub-band energy ratio, spectral entropy A’s VAR, DFFTcomponents and GPS speed similarity score using geometric template matching algorithm mean, VAR mean, root mean square, difference between max and min values maximum and minimum euclidean norm signal magnitude, coefficient of variance, counts per minute mean, min, max, SD mean, VAR, SD, correlation between axes, inter-quartile range, mean absolute deviation, root mean square and energy mean (axis, magnitude), SD (axis, magnitude), tilt, linear regression coefficients, wavelet coefficients A’s VAR, MFCC (Mel-frequency cepstral coefficient), RMS (root mean square), ZCR (zero-crossing rate) as acoustic features. peak, SD/mean, FFT energy 21 features, including mean, SD, min, max, 5 different percentiles and observations below/above these percentiles For details, refer to [67] Mean, VAR, magnitude, covariance, FFT energy and entropy For details, refer to [62] 9 features based on the auto-correlation function of accelerometer signals Auto-regressive coefficients SD and auto-regressive fitting of y-axis, correlation of x, y, z, signal magnitude area, mean, SD and skewness of the pitch Mean, VAR, zero crossing rate, 75th percentile, correlation, inter-quartile, signal energy, power spectrum centroid, FFT energy, frequency-domain entropy SD, min, max, the remainder between percentiles (10, 25, 75, 90), median, the sum, square sum and number of crossings of values above or below the percentile (10, 25, 75 and 90) mean, SD time gap peaks, mean, SD, A’s energy, Hjorth mobility and complexity Average period, VAR, average energy, binned distribution for each axis and correlation between y and z mean, SD, correlation, signal magnitude area, auto-regressive and moving average coefficients for A; altitude difference for pressure sensor; mean, VAR, min and max for audio sensor. mean, VAR. A’s mean, VAR, FFT coefficients and GPS speed SD (based on accelerometer magnitude). Data Features mean, SD, number of peaks. LG NEXUS 4. Samsung Nexus S. iPhone 4S HTC Nexus. Nokia N8, Samsung Galaxy Mini. Google Nexus S. Android smartphone. iPhone Nokia N95, Samsung Galaxy S2 Samsung Galaxy S2 Samsung Galaxy Y LG Nexus 4. Samsung Galaxy Mini, Nokia N8. HTC Hero. Android phone. HTC G11, Samsung i909. HTC Evo 4G. Nokia N95 Android phones Android Nexus One Android phones ZTE Blade Samsung Galaxy S Samsung Galaxy Gio. Nokia N95, iPhone. Motorola Droid. OpenMoko Neo Freerunner. Android Phone, Nokia N95 Nokia N95. Phone Nokia N95. Activities: walking, A1; standing, A2; sitting, A3; jogging, A4; running, A5; walking upstairs, A6; walking downstairs, A7; still, A8; biking, A9; driving a car, A10; in vehicle, A11; jumping, A12; using elevator up, A13; using elevator down, A14; vacuuming, A15; laying, A16; phone on table/detached, A17; washing dishes, WD; ironing, IR; brushing teeth, BT; hair drying, HD; flushing the toilet, FTT; boarding, BD; unknown. A1, A2, A3, A5, A9, A10. [70]. [58]. [66]. [53]. A1, A2, A3, A5, A6, A7, A9, A10, A12, A16 (prone, supine) A1, A5, A15, A16, WD, IR, BT, HD, FTT, BRD, unknown A1, A2, A4, A6, A7. A1, A5, A8, A9, A10 A1, A5, A8, A11 A1, A2, A3, A6, A9, A16, A17, phone in hand, typing text messages; talking on the phone. [59] [60]. [78]. Activities A1, A2, A3, A5. Study [56]. Table 2.6: Phones, activities and data features used in online activity recognition..

(38) A. Decision tree. Decision tree. Lara et al. [52]. Liang et al. [53] 7, 8, 10. 12.5. 6 11.3 8.3 16 15 6–8 24. Battery Lifetime (h). Decision tree Decision tree + DHMM Decision tree Fuzzy classification Naive Bayes QDA Naive Bayes, KNN clustered Decision tree. Symbian Symbian Symbian, iOS Debian Linux Android Symbian, Android Android Symbian. Platform. Phone Nokia N95 Nokia N95 Nokia N95, iPhone OpenMoko Neo Freerunner Android Nexus One Samsung Galaxy Mini, Nokia N8 Samsung Galaxy Gio Nokia N8. Implemented Classifiers. Decision tree Decision tree + DHMM Naive Bayes Naive Bayes, KNN clustered Decision table, naive Bayes, decision tree. Study. Miluzzo et al. [56] Reddy et al. [50] Lane et al. [11]. Google Nexus S. Samsung Galaxy Gio. Android. Android. Phone Nokia N95 Nokia N95 Android Nexus One. Symbian Symbian Android. Platform. A, M, G, linear acceleration, gravity. A. A A, GPS A. Sensors. Table 2.9: Studies with memory usage analysis.. Implemented Classifiers. Study. Kose et al. [68]. HTC G11, Galaxy S2. HTC Evo 4G. Nokia N95 Nokia N95 Nokia N95 Nokia N95 Android Nexus One Samsung Galaxy S Galaxy Mini, Nokia N8. Phone. Table 2.8: Studies with CPU usage analysis.. Miluzzo et al. [56] Reddy et al. [50] Lu et al.[54] Berchtold et al. [78] Lane et al. [11] Siirtola [70] Kose et al. [68] Siirtola and Roning. [57]. Martin et al. [55]. Symbian Symbian Symbian Symbian Android Android Android Symbian, Android Android. Platform. CPU Usage %. Memory Usage (MB) 34 29.64 14.74 12.6 (naive Bayes), 21.9 (KNN clustered) 16.5 (decision table), 0.00146 (naive Bayes), 0.8376 (decision tree). A A, GPS A A A A A A. Sensors. 20, 10, 2 Hz. 50 Hz. Various rates NA 32 Hz 32 Hz NA 90 Hz 40 Hz. Sampling Rates. 31 4.72 iPhone (0.9–3.7), N95 (1–3) 3.3 11 5 42 (Naive), 29 (KNN) 15. 1450, 1650. 1500. 950 950 950 950 1400 1500 1200, 1200. Battery Capacity (mAh). Table 2.7: Details of battery usage analysis. (QDA:quadratic discriminant analysis). A. A A A, GPS A A A A. Decision tree Decision tree Decision tree + DHMM Decision tree Naive Bayes Rule-based classifier KNN, QDA. Miluzzo et al. [56] Wang et al. [60] Reddy et al. [50] Lu et al. [54] Lane et al. [11] Guiry et al. [77] Siirtola, [70]. Sensors. Implemented Classifiers. Study.

(39) 2.3 Online Activity Recognition. 25. of hours a battery lasts as a resource metric [77, 52, 53, 11, 54, 70, 60], it has a drawback. Many of these studies use different mobile phones with different battery capacities, so this metric can be misleading. This can be seen in Table 2.7, where we added the battery capacities to see how different these batteries were. Therefore, watt-hour per hour is a better choice to use for battery usage, as it is independent of the battery capacity. Some of these studies [56, 50, 61] use both of these metrics. However, this is also not fully platform independent, as these metrics might be affected by other factors in different platforms, for example, the CPU speed. The CPU usage was reported in terms of percentages for which the CPU was occupied by the recognition process and memory used was reported in MBs (megabytes). It is difficult to compare these reported values due to their different experimental setups and evaluation methods. For example, it can be seen in Table 2.7 for battery usage how different various parameters are in these studies. These values are presented and discussed in different ways in various studies. The details about CPU and memory measurements are given in Tables 2.8 and 2.9, respectively. These values are hard to compare because they are reported in different experimental and evaluation setups. To show this, we present some additional information in these tables, such as classifier, platform, sensor and phone. Apart from this information, different data features are used, as shown in Table 2.6. In some cases, these values only represent specific parts in a complete application, which consists of other parts, too. For example, in [56], these values are only relevant when an accelerometer is being used by an activity classifier, because the activity classifier is a part of an application, which consists of other parts, too, such as a classifier for audio classification. For details on memory and CPU usage in this work, readers are referred to Table 4 in [56].. 2.3.5. Real-Time Assistive Feedback. Real-time assistive feedback is an important aspect of the healthcare and other context-aware applications built on top of activity recognition systems to improve people’s well-being. However, most of the studies on online activity recognition are missing this feature. There are only two [72, 11] out of 30 reviewed studies that provided the capability of real-time feedback for assisting people. In [72], basic activities are recognized on the mobile device using its accelerometer. These recognition results are sent to a server that used richer context information to give real-time audio feedback to help people with cog-.

No results found