Evaluation of a mHealth application for pregnancy-related work advice and information: The impact of user characteristics on Think Aloud method results

(1)

Evaluation of a mHealth application for

pregnancy-related work advice and information

The impact of user characteristics on Think Aloud method results

Liesbeth A. van den Berg 10763066

Thesis Master Medical Informatics University of Amsterdam

(2)

Student:

MSc. Liesbeth A. van den Berg Student number: 10763066

Email: liesbeth.vandenberg@amc.uva.nl

Place of the Scientific Research Project:

Academic Medical Center – University of Amsterdam (AMC – UvA) Department of Obstetrics and Gynaecology

Period: November 2016 – September 2017

Mentors:

MD M.D.M. van Beukering

Academic Medical Center – University of Amsterdam Department of Obstetrics and Gynaecology

Email: m.d.vanbeukering@amc.uva.nl

MD Dr. M. Kok

Email: m.kok@amc.uva.nl

MD A.V. Velu

Email: a.v.velu@amc.uva.nl

Tutor:

Dr. L.W.P. Dusseljee-Peute

Academic Medical Center – University of Amsterdam Department of Medical Informatics

(3)

Preface and acknowledgments

This thesis is the end product of my Scientific Research Project (SRP) and final product of the master Medical Informatics at the University of Amsterdam. The obstetrics department of the Academic Medical Center (AMC) commissioned this SRP, which started November 2016 and was finalised in September 2017. What attracted me to this project was that it allowed me to perform scientific research, but also work with potential end-users. This research project has challenged me to dive further into the field of usability evaluation methods. It allowed me to partake in a redesign process and find, sometimes creative, solutions to solve real-life issues. Upon reflection, this SRP has gained me insight into the processes of redesigning and the role of usability evaluation methods, scientific research about these processes, and how to manage a multidisciplinary project. I have learned a lot during these months and I hope that this thesis will provide you an adequate

representation of the work that I have conducted within this period.

There are number of people that deserve gratitude for their valuable contribution to this thesis. First, I would like to thank Linda Dusseljee-Peute. She encouraged me to do this master and her continued (mental) support during my (pre-) master and this SRP is beyond anything I could have asked for. I’m very grateful that you have guided me along this journey, your expertise and enthusiasm is very contagious and cannot be praised enough. Furthermore, I would like to thank my mentors from the ‘Zwangerschap en Werk’ project. Thank you for offering me this opportunity and entrusting me with this research. I would especially like to thank Monique van Beukering. Your enthusiasm about this SRP and your perseverance was very welcoming in sometimes stressful situations. I am going to miss our weekly skype calls where we would discuss SRP-related topics but also everyday life. I would also like to thank my friends and family for helping me during this SRP period. You were always there to help, listen or tell me to relax. My parents, however, deserve my utmost gratitude. Despite all trials and tribulations, the three of us will persevere and both of you help me strive to become the best person I can be. Finally, I would like to thank everyone who participated or contributed in anyway in this SRP. Without you this thesis would not have been possible.

Dear reader, I hope you will enjoy reading this thesis.

Liesbeth van den Berg

(6)

Summary

The obstetrics department of the Academic Medical Center has built a mHealth solution, called the ‘Pregnancy and Work’ app, which aims to provide evidence-based information and advice about work-related pregnancy risks. In recent years there has been a rise in mHealth applications, but the impact of these mHealth applications, especially the influence of human factors on these applications, is a subject that has not received a lot of attention. In this thesis multiple studies were conducted to assess the usability of the ‘Pregnancy and Work’ app and the influence of previous domain experience on Think Aloud outcomes.

The main study of this thesis is the performance of a Think Aloud study conducted with twelve potential end-users to assess the usability and participants’ evaluation of the app. Participants were tasked to talk aloud while performing nine tasks in the app during its first time use. Task completion rates and times were tracked through video recording. Participants were required to provide demographic information and evaluate the app through means of questionnaire questions, which included: Intrinsic Motivation Inventory value subscale, System Usability Scale, and grading the app. Usability problems that resulted from the Think Aloud study were rated via Nielsen’s severity rating scale. Participants identified 11 bugs and 101 usability problems. Out of these 101 usability problems, 40 were rated with a severity of 3 or 4 and require attention for future iterations of the app. The core purpose of the Pregnancy and Work app is to provide work advice to its end-users. However, due to multiple factors an accumulation of 5 of 12 (42%) participants will not act upon the work advice given to them. Overall the participants’ evaluation of the Z&W app was relatively positive.

In addition a second study was performed to re-analyse the results on a deeper level. The results of the aforementioned Think Aloud study were utilised to assess the potential influence of previous domain experience on these outcomes. The participants were divided into two groups according to their previous domain experience: novices (first time pregnancy) and experts (participants whom have experienced a pregnancy). The usability problems were categorised in accordance to the Usability Problems Taxonomy and rated for their severity with Nielsen’s severity scale. Lastly, the thoroughness, validity, and effectivity of both groups were measured. We found a strong indication that previous domain experience could influence Think Aloud outcomes. There was a significant difference between the expert and novice group in finding usability problems and task mapping usability problems. The experts and novices found 18% overlapping usability problems, 82% were identified by either one of the two groups. The performance measurements indicated that the expert group, overall, performed better than the novice group in identifying usability problems. However, based on this research, it is still recommended to utilise both experts and novice in usability evaluation methods. There were no significant differences between the two groups with regard to the participants’ evaluation.

Lastly, a case study was performed on the entire development process of the app to showcase a realistic utilisation of pragmatic usability evaluation methods in the redesign process of the ‘Pregnancy and Work’ app. Before the primary study took place, in line with a human factor development process, a usability study was performed by Medical Informatics students to inform a first redesign iteration. They applied three different usability evaluation methods. The primary Think Aloud study was conducted on the redesigned version. Even though all of the methods had their limitations, the results indicate that the redesign resulted in an improved app. Usability problems during the Think Aloud study focused more on content instead of visual aspects. The mean severity of usability problems was lower in the post-redesign phase and most (92%) of the identified usability problems were new.

The conducted research on the ‘Pregnancy and Work’ app indicate that participants have a positive view on the app, but experienced many (usability) problems. Especially participants not acting upon received work advice, is a subject that needs to be addressed in future iterations. The potential influence of previous domain experience opens doors for a future in-depth look on how this and other user

characteristics influence usability evaluation methods outcomes. Finally, this study showcases the value of assessing human factors through usability evaluations and that such evaluations should be implemented as soon as possible within a multidisciplinary design process.

(7)

Samenvatting

De afdeling verloskunde in het Academisch Medisch Centrum heeft een mHealth applicatie ontwikkeld, de ‘Zwangerschap en Werk’ app. Werkende vrouwen krijgen individueel advies over werkgerelateerde

zwangerschapsrisico’s, gebaseerd op een evidence-based richtlijn. De laatste jaren is er een stijging in aantal mHealth applicaties. De impact van dergelijke applicaties, in het bijzonder de invloed van menselijke factoren, is nog niet onderzocht. In deze thesis zijn er meerdere studies gedaan om de gebruiksvriendelijkheid van de ‘Zwangerschap en Werk’ app te evalueren, waarbij specifiek gekeken is naar de mogelijke invloed van voorafgaande domeinervaring op ‘Think Aloud’ (hardop denkmethode) resultaten.

Een ‘Think Aloud’ studie is uitgevoerd met twaalf potentiële eindgebruikers (participanten) om de gebruiksvriendelijkheid en bruikbaarheid van de ‘Zwangerschapen en Werk’ app te onderzoeken. Participanten moesten in lijn met de ‘Think Aloud’ methode hardop praten en negen taken uitvoeren terwijl zij de app voor de eerste keer gebruikten. Het voltooiingspercentage van de taken en de voltooiingstijd werden bijgehouden via video-opnames. Demografische informatie van de participanten en hoe zij de app evalueerden werd achterhaald via vragenlijsten. Deze vragenlijsten omvatten de: ‘System Usability Scale’, waarde subschaal van de ‘Intrinsic Motivation Inventory’ en een cijferbeoordeling. De ‘Think Aloud’ studie resulteerde in een lijst van gebruiks-vriendelijkheidproblemen die werden geprioriteerd op basis van ‘Nielsen’sseverity scale’. Participanten

ontdekten 11 bugs en 101 gebruiksvriendelijkheidsproblemen, 40 van deze problemen hadden een prioriteit van 3 of 4. Deze problemen vereisen aandacht voor toekomstige versies van app. Het hoofddoel van de app is om eindgebruikers van werkadvies te voorzien, maar vanwege meerdere factoren werd duidelijk dat 5 van de 12 (42%) participanten niet van plan waren iets met het advies te doen, of zich niet bewust was dat zij een werkadvies hadden gekregen. De evaluatie van de participanten over de app was over het algeheel redelijk positief.

De resultaten van de ‘Think Aloud’ studie werden gebruikt om de potentiële invloed van voorafgaande domeinervaring op deze resultaten te analyseren. De participanten werden onderverdeeld in twee groepen: novices (eerste keer zwanger) en experts (minimaal één eerdere zwangerschap doorgemaakt). De gebruiksvriend-elijkheidsproblemen werden gecategoriseerd via de ‘Usability Problems Taxonomy’ en beoordeeld op ernst via de ‘Nielsen’s severity scale’. Ten slotte werden bij beide groepen prestatiemetingen verricht aan de hand van: grondigheid, validiteit en effectiviteit. Wij vonden een sterke indicatie dat voorafgaande domeinervaring invloed kan hebben op ‘Think Aloud’ resultaten. Er was een significant verschil tussen de novices en experts in het vinden van gebruiksvriendelijkheidsproblemen en ‘task mapping’ gebruiksvriendelijkheidsproblemen. De experts en novices vonden 18% overlappende problemen, de overige 82% werden apart geïdentificeerd door één van de twee groepen. De prestatiemetingen gaven aan dat de experts globaal beter presenteerden dan de novices in het vinden van gebruiksvriendelijkheidsproblemen. Op basis van deze onderzoeksresultaten volgt de aanbeveling om zowel novices als experts in te zetten in een gebruiksvriendelijkheidsonderzoek. Wat betreft de participant-evaluaties is er geen significant verschil gevonden tussen de twee groepen.

Als laatste wordt een kleine case studie beschreven die is uitgevoerd om een realistische weergave te omschrijven over het gebruik van pragmatische gebruiksvriendelijkheidsonderzoeken binnen het

herontwerpproces van de ‘Zwangerschap en Werk’ app. Medical Informatics studenten hebben drie verschillende gebruiksvriendelijkheidsonderzoeken uitgevoerd wat leidde tot een herontworpen app. De voornoemde ‘Think Aloud’ studie maakte gebruik van deze vernieuwde app. Hoewel alle uitgevoerde onderzoeken limitaties hadden, gaven de resultaten weer dat het herontwerpproces leidde tot een verbeterde app. Gebruiksvriendelijkheids-problemen in de ‘Think Aloud’ studie waren meer gefocust op content in plaats van visuele aspecten. De gemiddelde ernst van de gebruiksvriendelijkheidsproblemen waren lager in de post-herontwerp fase en het merendeel (92%) van deze problemen waren nieuw.

Het onderzoek verricht in deze thesis op de ‘Zwangerschap en Werk’ app laat zien dat participanten een positieve houding hebben tegenover de app, maar dat zij wel tegen veel (gebruiksvriendelijkheids-) problemen aanliepen. Het feit dat een aanzienlijk deel van de participanten geen gebruik gaat maken van het werkadvies is een onderwerp dat veel aandacht verdient in volgende iteraties. Het potentiële effect van voorafgaande domeinervaring opent deuren voor toekomstig onderzoek naar hoe dit en andere gebruikerseigenschappen de resultaten van gebruiksvriendelijkheidsonderzoeken mogelijk beïnvloeden. Tot slot, deze studie toont de waarde van het evalueren van menselijke factoren, door middel van gebruiksvriendelijkheidsonderzoeken en dat zulke evaluaties zo vroeg mogelijk binnen een multidisciplinair designproces moeten worden geïmplementeerd.

(8)

1. General introduction

In recent years the accessibility of internet and mobile phones has resulted in a rise of mobile health applications (mHealth) (1). mHealth applications provide (public) health services and information via mobile technologies. mHealth has many benefits, one of which is the high accessibility of mHealth applications (2-5). However, mHealth also has its caveats. One caveat is that mHealth applications are not always of the best quality or devoid of evidence-based research (5, 6). Additionally, the impact of mHealth applications is only a subject studied for a couple of years (4, 7, 8).It is recommended that future research on the impact of mHealth applications (e.g., patient care) should incorporate analyses of the human factors influencing these effects (7). Such research will provide knowledge on human factors facilitating or hindering successful implementations and result in more effective mHealth applications.

The obstetrics department of the Academic Medical Center (AMC) has built a mHealth solution that aims to provide information and advice about work-related pregnancy risks, the ‘Pregnancy and Work’ app (Zwangerschap en Werk app) (9). The ‘Zwangerschap en Werk’ (Z&W) app’s content is derived from the evidence-based practice guideline: Pregnancy, Postpartum Period and Work (10). The creators of this

application expressed their desire for it to be tested via a usability evaluation method (UEM). This application, its development, and evaluations are at the centre of the research conducted within this thesis.

UEMs are a set of methods that can be utilised to assess the human interaction with a system for the purpose of identifying facets of this interaction that can be improved (11). The concept of UEMs arose around the 1980s, in the 1990s it further developed into how we know and view UEMs today (12). Usability is defined by the International Organisation Standardization (ISO) as: “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” (13). UEMs play a large role in the development and testing of all sorts of products in the health care domain. Such products include: electronic health records, medical devices, and mHealth solutions. Typically a design process of any health-related product is conducted in an iterative fashion. This entails the utilisation of mock-ups, multiple evaluations of the product and redesigned products until the final product is released. Utilising UEMs in such an iterative design process in the health care domain is especially important as poor design and usability of medical products can lead to harmful consequences (14-17). Therefore, the utilisation of UEMs during the development and testing process is widely recommended throughout research (14, 15, 18, 19). UEMs can be divided into two categories: usability testing and usability inspection (19, 20). Usability testing refers to measuring performance (by users) when testing a product, whereas in a usability inspection experts analyse products for potential usability problems. When evaluating user requirements on a health-related product there are three widely adopted methods (19, 21). These methods are: Cognitive Walkthrough (CW) and Heuristic Evaluation (HE), both usability inspection methods and the Think Aloud (TA) method, a usability testing method.

In this thesis the usability of the Z&W app was thoroughly tested via the TA method. The TA method requires participants to talk aloud (i.e., verbalise their thoughts) while performing or solving a task (22). By analysing the verbal data the cognitive processes that occur when performing or solving tasks can be

characterised. The analysis of the verbal reports results in insights into usability problems and its causes (19). Through analyses of the TA sessions and additional questionnaires we aimed to procure data on how first-time users experience, evaluate, and accept the Z&W app. Furthermore, an in-depth analysis was performed on the potential influence of certain user characteristics on these TA outcomes. The Z&W app is aimed towards all pregnant women that are working in the Netherlands and will therefore be used by pregnant women with different backgrounds. The impact of such different user characteristics on UEM outcomes, specifically TA outcomes, is a topic that has not attracted much research. In this thesis we analysed whether first-time pregnant (primigravida) women and women that have experienced a previous pregnancy (multipara) have different influences on the TA outcomes. For this particular user characteristic we coined the term ‘previous domain experience’.

There are advantages and disadvantages to each UEM, which UEM is used and when to use it is dependent on a number of factors. Most of these factors revolve around: time, costs, expertise issues, and the goal of the UEM. Whilst the need of UEMs in the health care domain is founded, the implementation is still somewhat lacking. Also during the design process of the Z&W app pragmatic methods were required and

(9)

implemented. These pragmatic methods resulted in students performing small usability evaluation studies. These studies included all three aforementioned UEMs. The utilisation of these studies resulted in an

additional small scale study which takes a look at a realistic approach of utilising pragmatic UEM studies within the redesign process of the app before launch.

Outline and research questions

This thesis is comprised of three studies. The outline of these studies and their position on the SRP’s timeline are depicted in figure 1.1.

Figure 1.1. Overview and timeline of the conducted studies within this SRP (November 2016 – September 2017). The main research question that will be answered in this thesis is the following:

Main research question

How do novice potential end-users evaluate the Z&W app during first-time use in a Think Aloud usability study and how does their previous domain experience influence these outcomes?

The answers to this main research question will be derived from study I and II, in chapter 2 and 3 respectively. Study I and II are the main focus of this thesis. Chapter 2 will elucidate how potential end-users evaluate the Z&W app in terms of usability (i.e., effectiveness, efficiency, and satisfaction). Light will be shed on the identified usability problems, their severity, completion rates and times, and the participants’ evaluations. This subject will mainly focus on a Think Aloud study design with end-users and will answer the following research questions:

Study I

 What are the usability problems in the Z&W app found through the Think Aloud method during first-time use?

- What are the severity ratings of these usability problems?

 What are the end-users’ task completion rates (effectiveness) and completion times (efficiency)?  How do users evaluate and accept the Z&W app (satisfaction)?

- What grade do the participants give the Z&W app?

- How do participants rate the Z&W app on the System Usability Scale?

- What is the perceived value of the Z&W app?

- Do the participants indicate an intention to use the work advice provided by the Z&W app?

Chapter 3 will entail the process of study II, in which we will delve into the potential influence of previous domain experience (primigravida versus multipara) on TA and user evaluation outcomes. We will look at the number, severity, and categorisations of the usability problems between the two groups. Additionally, a

Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Pilot Z&W app

Feb 8

Redesign Z&W app

Apr 2 End SRP Sep 8 UEMs performed pre-release by MI students Feb 13 - 27 Start SRP Nov 10 Usability report Mar 8 Development Z&W app

Focus group meetings & Guidelines Conductedby Z&W

project managers

Jan ‘16 – Feb ‘17

Study I TA sessions with end-users using the Z&W app

for the first time

Apr - Jun Study II Analysis of influence user characteristics on TA outcomes Jun - Aug Study III Value of UEMs in a redesign process Jun - Aug

(10)

comparison is made between the groups with regard to their thoroughness, validity, and effectiveness in detecting usability problems. Lastly, the potential difference between the two groups with regard to grades, usability scores, perceived value scores, and intention to utilise the work advice will be researched. The research questions of study II are described below:

Study II

 How does the difference in previous domain experience (primigravida vs. multipara) affect the number, categorisation, and severity of usability problems found in the Think Aloud sessions?  What are the thoroughness, validity, and effectiveness of both groups (primigravida vs. multipara) in

detecting usability problems?

 Does the difference in previous domain experience (primigravida vs. multipara) affect the users’ evaluation and acceptance of the Z&W app?

- What grades do both groups give the Z&W app?

- How do both groups rate the Z&W app on the System Usability Scale?

- What is the perceived value of the Z&W app in both groups?

- Do the two groups differentiate in intention to use the work advice provided by the Z&W app?

Chapter 4 will describe a small case study that looked into the potential value of pragmatic UEMs studies in a redesign process. Within this chapter the process, outcomes and comparisons are described between the pre-redesign UEMs and post-pre-redesign TA study. Within this chapter the following research question will be addressed:

Study III

 How can UEMs be utilised in a redesign process in a limited setting and what is the value of such a redesign process.

Lastly, in chapter 5 we will take a look at and discuss the overall research performed within this thesis.

References

1. Research2Guidance. The mHealth App Developer Economics 2016. Research 2 Guidance; 2016. 2. Free C, Phillips G, Watson L, Galli L, Felix L, Edwards P, et al. The effectiveness of mobile-health

technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Medicine. 2013;10(1):e1001363.

3. Akter S, Ray P. mHealth - an Ultimate Platform to Serve the Unserved. Yearbook of medical informatics. 2010:94-100. PubMed PMID: 20938579. Epub 2010/10/13.

4. Tamrat T, Kachnowski S. Special Delivery: An Analysis of mHealth in Maternal and Newborn Health Programs and Their Outcomes Around the World. Maternal and Child Health Journal. 2012 July 01;16(5):1092-101.

5. Majeed-Ariss R, Baildam E, Campbell M, Chieng A, Fallon D, Hall A, et al. Apps and Adolescents: A Systematic Review of Adolescents' Use of Mobile Phone and Tablet Apps That Support Personal Management of Their Chronic or Long-Term Physical Conditions. Journal of medical Internet research. 2015 Dec

23;17(12):e287. PubMed PMID: 26701961. PMCID: PMC4704897. Epub 2015/12/25.

6. de la Vega R, Miró J. mHealth: A Strategic Field without a Solid Scientific Soul. A Systematic Review of Pain-Related Apps. PLOS ONE. 2014;9(7):e101312.

7. Wildenbos GA, Peute LW, Jaspers MW. Influence of Human Factor Issues on Patient-Centered mHealth Apps' Impact; Where Do We Stand? Studies in health technology and informatics. 2016;228:190-4. PubMed PMID: 27577369. Epub 2016/09/01.

8. van Beukering MDM, Aaftink DK, Brand T, Broekhuijsen R, Hendriks K, de Lange CCM, et al. Practice Guideline - Pregnancy, Postpartum Period and Work. Utrecht: NVAB; 2007.

9. ZonMw. Project - De App ‘gezond werken tijdens de zwangerschap. Zwangerschap en geboorte Een impressie van het kennisnetwerk geboortezorg en onderzoeksprojecten Den Haag. 2014.

10. van Beukering MDM, Lebbink M, Bastiaanssen MHH. Samenwerking Bedrijfartsen en Verloskundigen - samenvatting en aanbevelingen. Stichting NVAB

(11)

11. Gray WD, Salzman MC. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human–computer interaction. 1998;13(3):203-61.

12. Dumas J. The great leap forward: The birth of the usability profession (1988-1993). Journal of Usability Studies. 2007;2(2):54-60.

13. Din E. 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs)–Part 11: Guidance on usability. International Organization for Standardization. 1998.

14. Patel VL, Kannampallil TG. Human factors and health information technology: current challenges and future directions. Yearbook of medical informatics. 2014 Aug 15;9:58-66. PubMed PMID: 25123724. PMCID: PMC4287067. Epub 2014/08/16.

15. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform. 2005 Aug;74(7-8):519-26. PubMed PMID: 16043081. Epub 2005/07/27.

16. Horsky J, Zhang J, Patel VL. To err is not entirely human: complex technology and user cognition. Journal of biomedical informatics. 2005 Aug;38(4):264-6. PubMed PMID: 15967732. Epub 2005/06/22.

17. Peute LW, Jaspers MW. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform. 2007 Feb-Mar;76(2-3):157-68. PubMed PMID: 16854617. Epub 2006/07/21. 18. Peute LW, Spithoven R, Bakker PJ, Jaspers MW. Usability studies on interactive health information systems; where do we stand? Studies in health technology and informatics. 2008;136:327-32. PubMed PMID: 18487752. Epub 2008/05/20.

19. Jaspers MWM. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics. 2009 2009/05/01/;78(5):340-53.

20. Khajouei R. Usability Evaluation of Healthcare Information Systems: Comparison of methods and classification of usability problems. Amsterdam: University of Amsterdam; 2011.

21. Nielsen J. Usability engineering: Elsevier; 1994.

22. Jaspers MWM, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. International Journal of Medical Informatics. 2004 2004/11/01/;73(11):781-95.

(12)

2. Usability evaluation of the ‘Pregnancy and Work’ application: a

Think Aloud study

Abstract

Introduction: A number of work conditions during pregnancy negatively influence pregnancy outcomes. Pregnant women are often unaware of the implications of these work conditions on pregnancy outcomes. A mobile application, the ‘Pregnancy and Work’ app, was created to inform and advise women about these implications. The aim of this study is to analyse how potential end-users experience and accept the app during first-time use through means of a Think Aloud study.

Methods: Potential end-users (n = 12) participated in Think Aloud sessions during which they were asked to perform nine tasks. All Think Aloud sessions were recorded, transcribed and co-analysed. The usability problems that emerged from the Think Aloud sessions were rated for their severity in accordance to Nielsen’s severity scale. The completion rates and times of tasks were also registered. Additionally,

participants were questioned on demographics, user characteristics, System Usability Scale, perceived value (Intrinsic Motivation Inventory: value subscale), and evaluation of the app (i.e., received work advice and grade).

Results: The Think Aloud sessions resulted in the identification of 82 real usability problems and 11 bugs. Forty usability problems were rated with a severity of 3 or 4. Half of the tasks were not completed by all participants and the first three tasks had the longest completion times. Three participants indicated that they had not received work advice, an additional two participants indicated that they would not utilise the provided advice. The mean perceived value of the app was a 4.8 out of 7, the overall mean system usability score was 68, and the mean grade given was a 6.7.

Conclusion: The main design issues of the app revolve around task completion rates, time spent on tasks, and severe usability problems found. The fact that 5 out of 12 (42%) participants will not act upon the provided work advice shows that the core purpose of the app is not fully achieved. Future development is needed to reduce the usability problems encountered and to improve the user’s evaluation and acceptance. Continuous research, user testing, and design improvements are facets that need to be addressed in order to realise a future version of the Z&W app as an appealing and valuable tool for informing and advising pregnant women about pregnancy-related work risks.

(13)

2.1 Introduction

A number of work conditions during pregnancy appear to negatively influence pregnancy outcomes (1-7). Examples of these conditions are: long working hours, working night shifts, physically demanding work, stress, and chemical or biological exposure. Those conditions can cause work-related pregnancy risks such as: low birthweight, stillbirth, preterm birth, and fetal abnormalities (7-10). Approximately 80% of Dutch pregnant women are in a position of employment (11). These work-related risks are often unknown by pregnant women (12). It is estimated that 25% of working pregnant women receive (obligatory) information about pregnancy related working conditions (13). The Pregnancy and Birth program, funded by ZonMw, consists of projects to improve these numbers (14). These projects aim to inform more pregnant working women about work-related pregnancy risks.

In recent years the accessibility of internet and mobile phones has resulted in a rise of mobile health applications (mHealth) (15). Especially women of reproductive age that are expecting a child are frequent consumers of online health information (16-20). In order to inform more working pregnant women on

pregnancy related work risks, it is a reasonable choice to implement a mHealth solution. One of the projects of the aforementioned Pregnancy and Birth program entails the development and implementation of a mHealth solution. mHealth has many benefits, one of which is the high accessibility of mHealth applications (21-24). However, mHealth also has its caveats. One caveat is that mHealth applications are not always of the best quality or devoid of evidence-based research (24, 25). Additionally, the impact of mHealth applications is only a subject studied for a couple of years (23, 26, 27). It is recommended that future research on the impact of mHealth applications (e.g., patient care) should incorporate analyses of the human factors influencing these effects (26). It is for, among other, those reasons that this mHealth solution underwent multiple (small) research studies described in this thesis. Within this chapter we will take an in-depth look at the evaluated usability and acceptance by potential end-users through a Think Aloud (TA) protocol.

2.1.1 ‘Pregnancy and Work’ project

The Pregnancy and Birth program, funded by ZonMw, consists of the project: The App ‘Pregnancy and Work’ (Dutch: ‘Zwangerschap en Werk’ app, abbr. Z&W app) (14). The aim of this project is to reduce work inflicted risks during pregnancy. Through an extensive research period the project supervisors aim to research, among other, the effect of the Z&W app on these risks. This mobile application that conforms to evidence-based guidelines (Practice Guideline – Pregnancy, Postpartum Period and Work) should inform and make pregnant women aware of these risks (27). The development process started with two multidisciplinary focus group meetings (28). The focus group meetings were conducted with pregnant women, occupational physicians, general practitioners, midwifes, obstetricians, and representatives of trade unions and employers’

organizations. These meetings resulted in 24 potential facilitators and 12 potential barriers for the use of the app. The three main facilitators were found to be: good user interaction with the app, the app should serve as a practical source of information, and the information should be understandable, evidence-based and well-dosed. The two main barriers were extensive memory and battery use and overuse of push-notifications. The results of these focus group meetings provided design instructions for the development of the Z&W app.

After the initial development process with constant feedback loops between the developer and project supervisors the Z&W app went through a small scale usability evaluation process, elucidated in chapter 4. Supported by the usability evaluations’ results, the iteration (updated app) that followed resulted in the first version of Z&W app that went live within the overall research project. It was with this version of the Z&W app that the TA study was conducted.

2.1.2 The Think Aloud method

The TA method is a usability inspection method to assess the usability of a product with (potential) end-users (29). The TA method requires participants to talk aloud (i.e., verbalise their thoughts) while performing a task or solving a problem. Through analysing the verbal data the cognitive processes that occur when performing or solving tasks can be characterised. The analysis of the verbal reports results in insights into the causes of usability problems (30). Additionally, through video capture the user’s actions can be subsequently analysed (31, 32). The TA method was chosen in this research as it appeared most suited to assess user specific usability

(14)

problems. Moreover, performing a TA also allowed for detailed qualitative user interaction data, which was very valuable for insights into the understandability of the Z&W app.

2.1.3 Research questions

The aim of this study is to provide an in-depth look at the usability of the Z&W app through means of a TA study. This chapter reports thorough qualitative data with regard to how users experience the Z&W app when using it for the first time. Additionally, this chapter will explore some of the user’s demographic data and their self-reported intention to use the provided work advice. These aims have led to the following (sub) research questions:

 What are the usability problems in the Z&W app found through the Think Aloud method during first-time use?

- What are the severity ratings of these usability problems?

 What are the end-users’ task completion rates and completion times?  How do users evaluate and accept the Z&W app?

- What grade do the participants give the Z&W app?

- How do participants rate the Z&W app on the System Usability Scale?

- What is the perceived value of the Z&W app?

- Do the participants indicate an intention to use the work advice provided by the Z&W app?

Lastly, this chapter will provide an overview of some general recommendations made for further development of the Z&W app, based on the gathered data.

2.2 Description ‘Pregnancy and Work’ application

The Z&W app is a webpage configured for mobile phone utilisation. Once the Z&W app will be available for Dutch public, time and effort will be spent on releasing a downloadable app available in app stores. The Z&W app requires the user to create an account to gain access to its content. After creating an account she has to fill in a questionnaire about the user’s (pregnancy related) medical condition(s) and work conditions. When completing this questionnaire the user will be directed to the home page of the app (figure 2.1.a). From this home page the user can navigate to all pages of the app. On the home page users can find small monthly pregnancy and work-related advice. When the user scrolls down she will find so-called baby messages (figure 2.1.b), these messages entail information about the growth of the unborn baby as the weeks pass by.

Underneath the baby message there is a video, made by the Academic Medical Center, with tips and information about pregnancy-related work advice.

a. b. c.

(15)

Near the bottom of the home page (figure 2.1.c) there is a menu to navigate to other pages; ‘All baby messages’, ‘Adjust your questionnaire’, ‘Your rights and tips for consultation’, and ‘More information’. At the bottom of every page the user can find the log out button and a button for English translations (the English version was not available during the UEMs performed in this thesis).

At the top of the home page (figure 2.2.a) the user can tap on the ‘Your work advice’ button to navigate towards that page (figure 2.2.a). The content of this page is set up according to what the user answered at the questionnaire page. The page is divided into five categories: work-related issues, medical history, symptoms previous pregnancy, symptoms at the start of current pregnancy, and symptoms pregnancy gestational age >20 weeks (figure 2.2.b-c). Based on the answers given at the questionnaire page the user will receive (general) advice about the user’s rights and actions she can undertake. If the user wants to create an overview of her advice she can scroll to the top of the page (figure 2.2.a) and there she can print or save (.pdf file) her advice.

a. b. c.

Figure 2.2.a-c. Screenshots of the ‘your work advice’ page of the Z&W app.

2.3 Method

2.3.1 Participants

Two obstetric care facilities, both located in Amsterdam, participated in this study. One was the obstetric department at the Academic Medical Center and the other was a midwifery practice. These two facilities were chosen because of their collaboration with the Z&W project, variety in patient groups, and accessibility for participant recruitment. Posters and flyers were distributed on both locations. Eligible participants were recruited in the waiting area. The inclusion criteria were: Dutch working women, whom were less than 20 weeks pregnant. These inclusion criteria were all a result of the Z&W app’s design set-up. Dutch was the only language available on the Z&W app, the app only provides services for women less than 20 weeks pregnant (this barrier was set up due to the Z&W project exclusion criteria), and was made to provide work advice. All participants that were included in the study received a gift card worth €15. After a recruitment period of several weeks a group of 12 female participants, aged between 28 and 41 (M = 32.9, SD = 3.8), had partaken in the TA sessions. None of the participants had utilised the Z&W app before this research.

2.3.2 Materials

The Z&W app was in this stage of the development process a webpage adjusted for mobile phone utilisation. This allowed for easy access to the Z&W app for the participants, regardless of their phone type or operating system. The System Usability Scale (SUS) was used to assess the perceived usability of the Z&W app (33). The SUS consists of 10 statements to which the participant must rate on a scale from 1 (strongly disagree) to 5 (strongly agree) as to which extent they agree with that statement. The value subscale of the Intrinsic

(16)

Motivation Inventory (IMI) was used to assess self-reported evaluation on how much the participant value the Z&W app (34). The value IMI subscale consist of 7 statements where the participant has to rate on a scale from 1 (do not agree) to 7 (strongly agree) to which extent they agree with that statement.

Additional questionnaires and questions were conducted to gain more insight into participants’ demographics, medical history related to pregnancy, working hours, and health literacy (NVS-D) (35-37). A more in-depth analysis of health literacy and previous domain experience will be discussed in chapter 3. Furthermore, questions were asked to provide an image on how often the participant utilises the most downloaded (pregnancy related) mobile applications. This was done to assess how well-known and practised participants are with (pregnancy related) mobile applications. Lastly, participants were tasked to give the Z&W app a grade on a scale from 1 to 10.

2.3.3 Procedure Think Aloud

A TA session comprised of three data collection stages (figure 2.3). Before the TA session took place the participants were informed about the content of the TA session and were asked to fill in an informed consent form.

Figure 2.3. Overview of study design.

Thereafter the participant would fill in a small survey about some characteristics to double check for exclusion criteria (i.e., employment, gestational age, possessing a mobile phone with internet access) and the NVS-D. After filling in the forms the participant would be prepared for the TA session (Appendix I). The participant had to practice how to ‘think aloud’ with example tasks given by the researcher. The participant was made aware that the researcher had no part in the development of the app and that she was solely interested in how the app performed not the participant herself. When the participant understood the procedure, the TA session would start. All TA sessions were recorded via means of a video camera (figure 2.4). The camera recorded what the participants were saying and recorded the screen of their mobile phone. The researcher would oversee whether the camera was still recording and functioning accordingly. Lastly, the researcher would only interrupt the participant to provide new tasks and to encourage her to keep talking in order to break long silences (29, 38).

Figure 2.4. Set-up TA session.

During the TA session the participant had to complete 9 tasks that were centered around the core purpose of the app. These tasks, their underlying motivation, and task achievement considerations can be found in table 2.1. The tasks were developed in collaboration with the developer and project supervisors of the Z&W project.

Stage I Stage II Stage III

 Informed consent  Exclusion criteria  NVS-D

 Think Aloud session  SUS

 IMI - value  Demographics  Grade Z&W app

(17)

Table 2.1. Participant tasks TA: description, achievement, and inclusion motivation

Task Achieved when Inclusion motivation

1 Create an

account.

Participant successfully created an account and can therefore utilise the app.

In order for the end-user to utilise the app she needs to be able to make an account.

2 Fill in the

questionnaire.

Participant successfully filled in the two pages of

questionnaires (mandatory) and gains access to the home page.

In order to use the app and receive work advice, the end-user needs to fill in the questionnaire.

3 Adjust answers

questionnaire.

Participant was able to adjust one of the answers in the aforementioned questionnaire.

The work advice provided in the Z&W app is based on the answers given in the questionnaires. If there is a change in either of these answers the user should be able to easily adjust her answers.

4

Find ‘Your rights and tips for consultation’ page.

Successfully found the ‘Your rights and tips for consultation’ page.

The ‘Your rights and tips for consultation’ page provides, what is considered by the project supervisor, essential information.

5 Find ‘baby

message(s)’.

Either located the ‘baby message’ on the home page or finds the ‘All baby messages’ page.

The baby messages were added by the developers to stimulate recurrent visits by the user.

6

Find the ‘Your work advice’ page.

Participant reached the ‘Your work advice’ page.

The main aim of the Z&W app is to provide end users with pregnancy related work advice. Finding this page is therefore vital for achieving this goal.

7

Find the ‘PRINT/SAVE’ button.

Participant located the ‘PRINT/SAVE’ button on the ‘Your work advice’ page.

The end-user should be able to print and save their work advice. This documentation can potentially be used for future discussion with an employer or care giver.

8 Find the goal of

the Z&W app.

Participant located the goal of the app on the ‘About us’ or ‘About this app’ page.

End-users will not use the app in a research setting, where the goal is explained beforehand. The end-user should therefore be able to locate the goal of the app on the app itself.

9 Log out of the

app.

Participant successfully logged out of the app.

Whether the participant can log out was added to simulate a normal session and to check whether users could find the log out button.

Next to completing tasks, the participants were also encouraged to explore the Z&W app. Participants were instructed to explore the page that they were tasked to find. They were told that they should browse through that what they would find particularly interesting. After the participant was done reviewing the page the next task would be given. The instruction to explore the content of a number of the pages allowed for more in-depth evaluation of what the participant thought of the content of the Z&W app. After completing the TA session the participant was then asked to fill in three questionnaires. These questionnaires were the SUS, IMI, and additional information where the latter comprised of: participant characteristics questions, mobile phone application utilisation, and grading the Z&W app.

2.3.4 Data collection and analysis

All questionnaires were filled in on paper and put in a .csv file. As mentioned earlier, all TA sessions were videotaped. This allowed for sessions to be reviewed multiple times and to be transcribed. All TA sessions were transcribed by the researcher. Each transcription comprised of text spoken by the participant, time stamps, actions, task completion, and completion times. Answers filled in by the participant in the Z&W app were also collected. Out of these data usability problems were found, categorised, and analysed. The severity of the usability problems was determined in accordance to Nielsen’s severity classification (39). Nielsen’s severity scale is a rating scale from 0-4 (figure 2.5) that allows for prioritization of usability problems that need to be revised in the development process. The usability problems overview was co-analysed and revised by a university-based human factor specialist (LDP).

(18)

0 – I do not agree that this is a usability problem at all.

1 – Cosmetic problem only: need not to be fixed unless extra time is available on project. 2 – Minor usability problem: fixing this should be given low priority.

3 – Major usability problem: important to fix, so should be given high priority. 4 – Usability catastrophe: imperative to fix this before product can be released.

Figure 2.5. Nielsen’s severity scale (39).

2.4 Results

2.4.1 Participants characteristics and task completion

The TA sessions with the participants (n = 12) took place in April until June 2017, the average duration of a TA session was 19:55 minutes (SD = 05:25). All participants had jobs, a gestational age less than 20 weeks (M = 15,

SD = 3), a smartphone (7 Samsung and 5 iPhone devices), and used their phone daily. The participants’ average

age was 33 (SD = 3.8) and they worked an average of 37 hours (SD = 6.15) per week. All participants had completed secondary school and varied in subsequenteducation levels (higher education = 8, intermediate vocational education = 4). The completion rates and times can be found in table 2.2. Task 2, 3, 5, and 9 were completed by all participants. Task 1, 4, 6, 7, and 8 were not. The first three tasks took, on average, the longest to complete ranging from 4 minutes to one minute and a half. Task 9 had the fastest mean completion rate of 4 seconds.

Table 2.2. Completion rates and times per task.

Task Completion rate Mean completion time (s) / (SD)

1. Create an account. 10/12 240 (SD = 83)

2. Fill in questionnaire. 12/12 179 (SD = 101)

3. Adjust answers questionnaire. 12/12 96 (SD = 74)*

4. Find ‘Your rights and tips for consultation’ page. 11/12 31 (SD = 38)

5. Find ‘baby message(s)’. 12/12 16 (SD = 10)

6. Find the ‘Your work advice’ page. 10/12 10 (SD = 8)

7. Find the ‘PRINT/SAVE’ button. 10/12 9 (SD = 9)

8. Find the goal of the Z&W app. 11/12 32 (SD = 18)

9. Log out of the app. 12/12 4 (SD = 4)

* Two participants initially did not understand this task.

2.4.2 Usability problems

The Think Aloud study identified a total of 101 usability issues, 82 of which were considered ‘real’ usability problems (i.e., severity >0). Additionally, the participants encountered 11 unique bugs when using the Z&W app. In the following sections the most frequent or severe usability problems will be elucidated per task or page location. An overview of these highlighted usability problems can be found in table 2.3. To view the entire usability problems overview, please contact the author, mentors or tutor of thesis. None of the participants found (severe) usability problems when completing task 5, 7, and 9, they are therefore not described in the following sections.

Table 2.3. Overview severe usability problems.

ID Usability problem Frequency Severity Location

2 Functionality [create an account] button misunderstood. 4 4 Log in

7-10 Password feedback pop-up unclear. 8 2-3 Create account

14 Layout problem month button. 12 4 Create account

15 System feedback issue due date entry field. 1 4 Create account 16 Terminology problem ‘terms and conditions’. 2 4 Create account 19 [Continue] button does not appear, participant presses the wrong link. 3 4 Create account

29 Terminology interpretation problem, Q1. 1 4 Questionnaire

30-31 Terminology interpretation problem, Q1.1. 2 4 Questionnaire

51 Location ‘rights and tips’ page unclear/unknown. 2 3 Home

55 Functionality [Your work advice] button unclear. 2 4 Home

(19)

76 Participant goes back to previous page instead of using the menu button. 4 4 Home 79 Inconsistency in information presentation on work advice. 1 4 Work advice

93 Participant cannot locate the goal of the app. 1 4 Home

98 Layout menu [stop participation] button. 1 4 Home

2.4.2.1 Task 1 – Create an account

ID 2. Functionality [create an account] button misunderstood. In order for the user to create an account she

needs to press the [create an account] button to be navigated to that respective page. However, 4 participants filled in a username and password on the log-in part of the page and then expected to have made an account when pressing the [create an account] button. This led to confusion later on when participants received a password mismatch feedback on the ‘create an account’ page, some thought that this was because that password didn’t match the password on the log-in page. This was not the case.

“Then the question is if it is a ‘mismatch’ with what I entered on the first screen (login page) or a ‘mismatch’ between these two.” – participant 6

ID 7-10. Password feedback pop-up unclear. The user is required to create a password and confirm this

password by entering it a second time. Underneath these two entry fields there was a password strength indicator which caused issues for participants. An example is that 4 participants did not understand that the pop-up provides real-time feedback. The participant would start typing in their confirmation password but the feedback of ‘mismatch’ would pop-up immediately, this confused a lot of participants in such a way that they retyped both password entry fields multiple times. Moreover, the meaning of this strength-indicator was unclear to some participants (ID 7 & 10). The app did not state that the password should conform to any rules, such an explanation would only appear when user has entered a ‘bad’ password when finalising the ‘create an account’ page.

“I think I’m turning mad (after retyping password data entries multiple times).” – participant 4

ID 14. Layout problem month button. When selecting the due date entry field, a calendar would appear

partially on screen (figure 2.6). This led to frustrated users and difficulty in finding the button to edit the month. Additionally, this was a small button to select.

Figure 2.6. Layout calendar.

ID 15. System feedback issue due date entry field. One user tried to manually type in her due date. The app did

not recognise the ‘-‘ symbol as a mean of differentiation between the day, month, and year. This resulted in the user thinking that the app only needed to know which year she was due to have her baby. Subsequently, this resulted in the absence of the [continue] button at the bottom of the page (ID 19).

ID 16. Terminology problem ‘terms and conditions’. Two participants stated that the ‘terms and conditions’

checkbox is written in English. One of the participants subsequently did not select this checkbox and encountered usability problem ID 19.

(20)

ID 19. [Continue] button does not appear, user presses the wrong link. When the user does not fill in all the

data entry fields on the ‘create account’ page the [continue] button does not appear. The user is unaware that such a button should appear and receives no feedback about not filling in all the data entries. As a

consequence the participants strayed further down the page where there is a link called: ‘Log in’ (figure 2.7).

Figure 2.7. ‘Log in’ link at the button of the page.

Four times participants pressed this button and were eventually navigated back to the login page. The lack of feedback resulted in participants being unaware that they failed to create an account. The consequence of this action is that the participants had to refill all their data entries on the ‘create an account’ page.

“I don’t think this went well. We have to fill everything in again. Now it asks me to create an account again, I think I have just done that.” – participant 7

2.4.2.2 Task 2 – Fill in questionnaire

For almost each question in the questionnaire (apart from Q3.) there were some issues regarding readability or interpretation of the terminology. However, a number of terminology interpretation issues were deemed more severe than others.

ID 29. Terminology interpretation problem question 1. Question 1 asks whether the user had problems during

her previous pregnancy. One of the participants is unsure whether ‘previous pregnancy’ implies her previous pregnancy, or those before. This participant in particular had no problems during her previous pregnancy, but did experience issues the pregnancy before that one. In the end she assumed it implies her direct previous pregnancy, which therefore resulted in missing information about her problems during one of her previous pregnancies.

ID 30-31. Terminology interpretation problem question 1.1. Question 1.1 is a follow up question of questions 1

when a user selects [Yes]. Question 1.1 asks the user which problem she encountered during her previous pregnancy. One participant felt that her problem is not on the list of options, she expected an extra answer option. She and another participant had trouble finding the best suited option to describe their problem.

“… But I do not know if that can put under ‘deceased child’ or ‘child born before a gestational age of 37 weeks’? You know what I mean?” – participant 3

ID 32. Terminology interpretation problem question 2. Question 2 asks the user if she had general

health-related problems before she got pregnant. One participant was conflicted on whether this was about her health status before her previous or current pregnancy.

40. Terminology interpretation problem question 9. Question 9 asks the user if she could be exposed to any

chemical agents in her work environment, followed by a list of examples. Several participants did not read the list of examples and answered no. As a result, the possibility remains that they might have overlooked the possibility that they are exposed to chemical agents. Moreover, 2 participants did not know whether an agent that they worked with should be considered chemical as it was not on the list of examples.

(21)

“… I’m having doubts. I work with laughing gas. That’s not very chemical, but… I don’t know whether I should answers yes or no.” – participant 11

2.4.2.3 Task 3 – Adjust questionnaire

ID 76. Participant goes back to previous page instead of using the [Adjust your questionnaire] button. After the

participants had a look at the home page they were instructed to adjust one of the answers on the questionnaire that they had previously filled it. Four participants utilised the [Previous] button instead of utilising the [Adjust your questionnaire] button in the menu at the bottom of the home page. Using the [Previous] button resulted in a reloaded questionnaire page, meaning that all the data entries were deleted. This caused confusion among some of the participants since they were unsure whether the app had stored the answers or not. In the end these four participants had fill in the questionnaire again in its entirety.

2.4.2.4 Task 4 - Find ‘Your rights and tips for consultation’ page

ID 51. Location ‘Rights and tips for consultation’ page unclear/unknown. Two participants expected to find

information about their rights and tips for consultation at the top of the home page. One of these participants proceeded to an external website and stated that she had completed the task. This participant never reached the ‘Your rights and tips for consultation’ page.

2.4.2.5 Task 6 - Find the ‘Your work advice’ page

ID 55. Functionality [Your work advice] button unclear. Two participants did not understand that the [Your

work advice] button was clickable and therefore sought work advice elsewhere or stated that they could not find it (figure 2.8).

Figure 2.8. Location ‘Your work advice’ button on home page.

ID 56. Participant believes that her work advice is depicted on the home page. On the home page there is a

small section with tips and information about pregnancy-related work advice. However, to some this was interpreted as their entire work advice. Two participants thought this was the case and therefore both missed the content of the ‘Your work advice’ page. This automatically meant that both these participants were unable to save or print their work advice. This function is only available on the ‘Your work advice’ page.

2.4.2.6 Task 8 – Find the goal of the app

ID 93. Participant cannot locate the goal of the app. There are multiple ways to find information about the goal

of the Z&W app. When the user goes to [More information] at the bottom of the home page, she will find at least two routes to find the goal of the app. However, one participant was unable to find this information. There is a high probability that the participant expected a distinct button/section titled: ‘Goal of the app’.

(22)

2.4.2.7 Home page

ID 98. Layout menu [stop participation] button. One participant nearly pressed the [stop participation] button

because it stands out from the other buttons.

2.4.2.8 Your work advice page

ID 79. Inconsistency in information presentation on work advice. The work advice page is filled in according to

what answers the user gives in the questionnaire page. This can results in some sections having no advice, but some do. One participant got no advice underneath the work header, however she did receive work advice with regard to issues during her previous pregnancy. When asked whether the participant received work advice she answered no.

“None? That’s easy. I don’t need to make any work adjustments. I don’t think so either, because I have an office job.” – participant 1

2.4.3 User evaluation

Three participants indicated that they had not received work advice from the Z&W app (table 2.4). However, this was not true, all participants received some form of pregnancy-related work advice. Among the 9

participants that did state that they received work advice, two indicated that they would not utilise the advice. The overall mean IMI value score was a 4.8 (SD = 0.9) out of 7. The overall mean SUS score was 68 (SD = 11) and the mean grade given to the Z&W app was a 6.7 (SD = 0.89). The sample size was not large enough to perform reliable correlation calculations between the completion times, SUS, IMI, and grades given by the participants.

Table 2.4. User evaluation.

Work advice? Use advice? IMI SUS Grade

1 No NA 5.57 85 8 2 No NA 4.29 77.5 7 3 Yes Yes 3.71 55 5 4 Yes Yes 5.00 77.5 7 5 Yes Yes 5.14 65 8 6 Yes No 4.43 77.5 6 7 Yes Yes 5.57 57.5 7 8 Yes No 3.00 70 7 9 No NA 4.29 75 7 10 Yes Yes 6.29 55 6 11 Yes Yes 5.29 50 6 12 Yes Yes 4.86 72.5 6

2.5 Recommendations

In this section we will provide general recommendations based on a majority of the usability problems and the user’s evaluation data. It must be noted that these recommendations serve as a translation of the issues found, this section will not provide content-related or design recommendations. Such recommendations should, with help of the data gathered in this study, be made by appropriate domain experts. Instead of tasks these recommendations will be categorised according to the location of the usability problem. Nevertheless, the order of these categories will share a resemblance to the order in which the tasks were executed.

2.5.1 Login page

Usability problems and bug

The TA results indicated that participants experienced 4 unique usability problems on this page (ID 1-3, 20). Multiple participants entered a username and password and either clicked on the [log in] button or [create an account] button (ID 1 & 2). Both usability problems indicate that they thought that by filling in the username and password they would create an account. Feedback provided by the app when clicking the [log in] button was written in English and did not always help the participant in navigating to the [create an account] page. Additionally, the [create an account] button was not always in view when opening the app. Another severe usability problem that occurred was that when participants failed to create an account and were redirected to

(23)

the login page, they had no knowledge that they failed to make an account (ID 20). Lastly, the log in page has a ‘Need help?’ feature, however this page has no content (ID B1).

Recommendations

The feedback provided on the login page should be expanded and be more thought through. Multiple times the participants were unaware of their status within the Z&W app. Additionally, the layout of the page should be reconsidered in order to prevent the user missing vital information or buttons.

2.5.2 Create an account page

Usability problems and bug

The ‘create an account’ page caused the participants 19 usability problems and 1 bug (ID 4-23). Moreover, participants spent most time on this particular task (µ = 4 minutes). The password entry fields caused a lot of difficulties. The participants thought they could click on the strength-indicator (ID 8), participants thought that ‘bad’ meant that they had not filled in all the other entries (ID 7), participants did not realise that the strength-indicator feedback was real-time (ID 9), participants were not aware of the consequences of a ‘bad’ password (ID 10), one participant said that the strength-indicator feedback scared her (ID 12), and the participants received poorly translated information about the password strength after they tried to create an account with a ‘bad’ password (ID 13). The due date entry field also resulted in some usability issues: when clicking on the entry field the calendar only appeared partially in frame (ID 14) and when trying to enter the due date manually the participant needed to use the ‘-‘ symbol (ID 15). The first due date related usability problem happened to all 12 participants and resulted in participants occasionally having difficulty finding the button to select the month. The second usability problem occurred once but ultimately resulted in the participant needing to re-enter all her data. The problem with ID 15 was that the participant did not receive adequate system feedback and therefore assumed that what she had filled in was correct. One of the biggest issues on the ‘Create an account’ page was that the [continue] button at the bottom of the page would not appear when the participant did not (correctly) complete all the entry fields (ID 19). The participants were unaware that this button should appear and therefore clicked on the ‘log in’ link at the bottom of the page (this would redirect them to the login page). This resulted in having to redo the entire ‘Create an account’ page. Moreover, participants were (initially) unaware that they had not successfully made an account. They tried (multiple times) to log in with their username and password, but to no avail. This caused frustration among the participants (ID 23). There was one bug that occurred during one of the TA sessions where the participant clicked numerous times on the [continue] button, but nothing happened (ID B2). This resulted in the

consequence of the aforementioned ID 19, where the participant went to the ‘log in’ link and had to redo her ‘create an account’ process.

Recommendations

Overall, the process of making an account should not take the end-user 4 minutes. It is recommended to provide more clarity and feedback on the password strength-indicator. The strength-indicator should not be removed as it forces the user to use a strong password. This feature was added after the first evaluation phase (chapter 4), when the data security heuristic was violated (40). Suggested improvements are a better layout and object visibility of the strength-indicator (i.e., in view and not appear as a button), information about password requirements, and delayed feedback (i.e., when user is finished typing the second password). The due date entry field should allow for ‘-‘ delineation and the calendar should show up fully in frame. The biggest issue that needs to be resolved is the [continue] button not appearing at the bottom of the page. This button should be there regardless whether the user (correctly) fills in all the entry fields and work properly. If the user made a mistake, bad passwords, etc. she should get an appropriate feedback message when clicking on the [continue] button.

2.5.3 Questionnaire page

Usability problems

Most of the 24 usability problems found in the questionnaire page are about readability and terminology issues (ID 24-47). Readability issues were found with questions; 4, 5, 7, and 9. Several participants were prone to answer before reading the entire question (ID 34, 36, 42). This might result in the user not answering

Evaluation of a mHealth application for pregnancy-related work advice and information: The impact of user characteristics on Think Aloud method results