Student Luna de Rooij
Student number: 10794557
Email: l.d.derooij@amsterdamumc.nl / lunaderooij@hotmail.com Place of the Scientific Research Project
National Institute for Public Health and the Environment (RIVM)
Department of National Coordination Centre for Communicable Disease Control (LCI) Period: December 2019 – July 2020
Mentors
F. Meerstadt-Rombach, MD
National Institute for Public Health and the Environment (RIVM)
Department of National Coordination Centre for Communicable Disease Control (LCI) Email: fleur.meerstadt@rivm.nl
M.E.J. Bongers, PhD
National Institute for Public Health and the Environment (RIVM)
Department of National Coordination Centre for Communicable Disease Control (LCI) Email: marloes.bongers@rivm.nl
Tutor
L.W.P. Dusseljee-Peute, PhD
Amsterdam UMC, University of Amsterdam Department of Medical Informatics
Content
Content ... 3
Preface ... 5
Alterations due to COVID-19 pandemic ... 5
Acknowledgements ... 5
General abstract ... 7
Samenvatting ... 8
General introduction ... 10
Outline and research questions ... 11
References ... 13
Theoretical framework ... 15
1. Conceptual thinking ... 15
2. User-centered design approach ... 18
2.1 Content analysis ... 19
2.2 Usability evaluation ... 20
2.3 Optimization ... 21
References ... 22
Study I Quality assessment of a health information website regarding infectious diseases during pregnancy: A thematic content analysis ... 25
Abstract ... 25
1. Introduction ... 26
2. Methods ... 26
3. Results ... 29
3.1 Overview navigational structure ... 29
3.2 Thematic content analysis ... 31
3.3 Relational model ... 34
3.4 Completeness of information framework ... 34
4. Discussion ... 36
4.1 Main findings and implications ... 36
4.2 Strengths and limitations ... 37
4.3 Comparison to other studies ... 38
4.4 Future research ... 38
5. Conclusion ... 39
References ... 40
Study II Usability evaluation of a health information website regarding infectious diseases during pregnancy: A mixed methods approach ... 43
Abstract ... 43
1. Introduction ... 44
2.1 Heuristic evaluation ... 45
2.2 Think aloud analysis ... 46
2.3 Online focus group ... 47
3. Results ... 49
3.1 Heuristic evaluation ... 49
3.2 Think aloud analysis ... 53
3.3 Comparison results heuristic evaluation and think aloud analysis ... 54
3.4 Online focus group ... 54
3.5 Comparison results heuristic evaluation, think aloud analysis, and online focus group ... 56
4. Discussion ... 57
4.1 Main findings and implications ... 57
4.2 Strengths and limitations ... 58
4.3 Comparison to other studies ... 59
4.4 Future research ... 59
5. Conclusion ... 59
References ... 60
Optimization of the website providing information on infectious diseases during pregnancy ... 62
Points of improvement ... 62 Missing information ... 62 Excessive information ... 62 Visualizations ... 62 Content description ... 63 Navigational structure ... 64 Titles ... 64 Target group ... 64 Errors ... 64 General discussion ... 67
Main findings and implications ... 67
Strengths and limitations ... 68
Future research ... 69
Overall conclusion ... 69
References ... 71
Appendix ... 72
A. Heuristics defined by Zhang et al. and Pierotti ... 72
B. Task scenarios for heuristic evaluation ... 73
C. Nielsen’s severity scale rating ... 74
Preface
This thesis is the final product of my scientific research project (SRP) of the master Medical Informatics at the University of Amsterdam. The National Institute for Public Health and the Environment (RIVM) commissioned this SRP, which started in December 2019 and was finalised in July 2020. My journey with RIVM started with the bachelor internship two years ago. During that internship, I researched the information needs of pregnant healthcare workers regarding infectious diseases. The results led to the organization of a hackathon with healthcare professionals and other stakeholders involved in pregnancy care or in the provision of online information a year later. The hackathon led to further research on infectious diseases during pregnancy. From the beginning I was engaged in this study and was allowed to determine and execute the following steps. This SRP has allowed me to dive further into the field of usability evaluation methods. Moreover, due to the COVID-19 pandemic I was challenged in the available options in terms of participants and suitable methods. This resulted in a flexible work attitude and an individual approach and implementation. I have learned greatly during these past months and this thesis represents the work that I have conducted with the support of my mentors and tutor.
Alterations due to COVID-19 pandemic
Due to the corona pandemic, my scientific research was altered from its original plan. The idea was to evaluate the usability of the RIVM website Pregnancy and infectious diseases with healthcare professionals. The preferred evaluation method was the think aloud analysis. However, healthcare professionals were otherwise occupied due to the COVID-19 pandemic. To perform think aloud sessions with healthcare professionals would take up too much of their time. We therefore focused on healthcare professionals in training, since medical internships were paused. Their characteristics were overall comparable to the characteristics of intended end users. Chosen was to invite these participants for the think aloud analysis. In addition, an online focus group with intended end users was performed in order to collect results from real users. Implementing (part of) the recommendations to test the redesign of the website was not possible yet. Therefore, this thesis provides recommendations for the RIVM website and an overall framework on how to approach redesign of health information websites in general.
Acknowledgements
First, I would like to thank my mentors from RIVM, CIb/LCI for offering me this opportunity and entrusting me with this study. During these months, I have learned deeply about all the aspects and process of (performing) a research, especially in time of a pandemic. Moreover, this made me learn to be more flexible and my creativity and solution-based thinking was challenged. I would especially like to thank Fleur Meerstadt-Rombach for her continuing support during this study. Your involvement and enthusiasm about this project were very welcoming. Mostly, your action-oriented approach was really nice to work with and gave me trust in time of stress. Marloes Bongers, thank you for your expertise and keeping an overview of this study. Furthermore, I would like to thank Linda Dusseljee-Peute, for your good advice and support during stressful situations. Your view on this project, provided me insight in the scientific knowledge of this study. Our weekly meetings (by phone) always gave me that ‘aha’ moment when I was stuck writing this thesis. I am very grateful that you all have guided me along this journey. You have made me excited about doing research. Finally, I would like to thank everyone who participated or contributed in anyway in this SRP. Without you this thesis would not have been possible.
General abstract
Introduction Infectious diseases can cause serious complications for women when acquired during pregnancy. The National Institute for Public Health and the Environment (RIVM) provides online information regarding infectious diseases during pregnancy as an element of the strategy for infection prevention and control. However, multiple internal studies of the RIVM concluded that the information is fragmented over the website and some information materials, such as brochures, are not up to date. Furthermore, the complexity of the RIVM website is high, due to the continuous addition of new information which limits its strategic communication purpose. The aim of this study was to evaluate the content quality and provide redesign recommendations for the RIVM website. In doing so, the overall study objective was to establish a User Centered Design (UCD) framework to evaluate content, quality, and usability of complex health information websites to inform redesign. Methods This study is derived in two studies based on the new framework. The first study assessed the information completeness of the website. A thematic content analysis was performed on 28 webpages providing information on infectious diseases during pregnancy and resulted in the identification of 13 categories. For each category the quality and information completeness provided on the website was then analysed based on the quality dimension framework by Tao et al. The second study evaluated the usability of the website by performing three pragmatic usability evaluation methods, namely a heuristic evaluation, think aloud analysis, and an online focus group. Based on the findings of both studies within the UCD cycle, recommendations are provided for website optimization.
Results In total, 13 information need categories were determined to cover the information on the website. Results indicated that the quality of the current RIVM website was inadequate, due to missing information in the categories: breastfeeding, immunity, symptoms, transmission, vaccination, and work policy. Sufficiently presented categories are health risks, measures, diagnostics, and treatment. There were numerous absent categories for bladder infection, cold sores, flu, genital herpes, group A streptococci, leptospirosis, and malaria, while chicken pox, hepatitis B, measles, and Q fever were adequately presented. The usability evaluation with expert evaluators and (potential) end users identified problems regarding consistency, content, navigation, ease of use, and design of the RIVM website. The combination of three usability evaluation methods enabled complete coverage of usability problems. Based on previous findings, recommended is to assess the relevance of information gaps, add a clickable content list, utilize clear and consistent titles, add visualizations, and utilize a clear distinction between information for different target groups.
Discussion Overall, the defined UCD approach to establish redesign recommendations for the website covered a triangulation of methods. The findings of the thematic content analysis were acknowledged in the following phase of the framework. The usability evaluation concluded that participants were missing information and encountered problems with the findability of information and navigation on the website. By following the phases in evaluating the content and usability of the website, action-oriented recommendations for optimization were drafted. However, actual content that is needed within certain categories needs further work. Preferably, a working group of field experts will be formed to address this. Further steps need to be taken to investigate the applicability of the new developed framework on other complex health information websites.
Conclusion In conclusion, the new developed framework offers structure for complex health information websites by evaluating the content, quality and usability of a website, which leads to suggestions for improvement. This case study specifically concerned a website that is part of a national information communication strategy for infection prevention and control. The website contributes to this strategy by providing information on infectious diseases during pregnancy. The application of the framework gave a complete overview of information provided on the website, its quality and findability. Recommendations for redesign aim to ensure effective information retrieval for healthcare professionals and public.
Samenvatting
Introductie Sommige infectieziekten kunnen negatieve gevolgen hebben voor de zwangere vrouw, het ongeboren kind of de pasgeborene als vrouwen hiermee in aanraking komen tijdens hun zwangerschap. Het Rijksinstituut voor Volksgezondheid en Milieu (RIVM) biedt online informatie over infectieziekten tijdens de zwangerschap als onderdeel van de strategie om zwangere vrouwen voor te lichten en het oplopen van infectieziekten tijdens de zwangerschap te voorkomen. Onderzoek naar deze informatievoorziening laat zien dat informatie echter versnipperd is over de website en niet alle informatiematerialen, zoals brochures, actueel zijn. Bovendien is de complexiteit van de RIVM-website hoog door het regelmatig toevoegen van nieuwe informatie, wat de vindbaarheid van informatie vermoeilijkt. Het doel van deze studie is om de kwaliteit te evalueren en om verbeterpunten voor de RIVM-website te bieden. Het uiteindelijke doel van deze studie was om een User Centered Design (UCD) framework op te zetten die de content, kwaliteit en bruikbaarheid evalueert van complexe zorginformatiewebsites.
Methoden Dit onderzoek is onderverdeeld in twee studies die de stappen van het framework volgen. De eerste studie focust op de informatiecompleetheid van de website. Een thematische content analyse van 28 webpagina’s op de RIVM-website over infectieziekten tijdens de zwangerschap resulteerde in dertien relevante categorieën. Deze categorieën werden toegepast in het Quality Dimension Framework opgesteld door Tao et al. om de huidige kwaliteit van de website te beoordelen. De tweede studie evalueerde de bruikbaarheid van de website aan de hand van drie gebruikersevaluatiemethoden, namelijk de heuristische evaluatie, de think aloud analyse en een online focusgroep. Gebaseerd op de bevindingen van beide studies in combinatie met de UCD-cyclus zijn aanbevelingen aangedragen ter optimalisatie van de website.
Resultaten In totaal werden 13 relevante categorieën geïdentificeerd die de content van de website omvatten. De resultaten tonen aan dat de kwaliteit van de huidige RIVM-website ontoereikend is door ontbrekende informatie in de categorieën: borstvoeding, immuniteit, symptomen, transmissie, vaccinatie en werkbeleid. Voldoende gepresenteerde categorieën zijn gezondheidsrisico’s, maatregelen, diagnostiek en behandeling. De webpagina’s over blaasontsteking, genitale herpes, griep, groep A streptokokken, koortslip, leptospirose en malaria hadden meerdere afwezige categorieën. Daarentegen waren hepatitis B, mazelen, Q-koorts en waterpokken adequaat gepresenteerd op de website. De gebruikersevaluatie met expert evaluatoren en (potentiële) eindgebruikers identificeerde problemen betreft consistentie, content, navigatie, gebruiksgemak en design van de RIVM-website. Gebaseerd op voorgaande resultaten wordt aangeraden om de missende categorieën te beoordelen op relevantie, een klikbare inhoudsopgave toe te voegen, gebruik te maken van duidelijke en consistente titels, visualisaties toe te voegen en om een duidelijk onderscheid te maken in de informatie bestemd voor de verschillende doelgroepen.
Discussie Het gedefinieerde UCD framework omvatte een triangulatie van methoden voor het opstellen van aanbevelingen. De bevindingen van de thematische content analyse werden bevestigd in de opeenvolgende stap van het framework. De gebruikersevaluatie toonde aan dat participanten informatie misten op de website en dat het vinden van informatie en navigeren op de website moeizaam is. Door de stappen van het framework te doorlopen, kunnen er goed onderbouwde en actiegerichte aanbevelingen worden opgesteld ter verbetering van de website. De benodigde content voor missende categorieën moet worden uitgewerkt door een werkgroep bestaande uit experts op het gebied van infectieziekten. Verder onderzoek is nodig om de toepasbaarheid van het framework te testen op andere complexe websites die zorginformatie bieden.
Conclusie Deze studie concludeerde dat het framework structuur kan bieden aan een complexe zorg-informatiewebsite door de kwaliteit betreft informatiecompleetheid en de bruikbaarheid van de website te evalueren. Deze evaluatie leidt tot duidelijke verbeterpunten ter optimalisatie van de website. In deze actuele casus gaat het specifiek om een website die onderdeel vormt van een nationale informatiecommunicatie-strategie om infectieziekten te voorkomen en te beheersen. De website draagt bij aan deze informatiecommunicatie-strategie door informatie aan te bieden over infectieziekten tijdens de zwangerschap. De toepassing van het framework bood een compleet overzicht van de content, kwaliteit en bruikbaarheid van de RIVM-website. De aanbevelingen voor optimalisatie verzekeren dat kennisoverdracht op effectieve wijze gebeurt.
General introduction
Emerging and re-emerging infectious diseases are a significant public health concern [1]. Infectious diseases are of concern specifically when it involves pregnant women. Some infectious agents cause mild-to-moderate diseases in healthy adults or children, however, can cause serious complications for both mother and unborn child if acquired during pregnancy [2-5]. Therefore, early detection and immediate response is crucial for disease control and long-term effects on pregnant women, unborn children, and/or newborn children [1]. Worldwide, governments and healthcare organizations develop contingency plans in preparation for emerging infectious diseases. These plans acknowledge that communication is vital in an event of an outbreak and include strategies for preparedness, response, and recovery [6]. Aside from a serious outbreak, information communication is vital when an infection causes a mild course of disease in the entire population, however, causes severe health risks for pregnant women. In the Netherlands, the strategy of the Centre for Infectious Disease Control (CIb) within the National Institute for Public Health and the Environment (RIVM) consists of four core tasks in order to manage the control of infectious disease: monitor, facilitate, fight, and protect. ‘Facilitate’ includes promoting expertise, development of directives, and innovative research. For example, the current situation with regard to the COVID-19 pandemic caused by Coronavirus outbreak showed us a process of growing insight and knowledge, and thus a continuing development of information for the public, protocols, and directives. This unique situation and health threat illustrate how important effective delivery of information is. Moreover, the COVID-19 pandemic demonstrated that countries have different strategies regarding measures to contain the virus and ways of communicating relevant information.
One of these mediums to deliver information in an effective manner is online via the internet, besides through stakeholders, guidelines, and education. As the development of online information technologies, web-based information has become increasingly important in health [1]. Both healthcare professionals and public have been extensively utilizing health information websites to fulfil their information needs regarding infectious diseases during pregnancy. Especially now, when health institutions use the web as medium for official communications [7,8]. Delivering online information regarding infectious diseases during pregnancy requires accurate and advanced information processing, evaluation, and strict compliance to guidelines. Communicating accurate health information in the correct presentation is critical to avoid misinterpretation of (mis)information as it may lead to harmful consequences [9]. For instance, online information can be misinterpreted due to unclear advice and causes a delay in a doctor’s visit, which possibly worsens the current condition of the pregnant woman. Therefore, health information websites have an important role in the national strategy for prevention of the burden of infectious diseases and for improvement of public health. The RIVM is the central point concerning infectious diseases in the Netherlands. Within the RIVM the National Coordination Centre for Communicable Disease Control (LCI) develops guidelines, provides support, coordinates outbreaks and operational crisis management on infectious diseases. Moreover, the centre provides information regarding infectious diseases during pregnancy for both professionals and public. The website Pregnancy and infectious diseases provided by the RIVM was selected for this case study, since multiple internal studies of the LCI concluded that the information is redundant and diffuse over the website. In addition, not all information materials, such as brochures, are up to date. Another problem is that the overview of content is disordered, due to the increasing complexity of the RIVM website as a consequence of continuous addition of new information. As a consequence, extraction of adequate information regarding infectious diseases during pregnancy requires more than casual browsing.
The aim of this study was to evaluate RIVM website regarding infectious diseases during pregnancy for redesign. In doing so, the overall study objective was to establish a User Centered Design (UCD) framework for optimization of complex health information websites by evaluating the content, quality, and usability. This way, the website forms to be an effective element in an information communication strategy.
Outline and research questions
The varied approach for this study consisted of four phases: 1. Literature study;
2. Content analysis; 3. Usability evaluation; 4. Optimization.
Figure 1 Overview of methods applied in this study.
The literature review formed the basis of a UCD approach of the new developing framework to evaluate the content, quality, and usability of the RIVM website. The relationships between the applied methods as a part of the UCD approach will be explained in more detail in the following chapter. The review resulted in a thematic content analysis of the current website Pregnancy and infectious diseases provided by the RIVM (step 1 and 2 in Figure 1). This analysis is focused on the content and navigational structures in general. The usability testing consists of three methods, namely an expert website evaluation, a user evaluation, and an online focus group (step 3 in Figure 1). The aim of usability testing is to gain insight into user interaction with the website and navigational paths to the provided information. The optimization of the website is based on the three previous phases (step 4 in Figure 1). In this study, the main research question is:
Main research question
How is a complex health information website evaluated on its content and usability to create a structured redesign in order to be an effective element in an information communication strategy?
The answers to this main research question are derived from the content analysis (study I) and usability evaluation (study II). Study I (step 2 in Figure 1) illustrates the available content on the website
Pregnancy and infectious diseases provided by the RIVM, including navigational structures. The aim is
to research which information is offered and in which way this is structured. Currently, extensive information is provided without clusters or clear target groups. The results provide us insight into availability, consistency and findability of information on the website. This study focuses on the themes acquired from the content analysis. The research question of study I is:
Study I research question
• How is the online information landscape within the RIVM regarding infectious diseases during pregnancy currently structured?
• To what extent is the information provided by the RIVM website Pregnancy and infectious
diseases complete?
Study II (step 3 in Figure 1) elucidates the multiple methods of usability testing of the website
Pregnancy and infectious diseases provided by RIVM. The first method, heuristic evaluation (HE),
focuses on the usability problems found by junior expert evaluators based on a set of heuristics. Furthermore, the website was evaluated by means of the think aloud analysis (TA). The detected usability problems in the HE were compared with the usability problems found during TA. The third method consists of an online focus group where healthcare professionals expressed their opinions according to predefined statements regarding the availability and efficiency of information on infectious diseases during pregnancy. The corresponding research questions of study II are as follows:
Study II research questions
• How do potential users of the website evaluate the usability of the website Pregnancy and
infectious diseases provided by the RIVM?
• What are the differences in results between the methods for usability inspection and user interaction testing?
• What is the opinion of healthcare professionals involved in pregnancy care with regard to the current design and available information of the website Pregnancy and infectious
diseases provided by the RIVM?
The recommendations for website optimization (step 4 in Figure 1) are based on the literature review on design principles, study I, and study II. Recommendations are described in detail, including visualizations in the form of single screen images and are admitted in Chapter 6.
Optimization research question
• How should the optimal website be structured based on previous findings and design principles for effective online knowledge transfer?
References
1. Choi, J., Cho, Y., and Woo, H. 2016. Web-based infectious disease surveillance systems and public health perspectives: a systematic review, BMC Public Health, 16, 1238.
2. Committee on Practice Bulletins. 2015. Cytomegalovirus, parvovirus B19, varicella zoster, and toxoplasmosis in pregnancy. Practice Bulletin No. 151. American College of Obstetricians and Gynaecologists, Obstet Gynaecol 2015, 125, 6, 1510–1525.
3. Racicot, K. and Mor, G. 2017. Risks associated with viral infections during pregnancy, The Journal of Clinical
Investigation, Mei 2017, 127, 5, 1591-1599.
4. Lyons, P. 2015. Infection in Pregnancy, Obstetrics in Family Medicine, 101-112.
5. Chin, T.L., MacGowan, A.P., Jacobson, S.K., and Donati, M. 2014. Viral infections in pregnancy: advice for healthcare workers, Journal of Hospital Infection, 87, 11-21.
6. Holmes, B. J. 2008. Communicating about emerging infectious disease: The importance of research, Health, Risk &
Society, 10, 4, 349–360.
7. Ebenezer, C., 2003. Usability evaluation of an NHS library website. Health Information & Libraries Journal, 20, 3, 134-142.
8. Usher, W., 2009. General practitioners’ understanding pertaining to reliability, interactive and usability components associated with health websites. Behaviour & Information Technology, 28, 1, 39-44.
9. Puspitasari, I. 2017. The impacts of consumer’s health topic familiarity in seeking health information online,
Proceedings - 2017 15th IEEE/ACIS International Conference on Software Engineering Research, Management and Applications, SERA 2017.
Theoretical framework
1. Conceptual thinking
In the Netherlands, the domain Centre for Infectious Disease Control (CIb) within RIVM coordinates the control of infectious diseases, including effective prevention, close vigilance and quick response in the event of an outbreak, and contributes to reducing health problems related to infectious diseases. The CIb advises the government and professionals on the control and prevention of infectious diseases. This includes clear and reliable communication for professionals and public, development of directives, and innovative research. Moreover, public communication belongs to one of the core tasks of the CIb. The messages and communication tools must align well with the needs of the target group(s). Besides transmitting information, the centre is exploring how they can incorporate the (information) needs of their users. A possible information path of reaching a target group is that the healthcare professional informs the patient, client or general public. In this specific case, the intended user of the CIb is the healthcare professional. However, other cases require that certain information reaches the public directly. Therefore, the information path of reaching the intended user is important for the delivery of information provided by CIb. The centre strives for optimal online presence (monitoring and webcare) and availability of information. In addition, the aim is to provide customized information for specific target groups. To effectively transfer the message of CIb, they progressively use serious games, apps, infographics, animations, and short videos on their website.
An important factor is that the online information provided by CIb (RIVM) reaches the end user in a productive manner. Proposed by Crucq and Knitel in their book “Conceptual Thinking” is that websites are an influenceable factor in the concept identity consisting of core values, communication goal, and mission [1]. The mission of RIVM is to work towards a healthy population in a sustainable, safe and healthy living environment. To contribute to this mission and the core tasks of the CIb, this study focuses on the website element of the information communication strategy. In Figure 1, the concept identity tree proposed by Crucq and Knitel is merged with the core tasks of the information strategy of the CIb.
Figure 1 Adjusted concept identity tree (design adapted and translated from Dutch) by Crucq and Knitel (2013) [1] merged with the information strategy of CIb.
A previous study concluded that midwives have (in general) insufficient knowledge regarding infectious diseases during pregnancy [2]. To enhance information communication in an effective way, the provided online information needs to be structured and presented in a way that healthcare professionals, such as midwives, can easily find relevant information regarding infectious diseases during pregnancy. Therefore, current literature was reviewed on how information is sufficiently and effectively communicated via a website. First, we searched for requirements of the available content. A study by Usher introduced a framework determining the definitions of relevant qualities regarding reliability, interactive, and usability components of health websites (see Table 1) [3]. Second, we focused on how users search for information online and how they react upon it. Kuhlthau proposed and developed a six-stage information search process (ISP) model, with each encompassing the user experience in the process of information seeking as a series of thoughts, feelings, and actions [4]. The model reveals a search process a person is following in the course of seeking information and is represented in Table 2. Third, insight was gained into the cognitive process when searching for information. Jaspers et al. presented a model generating the ongoing thought process during task performance (see Figure 2) [5]. Although these aspects are of influence on effective online knowledge transfer, there is no recent literature on requirements for informative health websites. The same applies for literature stating these aspects as converged aspects. Thus, additional research is needed on adequate online knowledge transfer regarding infectious diseases during pregnancy. Furthermore, the influence of design on information communication is researched.
Table 1 Defining qualities of each reliability, interactive and usability component by Usher [3]. Components Defining qualities
Reliability The quality and level of trustworthiness of information/material found on the website. Authority The extent to which material is the creation of a person or organisation that is recognised
as having definitive knowledge of a given subject area.
Accuracy The extent to which information is reliable and free from errors.
Objectivity The extent to which material expresses facts or information without distortion by personal feelings or other biases (sponsorship).
Currency The extent to which material can be identified as up to date.
Coverage The range of topics included in a work and the depth to which those topics are addressed. Intended
audience The group of people for whom material was created.
Confidentiality Confidentiality of data relating to individual patients and visitors to medical [health] website, including their identity, is respected by this website.
Justifiability Any claims relating to the benefits/performance of a specific treatment, commercial product or service will be supported by appropriate, balanced evidence.
Interactivity The quality and level of interactive components throughout a website.
Multimodality The level of such integration modalities as text, pictures, video, and various combinations, into a single modular unit.
Networkability The potential connections that computer users are able to make with other users and service providers.
Temporal
flexibility Asynchronous (email, chat rooms, discussion boards) or synchronous (interaction in real time) in nature, allowing users to interact with the content as the content is made available or to postpone the interaction desired.
Message-tailoring
capabilities Information can be crafted to suit preferences and characteristics of targeted individuals. Usability The level of ease of use, navigation and pleasantness throughout the website.
Health literacy Ease of text understanding by user, in terms of literacy levels and presentation. Navigation Ease of movement around interface and quickness of response.
Interface
engagement educational
The experience within the site encourages prolonged engagement within the domain.
Table 2 Model of the Information Search Process by Kuhlthau [4].
Initiation Selection Exploration Formulation Collection Presentation Assessment Feelings
(Affective)
Uncertainty Optimism Confusion Frustration Doubt Clarity Sense of direction/ Confidence Satisfaction or
Disappointment Sense of accomplishment Thoughts (Cognitive) vague focused increased interest Increased self-awareness Actions (Physical) seeking relevant
Figure 2 Model of the human cognitive system by Jaspers et al. [5].
2. User-centered design approach
In order for the website to be a successful tool that will be used in daily practice, it is essential to involve end users in the (re)design process to ensure that the tool is adapted to their needs [6]. Specifically, designing and developing websites based on the user’s information needs and to embody interaction design and usability principles is crucial for facilitating public health information [7]. User-centered design is one of the important factors for the success of website development or optimization and is therefore included in the standards of the International Organisation for Standardization (ISO) (ISO 13407:1999). For that reason, a mixed methods approach for a user-centered development process was considered as most suitable for redesigning the RIVM website providing information on infectious diseases during pregnancy. The user-centered design development lifecycle, described by Nielsen [8], is a cycle of iterative design activities based on the software development lifecycle [9,10]. The redesign process is tested as a new developing framework in a first case study for the RIVM website
Pregnancy and infectious diseases. For website evaluation, recommended is to perform a usability
study and include an information needs analysis as a first phase of the UCD cycle. In this manner, usability problems encountered during user interaction can be identified. However, usability studies do not offer insight into the content of a website. In this case study, the RIVM website already contains rich information regarding infectious diseases during pregnancy. In recent years, information is added on the website based on internal studies which includes the information needs of users. Therefore, chosen was to include a thematic content analysis in the framework to acknowledge existing information and to identify missing information, since the overview got lost due to continuous addition of new information. The thematic content analysis evaluates the information completeness and quality of the current available information on RIVM website. Consequently, a usability inspection method was executed for detecting usability problems by experts. Thereafter, usability testing with (potential) end users of the RIVM website was conducted. These three elements are of input for the optimization of the website in order to effectively align with the information communication strategy for infection prevention and control in the Netherlands. The three phases of the new framework are described in detail in the following paragraphs.
Figure 3 Applied UCD approach framework for evaluating complex health information websites.
2.1 Content analysis
Qualitative content analysis and thematic analysis are categorized under qualitative descriptive design and are used for analysis of textual data and elucidate themes [11]. The goal of content analysis is the systematic exploration of communicative material. The outcome of the analysis is categories describing the system. Commonly, the purpose of categories is to create a model, conceptual system or map [12]. Furthermore, content analysis can be used to develop an understanding of the meaning of communication [13]. Content analysis has been applied in the field of public health and nursing, since the multifaceted characteristics can be analysed [12]. An advantage of content analysis is that large numbers of textual data and different textual data sources can be dealt with. In this study, the theme development process defined by Vaismoradi et al. was applied (see Table 3) [14] combined with the coding process model defined by Thomas et al. [15].
Table 3 Phases and stages of theme development in qualitative content and thematic analysis by Vaismoradi et al. (2016) [5].
Phases Stages
Initialization • Reading transcriptions and highlighting meaning units • Coding and looking for abstractions in participant’s accounts • Writing reflective notes
Construction • Classifying
• Comparing • Labelling
• Translating & transliterating • Defining & describing Rectification • Immersion & distancing
• Relating themes to established knowledge • Stabilizing
Table 4 The coding process in inductive analysis by Thomas et al. (2006) [11]. Initial reading of
text data Identify specific text segments related to objectives
Label the segments of text to create categories
Reduce the overlap and redundancy among the categories Create a model incorporating most important categories Many pages of text Many segments of
text
30 to 40 categories 15 to 20 categories 3 to 8 categories
2.2 Usability evaluation
Usability is defined as the extent to which the product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [16,17]. Usability evaluation is a way of ensuring that interactive systems, such as websites, are adapted to users and their tasks [15]. Usability testing involves measuring the performance of users doing typical tasks [18-21]. The goal is to gain insight into the performance of a system to assess the usability. Previous works also mentioned that usability positively influenced the user satisfaction on mobile-based applications [22]. In mobile health applications, implementing adequate usability on its development is crucial to ensure the correct exchange of health messages [23].
There are multiple methods to evaluate systems on its usability, such as the heuristic evaluation (HE). A HE is an informal usability method in which experts evaluate whether user interface items of a system comply with a set of usability principles known as heuristics [8,18,19]. The aim of the HE is to detect serious usability problems that users might encounter. Nielsen developed the technique of heuristic evaluation and described ten heuristics that should be followed by good user interface design [8]. Shneiderman also described eight rules that all user interface designs should follow [24]. Based on the ten heuristics by Nielsen and the eight golden rules by Shneiderman, Zhang et al. stated fourteen heuristics [25]. The heuristics defined by Zhang et al. were applied in a study by Allen et al. for usability evaluation of paper-based webpages [26]. Chosen was the modified heuristics defined by Zhang et al. in this study. In addition, the usability checklist described by Pierotti were applied for usability evaluation [27]. This checklist includes 13 items, of which ten are the principles proposed by Nielsen [8]. Strengths of the HE method are that it is inexpensive, simple to conduct, and provides feedback promptly.
To supplement the limitation that usability problems found through HE are minor [28,29], think aloud analysis (TA) was performed in this study. The TA is a verbal report method and is originally from the field of cognitive psychology. During a TA session, participants interact with a system according to predefined tasks while they talk aloud their thoughts. Verbalizations are recorded and in combination with the screen recordings of the users’ website navigation patterns and their task performances, a deep insight can be gained into the system’s usability and the related usability problems [29]. For the TA, the model of verbal protocol analysis method proposed by Jaspers et al. is used as method [5]. The model consists of collecting TA protocols and analysing the protocols to obtain insight in the way participants perform certain tasks (see Figure 4).
Figure 4 Schematic overview of the verbal protocol analysis method by Jaspers (2004) [5].
HE has been applied in multiple studies to discover usability problems in healthcare websites [30-33], likewise in personal gadgets [34,35], medical devices, such as insulin pumps [36,37], and healthcare information systems, such as electronic health records [38-40]. These studies concluded that HE and usability testing are efficient, beneficial, and inexpensive. However, heuristic and usability evaluations have not been performed on health websites providing information on infectious diseases. The same applies for TA, since TA has been broadly applied in usability research. For example, for health websites on cancer [29,41,42], in the field of mental health [43,44], and for personal health records [45]. Yet, no studies using TA for analysis of websites providing information on infectious diseases (during pregnancy) have been published.
In addition to the HE and TA, an online focus group was used to evaluate the usability. The focus group was added, due to the types of participants in the HE and TA. Focus groups are frequently used as qualitative research to gain insight into the perceptions of purposely selected participants on a defined area of interest [46,47]. Focus groups have been used broadly in several disciplines including software engineering. This cost-effective and candid method is an effective way of evaluating the design of software products and webpages, since it provides quick information into participant’s behaviours, concerns, needs, and desires [46,48,49]. Focus groups can be used after formal usability studies as they provide discussion in a more relaxed setting than in one-on-one situations [48].
2.3 Optimization
Based on the findings of the previous phases in combination with design principles, a new (re)design of the website is proposed. Principles are general objectives guiding conceptual user interface (UI) decisions and reflect the knowledge around human perception, learning, and behaviour. Since, the main goal of the website is to inform users, these principles need to be honoured in order to effectively communicate information regarding infectious diseases during pregnancy. Usability principles and interaction design in a website is mostly conveyed via its interface design [50]. A proper interface enables users to perform their intended tasks effectively and efficiently. The heuristic evaluation provides the task-based component, while the think aloud evaluation and the online focus group provides insights into the component of user’s information needs.
References
1. Cruqc-Toffolo, G., and Knitel, S. 2013. Concept denken – Van slapend naar levend – Basisboek conceptontwikkeling.
2. Pereboom, M.T.R, Mannien, J., and van Almkerk, K.D.J. 2014. What information do Dutch midwives give clients about toxoplasmosis, listeriosis and cytomegalovirus prevention? An exploratory study of videotaped
consultations. Patient Education and Counseling, 96, 29-35.
3. Usher, W., 2009. General practitioners’ understanding pertaining to reliability, interactive and usability components associated with health websites. Behaviour & Information Technology, 28, 1, 39-44. 4. Kuhlthau, C. 1993. Seeking Meaning: A Process Approach to Library Information Services.
5. Jaspers, M.W.M., Steen, T., van den Bos, C., and Geenen, M. 2004. The think aloud method: a guide to user interface design. International Journal of Medical Informatics, 73, 781-795.
6. Jokela, T., Iivari, N., Matero, J., and Karukka, M. 2003. The standard of user-centered design and the standard definition of usability: analyzing ISO 13407 against ISO 9241-11. Proceedings of the Latin American conference on
Human-computer interaction.
7. Jaspers, M. W. M. 2009. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics, 78, 5, 340-353. 8. Nielsen, J., Mack, R.L., 1994. Usability Inspection Methods. New York: John Wiley & Sons.
9. Hartson, R., and Pyla, P.S. 2012. The UX Book: Process and guidelines for ensuring a quality user experience.
Elsevier.
10. Choi, J. and Bakken, S. 2010. Web-based education for low-literate parents in Neonatal Intensive Care Unit: development of a website and heuristic evaluation and usability testing, Int. J. Med. Inform., 79, 8, 565-575. 11. Forman, J. and Damschroder, L. 2008. Empirical Methods for Bioethics: A Primer In Jacoby L and Siminoff LA,
Advances In Bioethics, 11, 39-62.
12. Elo, S. and Kyngäs, H. 2008. The qualitative content analysis process. Journal of Advanced Nursing, 62, 1, 107-115. 13. Cavanagh S. 1997. Content analysis: concepts, methods and applications. Nurse Researcher 4, 5–16.
14. Dix, A., Finlay, J., Abowd, G.D., and Beale, R. 2004. Human-Computer Interaction. 3rd ed. England: Prentice Hall. 15. Bastien, J.M.C. 2010. Usability testing: a review of some methodological and technical aspects of the method.
Int J Med Inform, 79, 4, 18-23.
16. Fang Liu. 2008. Usability evaluation on websites, 2008 9th International Conference on Computer-Aided Industrial
Design and Conceptual Design, Kunming, 141-144.
17. Hasan, L., Morris, A., and Probets, S. 2012. A comparison of usability evaluation methods for evaluating e-commerce websites. Behaviour & Information Technology, 31, 7, 707–737.
18. Sharp, H., Rogers Y, Preece J, editors. Interaction Design: Beyond Human-Computer Interaction. 2nd. ed. New York, NY: John Wiley & Sons, Inc.; 2007.
19. Dix, A., Finlay, J., Abowd, G.D., and Beale, R. 2004. Human-Computer Interaction. 3rd ed. England: Prentice Hall. 20. Bastien, J.M.C. 2010. Usability testing: a review of some methodological and technical aspects of the method.
Int J Med Inform, 79, 4, 18-23.
21. Nielsen, J. 1993. Usability Engineering. San Diego, LA: Morgan Kaufman; 1993.
22. Kim, S., Lee, K.H., Hwang, H., and Yoo, S. 2015. Analysis of the factors influencing healthcare professionals’ adoption of mobile electronic medical record (EMR) using the unified theory of acceptance and use of technology (UTAUT) in a tertiary hospital, BMC Med. Inform. Decis. Mak., 16, 1, 12.
23. Georgsson, M. and Staggers, N. 2016. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics, J. Am. Med. Informatics
Assoc., 23, 1, 5-11.
24. Shneiderman, B. 1998. Designing the user interface (3rd ed.), Addison-Wesley, Reading, MA (1998)
25. Zhang, J., Johnson, T. R., Patel, V. L., Paige, D. L., and Kubose, T. 2003. Using usability heuristics to evaluate patient safety of medical devices. Journal of Biomedical Informatics, 36, 1-2, 23–30.
26. Allen, M., Currie, L. M., Bakken, S., Patel, V. L., and Cimino, J. J. 2006. Heuristic evaluation of paper-based Web pages: A simplified inspection usability methodology. Journal of Biomedical Informatics, 39, 4, 412-423. 27. Pierotti, D. 1995, Heuristic Evaluation – A System Checklist. Usability Analysis & Design. Xerox Corporation. 28. Nielsen J. Characteristics of usability problems found by heuristic evaluation
29. Peute, L.W., Knijnenburg, S.L., Kremer, L.C., and Jaspers, M.W.M. 2015. A Concise and Practical Framework for the Development and Usability Evaluation of Patient Information Websites. Applied Clinical Informatics, 6, 2, 383-399. 30. Cogdill, K. 1999. MEDLINEplus Interface Evaluation: Final Report. College Park, MD: College of Information Studies,
University of Maryland.
31. Nahm, E.S., Preece, J., Resnick, B., and Mills, M.E. 2004. Usability of health web sites for older adults: A preliminary study. Comput Inform Nurs, 22, 6, 326–334.
32. Britto, Jimison, Munafo, Wissman, Rogers, Hersh. 2009. Usability testing finds new problems for novice users of pediatric portals. Journal of the American Medical Informatics Association.
33. Lai, T.Y. 2007. Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing. Int J Med Inform.
34. Scandurra, I., Hgglund, M., Koch, S., and Lind, M. 2008. Usability laboratory test of a novel mobile homecare application with experienced home help service staff. The Open Medical Informatics Journal, 2, 117-128. 35. Treadwell, I. 2006. The usability of personal digital assistants (PDAs) for assessment of practical
performance. Med Educ., 40, 9, 855.
36. Zhang, J., Johnson, T.R., Patel, V.L., Paige, D.L., and Kubose, T.J. 2003. Using usability heuristics to evaluate patient safety of medical devices. Biomed Inform., 36, 1-2, 23-30.
37. Graham, M.J., Kubose, T.K., Jordan, D., Zhang, J., Johnson, T.R., and Patel, V.L. 2004. Heuristic evaluation of infusion pumps: implications for patient safety in Intensive Care Units. Int J Med Inform., 73, 11-12, 771-779. 38. Bright, T., Bakken, S., Johnson, S. 2006. Heuristic evaluation of eNote: An electronic notes system. AMIA. Annual
Symposium proceedings, 864.
39. Tang, Z., Johnson, T., Tindall, S., Zhang, J. 2006. Applying heuristic evaluation to improve the usability of a telemedicine system. Telemedicine Journal and E-health, 12, 1, 24.
40. Hyun, S., Johnson, S.B., Stetson, P.D., and Bakken, S. 2009. Development and evaluation of nursing user interface screens using multiple methods. J Biomed Inform., 42, 6, 1004-1012.
41. Bolle, S. Romijn, G., Smets, E.M.A., Loos, E.F., Kunneman, M., and van Weert, J.C.M. 2016. Works citing Older Cancer Patients’ User Experiences With Web-Based Health Information Tools: A Think-Aloud Study, J Med Internet
Res, 18, 7, 208
42. Smith, S. G., Vart, G., Wolf, M. S., Obichere, A., Baker, H. J., Raine, R., … von Wagner, C. 2013. How do people interpret information about colorectal cancer screening: observations from a think-aloud study. Health
Expectations, 18, 5, 703–714.
43. MacNeela, P., Clinton, G., Place, C., Scott, A., Treacy, P., Hyde, A., and Dowd, H. 2010. Psychosocial care in mental health nursing: a think aloud study. Journal of Advanced Nursing, 66, 6, 1297–1307.
44. Clarke, D. E., Boyce-Gaudreau, K., Sanderson, A., and Baker, J. A. 2015. ED Triage Decision-Making With Mental Health Presentations: A “Think Aloud” Study. Journal of Emergency Nursing, 41, 6, 496-502.
45. Segall, N., Saville, J.G., L'Engle, P., Carlson, B., Wright, M.C., Schulman, K., and Tcheng, J.E. 2011. Usability evaluation of a personal health record. AMIA Annu Symp Proc., 1233-1242.
46. Nyumba, T.O., Wilson, K., Derrick, C.J., and Mukherjee, N. 2018. The use of focus group discussion methodology: Insights from two decades of application in conservation. Methods Ecol. Evol. 9, 1, 20–32. 47. Calle-Jimenez T. et al. 2020 Improving Usability with Think Aloud and Focus Group Methods. A Case Study:
An Intelligent Police Patrolling System (I-Pat). Nunes I. (eds) Advances in Human Factors and Systems
Interaction. AHFE 2019. Advances in Intelligent Systems and Computing, 959.
48. Crowley, G.R., Leffel, D., Ramirez, J.L., and Armstrong, T.S., 2003. User perceptions of the library’s Web pages: A focus group study at Texas A&M University, Journal of Academic Librarianship, 28,4, 205-210. 49. Yen, P. Y., and Bakken, S. 2009. A comparison of usability evaluation methods: heuristic evaluation versus
end-user think-aloud protocol - an example from a web-based communication tool for nurse scheduling. AMIA Annual
Symposium proceedings. AMIA Symposium, 714–718.
50. Puspitasari, I., Indah Cahyani, D., and Taufik. 2018. A User-Centered Design for Redesigning E-Government Website in Public Health Sector. 2018 International Seminar on Application for Technology of Information and
Study I Quality assessment of a health information website
regarding infectious diseases during pregnancy: A thematic content
analysis
Abstract
Emerging and re-emerging infectious diseases are a significant public health concern, specifically for pregnant women, since infectious diseases can cause serious complications when acquired during pregnancy. In recent years, the availability of online information regarding infectious diseases has increased. However, health information websites are lacking coherence and becoming increasingly complex due to the frequent addition of new information. Therefore, the aim of this study was to analyse the online content concerning infectious diseases during pregnancy provided by the RIVM, including navigational structures on its completeness and information quality. A thematic content analysis was performed on 28 webpages providing information on infectious diseases during pregnancy and resulted in the identification of 13 categories. These categories were determined in order to assess the quality and completeness of information provided on the website based on the quality dimension framework. The results indicated that multiple webpages regarding a specific infectious disease during pregnancy are incomplete in terms of the quality driver ‘content’. Furthermore, in this study the information gaps were identified in order for the website to fulfil its completeness.
WHAT WAS KNOWN? WHAT DOES THIS CHAPTER ADD?
• Online information provision regarding infectious diseases is part of the information strategies for infection prevention and control.
• Currently, it is unknown if the provided information on the website provided by RIVM meets the information needs of users due to its high complexity. • Completeness of information is an
important factor in determining the quality of a website and therefore the trust and perceived usefulness. • A previous study established a
framework for completeness of information to determine the quality of a health information website.
• The quality of the current website Pregnancy and infectious diseases was inadequate, due to substantial amount of missing or irrelevant information (information gaps) in the following categories: breastfeeding, immunity, symptoms, transmission, vaccination, and work policy. • Sufficiently presented categories on the website
are risks, measures, diagnostics, and treatment. • Numerous absent categories for bladder
infection, cold sores, flu, genital herpes, group A streptococci, leptospirosis, and malaria, while chicken pox, hepatitis B, measles, and Q fever were adequately presented.
• The quality drivers of the information completeness framework were largely met; however, content was substandard, due to the identified information gaps.
• In order for the information on the website to be complete, identified information gaps must be evaluated and fulfilled if needed, and further research on usability is required.
1. Introduction
Emerging and re-emerging infectious diseases are a significant public health concern [1]. Specifically, infectious diseases are of concern when it involves pregnant women. Infectious agents that cause mild-to-moderate diseases in healthy adults or children, can cause serious complications for both mother and unborn child if acquired during pregnancy [2-5]. Therefore, early detection and immediate response is crucial for disease control [1]. To achieve this, one of the aspects of the information strategy for prevention of infectious diseases of the National Institute for Public Health and Environment (RIVM) is facilitating online information for both healthcare professionals and public, since they are known users of the internet for access to relevant health information [6,7]. Moreover, information communication technologies, such as websites, improve public health in terms of increasing life expectancy and decreasing infant mortality [8].
Due to the growing interest in using the internet for seeking health information, the quality of online health information was questioned [9-13]. Multiple studies indicate that the quality of health information websites is vital [14-16]. A study by Fennel et al. defined the quality of health websites as a well-designed website that leads users to better recall and recognize a positive attitude toward a website with relevant, readable, and effective information [17]. However, previous studies have found a high variance and dynamic in the quality of content [18-22]. Furthermore, various users became aware that not all online health information is reliable [23]. Consequently, the quality of health information websites influences the trust of users and their intention to use the website.
The availability of online information regarding health topics, among infectious diseases, has increased in recent years. Since new information regarding emerging and re-emerging infectious diseases during pregnancy is gathered continuously, website providers proceed adding information on their websites. However, the frequent extension of information develops a higher complexity of the website and therefore the findability of information evolves to be more difficult. Additionally, the quality of this online health information remains unknown.
This study is the first phase of the new developing framework and focuses on the quality of the RIVM website. The aim of this study was to analyse the online content and quality concerning infectious diseases during pregnancy provided by the RIVM, including navigational structures. In this manner, an overview can be created of the availability and continuity of information and to identify possible improvement areas for effective retrieval of information. This way, health complications for both mother and (un)born child, such as premature birth, may be prevented.
2. Methods
Qualitative content analysis and thematic analysis are categorized under qualitative descriptive design and are used for analysis of textual data and to elucidate themes [24]. In this study, the two types of analyses were combined. Firstly, an overview of the (navigational) structure of the RIVM website
Pregnancy and infectious diseases was created for content selection. The overview provides insight
into how users browse over the website and how the (related) pages of the website are structured. Secondly, content was analysed according to thematic analysis defined by Vaismoradi et al. and Thomas [25,26]. Furthermore, the health information quality concept framework with quality dimensions and associated drivers developed by Tao et al. was applied in this study [27].
The website Pregnancy and infectious diseases maintained and drafted by the RIVM was selected as source document. This website consists of 57 referenced webpages and provides general information regarding infectious diseases during pregnancy and links to illness specific information, amongst other relevant topics such as pregnancy and work. The illness specific webpages (28) were selected for
thematic content analysis, since these webpages are centred around the core purpose for visiting the website. For example, the webpage Measles and pregnancy provides information regarding health risks, advice for pregnant women, and advice for healthcare professionals. Excluded were Coronavirus, whooping cough, and Zika, due to the deviating information provided and its structure. However, these three topics are included in the optimization phase. Included were the following illness specific topics: bladder infection, Candida, Cytomegalovirus (CMV), shingles (herpes zoster), influenza, group A streptococci, group B streptococci, hepatitis A, hepatitis B, hepatitis C, impetigo, cold sores, leptospirosis, Listeria, Lyme disease, malaria, Methicillin-resistant Staphylococcus aureus (MRSA), Q fever, rubella, Scarlet fever, Toxoplasma, tuberculosis, fifth disease (parvovirus), chicken pox, chlamydia, gonorrhoea, genital herpes, genital warts caused by human papillomavirus (HPV), and syphilis.
The coding process was based on The Coding Process in Inductive Analysis adapted by Thomas [26]. General inductive approach for raw data analysis was adopted. The text provided on the selected webpages was copied and put into an excel sheet, whereas each sentence was one cell in the excel file. This way, codes could be matched with one sentence of a specific infectious disease. No categories or labels were developed before coding. Codes were directly derived from text data. This way, frequently occurring themes and categories were identified. Theme development was based on four phases: initialization, construction, rectification, and finalization proposed by Vaismoradi et al. [25]. These phases were used as columns in the excel file. All webpages were coded by one researcher (LR); four webpages were coded by two other researchers (FM, LP) in order to minimize researcher bias. The results were compared, and consensus was reached by discussion. After finalization of the categories, a model was created to link the relationships between categories. Besides the relationships between categories, the number of occurrences of a category in relation to all categories was analysed and resulted in coverage percentages.
A part of the health information quality concept framework with quality dimensions and associated drivers developed by Tao et al. was applied in this study [27]. This framework includes five dimensions: completeness of information, understandability of information, relevance of information, depth of information, and accuracy of information. These five dimensions are supported by the following main driver themes: content, design, links, consumer resources, search functionality, supporting references, user focus, content frequently asked questions (FAQ), open access, policy statements, and site performance. Selected for this study was the dimension of completeness of information, including its associated drivers (see Figure 1).
Figure 1 Quality driver theme and detailed drives for completeness of information on health websites [27].
3. Results
3.1 Overview navigational structure
The main webpage Pregnancy and infectious diseases (source document, see Figure 2A) contains 62 direct links, in which headers (including menus) and footers are excluded. Of these 62 links, fourteen links are grouped among “other information sources” and were excluded from this study. Of the remaining 48 links, five links are mentioned twice, resulting in 43 unique links the main webpage is referring to. In Figure 3, an overview is created of the navigational structure of the website. The main webpage provides links to 37 internal webpages. An internal webpage is defined as a webpage hosted by RIVM in the same design and layout as the main webpage (see Figure 2B). Three links are referring to indirect internal webpages within the RIVM. An indirect internal webpage implies that the webpage is hosted by RIVM, however, in another design and layout, due to a different focus of the website (see Figure 2C). The main webpage links to two external information sources, namely vSKBN, the association for city and children’s farm in the Netherlands and the nutrition centre (see Figure 2D). The last link is a dead link, since an error message is provided stating that the webpage is not found.
Figure 3 Overview of webpages the main page Pregnancy and infectious diseases directly refers to, including legend.
3.2 Thematic content analysis
In total, the content of 28 webpages regarding specific information on infectious diseases during pregnancy provided by the RIVM was analysed and resulted in 13 categories. Descriptions of categories are shown in Table 1, including reference code and coverage percentage. Most webpages are structured with firstly stating general information followed by the risks of an infectious diseases during pregnancy. Subsequently, information regarding prevention, immunity, transmission and symptoms of an infectious diseases is cited. Advice on what to do after having been in contact with an infectious disease is discussed. After the advice is presented, information for professionals is provided regarding diagnostics, screening, vaccination, and treatment. Lastly, information on breastfeeding and work policy is presented. In Figure 4, an overall heatmap of information fulfilment of the categories is presented. Based on the heatmap, the coverage percentage was calculated. In case of a missing category (group 1: light grey), the infectious disease was included as 0, an incomplete category (group 2: medium grey) was included as 0.5, and a covered category (group 3: dark grey) as 1.0 in the total amount. The total amount was divided by the total number of included infectious diseases (28). The 13 categories derived from the webpage content transcripts are presented below in the order of coverage percentage (low to high). A coverage percentage under 40% was deemed as low, between 40% and 60% was deemed as medium, and over 60% was deemed as high.
Table 1 General category description of emergent coding themes.
Code Category Description Information
coverage in %
BREA Breastfeeding Breastfeeding advice when the mother is infected 36%
DIAG Diagnostics Diagnostics of an infectious disease 70%
IMMU Immunity Immunity to infectious disease 29%
INFO Information General information on infectious diseases including background information, causing agent of the infectious diseases, and links to relevant information sources.
54% MEAS Measures Measures after contact with an infectious disease and measures
to perform when a pregnant woman is infected
64%
PREV Prevention Advice on how infectious diseases can be prevented 52%
RISKS Risks Risks of infectious diseases for pregnant woman, unborn child
and newborn child 75%
SCREE Screening Advice on screening of a pregnant woman for infectious diseases 45%
SYMP Symptoms Symptoms of infectious disease 25%
TRANSM Transmission Transmission of infectious diseases from mother to child and
what the chances and risks are of transmission 32%
TREATM Treatment Treatment of infectious diseases 70%
VACC Vaccination Vaccination advice for pregnant women 32%
WORK Work policy Work policy regarding infectious diseases for a pregnant
