Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=rics20
ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/rics20
Development and validation of the Internet of
Things Skills Scale (IoTSS)
Alexander J. A. M. van Deursen, Alex van der Zeeuw, Pia de Boer, Giedo
Jansen & Thomas van Rompay
To cite this article: Alexander J. A. M. van Deursen, Alex van der Zeeuw, Pia de Boer, Giedo Jansen & Thomas van Rompay (2021): Development and validation of the Internet of Things Skills Scale (IoTSS), Information, Communication & Society, DOI: 10.1080/1369118X.2021.1900320
To link to this article: https://doi.org/10.1080/1369118X.2021.1900320
© 2021 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
View supplementary material
Published online: 25 Mar 2021.
Submit your article to this journal
Article views: 249
View related articles
Development and validation of the Internet of Things Skills
Scale (IoTSS)
Alexander J. A. M. van Deursen , Alex van der Zeeuw, Pia de Boer, Giedo Jansen and
Thomas van Rompay
Department of Communication Science, University of Twente, Enschede, the Netherlands
ABSTRACT
The Internet of Things (IoT) is expected to have a massive impact on people’s lives. However, the system’s complexity is also likely to make it an important topic of investigation in digital inequality research. Those who have the skills to use the IoT to its full potential and gain maximal benefits have a technology at hand that will have the power to increase their (already privileged) positions. Prerequisites for and impacts of user (consumer) engagement with the Internet of Things are becoming increasingly recognized as an important study area. To support related research and policy development, there is a need for more theoretically informed, reliable, and valid instruments that are able to measure what people do and gain from the IoT. In the current contribution, we focus on a key component in digital inclusion debates: digital skills. The development of the IoT Skills Scale (IoTSS) started with examining existing digital skills theory which led to afirst instrument. We used a threefold approach to test the validity and reliability of the latent skill constructs and the corresponding items: cognitive interviews, followed by afirst survey of IoT skills. During the final step, we examined the consistency of the IoT skills scales and their characteristics when measured in second survey among a representative sample survey of Dutch Internet users. The result is a theoretical and empirical consistent framework consisting of two types of IoT skills: operational and data IoT skills and strategic IoT skills.
ARTICLE HISTORY Received 28 May 2020 Accepted 1 March 2021 KEYWORDS
Internet of things; digital skills; digital inequality; survey instrument
Introduction
The Internet of Things (IoT) can be defined as a system that contains ubiquitous every-day‘smart’ objects accessible through the Internet and equipped with (1) sensing (e.g., sound, movement, temperature), storing, and processing capabilities, and (2) identifying and networking capabilities (Van Deursen & Mossberger,2018). Examples of these smart objects are wearables and medical devices (from glasses,fitness trackers, to baby socks), smart home appliances, (e.g., digital assistants, smart thermostats, lightning, security,
© 2021 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way.
CONTACT Alexander J. A. M. van Deursen a.j.a.m.vandeursen@utwente.nl Faculty of Behavior, Management and Social Sciences, University of Twente, Enschede, Netherlands
fridges, TV), smart energy networks, or smart traffic systems. Most frequently used and popular consumer IoT devices are wearables for health and lifestyle purposes, and devices for energy monitoring (Sri et al., 2016; Van Deursen et al., 2019). With the IoT, the Internet used for connecting users has shifted to an Internet used for connecting physical devices that communicate with each other and/or with humans to offer a given service. The devices and persons involved create a complex omnipresent networked sys-tem in which devices might go unnoticed and make autonomous decisions. The links between the actors and purposes are concealed and for ordinary users it is difficult to
comprehend how dataflow is handled across IoT devices and who controls the data.
In addition to being complex, the potential impact of the IoT on people’s lives is direct (e.g., it affects hospital treatment, medication intake, insurance allowance, or energy con-sumption) and it is difficult to opt out of these services (e.g., utility companies roll out smart energy meters to entire areas). Because of the potential impact, the IoT should be recognized as a technology that requires attention in the realm of digital inequality (Bauer, 2018; Ransbotham et al., 2016; Riggins & Wamba, 2015; Van Deursen et al.,
2019; Van Deursen & Mossberger, 2018). Despite the importance to understand who
will be able to take part in an IoT-enabled society, behavioral factors necessary to under-stand the impact of the IoT have been largely ignored, and we know little about the key driver of (optimal) IoT usage: user skills (Van Deursen & Mossberger,2018). Because of the complexity and the potential impact, comparative advantages are likely to increase, and smaller groups of people will be able to benefit (Van Deursen et al.,2019; Van Deur-sen & Mossberger,2018). Furthermore, people with insufficient skills will be more likely
to be disadvantaged, for example, when personal data are used by people with malicious intentions. Paying attention to what the IoT exactly demands of its users is therefore necessary.
From Internet research, we know that digital skills are the main requirement for con-ducting capital-enhancing activities online (e.g., Helpser & Eynon,2013; Litt2013), for gaining positive outcomes from Internet use (e.g., Van Deursen et al.,2017), and for full participation in the Information Age (e.g., Ananiadou & Claro,2009). While research on Internet skills has proliferated in recent years, it is still unclear what types of skills are required by the general IoT user. There is a strong need for an operational framework that can support future research. The purpose of this article is to provide a theoretical and methodological contribution to IoT skills research. To develop focused strategies to improve IoT skills among those who are most in need, thefirst goal is to obtain a better understanding of which user skills are required for IoT usage. This involves a conceptual justification and inclusion of the core skills required for IoT use. The second goal is to develop a set of reliable measures for use in research, practice, and policy impact evaluation.
To achieve these objectives, we started with a literature review to develop a conceptual
IoT skills framework and propose afirst associated instrument (Section 2). Second, we
conducted cognitive interviews to test for face and content validity of the proposed instrument by reviewing clarity, readability, relevance, and credibility (Section 3).
Third, a first online survey of the proposed instrument was conducted in a general
IoT user sample to explore the underlying factor structure of the observed skill items and assess construct validity (Section 4). Finally, a second online survey was conducted to confirm the factor structure of the instrument (Section 5). The cognitive interviews
and two surveys were conducted in the Netherlands, a country where the Internet con-nection rate is saturated (98% in 2020) and where IoT is available for the general public, especially in relation to smart home and health appliances.
Initial instrument development
Conceptualization of IoT skills
To develop a framework of IoT skills, we started out from what we have learned from Internet skills research. Here, several concepts have emerged, mostly centered around operational or technical skills (e.g., Litt, 2013; Van Dijk & Van Deursen, 2014; Van Laar et al.,2017), information skills (e.g., Ferrari,2013; Litt,2013; Van Dijk & Van Deur-sen,2014), communication skills (e.g., Jenkins et al.,2009; Van Laar et al.,2018; Van Dijk & Van Deursen,2014), content creation skills (e.g., Ferrari,2013; Van Dijk & Van Deur-sen,2014), privacy- and security-related skills (e.g., Büchi et al.,2017; Hargittai & Litt,
2013), and strategic skills (e.g., Ferrari, 2013; Martin & Grudziecki,2006; Van Dijk & Van Deursen,2014). The available measures for Internet skills typically face challenges ranging from problems of incompleteness and oversimplification (for example, a strong focus on technicalities as opposed to a broad range of skills), conceptual ambiguity (for example, when skills are put in par with usage), or the use of self-reports that easily lead to over- or underrating of skill levels (Merritt et al.,2005). To overcome such challenges, Van Deursen et al. (2016) proposed the Internet Skills Scale (ISS), an elaborate concep-tualization of Internet skills that includes a set of reliable measures for use in research, practice, and policy impact evaluation. Departing from the ISS and other conceptualiz-ations, we started our quest for the Internet of Things Skills Scale (IoTSS) with a
differ-entiation betweenfive core skills that should be considered important to the IoT user
population (i.e., gender neutral, not specific to subgroups): operational, information, communication, privacy, and strategic IoT skills. For each of thesefive skills, we adapted and derived items from previous research and added newly created items. We sought to include independent items with varying levels of difficulty.
Operational IoT skills
In general, operational skills are considered the starting point for using technology and refer to a set of technical competencies (Van Deursen et al.,2016; Van Dijk & Van Deur-sen,2014). Operational skills are fundamental in relation to the Internet, as without these skills, Internet use is not possible, and one will not come to perform actions that involve other more advanced skills, such as information- or communication-related Internet skills (Van Deursen et al.,2011). However, for the IoT system, operational skills might be less important as a large part of the IoT works behind the screen and autonomously (Van Deursen & Mossberger,2018). The IoT system has embedded the detection, sto-rage, processing, identification, and networking capabilities in ubiquitous ‘everyday’ devices and is increasingly integrated into everyday life. Operational skills will be needed less frequently as the system reduces the number of decision points where operational skills are applied. While previous technologies typically required a fully aware user to operate a device, in the IoT system people are relatively passive and less aware of what
wearable automatically results in a large collection of physiological data and follow-up recommendations. Likewise, a smart thermostat will turn devices off automatically when everybody left the house. However, operational skills are likely to play an important role in the initial setup when new devices are installed and connected to a network (Van Der Zeeuw et al.,2020). Setting up devices and services while integrating them into an operating system requires considerable effort.
Information IoT skills
The IoT is prone to deceptive ease as it creates an abstract and autonomous system that is quite different from traditional media characterized by user-device interactions. As a result, information skills are considered less prominent in comparison to the Internet (where they are applied in search engines or browsing). Traditionally, information skill studies consider a sequence: choosing a search engine, defining keywords, selecting information, and evaluating information (Ala-Mutka,2011; Eshet,2004; Ferrari,2013). In the IoT system, information skills can be linked directly to the data that the devices collect. Think of data concerning one’s energy consumption, activity, or medical con-ditions. Users will increasingly make use of this vast and detailed information generated by IoT devices and need the skills to interpret it, to present it in an understandable format (for example, by visualizing data infigures), or to compare the collected data with earlier data (for example, to see what progress has been made and whether adjustments are required). Furthermore, the collected data should be evaluated based on criteria such as reliability, validity, preciseness, completeness, and accuracy.
Communication IoT skills
Digital inequality studies that focus on‘traditional’ Internet skills mostly link communi-cation skills to the use of social media. In this respect, communicommuni-cation skills involve encoding and decoding online messages, managing online contacts, online profiling, and collaboration between Internet users (Ferrari, 2013; Hobbs, 2010; Jenkins et al.,
2009; Van Dijk & Van Deursen, 2014). In the IoT, communication skills are mostly prevalent in relation to orientation to other parties, such as friends or companies. For optimal IoT performance, users need to be able to share and compare their own data (determined while applying information skills) with the performance of other IoT users. But also, they need to understand how devices communicate with other devices and people and vice versa, and what the consequences are of certain behaviors or expressions towards an IoT device to which others are also connected.
Privacy IoT skills
The orientation towards others in communication skills is closely linked to privacy-related skills (boyd & Hargittai,2010; Büchi et al.,2017; Hargittai & Litt,2013). The latter involves understanding what information is being collected and who will have access to it. While in prior technologies a fully aware user typically operates a device to achieve a certain goal, in the IoT humans are mostly passive and unaware what is happening. An IoT system typically collects a large amount of sensitive personal data, which calls for enabling users to protect themselves from potential harm. Privacy is increasingly at risk with automatic communication of IoT objects. Because IoT works in the back-ground, the risks associated with IoT usage are often unclear to users. Furthermore,
the collection, analysis, and use of collected data is often not transparent to users, making it more difficult to make decisions about whether or not to use a smart device. It will be difficult for regular users to understand whether security measures are reliable and ade-quate. Privacy IoT skills are required to decide how data that potentially reveals when a person is home, how often they cook, clean, shower, watch television, or use exercise equipment, is used. Such intimate information could be of interest to insurance compa-nies, employers, creditors, and law enforcement. Emerging literature on the IoT has identified privacy and security issues as being among key inhibitors of the widespread acceptance of the IoT (Fowler et al.,2013; Hsinchun et al.,2012). Besides skills for under-standing how data will be used, recent literature highlights the importance of algorithmic skills. As IoT developers are not always open about the details of the algorithms their devices apply, for users it is difficult to know for certain how and why particular outputs or outcomes are produced (Klawitter & Hargittai,2018; Pasquale,2015; Cotter & Reis-dorf,2020).
Strategic IoT skills
Ferrari (2013), Martin and Grudziecki (2006), and Van Dijk and Van Deursen (2014) consider strategic skills as the means to reach a particular goal by one’s own initiative.
Van Dijk and Van Deursen adapted a process proposed by Miller (2006) that consists
of four distinct steps in making effective use of technology: Orienting goals, taking
required actions, making decisions, and implementing decisions. These steps result in benefiting from technology. As in the ubiquitous IoT system devices go unnoticed and make autonomous decisions, the number of decision moments is reduced. While for some this might support the decision-making process, strategic IoT skills are required to take full advantage of the IoT and reach one’s personal goals (e.g., in health behavior or energy use) or overcome encountered problems. Users need strategic IoT skills to translate their preferences and needs in making the IoT-system reason in a distributed cooperative way so correct decisions concerning the resources under control are made (e.g., switch off lighting, heating).
Scales used to measure IoT skills
Despite the needed attention on IoT skills, to the best of our knowledge, there have been only few earlier attempts to measure IoT skills (see De Boer et al.,2019; Van Deursen et al.,2019). The current contribution is more sophisticated and should be considered
a first exploration towards an IoT skills scale. In the two earlier investigations, we
used a short scale adapted from the Internet Skills Scale (ISS) (Van Deursen et al.,
2016). However, we were unable to identify different skill dimensions that the guiding
skill theory suggests and IoT skills were used as a unidimensional construct. In the cur-rent contribution, we started with a larger variety of items linked to the diffecur-rent concep-tual IoT skills. We followed the ISS approach and used a Likert-type format to allow
participants response flexibility. The response items focused on truth claims (‘not at
all true of me,’ ‘not very true of me,’ ‘neither true nor untrue of me,’ ‘mostly true of me,’ and ‘very true of me’), as research has showed that the wording in this type of scale invites a neutral and objective response from participants, especially compared to
et al.,2016). Such self-reports may be more reflective of confidence rather than skill and might lead individuals from advantaged backgrounds to overrate and individuals from more disadvantaged backgrounds underrate their own skill levels (Hinostroza et al.,
2015; Huang et al., 2017; Shank & Cotten, 2014). Furthermore, the truth claims scale encourages respondents to reflect on themselves, rather than using terms that evoke com-parisons with others (e.g.,‘expert’). In the latter case, items would be dependent on the context the participant is situated in, who they compare themselves to, or who they are surrounded by (Van Deursen et al.,2016).
Cognitive interviews
The second step of the process involved cognitive interviews. Cognitive interviewing is a
common method for improving instrument design by assessing respondents’
under-standing of the survey’s items (Knafl et al.,2007). Cognitive interviews serve an explora-tory function by explaining people’s responses. Furthermore, they help to identify which items may be omitted or represent incomplete or misleading views (Desimone & Le Floch, 2004). In line with this technique, a trained interviewer askedfive participants whether they understood the items (did the intended meaning align with the meaning as described in their own words), what they think the item meant, found the item rel-evant, could provide an example and would be able to answer the questions with the pro-vided answer scale. Notes were made about the responses to the items. Our aim was to understand whether what we intended an item to mean corresponded with the interpret-ation of the participant (content validity). Thefive participants were recruited through convenience sampling with a focus on different gender, age, and educational level. Fur-thermore, we ensured that the participants used at least one smart device. SeeTable 1for an overview of the sample. The interviews took place in January 2020 and on average took 38 min (min. 25 and max. 59 min).
The feedback from the interviews resulted in the adjustment of several items as pro-blems in the conveyance of meaning and understanding of these items appeared. For
example,‘I know how to visualize the data collected by my smart devices’ was changed
as there were comprehension problems with the word‘visualization.’ Two lower
edu-cated respondents did not understand what was meant with visualization. The item was changed to‘I know how to clearly present data collected by my smart devices.’ Fur-thermore, we decided to remove some items. For example, three interviewees remarked that the item‘I know how to share data that my smart devices collected online’ is unclear. Ambiguousness appeared as two interviewees thought that‘online’ referred to where the data was collected, while the three others understood it as the place where it is shared. In the case of the latter, sharing data online seems to be a more substantial part of Internet skills than it is for IoT skills.
Table 1.Gender, age, and education of the cognitive interview sample.
Participant Gender Age Education
1 M 28 High
2 M 49 Low
3 F 26 High
4 F 65 High
Based on the cognitive interviews, some items were adjusted or excluded as they showed confusion and misinterpretation. Before starting the online pilot surveys, we ensured that all problems were corrected (although corrected items were not checked with the interviewees).Appendix Adisplays all the skill items as adjusted after conduct-ing the cognitive interviews. This list of items was used in the next step.
Survey 1 results
Sample
The third step of developing the instrument was testing the instrument among Dutch IoT users. Respondents were recruited using the survey respondent platform of PanelClix, a market research company with a representative panel sample in the Netherlands. Poten-tial panel members were chosen via a statistically valid sampling method. PanelClix applies targeted automatic invitations where participants are sent invitations depending on the sampling and criteria of the study. Potential respondents were sent one email invite. Responses were collected through Qualtrics and answers could not be linked to individual information. PanelClix pays their members a small incentive of a few cents for every survey question they answer. The survey was conducted in March 2020.
The aim of thisfirst survey was to test the reliability of the constructed scales. The tar-get sample was set at 150–200 respondents including differences in gender, age, and edu-cation, characteristics that are likely to make a difference in how individuals interpret and engage with IoT skills measures. Of the 1,123 respondents who participated in the pilot,
178 used at least one smart device and were included in thefinal sample. Used devices
were smart home appliances, wearables and activity trackers, and smart assistants.
Table 2summarizes the demographic characteristics.
Exploratory factor analysis
First, we ran an analysis for means, standard deviations, skewness, and kurtosis of the 36
candidate IoTSS items (Appendix A) that resulted from the cognitive interviews. As
skewness and kurtosis values ranged from−1.07 to −0.29 and −0.72 to 0.57 respectively, all 36 items were deemed good. They were then subjected to exploratory factor analysis using squared multiple correlations for prior communalities. The method of maximum likelihood extraction was used, followed by varimax rotation to allow for correlation Table 2.Demographic profile of the Dutch IoT user sample.
N % Gender Male 109 61.0 Female 69 39.0 Age 16–30 51 28.7 31–40 47 26.4 41–50 43 24.2 50+ 37 20.9 Education level Low 94 52.8 High 84 47.2
between the factors and to obtain an interpretable structure. No restriction was applied to the number of factors to be estimated, and the maximum likelihood method was used to make use of goodness offit indices not available with other extraction methods (Fabrigar et al.,1999). We sought optimal weightings of the measured skill factors so that the list of items could be reduced to general summary scores with maximal variability and reliability. Each item was to be loaded highly on as few factors as possible. In the case of secondary loadings, we checked for substantive interpretations. Items that loaded on a different factor than we expected were deleted. We based the factor solutions on the eigenvalues (>1), the percentage of the variance accounted for by the factors, and the cohesiveness of the items within the identified skill factors. In this exploratory phase, factor loadings exceeding .50 were considered significant for inclusion.
In the initial run, sixteen items either failed to load substantially on one factor (i.e., factor loading less than .40), or loaded strongly on two or more factors. We deleted these items and attempted to derive a solution based on the remaining 20 items. The second run resulted in a two-factor solution based on an examination of the scree plot and preliminary eigenvalues >1.0. However, six items loaded high on both factors and were excluded. In the third run, an acceptable two-factor solution emerged, accounting for 64% of the explainable variance. We labeled these two factors as operational and data IoT skills and strategic IoT skills. The operational and data IoT skill construct represents items of the theory-driven operational, information, communication, and privacy con-structs. The common delineator for this new construct is handling data. Therefore,
although the five-factor solution as conceptualized was not confirmed, a conceptually
sound alternative emerged.Table 3shows the scale characteristics for the two IoT skills and the factor loadings.
Survey 2 results
Sample and setting
Thefinal step of developing the instrument was conducting an online survey in the Neth-erlands over a period of one week in May 2020. To obtain a representative sample of the Table 3.Scale characteristics (overall N = 178) and factor loadings.
Skill type α M SD Loading
Operational and Data IoT skills (8) 0.92 3.63 1.02
I know how to set up smart devices for different users 3.20 1.26 .76 I know how to view data my smart device collected 3.77 1.25 .70 I know how to compare data from my smart device to data other users collected 3.61 1.28 .68 I know how to check if the collected data from my smart device is correct 3.47 1.17 .65 I know how to connect smart devices to my network 3.97 1.14 .60 I know how to adjust the privacy settings of my smart device 3.60 1.30 .58 I know how to reset smart devices to the original settings 3.85 1.26 .52 I know how to display data from my smart device in a chart 2.92 1.39 .50
Strategic IoT skills (6) 0.89 3.55 0.94
I know when to adjust my smart device to achieve my goal 3.56 1.19 .84 I know how I perform best with my smart device 3.72 1.23 .77 I know how to see if I am making progress with my smart devices 3.70 1.17 .74 I know what actions to take based on data from my smart device 3.55 1.28 .73 I know how to make better decisions with my smart device (e.g., about energy or health) 3.63 1.12 .68 I know how to achieve my intended goals with my smart device 3.62 1.18 .67
Dutch population, we again used PanelClix. Invitations were sent out in two waves to
ensure that thefinal sample represented the Dutch population in terms of gender, age,
and education level distributions. In total, we obtained valid responses from 1,667 indi-viduals, of which 234 owned and used at least one smart device. The time needed to answer the survey questions was approximately 18 min (as the survey also addressed
other IoT-related questions). Table 4 summarizes the demographic characteristics of
the respondents.
Confirmatory factor analysis
Confirmatory factor analysis (SPSS Amos v24) was used to evaluate whether the factor model that emerged from the exploratory analysis provides a goodfit to the data. Confi-rmatory factor analyses assess goodness offit based on the variance remaining after the factors are taken into account. Goodness offit can be determined with the following
indi-ces (Byrne, 2001): chi-square tests (χ2), root-mean-square errors of approximation
(RMSEA), the comparativefit index (CFI), Akaike’s information criterion (AIC), and
the Tucker–Lewis index (TLI). The results obtained from the confirmatory factor analysis indicate a good fit: χ2(76) = 176.7; χ2/df = 2.31; AIC = 261.79; TLI = .93; CFI = .95; RMSEA = .08 (90% confidence interval = .06, .09).
Factor loadings were statistically significant and at least moderately large in magni-tude, ranging from .57 to .88. Scale reliability was assessed in several ways. Item variance, indicated by the squared correlation between matched items and factors, ranged from .30 to .75. The proportion of item-level variance to measurement error was .57 for
Oper-ational and Data IoT skills and .27 for Strategic IoT skills. Table 5 shows that the
reliability estimates, analogous to a coefficient alpha for internal consistency, were high for both Operational and Data IoT skills and Strategic IoT skills.
External validity
To examine external validity, that is, whether the scales have similar characteristics inde-pendent of the context or the population they are in, we took a threefold approach: (1) examining descriptive information on the scale averages for different groups, (2) testing for convergent and discriminant validity, and (3) testing whether the scale characteristics were consistent through random resamples of the population.
Table 4.Demographic profile of the Dutch IoT user sample.
N % Gender Male 138 59.0 Female 96 41.0 Age 16–30 61 26.1 31–40 65 27.8 41–50 46 19.7 50+ 62 26.5 Education level Low 116 49.6 High 118 50.4
Examining descriptive information across sociodemographic groups
When constructing scales, it is important to explore the extent to which they are reliable across different groups. In digital inequality literature, a few key predictors have been described for the level of digital skills an individual professes to have. Here, we look at
gender and education. Overall, Table 6 shows that the scales are similar in their
reliability. Men rate themselves significantly higher than women on operational and data skills (F(1,232) = 10.91, p < .001) but not on strategic IoT skills (F(1,232) = 0.59, p = .45). The comparisons of the means for the two education levels reveal differences for strategic IoT skills (F(1, 232) = 4.62, p < .05), but not for operational and data skills (F(1,232) = 0.56, p = .45).
Differences across gender are in the direction that might be expected by the litera-ture; that is, men estimate their own operational and data skills higher than women, which as prior research has shown is particularly the case for operational skills (Van
Dijk & Van Deursen, 2014). Furthermore, education plays a role in more strategic
uses of the internet. This finding shows consistency with previous general research
and theoretical thinking around how Internet skills differ between sociocultural groups (Scheerder et al., 2017).
Convergent and discriminant validity
The discriminant validity of the two-factor model was first assessed by a chi-square
difference test, which compares the difference between the current two-factor model and one in which the interfactor correlation isfixed to 1. The test was statistically signifi-cant,χ2(1) = 332.24, p < .001, indicating that a unidimensional model would be inferior to the current two-factor model. Second, the 95% confidence interval for the correlation between operational and data IoT skills and strategic IoT skills (r = .76) did not include a value of 1 (.67–.84) providing further support for two distinct but related latent con-structs. Third, discriminant validity was demonstrated because for both skill constructs, the variance extracted index exceeded the square of the interfactor correlation (.58). The set of IoT skills items with factor loadings and reliability estimates are presented inTable 7. Convergent validity was demonstrated because all t-tests for associated factor loadings are significant.
Table 5.Scale characteristics.
Skill type M SD Variance α
Operational 3.75 0.93 0.87 .89
Strategic 3.80 1.03 1.05 .92
Table 6.Means, standard deviations and reliability measures (M (SD);α) in the different groups.
Operational Data Strategic
Men 3.90 (0.79); 0.90a 3.86 (0.92); 0.85a
Women 3.52 (1.07); 0.92b 3.69 (1.06); 0.86a
Low education 3.65 (0.95); 0.92a 3.72 (1.10); 0.86a
High education 3.83 (0.91); 0.92a 3.86 (0.95); 0.85b
Note: For gender and education, within each column, means with noncommon subscripts are significantly different (p < .05).
Consistency in the scale characteristics through random resampling
Finally, to understand whether the factor solution was stable, a Bollen–Stine test (Bollen & Stine, 1992) was conducted for the full factor model. In 1895 of 2000 bootstrap
samples, the fit was better than in the original model (p = .05). The model shows a
goodfit and is a stable solution for the full population.
Discussion
Mainfindings
With the introduction of the Internet of Things, we are entering the era of Web 4.0 which marks important differences in the way we interact with Internet technology. As the IoT is expected to have a massive impact on people’s lives, those who have the skills to use it to its full potential will have the power to increase their (already privileged) positions (Van Deursen & Mossberger,2018). This in combination with the complexity of the sys-tem is likely to make it an important topic of investigation in digital inequality research. Scholars are now starting to think beyond the rectangular confines of desktop, laptop, tablet, and smartphone computers, and prerequisites for and impacts of user (consumer) engagement with the Internet of Things are increasingly recognized as an important study area. To further support research and policy development, there is a need for more theoretically informed, reliable, and valid instruments that are able to measure
what people do and gain from the IoT. We propose a first version of the IoT Skills
Scale (IoTSS):
Operational and data IoT skills:
1 I know how to set up smart devices for different users 2 I know how to reset smart devices to the original settings 3 I know how to connect smart devices to my network 4 I know how to view data my smart device collected
Table 7.Confirmatory factor analysis: factor loadings and reliability estimates.
Factors and items
Standardized
loading t test
Item reliability Operational and Data IoT skills (8)
I know how to set up smart devices for different users .79 8.55 .60 I know how to reset smart devices to the original settings .66 9.23 .52 I know how to connect smart devices to my network .77 8.72 .59 I know how to view data my smart device collected .73 9.00 .53 I know how to check if the collected data from my smart device is
correct
.77 8.47 .62
I know how to display data from my smart device in a chart .55 9.59 .30 I know how to compare data from my smart device to data other users
collected
.70 9.18 .49
I know how to adjust the privacy settings of my smart device .77 8.67 .57 Strategic IoT skills (6)
I know how to achieve my intended goals with my smart device .66 9.20 .43 I know what actions to take based on data from my smart device .76 9.03 .58 I know how to see if I am making progress with my smart devices .86 8.17 .73 I know when to adjust my smart device to achieve my goal .86 7.44 .75 I know how to make better decisions with my smart device (e.g., about
energy or health)
.82 8.58 .66
5 I know how to check if the collected data from my smart device is correct 6 I know how to display data from my smart device in a chart
7 I know how to compare data from my smart device to data other users collected 8 I know how to adjust the privacy settings of my smart device
Strategic IoT skills:
1 I know how to achieve my intended goals with my smart device 2 I know what actions to take based on data from my smart device 3 I know how to see if I am making progress with my smart devices 4 I know when to adjust my smart device to achieve my goal
5 I know how to make better decisions with my smart device (e.g., about energy or health) 6 I know how I perform best with my smart device
The development of this instrument began with a critical look at the existing literature. We aimed to develop an instrument that can be used for a large variety of IoT devices, aimed at the general IoT user. The items use a scale that give statements about IoT-related actions (smart devices) that a person is able to perform, with answer formats ran-ging from‘not at all true of me’ to ‘very true of me,’ including a ‘I do not understand what this means’ option. After the development of a first survey instrument, we used a three-fold approach to test the validity and reliability of the latent skill constructs and the cor-responding items. Thefirst step consisted of cognitive interviews. Based on the results, we made several amendments to the proposed skill items to improve clarity. The second step consisted of a survey of IoT skills to explore the factor structure of the instrument that resulted from the cognitive interviews. During thefinal step, we examined the consist-ency of the IoT skills scales and their characteristics in a second survey of Dutch Internet users. The reliability and validity of the scales as well as indicators of convergent and dis-criminant characteristics were good. This allows us to recommend a theoretical and empirical consistent framework consisting of Operational and data IoT skills, and stra-tegic IoT skills scales for use in general population research.
The IoTSS is thefirst of its kind and we consider it a valuable contribution to survey research in thefield of digital inclusion as it provides a useful set of items that others can use as a starting point for survey research on IoT-skills. However, we do not want to suggest that this is the ultimate instrument. In the coming years, we will work on further improving the IoTSS by refining the items and testing additional ones, ideally among the five theoretically distinct skills types identified conceptually. In our study, we were able to single out strategic skills, but operational, information, communication, and privacy skills surfaced as one factor. Conceptually, the binding denominator is that all these skills center around (the operational use of) data. It might be easier to distinguish between the different concepts in relation to the Internet, as information skills involve the process of seeking information using search engines, and communication skills span a range of skills from encoding and decoding messages, profiling, and contact management. With the IoT, some of these indices disappear, become less prominent, or maybe even are less distinctive. Similarly, studies on Internet skills cluster skills in two main concepts on a higher level, medium- and content-related skills (Van Dijk & Van Deursen,
2014). With data becoming so distinctive and prominent in the IoT, skills related to
handling data (information, communication, and privacy) also become increasingly intertwined. This set of skills is then required to apply strategic skills (Van Dijk &
Van Deursen,2014). Another explanation for the appearance of a twofold definition is
and knowledge of IoT devices and their workings and potential risks might make it difficult for people to apply different (sub)skills.
Limitations
All phases in the development of the IoTSS took place in the Netherlands, a country with high household Internet penetration and a forerunner of technology trends. On the one hand this makes the Dutch context appropriate for research on IoT skills, as several smart devices have entered the consumer market in the past years and it becomes increasingly clear what types of skills are required. On the other hand, we do not yet know to what extent the Dutch-centric interpretation of IoT-skills reflects the skillset needed for IoT-engagement in other contexts or countries. However, as the items proposed relate to general IoT use, applicable to many devices, it is likely that also in other countries the IoTSS can be applied.
The fact that we focused on general IoT use can also be considered as a limitation.
Because we created items that fit a large array of smart devices, they do not reveal
much details of the skills that are required for specific appliances, further complicating
reflection on the IoT as a unified category and blurring distinctions between different
skill types. For example, smart lighting systems at home might request some specific skills different than the skills required for activity trackers. When details are required for a specific device, the IoTSS can be adjusted and extended for the use of a specific device in future investigations.
Disclosure statement
No potential conflict of interest was reported by the authors.
Funding
This work was supported by Nederlandse Organisatie voor Wetenschappelijk Onderzoek [grant number 452-17-001].
Notes on contributors
Alexander J. A. M. van Deursen is Professor and chair of the Department of Communication Science at the University of Twente in the Netherlands. Most of his research focuses on three lines of research with the overarching theme of digital inequality. He maps barriers of online engagement and explains differences in outcomes from Internet (of Things) use. Research projects Alexander leads are Digital Inequality in the Netherlands, twenty-first century digital skills in the creative industry, inequality in Internet of Things skills and studying digital inequality in the social context of the home. Alexander holds Visiting Scholar positions at the London School of Econ-omic and Political Science and Arizona State University.
Alex van der Zeeuwis a PhD candidate at the University of Twente at the department of Com-munication Science. He is currently involved in a project on social contextual analyses of Internet (of Things) use and outcomes. He addresses the transmission and development of skills for using the Internet of Things in human–machine figurations in the domestic sphere.
Pia de Boeris a PhD candidate at the University of Twente at the department of Communication Science. She is currently involved in a project on Internet of Things skills. She measures these skills in performance tests in which people actually use IoT devices.
Giedo Jansenis an assistant professor at the University of Twente, Institute for Innovation and Governance Studies. His research is on the intersection of political science, sociology, and labor relations. He has recently published in these areas in journals such as the American Journal of Sociology, Industrial and Labor Relations Review, Electoral Studies, Social Science Research, and West European Politics. Currently, he works on a research project on self-employment and political alignments, based on a VENI grant from the Netherlands Organization for Scientific Research (NWO).
Thomas van Rompayis an associate professor at the Department of Communication Science of the University of Twente and a fellow at the UT’s DesignLab. He has a background in cognitive psy-chology. He studies design experience from an embodied cognition perspective, investigating how design communicates meaning and affect. His current research projects take place on the threshold of design and psychology where he studies influences of environmental design and technology on health and wellbeing.
ORCID
Alexander J. A. M. van Deursen http://orcid.org/0000-0002-0225-2637
References
Ala-Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding. Institute for Prospective Technological Studies.
Ananiadou, K., & Claro, M. (2009). 21st Century Skills and competences for New Millennium lear-ners in OECD countries. Organization for Economic Cooperation and Development (EDU Working Paper No. 41).http://www.olis.oecd.org/olis/2009doc.nsf/linkto/edu-wkp(2009)20 Bauer, J. M. (2018). The Internet and income inequality: Socio-economic challenges in a
hypercon-nected society. Telecommunications Policy, 42(4), 333–343. https://doi.org/10.1016/j.telpol. 2017.05.009
Bollen, K. A., & Stine, R. A. (1992). Bootstrapping goodness-of-fit measures in structural equation models. Sociological Methods & Research, 21(2), 205–229. https://doi.org/10.1177/ 0049124192021002004
boyd, d., & Hargittai, E. (2010). Facebook privacy settings: Who cares? First Monday 15(8).http:// firstmonday.org/htbin/cgiwrap/bin/ojs/index.php/fm/article/view/3086/2589
Büchi, M., Just, N., & Latzer, M. (2017). Caring is not enough: The importance of Internet skills for online privacy protection. Information, Communication & Society, 20(8), 1261–1278.https:// doi.org/10.1080/1369118X.2016.1229001
Byrne, B. M. (2001). Structural equation modeling with AMOS: Basic concepts, applications and programming. Erlbaum.
Cotter, K., & Reisdorf, B. C. (2020). Algorithmic knowledge gaps: A new dimension of (digital) inequality. International Journal of Communication, 14, 745–765. https://doi.org/1932–8036/ 20200005
De Boer, P. S., Van Deursen, A. J. A. M., & Van Rompay, T. J. (2019). Accepting the internet-of-things in our homes: The role of user skills. Telematics and Informatics, 36, 147–156.https://doi. org/10.1016/j.tele.2018.12.004
Desimone, L. M., & Le Floch, K. C. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22.https://doi.org/10.3102/01623737026001001
Eshet, Y. (2004). Digital literacy: A conceptual framework for survival skills in the digital era. Journal of Educational Multimedia and Hypermedia, 13(1), 93–106.
Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.
https://doi.org/10.1037/1082-989X.4.3.272
Ferrari, A. (2013). DIGCOMP a framework for developing and understanding digital competence in Europe. Publications Office.http://dx.publications.europa.eu10.2788/52966
Fowler, D., Pitta, D., & Leventhal, R. C. (2013). Technological advancements and social challenges for one-to-one marketing. Journal of Consumer Marketing, 30(6), 509–516.https://doi.org/10. 1108/JCM-05-2013-0549
Hargittai, E., & Litt, E. (2013). New strategies for employment? Internet skills and online privacy practices during people’s job search. IEEE Security & Privacy, 11(3), 38–45.https://doi.org/10. 1109/MSP.2013.64
Helsper, E. J., & Eynon, R. (2013). Distinct skill pathways to digital engagement. European Journal of Communication, 28(6), 696–713.
Hinostroza, J. E., Matamala, C., Labbe, C., Claro, M., & Cabello, T. (2015). Factors (not) affecting what students do with computers and internet at home. Learning Media and Technology, 40(1), 43–63.https://doi.org/10.1080/17439884.2014.883407
Hobbs, R. (2010). Digital and media literacy: A plan of action. A white paper on the digital and media literacy recommendations of the knight commission on the information needs of commu-nities in a democracy. Aspen Institute.
Hsinchun, C., Chiang, R. H. L., & Storey, V. C. (2012). Business intelligence and analytics: From Big Data to Big Impact. MIS Quarterly, 36(4), 1165–1188. https://doi.org/10.2307/ 41703503
Huang, K. T., Cotten, S. R., & Rikard, R. V. (2017). Access is not enough: The impact of emotional costs and self-efficacy on the changes in African-American students’ ICT use patterns. Information Communication & Society, 20(4), 637–650. https://doi.org/10.1080/1369118X. 2016.1203456
Jenkins, H., Purushotma, R., Weigel, M., Clinton, K., & Robinson, A. J. (2009). Confronting the challenges of participatory culture. Media education for the 21st century. The MIT Press. Klawitter, E., & Hargittai, E. (2018).“It’s like learning a whole other language”: The role of
algo-rithmic skills in the curation of creative goods. International Journal of Communication, 12, 3490–3510.doi:1932–8036/20180005
Knafl, K., Deatrick, J., Gallo, A., Holcombe, G., Bakitas, M., Dixon, J., & Grey, M. (2007). The analysis and interpretation of cognitive interviews for instrument development. Research in Nursing & Health, 30(2), 224–234.https://doi.org/10.1002/nur.20195
Litt, E. (2013). Measuring users’ internet skills: A review of past assessments and a look toward the future. New Media & Society, 15(4), 612–630.https://doi.org/10.1177/1461444813475424
Martin, A., & Grudziecki, J. (2006). Digeulit: Concepts and tools for digital literacy development. Innovation in Teaching and Learning in Information and Computer Sciences, 5(4), 249–267.
https://doi.org/10.11120/ital.2006.05040249
Merritt, K., Smith, D., & Renzo, J. C. D. (2005). An investigation of self-reported computer literacy: Is it reliable. Issues in Information Systems, 6(1), 289–295.doi:10.48009/1_iis_2005_289-295
Miller, K. (2006). Organizational communication, approaches and processes. Thomson Wadsworth. Pasquale, F. (2015). The black box society: The secret algorithms that control money and
infor-mation. Harvard University Press.
Ransbotham, S., Fichman, R. G., Gopal, R., & Gupta, A. (2016). Special section introduction —ubi-quitous IT and digital vulnerabilities. Information Systems Research, 27(4), 834–847.https://doi. org/10.1287/isre.2016.0683
Riggins, F. J., & Wamba, S. F. (2015). Research directions on the adoption, usage, and impact of the internet of things through the use of big data analytics. In 48th Hawaii international conference on system sciences (pp. 1531–1540). Kauai, HI, USA: IEEE.
Scheerder, A., Van Deursen, A. J. A. M., & Van Dijk, J. A. G. M. (2017). Determinants of Internet skills uses and outcomes. A systematic review of the second-and third-level digital divide. Telematics and Informatics, 34(8), 1607–1624.https://doi.org/10.1016/j.tele.2017.07.007
Shank, D. B., & Cotten, S. R. (2014). Does technology empower urban youth? The relationship of technology use to self-efficacy. Computers & Education, 70, 184–193.https://doi.org/10.1016/j. compedu.2013.08.018
Sri, T. S., Prasad, J. R., & Vijayalakshmi, Y. (2016). A review on the state of art of Internet of things. International Journal of Advanced Research in Computer and Communication Engineering, 5(7), 189–193.doi:10.17148/IJARCCE.2016.5738
Van Der Zeeuw, A., Van Deursen, A. J. A. M., & Jansen, G. (2020). How to apply IoT skills at home: Inequalities in cultural repertoires and its interdependency chains. Poetics, 83, 101486.
https://doi.org/10.1016/j.poetic.2020.101486
Van Deursen, A. J. A. M., Helsper, E. J., & Eynon, R. (2016). Development and validation of the Internet Skills Scale (ISS). Information, Communication & Society, 19(6), 804–823.https://doi. org/10.1080/1369118X.2015.1078834
Van Deursen, A. J. A. M., Helsper, E. J., Eynon, R., & Van Dijk, J. A. G. M. (2017). The compound-ness and sequentiality of digital inequality. International Journal of Communication, 11, 452– 473.
Van Deursen, A. J. A. M., & Mossberger, K. (2018). Anything for anyone? A new digital divide in internet-of-things skills. Policy & Internet, 10(2), 122–140.https://doi.org/10.1002/poi3.171
Van Deursen, A. J. A. M., Van Der Zeeuw, A., De Boer, P., Jansen, G., & Van Rompay, T. J. (2019). Digital inequalities in the Internet of things: Differences in attitudes, material access, skills, and usage. Information, Communication & Society, 24(2), 258–276. https://doi.org/10.1080/ 1369118X.2019.1646777
Van Deursen, A. J. A. M., Van Dijk, J. A. G. M., & Peters, O. (2011). Rethinking Internet skills: The contribution of gender, age, education, Internet experience, and hours online to medium-and content-related Internet skills. Poetics, 39(2), 125–144. https://doi.org/10.1016/j.poetic.2011. 02.001
Van Dijk, J. A. G. M., & Van Deursen, A. J. A. M. (2014). Digital skills, unlocking the information society. Palgrave Macmillan.
Van Laar, E., Van Deursen, A. J., Van Dijk, J. A, & De Haan, J. (2018). 21st-century digital skills instrument aimed at working professionals: Conceptual development and empirical validation. Telematics and Informatics, 35(8), 2184–2200.
Van Laar, E., Van Deursen, A. J. A. M., Van Dijk, J. A. G. M., & De Haan, J. (2017). The relation between 21st-century skills and digital skills or literacy: A systematic literature review. Computers in Human Behavior, 72, 577–588.https://doi.org/10.1016/j.chb.2017.03.010
Appendix A. Items per skill after the cognitive interviews
Operational IoT skills
. O01 I know how to set up smart devices for different users
. O02 I know how to reset smart devices to original settings
. O03 I know how to connect smart devices to my network
. O04 I know how to coordinate my smart devices
. O05 I know how to give different users access to my smart device
. O06 I know how to tailor my smart device to my own preferences
Information IoT skills
. I01 I know how to view data my smart device collected
. I02 I know how to adjust what data is collected on a smart device
. I03 I know how to check if the collected data from my smart device is correct
. I04 I know how to display data from my smart device in an understandable way
. I06 I know how to display data from my smart device in a chart
. I07 I know how to clearly display collected data from a smart device
Communication IoT skills
. C01 I know how to compare data from my smart device to data other users collected
. C02 I know how to share data that my smart devices collect
. C03 I know how to share collected data from smart devices on the internet
. C04 I know how I can make contacts with other users with my smart devices
. C05 I know how I can give other users feedback on the data they have collected with their smart device
. C06 I know how to set up with whom I can share the collected data from a smart device
Privacy IoT skills
. P01 I know how to block access to my smart devices for others
. P02 I know how to shield data that my smart devices collect from others
. P03 I know how to adjust the privacy settings of my smart device
. P04 I know how to delete personal data from smart device
. P05 I know how to customize on smart devices with whom I share data
. P06 I know how to set which data can be shared from my smart device
Strategic IoT skills
. S01 I know how to make better choices with my smart device
. S02 I know how to achieve my goals with my smart device (eg energy savings, better security, or healthier life)
. S03 I know how to get my smart devices working together to get the most out of it
. S04 I know how to set realistic goals with my smart device
. S05 I know how to make my smart devices work best together
. S06 I know how to achieve my intended goals with my smart device
. S07 I know what actions to take based on data from my smart device
. S08 I know how to see if I am making progress with my smart devices
. S09 I know when to adjust my smart device to achieve my goal
. S10 I know how to make better decisions with my smart device (eg about energy or health)
