Supervisor Dr. André Nusselder University of Amsterdam Second Examiner Dr. Frank Nack University of Amsterdam
Digital Signatures from a User Perspective: Trust and the
Technology Acceptance Model
SUBMITTED IN PARTIAL FULLFILLMENT FOR THE DEGREE OF
MASTER OF SCIENCE
JORDI LANGEN
6131646
MASTER INFORMATION STUDIES
HUMAN-CENTERED MULTIMEDIA
FACULTY OF SCIENCE
UNIVERSITY OF AMSTERDAM
Digital Signatures from a User Perspective: Trust and the
Technology Acceptance Model
Jordi Langen
University of Amsterdam, Graduate School of Informatics
Science Park 904, Amsterdam
jordi.langen@student.uva.nl
ABSTRACT
An electronic signature is a digitized image of a handwritten signature. Digital signatures differ from this by requiring a digital certificate, such a certificate is needed to be able to place a digital signature. When comparing both, a digital signature has several advantages over an electronic signature, the most important one being the ability to verify the signature through a trusted third-party.
An EU regulation, Electronic IDentification and Authentication Services (eIDAS) went into effect 1st July 2016. The main purpose
of this regulation is to create mutual recognition of digital signatures across all member states. To lower (online) trade barriers and eliminate the time-consuming process of posting documents for businesses, persons and public authorities throughout the EU. This research dives into digital signatures from a user perspective. Would an average person make use of digital signatures and what is their perception of the technology after using it? The user perspective will be explored through multiple metrics, namely trust and the Technology Acceptance Model use-antecedents perceived usefulness and perceived ease of use. First, through relevant studies on trust, the concept of trust will be defined and divided into three categories, and there will be looked into the third-party trust model. After that the Technology Acceptance Model will be explained. The first user testing consisted of the task for participants (N=10) to place a digital signature on a PDF document, using three different PDF readers. Afterwards they were asked several questions on trust, usefulness and ease of use. This resulted in a set of factors that influences trust and specific functional requirements for digital signing software.
Based on the outcome of the first test session a digital prototype was made, designed for the task of digitally signing a PDF file. A second user testing was conducted with 10 participants, 5 persons from the first test and 5 new participants. The results from the second user testing showed that the prototype was perceived as easier to use than the highest on usability rated PDF reader from the first test.
Overall, the findings show that the ease of use of digitally signing files can be improved, and that when people would (want to) start using digital signatures there are several important factors that build and maintain trust, and influence people their behavioral intention to use digital signatures.
Keywords
Digital signatures; Trust; Perceived Ease of Use; Perceived Usefulness; Technology Acceptance Model
1. INTRODUCTION
A digital signature is an identity seal on digital data and it can be seen as a digital identification method on documents such as PDFs. Digital signatures make use of digital certificates; hence they are sometimes also called certificated-based signatures. These certificates always contain information about its owner’s identity and are unchangeable after issuance, therefore such certificates can be seen as a digital ID.
Like a conventional handwritten signature, a digital signature identifies the person signing a document. A digital version of a handwritten signature, i.e. a digitized image of a handwritten signature, is called an electronic signature. The difference between this and a digital signature, is that an electronic signature is solely a digital image, there is no extra security behind it. Everyone on the internet can find and copy the electronic signature of, for example, Barack Obama [3].
Digital certificates make use of public-key cryptography, making it impossible to duplicate or hack these certificates, and therefore it is impossible to forge digital signatures.
A certification authority (CA) is an entity that issues (and can revoke) digital certificates. An issued certificate always has an end date; it is only valid for a specific period of time. A CA is the trusted third-party, in order for the trust model to work the CA should be trusted by the owner of the certificate, but also by the party relying upon the certificate. It means CAs are trusted to ensure independent identity validation, that the validation process is secure and uncompromised, and that the CA has proper certificate revocation policies put in place.
When documents are distributed digitally, it is important that recipients can: verify document authenticity (confirming the identity of each person who signed the document), and verify document integrity (confirming that the document has not been altered in transit). Digital signatures provide both of these security services.
Many business transactions, including financial, legal, and other regulated transactions, require a high level of assurance when signing documents. That is why businesses and governments can benefit from, or are already using, a digital certificate infrastructure within their organisation.
Digital signatures can be viewed as a tool for identity services and authorisation services. Both are examples of trust services [2]. Consumers' lack of trust in e-commerce is often assumed to be one of the main reasons for the disappointing development of B2C e-commerce [31]. Baldwin et al. state that: “Trust services are an emerging enabler for e-commerce” [2, p. 1].
It is inevitable that in the near future hand-signed documents will become old-fashioned and inefficient to use. The use of digital
signatures will most likely keep increasing, also due to the introduction of the new eIDAS regulation mentioned earlier [10]. According to Gupta el al. [18] digital signatures are already or will be soon adopted by organizations to replace traditional ways of signing documents, especially when they are send over the internet. To support their claim, they listed a number of advantages of digital signatures:
There is no need to print out documents before signing, meaning a printer, paper and ink are not necessary anymore.
Companies no longer need a big archive room, as digital documents only take up server space.
It is easier to manage and access (anytime/anywhere) digital documents compared to paper documents. From a digitally signed document an unlimited number
of copies can be made, all bearing the same legal force. Eliminating the worry of securely archiving the only copy.
Better security of document transmission. No one can longer possibly intercept the (physical) document, because there is no printed document.
No need to fax or mail documents. They can be delivered by e-mail, reducing cycle time.
Since societies (and therefore) economies will turn more towards online communications, it can be seen that building online trust is key to economic development. One of the most essential parts of a functioning digital society is having a secure digital identity. Government and businesses need to know who is using their online services. From the other side, citizens need to know and trust that their identity is safe and being protected, that no one can steal their digital identity. Estonia is an example of a country that has set up a digital infrastructure for the use of digital signatures. They hand out mandatory ID-cards to citizens, ID-cards with a built-in chip (containing the certificate) which can be used to identify themselves online and digitally sign documents [9].
What if companies will start using digital signatures? How will persons deal with it? People should trust the technology and be able to work easily with software that makes use of the technology. When companies or persons will switch to using digital signatures this might bring (initial) issues.
One of those issues will be trust. How will people be assured that a digital signature was actually placed by the person the signature displays, and which factors play a role in establishing their trust level? For example, an electronic signature, for some people, it might be more trusted as it is directly similar to a ‘real life’ signature. A digital document could just show the text line ‘This document is signed by person X’, backed by a digital certificate, that could be less trustworthy to an average user than the ink-on-paper signature.
Another issue is acceptance of the technology. The technology, together with the software used to work with the technology, should be perceived as useful and easy to use. The technology acceptance model (TAM) is a model for predicting acceptance and use of technologies via the metrics perceived ease of use and perceived
usefulness [8]. These two metrics are the main components of the
TAM. The terms ‘usability’ and ‘ease of use’ will be used as synonyms throughout this research.
In this research the focus was on these two perspectives: 1) the trust in the technology and certification authorities; 2) how people perceive digital signatures on the level of usefulness and usability. Potential barriers withholding people from using, or barriers while using, digital signatures were investigated. Coming up with solutions for any barriers that were found could be beneficial to the increase in the use of digital signatures.
First existing literature on (online) trust and TAM was analysed. Using the related work an interview procedure was made, this procedure involved participants testing software which can digitally sign (and verify) PDF files. Findings were structured into factors influencing trust and a set of functional requirements for digital signing software. Using these requirements, a prototype was made that can place digital signatures on documents. Lastly, the prototype was evaluated through a second user testing round.
2. RELATED WORK
2.1 Trust
2.1.1 Scientific research
There has been little research on the combination of digital signatures and trust. A lot of research focuses on online trust in general, and implications for e-business. Multiple studies were analysed and relevant factors have been written down.
First the concept of trust had need to be defined. A long-time used definition in psychology, by Rotter (1971), of trust is “an expectancy held by an individual or a group that the word, promise, verbal or written statement of another individual or group can be relied on” [33, p. 444].
The concept of trust is multifaceted, to clarify findings many studies made the distinction between three categories of trust: institutional trust, social trust and technological trust [23], [24] and [28]. Institutional trust is an expectation that your vulnerabilities will not be exploited [25] and [26]. In the context of digital signature this can be seen as the trust people have towards certificate authorities. Social trust has been defined as the disposition to trust and a feeling that the person you are trusting is of a good disposition [6]. The belief that other people are generally trustworthy and the trustor’s willingness to be dependent on others, having a trusting stance and faith in humanity [27]. Technological trust is the dimension of trust that consists of technology-related trust issues and the individual perceptions and assessments of these [23]. In this research it was seen as the people’s trust in the technology behind digital signatures, which entails users’ trust that a digital certificate is secure and cannot be hacked, and that a signed document really was signed by the displayed signer.
The development of online trust can be influenced by multiple factors. Results from research by Beldad et al. have described their results by dividing them into three clusters [4], being:
Client-based trust antecedents:
Users’ experience with the technology used for the task or transaction
Users’ tendency to trust Web-based trust:
The quality of the website used for the task or transaction Having security assurances displayed
Company/organisation-based trust:
Prior experiences with online companies/organisations The reputation of those organizations
Referring at the three trust categories that were established earlier on, ‘Users’ experience with the technology used for the task or transaction’ can be seen as perceived ease of use. ‘Users’ tendency to trust’ can be seen as social trust. Web-based trust can be seen as technological trust. Company/organization-based trust is the counterpart of institutional trust.
Most interface elements of software can be seen as trust qualifiers: they won’t get people over the trial-threshold, but if interface elements are not looked after properly they can be great trustbusters [32]. Such trustbusters are poor usability, inconsistent design or technological failures. In e-commerce the strongest trust builders are factors outside the interface, which are brand and reputation [32].
2.1.2 Certificate authorities
A certification authority is a trusted third-party company that issues digital certificates, these digital certificates are needed to create digital signatures. The role of a CA is to guarantee that the person using a certain certificate is, in fact, who he or she claims to be. A root CA is the trust anchor for a digital signature, it is accountable for all the tranches below them, see figure 1. If a root CA no longer accounts for an intermediate CA, it will revoke that certificate and thereby break the link with that intermediate CA and all the underlying certificates of end-users underneath it.
Figure 1. Chain of trust
When a digital signature is presented to a person as a means of identifying the certificate holder, it is useful only if the person receiving the certificate trusts the issuer (the CA). When you trust a CA, that means you have confidence that the CA has the proper policies in place when evaluating certificate requests and will deny certificates to any entity that doesn’t meet those policies. In addition, you trust that the CA will revoke certificates that should no longer be considered valid.
A certificate always shows the issuer, for example Symantec [38]. Basically it comes down to: ‘I know this document is signed by person X, because the trusted third-party, Symantec, vouches for it.’ The verification process is always bottom-up and the trust is top-bottom, see figure 2.
Figure 2. Certificate path
Symantec is an example of a commercial company providing certificate services where they act as the root CA. Root CA’s can be commercial or governmental. An advantage of having a governmental root CA is not being dependable on (possible foreign) commercial companies.
2.2 Technology Acceptance Model
The interface and (work)flow of software has to provide enough means for people to establish trust into a back-end system that most users do not understand [17] and [19].
The technology acceptance model is an information systems theory that models users’ acceptance and use of technologies via the metrics perceived ease of use (PEOU) and perceived usefulness (PU). These two metrics are the main components of the TAM. TAM is based on the theory of reasoned action (TRA), a psychological theory that seeks to explain behaviour [11]. TAM is useful for predicting whether users will adopt new information technologies. The TAM is depicted in figure 3.
Figure 3. Technology Acceptance Model by Davis et al. [8] There are a number of factors, suggested by the model, that affect how a user will feel about how and when they will use the technology. Over time, users will develop perceptions on the usefulness and ease of use of a technology. This perceived usefulness and perceived ease of use affect the acceptance and intended use of IT systems, the behavioural intention. TRA assumes this behavioural intention is closely linked to actual behaviour.
Venkatesh and Davis proposed an extension to the model, the Technology Acceptance Model 2 (TAM2) [37]. With this extension they tried to provide a stronger model than the original TAM. The extension consists of, amongst other things, the addition of the factor ‘result demonstrability’ as an influence on the perceived usefulness. Next to that they omitted the construct attitude towards
using (ATU) because it was a weak predictor of the behavioural
intention to use [35] and [36]. Therefore, it was decided to not use ATU in this research.
External variables, or external prior factors, are factors such as prior usage or experience, trust or result demonstrability.
There are a few different occasions where trust is considered crucial. Business and social interactions are often ruled by trust and the dependency on another party. In combination with dependency, lack of control is also a crucial factor [13]. In these cases, trust determines, among other things, the expected utility people expect to gain from the business interaction [14]. From this point of view, you can see that trust influences the perceived usefulness. PU is the extent to which people believe that the technology will help them perform (their job) better, while PEOU is how effort free a person believes that using the specific system would be [7]. Usefulness is seen as a function of task/tool fit, while EOU is viewed more as a task-independent construct reflecting intrinsic properties of the user interface [22].
Research suggests that the degree and impact of trust, perceived usefulness, and perceived ease of use change with experience [15]. Perceived ease of use is less likely to be an indicator of behavioural intention to use according to studies of mobile commerce [39], online banking [30] and telemedicine [20], This supports the before mentioned concept that the main trust factors are outside of the interface. Usefulness and trust are more important towards usage intention; ease of use is mainly just a possible trustbuster when not perceived as high enough.
Research by Gefen et al. [16] on online shopping shows that consumer trust is as important to online commerce as the TAM use-antecedents, perceived usefulness and perceived ease of use. The study indicates that building online trust depends on:
A belief that the other party has nothing to gain by being deceitful.
This falls in the category of institutional trust.
A belief that there are safety mechanisms built into the website.
This falls into technological trust. By having a typical interface. A website that is easy to use.
These two points could be placed under the construct PEOU.
3. REQUIREMENTS ANALYSIS
Via user testing, participants were questioned in order to gather their opinion on digital signatures and digitally signing PDF files. Potential issues related to trust, PEOU and PU were listed and analysed.
The questionnaire incorporated multiple factors. These constructs were social trust, prior experience, perceived ease of use, result demonstrability, perceived usefulness, technology trust, institutional trust, and behavioural intention to use. Most of the used questions were derived from the TAM and TAM2 questionnaire, with some questions being slightly edited.
The full questionnaire is listed in the appendix.
3.1 Method
The used method was an assisted survey, where the researcher was present to help and explain where needed.
First the participant’s age, gender and highest level of education were noted. Then they were given an explanation on digital signatures and the difference in comparison to electronic signatures. The different representation possibilities of a digital signature (in a PDF file) were shown. next to that a signed PDF file and a PDF file with a non-verifiable (non-valid) signature were shown. After that participants were presented with the social trust and institutional trust questions. They were also asked if they had any prior experience with digital signatures. Questions needed to be answered on a 7-point Likert scale, ranging from 1: strongly disagree to 7: strongly agree, except for the prior experience question.
Then the participants were given the task of digitally signing a PDF file, with a pre-set (self-issued) certificate. This needed to be done three times, for each of the three PDF readers: Adobe Acrobat Reader [1], Foxit Reader [12], and Nitro Pro 10 [29]. Participants received assistance if they weren’t able to work out how to initiate the digital signing process.
3.2 Participants
Participants were recruited among family and friends. For this user testing ten participants were used, 4 males and 6 females. Looking at the level of education of the participants: one person attended intermediate vocational education, six persons were studying on the level of, or finished, university of applied sciences. Three persons did go to university. The age range was 20 to 59 years, with the average age being 31.2.
3.3 Results
The scores for the TAM and trust constructs were visualised through scatter plots. Followed by sections on the specific trust requirements and functional requirements outcomes.
3.3.1 Scatter plots
Scatter plots were used to visualise the data outcomes. Linear regression was used to add trend lines. These trend lines were based on the formula of y = a + b*x. Where x and y are the variables, a is the intercept point of the regression line and the y-axis. b is the slope of the regression line. Linear regression does not answer questions of causality directly though. Adding the fact that the sample size of the user testing was relatively low, making it that the regression won’t proof causality but might give an indication of it. The average score of the social trust questions in comparison to the average scores of the technology trust questions and the institutional trust questions is shown in figure 4.
Figure 4. Social trust in relation to technological trust and institutional trust
Without the trend line it is not directly apparent that there is a correlation between social trust and technological trust, but when a trend line was added it seemed clearer. Comparing both trend lines it seems that the positive correlation of social trust on trust in the technology and institutional trust is almost similar.
When looking at technology trust in relation to the perceived ease of use and the perceived usefulness, in figure 5, it shows that the slope of the regression line is quite similar. The difference is easy to notice: the perceived usefulness was awarded around 1 point higher on average than the perceived ease of use.
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Tech n o lo gy tr u st ( b lu e ) In sti tu ti o n al tr u st ( re d ) Social trust
Figure 5. Technology trust in relation to PEOU and PU Figure 6 showed that institutional trust is less of a determent of perceived usefulness than technology trust and that it is not a strong influence on PU in general. Figure 6 also displays that institutional trust seemed to be of no influence on the perceived ease of use.
Figure 6. Institutional trust in relation to PEOU and PU Technology trust and institutional trust their relation to
behavioural intention is shown in figure 7. Due to some outlying values the trend lines are quite skewed, without those lines it seemed that both trust factors are not predictors of intended usage.
Figure 7. Technology trust and institutional trust in relation to behavioural intention to use
Figure 8 gives the perceived ease of use against the trust in the technology. It quite noticeably gives the impression that the
perceived ease of use has a positive correlation with the trust users will have in the technology.
Figure 8. PEOU in relation to technology trust Figure 9 gives a slight indication that a higher value of perceived usefulness relates to a higher trust in the technology.
Figure 9. PU in relation to technology trust If you look at the technology acceptance model, it shows an outgoing line from perceived ease of use towards perceived usefulness. Meaning that PEOU can be of influence on PU. Figure 10 depicts the relation between the two. From the data it seemed that the value of PEOU has a slight positive correlation with PU.
Figure 10. PEOU in relation to PU
Result demonstrability is a factor from TAM2, where it is shown as an influence on the perceived usefulness. Figure 11 shows the user testing results of the two metrics. In this graph the trend line is highly skewed due to one outlying value. When ignoring the trend line there seemed to be not much of an effect of result demonstrability on the perceived usefulness.
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Pe rc e iv e d e ase o f u se ( b lu e ) Pe rc e iv e d u se fu ln e ss (re d ) Technology trust 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Pe rc e iv e d e ase o f u se ( b lu e ) Pe rc e iv e d u se fu ln e ss (re d ) Institutional trust 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Behav io u ral in te n ti o n to u se
Technology trust (blue) Institutional trust (red)
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Tech n o lo gy tr u st
Perceived ease of use
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Tech n o lo gy tr u st Perceived usefulness 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Pe rc e iv e d u se fu ln e ss
Figure 11. Result demonstrability compared with PU Finally, figure 12 depicts the perceived ease of use and the perceived usefulness values on the x-axis, with the y-axis being the BI values. Due to the score of ‘1’ for behavioural intention of use, the trend line for perceived usefulness is highly misleading. If you would remove that single BI score, both trend lines will be near horizontal, indicating that the results for both PEOU and PU seemed to not be a good predictor of intention to usage.
Figure 12. PEOU and PU in relation to BI
3.3.2 Trust requirements
Most of the trust related results were derived from analysing the scores that participants gave on the relevant trust questions. Some results were also based on remarks made by participants during the testing procedure.
One particular result was that participants didn’t differentiate much between trusting government root certification authorities or commercial root CAs, both were given an average of 4.3 points. Three persons slightly trusted commercial CAs more, three persons did trust governmental CAs a bit more, four persons gave an equal score to both. A given score of 4 means being neutral (neither disagree nor agree) so it is likely that some participants didn’t fully understand the difference yet, or didn’t care about the difference. When looking at the three different PDF readers, Adobe Reader was voted as the most trustworthy by the largest group, 50% of the participants thought so. 30% had no opinion, and 10% and 10% voted for respectively Foxit Reader and Nitro Pro 10. Reasons that people gave for trusting Adobe the most, was that Adobe is a known brand and therefore more trustworthy to them.
An important factor influencing trust was having knowledge about the technology, for example knowing that the technology is safe and is already established in other businesses, being assured about the safety of the technology. 4 out of the 10 participants explicitly
stated that their trust is linked with knowing the technology is safe, that abuse by third-parties should be (near) impossible. So they endorsed that guaranteeing and emphasizing on security/safety should be an important item. In addition to that it should be clear that when requesting certificates your personal details are processed safely and stored secure.
Another important aspect was prior experience with digital signatures. Quotes from multiple participants were “If I knew it (= the technology) better and had more experience using it, I would trust it more.”
Overall, the main points affecting trust were:
Brand. Adobe was the biggest name and was because of that rated as the most trustworthy.
A person’s disposition to trust. The higher this is, the more likely institutional and technological trust will be higher. (see figure 4)
Assuring the technology is secure/safe.
Related to the point above, being assured that your personal details are handled with care by certificate authorities.
Personal experience with the technology. The more people are known with using (or know about) digital signatures, the more trust they will place in it.
3.3.3 Functional requirements
Most functional requirements were gathered through answers on the two open questions, next to that additional notes were made by the researcher. These notes would be either comments made by the participant or observations from the researcher.
The user testing showed that the largest group (50%) of the respondents found Foxit Reader the most appealing software, meaning it looked the most inviting to work with.
The software with the highest PEOU was Foxit Reader, participants found this system the easiest to use with an average given score of 5 points. Nitro scored an average of 4 and Adobe a 3.7. Foxit and Nitro scored higher on the area of ease of use due to the tab structure both interfaces have. Participants liked this tab structure because
they were familiar with it by using Microsoft Office. The average PEOU of the three PDF readers combined was a score
of 4.23. This shows that there is room for improvement on the side of current PDF readers, to make it easier for people to place digital signatures through such software.
All participants did found it hard to find the digital signature option in the PDF readers, especially in Adobe Reader. One of the mistakes participants made was that they clicked on sign options which where electronic signatures, placing an electronic (ink) signature in the document, instead of a digital signature. Another issue was that people didn’t understand that they had to draw a rectangle to select the area where the signature will be put. The programs didn’t notify users that they had to do this after clicking the ‘sign’ button.
Adobe Reader and Foxit Reader both have just one overview screen with all the signing options before placing the digital signature. Nitro Pro had divided this screen into three steps. All participants preferred to have just one screen and not to have multiple screens. There is the option of signing documents with a non-visible digital signature. All participants made it clear that they preferred a visible signature in the document itself.
A digital signature in the document can have different layouts. For example, it can contain a photo of the signer, or partly consist of a
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Pe rc e iv e d u se fu ln e ss Result demonstrability 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Behav io u ral in te n ti o n to u se
Perceived ease of use (blue) Perceived usefulness (red)
handwritten signature image. 8 out of the 10 participants preferred for the layout to only display the name, date and time.
To sum up the specific functional requirements:
Put all the sign options under one tab named ‘sign’. Make a clear distinction in the interface between the
forms of signing, between placing an electronic (ink) signature or a digital signature.
(After clicking the button to initiate digitally signing) show a pop-up notifying people that they have to draw a rectangle at the location in the document where they want to place the signature. (with the option to hide this pop-up after the first time).
The ‘Sign Document’ screen should have all the options/settings in 1 screen, not consist of multiple screens.
The Digital signature should be visible in the document. Standard signature layout should be name, date and time.
4. PROTOTYPING
Using the results from the first user testing, and the functional requirements derived from it, a prototype of signing software was made. Software with a workflow that tries to overcome the usability issues that were found earlier on, the goal was to increase the ease of use.
As stated earlier on, interface elements can be trustbusters if not designed or working properly. Such trustbusters are poor usability, inconsistent design and long system response time. During the design process these aspects, amongst other factors, needed to be kept in mind. On the area of consistency, a ribbon structure alike in Microsoft Office programs was used.
4.1 Paper Prototyping
Paper prototyping was used as a start, by iteratively sketching interfaces and creating a general first idea. From the functional requirements listed in section 3.3.3, all points have been taken into the process of designing.
The paper prototypes are listed in the appendix.
4.2 Digital Prototyping
A digital prototype was made using Justinmind Prototyper [21]. Six screens were made, interlinked by clickable elements. The digital prototype was designed solely for the task of placing a digital signature. Some buttons that had nothing to do with this task were created, but nothing would occur if clicked on.
All the figures (screenshots) of the digital prototype are listed in the appendix.
The task scenario of the prototype is as follows: 1. User begins at the start screen (figure 17)
2. By clicking on the ‘Sign’ tab, the options in the second ribbon bar will change (figure 18)
3. By selecting the option ‘Place digital signature’ a pop-up message will appear (figure 19)
4. When the user clicks in the designated signing area in the document, the digital signing settings screen will show (figure 20)
5. If a user selects ‘Sign’ in that settings screen the digital signature will be placed (figure 21)
6. When a user clicks on the digital signature in the document a pop-up will show displaying that the signature is valid (figure 22)
5. PROTOTYPE EVALUATION
After finishing the digital prototype, it was time to evaluate it in order to try to validate the implemented functional requirements. The primary goal of the prototype was to improve the ease of use of placing digital signatures, in that area it was compared to Foxit Reader. Foxit Reader was chosen as a comparison, ‘the one to beat’, since it scored the highest on usability in the previous test. As stated in the related work section, software can be a trustbuster when the usability is below par. But since earlier results showed that the ease of use positively correlates with trust in the technology, indirectly the prototype also might attribute to higher trust of users.
Comparing the prototype to Foxit Reader, on the area of trust, would be an uneven comparison and might give biased trust scores. Asking participants any trust related questions about the prototype most likely would be influenced by the fact it is a (single task) prototype and not a full functioning (nor fully designed) program. On the area of usefulness, it is possible to perform the same exact task with Foxit and the prototype, so it wouldn’t have been any helpful to ask any question about this.
5.1 Method
The prototype was evaluated by comparing the digital prototype with Foxit Reader, by having participants perform the task of signing a PDF file with both systems. With the task specifically being: ‘Sign the test PDF file with a digital signature’.
New participants were given an explanation on digital signatures
and the difference in comparison to electronic signatures. All participants received a short explanation on the prototype, that it was just a demo aimed at one task and not a fully functioning program.
After first performing the task using Foxit Reader, participants had to give a score. Followed by performing the task on the digital prototype and grading it again. Both software needed to be graded by the participants using the Single Ease Question (SEQ). This SEQ was ‘Overall, how difficult or easy did you find this task?’. It needed to be answered on a 7-point rating scale, ranging from 1: ‘very difficult’ to 7: ‘very easy’. Despite its simplicity, the SEQ performs about as well or better than more complicated measures of task-difficulty [34].
Next to that all participants were asked on each specific functional requirement. For each requirement (as listed in section 3.3.3) they were asked if they found it helpful in the prototype, along with any possible comments they had.
5.2 Participants
Prototype evaluation was performed by 2 user groups. The first group consisted of 5 people that had participated in the first user testing, the second group is a group of 5 new participants. Of the new participants, the age, gender and highest level of education was noted. Next to that they were asked if they had any prior experience using digital signature.
In total, the age range was 20 to 54 years, with the average age being 30.3. The group consisted of 8 males and 2 females. Level of education of the participants was 2 times intermediate vocational education, 5 persons attended a university of applied sciences, and 3 participants on university level.
5.3 Results
The first group of participants were the same ones that participated earlier on. The SEQ mean score for Foxit Reader was a 4, for the
prototype the mean score was a 6.4. For the second group, the new group of participants, the average score for Foxit Reader was a 2.4, for the prototype this was a 5.8. One reason that likely contributed to this difference between the two groups was the fact that the first group had used Foxit Reader before (at least once) during the user testing earlier on in the research.
Having the two groups combined the mean score for Foxit was a 3.2, and for the prototype it was a 6.1. Overall, it shows the prototype had a higher perceived ease of use compared to Foxit Reader.
When the participants were asked about each specific functional requirement that was implemented the answers were very positive. For every requirement 9 or 10 out of the 10 participants thought that it was helpful and improved the usability. The only exception was related to the pop-up and selection of the area where a digital signature would be placed. Due to a limitation in the prototyping software it was not possible to create the possibility for participants to draw a rectangle to select the area where the digital signature will be placed. Therefore, the text on the pop-up was changed to ‘select the location […] by clicking on an area in the document’. Half of the participants preferred to be able to select an area by drawing a rectangle, instead of clicking.
Next to that two persons mentioned that they would like the option of ‘Lock document after signing’ in the signing option screen, which Foxit Reader does have. One person also would have liked to have a ‘reason’ field added to the options.
A comment made by another participant was that if the prototype aims at digital signing, the option ‘place digital signature’ should be the first option on the left in the ribbon bar.
Two participants made it clear that they disliked the ‘place ink signature’ button, they would rather have it being ‘place electronic (handwritten) signature’. Or that there would be more buttons present, like ‘draw signature’, ‘place electronic signature’ and ‘place digital signature’.
6. DISCUSSION
The surprising result of the first test was that both perceived usefulness and perceived ease of use didn’t appear to be indicators of intended usage. This undermines the Technology Acceptance Model.
Other metrics seemed better indicators though, like result demonstrability from TAM2 [37] showed to be a positive influence on perceived usefulness. PU is the extent to which people believe that the technology will help them perform (their job) better [7], figure 9 shows that if your perceived usefulness is higher, your trust in the technology probably will be higher too. The other way around though gives a strong indication of correlation, as can be seen in figure 5.
For the prototype evaluation only the Single Ease Question was used. The plan was first to use the System Usability Scale (SUS) [5], a widely used usability evaluation questionnaire. But in the end the SUS deemed to be too extensive for the prototype. The SUS is meant for people to browse through systems, explore options, etc. The prototype that was build was only designed to be able to perform one task.
One the trust requirements that came forward was assurance of the safety of technology. In the case of digital signatures, such assurances should not be in the software used to digitally sign. People need to gain the trust to (want to) use digital signatures before, or during, the request of a digital signature. The security assurances should be handled by a company itself when it would
start using digital signatures, or by a certificate authority when requesting certificates. If you would be using digital signatures you should trust it sufficiently already.
A possible limitation of this study was that most of the participants of both user testing rounds had received a level of education at university of applied sciences or higher, this might give the problem that the research doesn’t sufficiently surfaces any problems lower educated people might have. The average person or employee might not have received the same education level as the participant sample group.
Another limitation was that trust perception depends on personal and cultural factors. Age, gender and highest level of education were asked of the participants but not looked into, mainly because the sample size would be too small to make valid conclusions when splitting up the participants. Cultural factors were not taken into consideration in this research, but could be an interesting addition for future research.
7. CONCLUSION AND FUTURE WORK
In this research on digital signatures the focus was on trust and the technology acceptance of (potential) users. Integrating these two perspectives and examining the factors that build trust, advanced our understanding of these constructs and their linkages to intended use of digital signatures.Technological trust showed to have a positive correlation with perceived ease of use, and also with perceived usefulness. On the other hand, perceived ease of use proved to be not just a trustbuster but the findings implied that a higher value of perceived ease of use would raise trust in the technology. It appeared that the relationship between technology trust and usability is not one-way, it plays out in both directions.
The first user testing resulted into factors affecting trust such as brand, safety assurances, having knowledge about the technology and having personal experience with it. The second user testing showed that the PEOU of signing software can be improved by using the functional requirements that came forward from the first round of user testing. Next to that some new issues and comments did rose up during the second testing, which gives possibilities for further research.
Further research could be performed in multiple directions, one could be the process of building a fully working, high fidelity prototype that would look like ‘real’ software, incorporating the functional requirements that were validated, but also the improvement points that came forward from the second test. In this way multiple metrics of the prototype could be examined instead of just the PEOU.
Another dimension which could be interesting for future research are certificate authorities. Performing a study that dives into certification authorities and the process of requesting certificates.
8. ACKNOWLEDGEMENTS
I would like to express my gratitude to André Nusselder for his help and advice, for his guidance throughout the thesis process. I would also like to thank Frank Nack for being my second examiner.
9. REFERENCES
[1] Adobe Acrobat Reader DC (n.d.) Retrieved from https://acrobat.adobe.com/uk/en/acrobat/pdf-reader.html [2] Baldwin, A., Beres, Y., Mont, M. C., & Shiu, S. (2001). Trust Services: A trust infrastructure for e-commerce. HP
[3] Barack Obama signature [Illustration]. (2008, February 23). Retrieved from
https://upload.wikimedia.org/wikipedia/commons/1/11/Barac k_Obama_signature.svg
[4] Beldad, A., De Jong, M., & Steehouder, M. (2010). How shall I trust the faceless and the intangible? A literature review on the antecedents of online trust. Computers in
Human Behavior, 26(5), 857-869.
[5] Brooke, J. (1996). SUS-A quick and dirty usability scale.
Usability evaluation in industry, 189(194), 4-7.
[6] Castelfranchi, C. and Falcone, R. (2001). Trust and
deception in virtual societies. Norwell, USA.
[7] Davis, E. (1989), "Perceived usefulness, perceived ease of use, and user acceptance of information technology," MIS Quarterly, 13(3), pp. 319-339.
[8] Davis, F. D.; Bagozzi, R. P.; Warshaw, P. R. (1989), "User acceptance of computer technology: A comparison of two theoretical models", Management Science 35: 982–1003 [9] Digital Signature. (n.d.). Retrieved from
https://e-estonia.com/component/digital-signature/
[10] eIDAS regulation. Official Journal of the European Union. Retrieved from
http://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32014R0910&from=EN [11] Fishbein, M. Ajzen, I. (1975) Belief, Attitude, Intention, and
Behaviour: An Introduction to Theory and Research. [12] Foxit Reader (n.d.) Retrieved from
https://www.foxitsoftware.com/products/pdf-reader/ [13] Fukuyama, F. (1995) Trust: The Social Virtues and the
Creation of Prosperity. New York: Free Press, 1995. [14] Ganesan, S. (1994) Determinants of long-term orientation in
buyer-seller relationships. Journal of Marketing, 58, 2 (1994), 1-19.
[15] Gefen, D., Karahanna, E., & Straub, D. W. (2003). Inexperience and experience with online stores: The importance of TAM and trust. IEEE Transactions on
engineering management, 50(3), 307-321.
[16] Gefen, D., Karahanna, E., & Straub, D. W. (2003). Trust and TAM in online shopping: an integrated model. MIS
quarterly, 27(1), 51-90.
[17] Grabner-Kräuter, S., & Faullant, R. (2008). Consumer acceptance of internet banking: the influence of internet trust.
International Journal of bank marketing, 26(7), 483-504.
[18] Gupta, A., Tung, Y. A., & Marsden, J. R. (2004). Digital signature: use and modification to achieve success in next generational e-business processes. Information &
Management, 41(5), 561-575.
[19] Hertzum, M., Jørgensen, N., & Nørgaard, M. (2004). Usable security and e-banking: Ease of use vis-a-vis security.
Australasian Journal of Information Systems, 11(2).
[20] Hu, P. J.; Chau, P. Y. K.; Sheng, O. R. L. (1999), "Examining the technology acceptance model using physician acceptance of telemedicine technology.", Journal
of Management Information Systems 16 (2): 91–112
[21] Justinmind Prototyper (n.d.) Retrieved from http://www.justinmind.com/
[22] Keil, M., Beranek, P. M., & Konsynski, B. R. (1995) Usefulness and ease of use: field study evidence regarding task considerations. Decision Support Systems, 13(1), 75-91. [23] Leppänen, A. (2010). Technology trust antecedents: Building
the platform for technology-enabled performance. [24] Lippert, S. K., & Swiercz, P. M. (2005). Human resource
information systems (HRIS) and technology trust. Journal of
information science, 31(5), 340-353.
[25] Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995) An Integrative Model of Organizational Trust. Academy of Management Review 1995.20(3), pp 709-734.
[26] McAllister, D.J. (1995) Affect- and Cognitation-based Trust as Foundations for Interpersonal Cooperation in
Organizations. Academy of Management Journal 1995.38(1), pp 24-59.
[27] McKnight, D. H., Cummings, L. L. & Chervany, N. L. (1998). Initial trust formation in new organization
relationships. The Academy of Management Review. 23(3)
473-490.
[28] Misiolek, N. I., Zakaria, N., & Zhang, P. (2002, November). Trust in organizational acceptance of information
technology: A conceptual model and preliminary evidence. In Proceedings of the decision sciences institute 33rd annual
meeting (pp. 1-7).
[29] Nitro Pro 10 (n.d.) Retrieved from https://www.gonitro.com/pro
[30] Pikkarainen, T.; Pikkarainen, K.; Karjaluoto, H. (2004), "Consumer acceptance of online banking: An extension of the Technology Acceptance Model.", Internet
Research-Electronic Networking Applications and Policy 14 (3): 224–
235
[31] Reichheld, F. F. & Schefter, P. (2000). E-Loyalty: Your Secret Weapon on the Web. Harvard Business Review, 78(4), 105-113.
[32] Riegelsberger, J., & Sasse, M. A. (2002). Trustbuilders and trustbusters. InTowards the E-Society (pp. 17-30). Springer US.
[33] Rotter, J.B. (1971) Generalized expectancies for
interpersonal trust. American Psychologist, 26, 5 (May), 443-450.
[34] Sauro, J., & Dumas, J. S. (2009, April). Comparison of three one-question, post-task usability questionnaires. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1599-1608). ACM.
[35] Taylor, S., & Todd, P. (1995). Assessing IT usage: The role of prior experience. MIS quarterly, 561-570.
[36] Taylor, S., & Todd, P. A. (1995). Understanding information technology usage: A test of competing models. Information
systems research, 6(2), 144-176.
[37] Venkatesh, V., & Davis, F. D. (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management science, 46(2), 186-204.
[38] VeriSign-branded CAs and RAs within the Symantec Trust Network (STN). (2015, September 04). Retrieved from https://www.symantec.com/page.jsp?id=ca-ra
[39] Wu, J. H.; Wang, S C. (2005), "What drives mobile commerce? An empirical evaluation of the revised technology acceptance model.", Information and
10. APPENDIX
10.1 First User Testing Questionnaire
Age Gender
Highest level of education
Social trust (ST)
I generally trust other people I tend to count upon other people I generally have faith in humanity I feel that people are generally reliable
I generally trust other people unless they give me reason not to Institutional trust (IT)
I tend to trust large companies or organisations I trust certificate authorities to be secure and thorough I place trust in Root CA's that are governmental
I place trust in Root CA’s that are commercial companies Prior experience
I have prior experience in using digital signatures (Answer possibilities are ‘Yes’ or ‘No’)
(After performing the tasks) Compare the three PDF readers:
Which program did you find the most appealing? (give the name of the program)
Which program did you find the most trustworthy? (give the name of the program)
Perceived ease of use (PEOU)
I find it easy to get the system to do what I want it to do (answer this question three times, one time for each program) Perceived usefulness (PU)
I find digital signatures to be useful.
I think digital signatures have strong advantages compared to electronic signatures
Technology trust (TT)
I trust that a PDF file cannot be changed after it is signed
I trust a PDF file to be actually signed by the person that the certificate shows
Result demonstrability (RD)
I have no difficulty telling others about the results of using digital signatures.
I believe I could communicate to others the consequences of using digital signatures.
The results of using digital signatures are apparent to me.
I would have difficulty explaining why using digital signatures may or may not be beneficial.
Behavioural intention to use (BI)
I would use digital signatures (in a work environment) Two open questions:
Which layout of the digital signature do you prefer? What should be visible?
Do you have any factors that are not mentioned yet, which can influence your trust or user experience related to digital signatures?
10.2 Paper Prototypes
Figure 13. Start screen
Figure 15. After clicking on ‘digital signature’
10.3 Digital Prototype Screenshots
Figure 17. Start screen
Figure 19. After clicking on ‘place digital signature’
Figure 21. Digital signature is placed
