3 Pilot prototype
3.5 Pilot study
3.5.2 Results
At the end of the pilot period a total of 59 people were registered in the 1 MORE
database, 18 of which still had a UserLevel of O. These people either never called to the 1 MORE service or never got any further than the 'first time information' prompt. The other 41 users successfully connected 242 times to the service. This is an average of 5.9 logons per active user or 11.5 calls per day.
A user was connected to the InterfaceOverflow menu (i.e. another user occupied the system) 15 times. Not once did a user receive a busy signal because of the unavailability of an InterfaceOverflow object.
A total number of 24 calls had to be rejected because they were performed with an invalid phone: either the user's CLIP (Calling Line Identification Presentation) was off (14 times), the user was using a fixed phonenumber (10 times) or the mobile caller was not
subscribed yet (7 times).
A total of 13 times some sort of error occurred during the 1 MORE service. In five cases, all of which occurred in the first 48 hours and three of which were debug sessions
performed by a Garage member, the system was unable to create TIS (Text-To-Speech) data because of a programming error. After this bug was fixed, the problem never occurred again. All other errors were due to incorrect Favo Friends information. In one case a user had created a profile but no friends or folders. The system reacted by rejecting the Friends objects results and disconnecting the user. The other cases were all due to a name with a digit, which is rejected by the speech recogniser. The 1 MORE homepage did not check for digits in names or folders, but this would have prevented these errors. Aside from the TTS error, the program is bug-free. It shows no unexpected behaviour and does not appear to have any memory leaks. It is able to run continuously for more than 21 days without any problems.
The table below recapitulates these results. Since the application is bug-free (at least during most of the pilot) the results from the pilot can be considered valid.
Table 3-3 Results from the pilot study
Total calls Busy Rejected Invalid data Error Successful 289 15 (5.2%) 24 (8.3%) 8 (2.8%) 5 (1.7%) 242 (84%)
If all KPN Research employees, including Garage-members, are removed from the pilot data only 27 active subscribers are left. These subscribers made a total of 101 successful calls that lasted for more than 20 seconds; shorter calls were removed from the results, as they were useless. Eight persons called only once and two people called 10,
respectively 15 times. If the last two are excluded from the data, the average is 3.0 (76 calls), see Figure 3-9. This means that the average user called 1 MORE once a week!
Figure 3-10 shows that a 1MORE call takes about 3 to 4 minutes; 22 conversations took more than 5 minutes, whereas 56% of the conversations were shorter than 2% minutes. If a user calls 1 MORE to discuss a time for an appointment, the conversation mostly last shorter than 5 minutes. If 1 MORE is used for a 'social call', conversations of 10 minutes or more are no exception.
P.P.S. Giesberts 35/128
ICS/EB 750 1MORE November 2000
Figure 3-11 shows that in a relatively large part of the conversations the user was not connected to any friends (conference size
=
1) or to just one (size=
2). If the user was not connected at all, he might have been just trying the service, using the Help menu or he may have called one or more people that could not be reached. In the case of the two-party conference, the user may have used 1 MORE as a voice dialling system or he may have called additional people that were not available. If the size-one calls are eliminated, the average number of conferees per conversation is 3. There were no calls with more than 5 conferees, whereas the system could handle up to 8!Figure 3-9 Number of sessions per person
~~ .---.
Figure 3-10 Duration of a session
~ 40
Figure 3-11 Size of a conference
P.P.S. Giesberts
1MORE November 2000
leS/ES 750
On average the service is used 4 to 5 times a day (Figure 3-12). If the first days are excluded, the average is almost 4 sessions a day. There is no difference between the use in the weekend or during weekdays. This is a remarkable result; the service was
expected to be used particularly in the weekend, since this is the time most youth go out to meet each other. p.m. This is also rather unexpected; the team had expected the service to be used mainly from 6 to 9 p.m.
Figure 3-13 Usage per time of day
21 to 24
After the pilot period several subscribers were asked for their experiences with 1 MORE.
Most people's view of 1 MORE was either positive or very positive. It is considered user-friendly, but this depends heavily on the quality of the speech recognition. The speech recognition is good, except when there is a lot of background noise. This is especially annoying when other conferees continue their conversation during the recognition phase.
This could be solved by temporarily disconnecting the initiator from the conference while the IVR system is active.
The fact that the user cannot disconnect a conferee is, as expected, seen as a more important problem. In particular the fact that friends are saddled with minute-long meaningless voicemails is a critical inadequacy.
Some users reported bothersome tapping during the conversation. The taps are created by the conference card and are typical to the PBX-solution. From time to time, the audiofiles that are a part of the IVR menus suffer slightly from noise on some types of telephones, both mobile and fixed. This probably has something to do with the codecs used in the different phones, but might be avoided by altering the format and quality of the audiofiles.
P.P.S. Giesberts 37/128
ICS/EB 750