Web based interface for a smart home

(1)

Web based interface for a smart home

by

Sardar Yumatov

in Partial Fulfillment of the Requirements for the Degree of Bachelor of Science in Computer Science

at the University of Groningen

Faculty of Mathematics and Natural Sciences November 30, 2011

Thesis Supervisors:

Dr. Alexander Lazovik, Eirini Kaldeli, Ehsan Ullah Warriah

(2)

Abstract

The technical design and the implementation of a graphical user interface is of particular importance to the fulfillment of user-centric and adjustable smart homes, and has to address a number of challenges, including support for platform independence, usability, and integration with the devices available at the middleware. In this thesis, the design and implementation of a user interface is described, which has been integrated in the service-oriented platform developed in the context of the European project SM4ALL (Smart Homes for ALL).

The user interface should be available on a broad range of devices ranging from the mobile phones and portable computers to traditional desktop PC’s. The software should be capable to run on various software platforms, including iOS, An- droid, Windows, Windows Phone and Linux platforms. After comparing various methods, this paper will focus on the web-based implementation, that looks as the most platform independent approach at this moment.

The usability is one of the main issues addressed by this paper. The quick access to the frequently used items through the bookmarks, the filter of irrelevant information and the support for various screen sizes will be discussed in the context of usability. However, this paper will not discuss the proper graphical design, which should be the topic of a separate research.

One way to control the smart home is to declare the desired effects and let the SM4ALL middleware figure out necessary actions needed to achieve them. The set of declared effects is called a goal. The graphical goal editor is part of the user interface described in this paper. The goal editor should encourage the users to automate their daily tasks, so it should be simple and user friendly. This paper will discuss the requirements of such a goal editor and the proposed web-based implementation in detail.

The evaluation will focus on the implemented features, which include the screen calibration, filter by relevancy and the goal editor. Unfortunately, the prototype shipped with this paper is not fully integrated with the SM4ALL services, so not all of the planned features are implemented and tested. The dropped features include the direct control of the devices and showing the actual state of the devices to the user. These features were dropped because the integration with the services related to pervasive layer, such as COPAL, is missing. The future software releases will be fully implemented and properly integrated with SM4ALL services once these services are in turn fully implemented and brought together.

(3)

Acknowledgements

I am grateful to prof. Marco Aiello for letting me make this project. Many thanks also to dr. Alexander Lazovik for help and patience. I am most indebted to Eirini Kaldeli for helpful comments on the first draft of this paper. Many features would not have been implemented in the software without her help. Special thanks to Ehsan Ullah Warriah for help with the early draft of this paper. My colleagues at Getlogic were great source of experience regarding the initial prototype, so I would like to express my appreciation to them too.

(4)

Declaration

I declare that this thesis was composed by myself, that the work contained herein is my own except where explicitly stated otherwise in the text, and that this work has not been submitted for any other degree or professional qualification except as specified.

Groningen, November 30, 2011

Sardar Yumatov

(5)

(6)

List of Figures

1.1. Architectural overview of SM4ALL network . . . 2

2.1. The AAI and related services . . . 10

4.1. The sign-in view. . . 22

4.2. The sign-out view. . . 22

4.3. The screen calibration view. . . 23

4.4. The list of rooms . . . 24

4.5. The list of goals . . . 24

4.6. The profile view page . . . 25

4.7. The profile edit page . . . 26

4.8. The list of goals . . . 27

4.9. The screen confirming deletion of the goal . . . 27

4.10. The properties editor . . . 28

4.11. The editor’s canvas . . . 29

4.12. The dialog with all language constructs . . . 30

4.13. The dialog with all services . . . 31

4.14. Forcibly generated invalid goal. . . 32

4.15. The TV service with its state variables . . . 32

4.16. The power states of the TV . . . 33

4.17. Advanced expression editor . . . 34

4.18. Simple and complex predicates . . . 34

4.19. Two optional conditions chained together. . . 35

4.20. Generic system message, in this case confirming the goal change. . . 35

4.21. The home page of the administration panel. . . 36

4.22. The list of all state variables. . . 36

4.23. The edit screen of a state variable. . . 37

6.1. The overall architecture of WebUI . . . 43

6.2. The architecture of the UI server . . . 46

6.3. The architecture of the initial prototype . . . 47

6.4. The value picker of the integer domain handler. . . 51

6.5. The value picker of the choices, scoped to the Lamp state. . . 51

6.6. The advanced expression editor. . . 53

7.1. The Reading goal . . . 59

7.2. The Watch TV goal . . . 60

(9)

(10)

1. Introduction

A world full of computers which you can’t understand, Can’t fix and can’t use is a world controlled by machines.

- Eben Moglen, 2006 - The Smart Homes For All (SM4ALL¹) is a design of a middleware platform for cooper- ative working of various services, together realizing a domotic infrastructure that is highly interactive and adaptive to different houses and users. It makes use of dynamic service composition and semantic techniques in order to guarantee dynamicity, dependability, and scalability, while preserving the privacy and security of the home and its users.

The SM4ALL network can be architecturally split into three macro layers as shown in Figure 1.1. At the bottom is the pervasive layer, where the heterogeneous sensors, actuators and mobile devices of the house live. In the middle sits the composition layer, which is responsible for registering and inferring the state of the home, as well as coordinating it on behalf of the user. On top is the user layer which provides the interface to the home.

The user interface, clear and easy to use for non-experts, is one of the many challenges SM4ALL project is facing. There are various kinds of user interfaces possible, including brain-computer interface [1] and speech-recognition based interface. The aim of this work is the design and the implementation of a graphical user interface, that is capable to present the user various services a Smart Home can provide in a usable and unobtrusive way. The various aspects will be discussed through the paper, including platform independence and usability issues further described in Chapter 3 and the integration with the existing services of SM4ALL middleware explained in Chapter 5. The main focus will be on scalability, availability and security of the web based software, leaving the issues of the proper graphical design open. The prototype, that is shipped with this paper, is using a simple grid-based graphical design that will be further discussed in Chapter 4.

This paper will discuss the graphical user interface, accessible via modern mobile devices like smart-phones and tabled devices and via traditional desktop PC. The variety of hardware platforms makes the design and the implementation of the common graphical user interface rather challenging task. The hardware varies greatly in features, most notable are the screen size and networking capabilities. The programming is often vendor specific and the publication of the software is further hindered by the vendor’s 3rd party software policies. For example, iOS based devices, like iPhone or various Windows Phone 7 devices can not run the software not explicitly approved by the vendor. It is often possible to crack the phone, but doing this will render the product’s warranty void. Another option is to buy the developer’s license, but this can be an unacceptable requirement for just running the software. After comparing advantages and disadvantages of various approaches this paper will focus on the web-based implementation. The technical decisions will be discussed in detail in Chapter 6.

1Smart hoMes for All http://www.sm4all-project.eu

(11)

1. Introduction

Figure 1.1.: Architectural overview of SM4ALL network

A working prototype will be demonstrated and the results will be discussed in Chapter 7.

All implemented features like user profiles, relevancy filters, screen calibration and goal editor will be discussed and tested. The integration with the planner service will be also discussed. There is a room for improvements which will be discussed in Chapter 8. All missing features that are described in Chapter 6 will be eventually implemented and tested in the future software releases.

(12)

2. Problem statement

The aim of this project is to design and create a working implementation of the graphical user interface running on the common mobile devices like cellphones and tabled devices.

The interface should be usable on the desktop PC too. The main focus is on the overall architecture of the web-based implementation, addressing the security and scalability issues and interoperability with other SM4ALL services. The proposed implementation is a web application running in the client’s web-browser. This decision will be further explained in the following sections, together with other aspects related to the resulting software design.

2.1. User interface

The ultimate goal of the user interface is to offer the user a simple, unobtrusive and user-friendly access to the functionality the smart home services provide. The typical set of features, that user interface should support, contains the following list:

• Device explorer - offers the user-friendly navigation within the possibly large set of devices the user has access to.

• Direct interaction with the devices - the possibility to directly control the devices.

For example, to set the lamp on.

• Graphical representation for all available devices that corresponds to the function and the current state of the related device.

• User-friendly dialogue between the user and the SM4ALL middleware.

• Goal editor - used to express and record complex tasks in a user-friendly way, which will encourage the users to automate their daily needs.

The following subsections will further discuss each of these features in details.

2.1.1. Device explorer

The smart home might contain hundreds of devices directly available to the user, ranging from the simple actuators, built in the doors and windows, to the dynamically discovered mobile devices, like cellphones. Some of the devices might be the source of the information only. For example, a clock or a fire alarm. Other devices might be controllable by the user, like an electronic chair or a TV. The primary job of the device explorer is to offer all of these devices to the user in a user-friendly way. The proper graphical design of such device explorer is not discussed in this paper as it falls beyond the scope of this thesis.

However, the key-features of a user-friendly device explorer will be discussed, which are:

(13)

2. Problem statement

• Device catalogue - groups the devices by function, location or by any other common property.

• Bookmarks - one of the methods to optimize the access to frequently used functionality.

• Relevancy filter - the devices that are not relevant to the user due to personal disabilities should be hidden by default.

The explorer should group the devices by location, function or by any other property, so the users will be able to reach any device quickly. Frequently used devices can be added to the bookmarks, which should be location specific.

The user might have personal disabilities that can prevent the effective usage of some devices. For example, the user with the hearing problems might be not interested in a stereo system. The user might declare such disabilities, which will help the user interface to hide irrelevant devices. However, it should always be possible to ignore the relevancy filters and reach any device if the user explicitly wants to.

2.1.2. Direct interaction

The users should be able to directly control any device available to them from any place in the house. For example, it should be possible to switch off the light in the kitchen while sitting in the living room. The devices are represented by the services, which are further described by the service definitions stored in the central semantical repository [3].

The definition of the service includes the state variables that represent the state of the device and the actions that make it possible to change the device state. The ability to directly drive the device is challenged by the fact, that the low-level actions of the device do not always need to correspond to the high-level wishes of the user. A simple device might have simple actions, for example, a lamp might have simple setState(ON|OFF) action. However, complex devices might expose low-level actions, that should be properly composed together. The goals might be used in this situation, which are briefly introduced in Section 2.1.5.

2.1.3. Graphical representation

The graphical representation of the device should not only correspond to its function, but also to its current state, which should be always kept synchronized with the state of the real device. The changes in the device state should be propagated as quickly as possible.

This requires the proper integration with the services related to the pervasive layer. The device context should contain detailed meta-information about the state of the device, which might be organized hierarchically. For example, the ON state of the TV might further contain information about the current program, volume settings and how long the TV is already on. The information about the program might further contain details about that program, including the icons, description, rating, age restrictions and other suggested programs. This state information should be exchanged in a efficient way, allowing the state change events to contain the most basic information only. The more detailed information, related to the state change event, should be available on demand. This design will allow a fully featured graphical user interface to obtain and display the user the most of the

(14)

2.1. User interface

relevant information, while other user interfaces, like the brain-computer interface, might use only the basic part. The integration with the SM4ALL services is further discussed in Chapter 5.

2.1.4. User-friendly dialogue

The graphical user interface can be accessed from different devices, including the mobile phone carried by the user and a wall-mounted touch-panel. In both cases the user might be not available for the feedback when some command is being executed. Once the command is sent, the user will often walk away from the wall-mounted touch-panel or just close the interface running on the phone, ignoring the fact that his/her command may fail. When a goal or a command is sent to the composition layer, the SM4ALL middleware tries to find out how to achieve it [3]. If the chosen solution can not be carried out, for example, the hardware failure had happened, the composition layer will backtrack and try to find another solution. The new solution may ask some extra information from the user, for example, to choose another topping for pizza if previously chosen one is not available.

The composition layer must be designed to handle the situation when the user is not directly available for the feedback. The graphical user interface must be designed to queue and keep persistent the messages coming from SM4ALL network in order to present them to the user when he/she comes back. However, this kind of interaction requires the properly defined protocol between the graphical user interface and the composition layer.

The protocol should not only allow the SM4ALL network to notify the user via messages and query any kind of additional information, it should also be possible to cancel and clear the previously sent requests, if they become not relevant anymore. For example, the query for a pizza topping should be canceled, if the goal to order the pizza fails after 10 minutes waiting for the user’s response. The user should be only notified that his/her goal was canceled with a brief explanation.

2.1.5. Goal editor

Driving single devices through a user-friendly interface is not the primary goal of the Smart Homes for All project. The idea is to let the users declare the effects and results they want and let the system find out how to achieve them [3]. The goal editor must be very user friendly to encourage the users to automate their daily tasks.

Every device in the smart home can be represented by a set of state variables and actions, closely related to UPnP standard¹. A device in a desired state can be represented by a predicate declaring all state variables with specific values. For example, a switched on lamp can be represented by lamp-id:light=ON.

The goal is a formal declaration that describes one or more devices in a specific state.

The composition layer figures out what actions should be performed to bring the devices to the declared state. For example, the single action switch on lamp-id might be the result of a simple goal Achieve(lamp-id:light=ON).

The goal editor is a graphical tool that helps the users define the goals. The main building blocks defined by the language [3] are the achieve, find-out, condition and condition- optional predicates. The condition and condition-optional predicates enclose other pred-

1Universal Plug and Play http://www.upnp.org

(15)

icates including themselves, which can be unlimitedly deep nested. The syntax can be parsed into the usual abstract syntax tree that can be represented graphically using nested blocks. The goal editor is basically a canvas that renders the goal as the set of nested blocks and makes it possible to edit the goal by adding and removing blocks.

The achieve and find-out predicates enclose the state-variable expressions of the form:

<state-variable> <comparison> <value-expression>

The state variable is compared against the value-expression using common comparison operators: equal, greater, smaller-or-equal etc. The value-expression consist of operators, constant values an references to other state variables. The state-variable expression can be represented by a block enclosing the related value-expression. The expression of the form state-variable = constant-value is a special case. Instead of showing the expression in a generic way, it can be replaced by the icon related to the constant-value.

Chapter 4 discusses further the graphical aspects of the goal editor.

2.2. Software design decisions

The device and platform independent software can be implemented using various approaches. All of them have their advantages and disadvantages. The proposed implementation is web-based, however this decision will be further justified in the following sections.

2.2.1. Standalone vs client-server application

The SM4ALL middleware is designed as a set of services available via SOAP, REST and similar protocols. These services can be accessed though the gateway server or directly from the client device, that is running standalone client application. When all of the logic resides on the client device, the software is forced to handle all possible application level protocols the SM4ALL services use. The protocols and services do change often because SM4ALL middleware is still in development. The standalone version is forced in this case to remain being constantly in development. This issue increases the maintenance costs, because constantly updating lots of different versions of the standalone application, used by many different users, is a hard task that requires full time work of a small team of developers. The people usually do not like to update the software they have once installed until they get frustrated by the failures of the current version.

With the client-server approach the most of the logic reside on the gateway server. The client application becomes much simpler handling graphical and networking aspects only.

The changes of SM4ALL middleware in many cases will not affect the client application, only the gateway server needs to be updated. The maintenance complexity and the costs will be greatly reduced because the gateway server is much easier to update than the large number of client devices owned by various people.

Many mobile devices have a Wi-Fi and similar networking interface used to connect to the local network. Such interface will make the communication between SM4ALL network and the device possible, assuming the user will enter the Smart Home. The devices that have only internet connection² can use a public secure gateway to the SM4ALL network.

2Devices with 3G or similar network only. Currently most of the smart-phones are such devices.

(16)

2.2. Software design decisions

The gateway server can implement all needed security, preventing the SM4ALL services being exposed to the open Internet. Without the need to handle security issues the implementation of SM4ALL services can be greatly simplified.

2.2.2. Native vs. web-based application

Making the software for one particular device, especially for one vendor exclusively, is relatively easy because the standard toolkits are available for common programming idioms, that are implemented in a way specially tuned for the chosen device. This is most important for the GUI as it should preserve the look-and-feel of the host operating system. The various GUI builders, both from the device and 3rd party vendors, make the job much easier.

There are three major disadvantages of a native application running on the mobile device:

• It must be installed manually on every device used to access SM4ALL network.

This raises issues with the hardware compatibility, software platform compatibility, software updates etc.

• The device vendor may have special policies restricting the usage of 3rd party software.

• It might be more vulnerable to software patents in the countries where such patents are applicable.

There are few cross-platform frameworks that support the required broad range of devices and support look-and-feel of the host platform. Traditionally Java³ is used when cross-platform software is needed. However, it might be easier and better to drop the cross-platform support and write the truly native application for every family of devices.

Fortunately, most of the modern mobile devices run iOS⁴, Android⁵or Symbian⁶operating systems. By limiting the development for these three platforms the majority of the mobile devices market will be covered. The native application can handle the device sepcific input hardware better. This might be an important feature on mobile devices where the support for common gestures on a touch screen is expected. This is an essential requirement to support the look-and-feel of the host operating system.

The client for a desktop PC can be built on top of Java, Qt⁷ or any other cross-platform framework. All of these platforms have great support for the look-and-feel of the traditional desktop OS like Windows, Linux flavors or Mac OS X.

The device vendor may require all of the software to be installed from the vendor’s software distribution networks. This implies that SM4ALL UI should be published on all of such networks. This does not only increase the management costs as it requires time to create a package according to vendor’s specific rules, a fee might be required too for the publication of the software. Apple requires approval before the software can be published

3Java Platform http://java.com

4Apple iOS http://www.apple.com/ios/

5Google Android http://www.android.com/

6Nokia Symbian http://symbian.nokia.com/

7Cross-platform application and UI framework http://qt.nokia.com

(17)

on their AppStore⁸ and is known to block the software⁹ they do not like. Microsoft’s Windows Phone 7 based devices have the same restrictions too¹⁰. Usually the device vendors simply reject the software not targeted to the broad public. The SM4ALL user interface is exactly the kind of software usable only in a very specific environment – the SM4ALL network. Direct installation is often not possible or renders the device warranty void. Apple and Microsoft require the developer license to install the software form other sources than their distribution networks.

The native application, running on the mobile device, might employ obvious techniques like automatic software updates¹¹or inclusion of 3rd party content like streaming a video inside a widget¹² representing a TV. In some countries, most notably the United States, such techniques are usually patented. Even support for multi-touch gestures¹³is patented, including popular and obvious gestures. This is still possible because the mobile software platforms are young, there are few open industrial standards protecting the software and developers from the patent trolls. By using a web-based approach the application is just a dynamic content running in a web-browser. Other patents may apply in this situation, but brief look on the latest news of 2006-2011 years does not reveal many of them.

Most of the mobile devices include a web-browser in the default collection of the software distributed with the device. The web-browser gives the access to the web applications published on the internet. The modern web-browsers support very broad set of graphical capabilities, making it possible to create a graphical user interface of any complexity. The traditional features of HTML¹⁴ and CSS¹⁵are enough to properly style the graphical interface, while the HTML5¹⁶features like canvas¹⁷make it possible to build sophisticated widgets, that can display anything including a video stream. Implementation of the same graphical features in a native application often requires much more effort.

The modern web-browsers support advanced networking using WebSockets¹⁸, which allow the web-based client to keep a persistent connection for two-way communication with the remote host. More traditional methods like JSONP¹⁹and plain old page refresh can be used at the cost of added latency if more advanced methods are unavailable.

The available graphical and networking features make it possible to implement the UI entirely as a web-page. The advantages of the web-based approach are:

• Requires no installation, any device with a modern web-browser available is supported.

8Apple’s AppStore http://store.apple.com/

9For example Opera Mini http://operawatch.com/news/2008/10/

apple-blocks-opera-mini-on-iphone.html

10Microsoft’s Marketplace http://www.windowsphone.com/marketplace

11United States Patent 5,764,992 June 9, 1998, granted to Apple Computer, Inc.

12Search Google for "Streaming video patent", they cover practically all sensible use for video streams.

13United States Patent 7,966,578 June 21, 2011, granted to Apple Inc.

14HTML 4.01 specification http://www.w3.org/TR/1999/REC-html401-19991224/

15Cascading Style Sheets Level 2 http://www.w3.org/TR/CSS21/

16HTML 5 specification draft http://dev.w3.org/html5/spec/

17The canvas element http://www.whatwg.org/specs/web-apps/current-work/

multipage/the-canvas-element.html

18The WebSocket API http://www.w3.org/TR/websockets/

19Cross-domain AJAX http://www.json-p.org/

(18)

2.2. Software design decisions

• Client logic is very light, implementing graphical and basic networking features only.

The complex logic required to communicate with other SM4ALL services can be implemented on the server.

• The traffic between the client devices and the SM4ALL services can be minimized resulting in the longer battery life.

• The graphical user interface can be of any complexity and of any style.

• Desktop integration is possible for modern mobile platforms.

The first two advantages mentioned above make a web-based UI really stand out comparing to traditional approaches. The lack of software installation makes it possible to use the UI right away for any guest entering the house, assuming he or she has all required permissions. There is no need to support specific devices, instead only a limited set of layout rendering engines and screen sizes should be supported. A web-page is scalable by design, so handling varying screen sizes is relatively easy. Modern layout engines, like Gecko²⁰, used by Firefox and Seamonkey, or WebKit²¹, used by Chrome and Safari, follow open web standards²², that simplify the development. In fact, the reference implementation shipped with this paper is developed entirely using open web standards only, without relying on the layout engine specific features.

The Web UI²³is intended to run in a web-browser. The modern mobile platforms often offer the web-browser as an embeddable component²⁴. With minor changes the Web UI can be loaded and offered directly like a native application without the need to explicitly start a web-browser.

The possible disadvantages of using web-based UI can be:

• The look–and-feel of the web-based UI does in most cases not match the look–and- feel of the host operating system.

• The user experience depends on how user friendly is the web-browser, that is installed on the mobile device. Some devices lock all the features while the web- browser is active. Switching from the web-based UI to something else may require starting and stopping the web-browser. The user will likely be frustrated by such experience. Fortunately, such user unfriendly devices are rapidly decreasing minority on the market.

• Very old or very rare web-browsers might not be supported. However, this is very unlikely to occur in practice as custom web-browsers tend to use popular layout engines and most of them are supported.

The various devices and software platforms use different styling and input methods.

The users themselves my install custom themes and further tweak the graphical design of

20Gecko layout engine https://developer.mozilla.org/en/Gecko

21WebKit layout engine http://www.webkit.org

22The World Wide Web Consortium http://www.w3.org

23The graphical user interface shipped with this thesis will be further called Web UI.

24Google Android and iOS (iPad, iPhone) platforms offer WebKit as a WebView/UIWebView components that allow building an application almost entirely with HTML and JavaScript.

(19)

Figure 2.1.: The AAI and related services

the host operating system. Matching the look-and-feel of the host platform is impractical.

The web-based UI will look much simpler and less animated than the native UI. However, if the UI is clean and usable, then matching host’s look–and-feel will probably remain a least-priority issue.

The web-browsers evolve very rapidly, already supporting hardware accelerated graphics and video. The JavaScript performance is boosted thanks to the modern engines supporting JIT, that allows very complex and large scripts to run comfortably fast. This opens a large room for improvements and new features, that can be added later to the reference implementation provided with this paper.

2.3. Existing solutions

An attempt to design the user interface for SM4ALL project is described in [2]. The authors proposed the middleware called Abstract Adaptive Interface, designed to assist different kinds of user interfaces with the information about the available services and allow the UI to trigger actions. Figure 2.1 shows the proposed setup. The User Interface registers itself by AAI and queries the available devices and actions. The AAI queries the user’s profile from the Composition Layer (CL), then decides what devices and actions are permitted for current user. Every time a service is started, stopped or changes its state, the Composition Layer notifies AAI about the changes. The AAI maintains the list of available services and their states, updating it every time the new event is received from CL.

The initial idea was to re-use the functionality AAI provides and extend the UI component with the new features. However, there are major design limitations discovered, which have lead to the development of an independent solution.

The AAI is supposed to provide the UI all currently relevant actions ordered by priority.

Therefore, the corresponding API call AAI.getAvailableActionPartialOrder(UserID) should be used. The resulting list of actions is annotated with the location information.

The UI is expected to filter out the actions that do not match the current user’s location.

Every time the state of any device changes, the AAI will query all connected UI’s to update the lists of actions presented to the user by calling UI.updatePartialOrder().

All UI’s will then re-fetch the list of available actions from AAI. The UI is unable to fetch only the actions related to the current location of the user. The AAI is not allowed to filter the list of available actions according to the current location of the user, obtained from any tracking device that might be installed in the smart home or worn by the user.

Otherwise, it will be not possible for the user to reach the devices in other rooms remotely.

This limitation will conflict with the requirement to control any device in any room while sitting in any other room, declared in Section 2.1.2. So, the AAI is forced to always dump the actions of all available devices to all connected UI’s every time any device changes its

(20)

2.3. Existing solutions

state. If there are lots of devices, then AAI might be constantly busy with sending the big list of actions to all UI’s individually, causing high traffic in the SM4ALL network, which will lead to the reduced battery life of the connected mobile devices, especially if Wi-Fi is used for networking.

Given the current API, the user can not declare his/her own preferences for specific devices. For example, the user with limited hearing might be not interested in a CD player or the user frequently watching TV might like to have the TV controls sorted first in the list of available actions. This limitation is more related to the API design, than the overall architecture. The AAI might get all of the user’s capabilities with the profile obtained from the Composition Layer. How the CL gets this profile is the open question.

The UI might further sort the obtained list of services according to the user’s preferences and usage statistics. However, this will introduce more complexity to UI and reduce the need for AAI.

The UI can invoke any action using the API call AAI.actionHandler(actionID, parameters). The current implementation of AAI accepts single real number only as the parameters argument. The future releases might implement the possibility to provide more parameters and of various types. However, currently, this limitation greatly reduces the set of supported devices.

According to [4] the composition layer may ask the user interface to resolve problems in case of a failure. The execution can recover, if failed action can be replaced by another action of similar effect. However, there can be interaction with the user involved, if the replacement action requires additional information or might be less convenient to the user. The current API proposed in [2] does not allow the SM4ALL services to query any information from the user.

The choice of a SOAP based event dispatcher virtually limits the number of UI’s that can be concurrently processed. Every UI must have a persistent SOAP connection to AAI in order to receive the notifications. The AAI is forced to handle every UI separately. A message broker, for example, any flavor of *MQ²⁵, might be more efficient. This is not a design issue, it is an implementation issue. Future prototypes can use other kinds of communication between the services.

The underlying assumption on which AAI relies on is that devices fully expose their state by a finite state graph. Every state is then labeled with an unique integer. Every time the device state change, the AAI expects to be notified via AAI.updateService(serviceId, newStatusID) message. This may become problematic in cases where the device can have virtually unlimited number of states. The state of the smart-phone varies with the software being currently executed, the state of the TV varies with the available programs.

The single integer representing the new state of the device is unable to provide enough information.

There are other issues related to the current API, but not to the overall design. Some to mention are:

• The devices are represented by the actions labeled with the unique integers. A device with multiple actions, like a TV with on/off switch and a channel selector, will be represented by two actions grouped by the service id. The current UI implementation is forced to place these two actions in different cells on the grid, not necessarily close to each other.

25For example, Apache’s ActiveMQ or RabitMQ.

(21)

• The services are labeled with the unique integers, which are less flexible than the textual identifiers the rest of SM4ALL network may use. At this moment the devices tend to be referred by UPnP like identifiers. A separate mapping is therefore needed.

• The AAI requires a user profile labeled with the unique integer. There can be a special profile for anonymous users. It is not known, how a user can get to know his/her profile ID, the UI is assumed to know it somehow. It is not defined how the UI can let the user modify own profile and store the changes, such that they will be propagated to CL, which in turn makes it available to AAI.

Taking in consideration the issues mentioned above, the graphical user interface implemented for this project is not using AAI and has its own design described in Chapters 3 and 6.

(22)

3. Requirements

The graphical user interface for the SM4ALL project should meet the requirements that ensure the resulting software will be user friendly, run on common mobile devices and will have efficient and clean software design. The resulting set of requirements can be used for quality control of the final implementation. This chapter addresses non-functional and functional requirements the software must comply to.

3.1. Usability

The smart phones, laptops, tabled and other mobile devices share the common property: they are usually designed for point-and-click access. The traditional graphical user interface may fit perfectly on these devices. Some of the possible difficulties are:

• Varying screen size. The interface must adapt itself to available screen space.

• Varying gesture support, from a simple click to more advanced multi-touch gestures.

• Varying host’s look-and-feel rules and common usage patterns.

The graphical user interface must be scalable to utilize all available screen space. The web page layout is scalable by design. The text can be made scalable if it is not bound to pixel sizes¹. Scaling of the embedded graphics is possible with pure HTML/CSS. However, to prevent unnecessary network traffic and ensure proper quality of the embedded images, they must be scaled on the UI server. Because the images can differ in size based on user’s preferences, the UI server should be capable to produce quality image thumbnails and cache them properly for efficiency reasons.

Because the devices can vary greatly in supported gestures, the most of the gestures are not required to be implemented in the first prototype. Only the traditional point and click is required to be supported. Future web-based UI releases may employ multi-touch gestures, for example, a pinch may be used to go to the previous screen. If navigation follows drill-down approach, then pinch is usually associated with going back/up.

The UI should be capable to show the current set of the goals and services ready to be invoked by the user. The state of the services should be shown too. However, this might be less important because the user can usually observe the changes directly from the environment.

The services and goals should be organized in such a way, that more relevant information and controls are displayed first. Because there can be a huge amount of devices and

1The viewport-height (vh) and viewport-width (vw) units are the best option to bind the font- sizes to viewport size, but they are currently not supported by any browser except Internet Explorer 9. The draft of the standard covering them is CSS Values and Units Module Level 3 http://www.w3.org/TR/css3-values/

(23)

3. Requirements

goals, the UI should implement a bookmarking functionality. The services and goals, added to the bookmarks, should be shown on the first (default) screen, directly accessible without any navigation effort. All of the services and goals should be accessible from other supporting screens, where they can be bookmarked if used often.

To limit the visual clutter, the bookmarks, services and goals should be grouped by the room where they are logically available. The user should be able to select another room at any time, for example, to switch off the lights in the kitchen while sitting in the living room. The touch-screens mounted on the wall should by default show the room where they reside. Mobile devices, like cellphones, can select the room corresponding to the user’s current position. The rooms are in turn grouped by sections. A section can represent a building level or a section of a house. However, the rooms and sections are not required to represent the physical world, the division is purely logical.

By default most of the people are assumed to be interested in all services and goals available. However, some people may not be capable to use all kind of devices, for example, deaf people will probably be not interested in a stereo system. The registered user should have a profile where all his/her preferences will be stored, including possible disabilities.

The disabilities may filter out not relevant devices and global goals, that are created for everybody. The goals can take user disabilities into account too, when being executed. It is important to still let the user view all the goals, regardless of possible disabilities, when user explicitly wants it. The filter by disabilities has a suggestive role, it is not intended to completely prevent the user to access possibly irrelevant goals.

The disabilities may initially be implemented as a simple set of tags. The tag is a known label, that may be attached to one or more related objects. The objects can then be checked for a specific tag or the group of related objects can be searched by a common tag. More sophisticated meta-information can be used in later implementations.

3.2. Goal editor

The goal editor should be usable on mobile devices with the small screen space and on a regular personal computer with the screen large enough to contain the whole goal in the view. The editor’s canvas can not be simply zoomed out because the language constructs and the text will become unreadable. 2D free scrolling is not user friendly, so one of the dimensions should be limited. The initial prototype shipped with this paper virtually limits the width of the editor’s canvas to the physical screen width. The layout engine, used by the canvas to render the language constructs, prefers to stack the items vertically.

The vertical scrolling should be offered once the canvas becomes longer than the physical screen. The horizontal scrolling should be offered only if the layout engine fails to fit to large language construct into the screen.

The canvas should have as less controls as possible. A single button can bring the dialog window with all available language constructs. Every language construct should have the context specific dialog window with controls limited to the current context only.

The pop-up dialogs with very limited set of controls have a high chance to fit in the small screen of a mobile phone, reducing the need for scrolling.

The achieve and find-out predicates [3] use the state variables declared by the known services. Because the goal editor is not bound to currently selected room, all services should be shown at once. The dialog, used to select the service, should group all services

(24)

3.3. Device independence

by the type. Once the user has selected the service, the dialog with all available state variables should pop up. The services and the state variables should be annotated with the icons to help the user quickly find desired service and state variable.

The state variable expression, described in Section 2.1.5, can refer to other state variables. The state variables have a type that defines all possible values. When selecting a state variable for the expression, the dialog should fade-out variables of incompatible type. The services that do not have at least one variable with compatible type, should be faded-out too. A faded out service and variable can be selected, but the user should be warned that the resulting goal might be invalid. The language construct rendered on the editor’s canvas should highlight such invalid expressions.

The goals with similar purpose should be grouped together. For example, there can be three goals handling illumination: maximum light, for reading and dimmed light for watching TV and relaxing. The user can color the goals of similar purpose with the same color. The colors will help the user quickly find the goal of a specific function.

3.3. Device independence

The Web UI consist of the client software and the UI server. The client software should run on all modern mobile devices that have the web-browser installed. This can be achieved by supporting major layout engines:

• WebKit² – used by Safari³, Nokia S60 browser⁴, Blackberry Browser⁵, Chrome⁶ and other web-browsers for both mobile and PC platforms.

• Gecko⁷ – used by Firefox⁸ and other web-browsers.

• Any browser supporting HTML 4⁹ and CSS 2¹⁰. Some HTML 5¹¹ related features like WebSockets¹² may be used for better performance if supported by the host platform.

The user experience should degrade gradually with the missing features of the host platform. If the screen size is really to small, then scrolling should be possible to view everything. If embedded graphics is not rendered, then there should be enough alternative information to describe the objects being viewed. The basic CSS styles should ensure the grid layout is kept intact even by the oldest browsers. Other graphical features, like rounded corners, may be disabled, if the client browser does not support them.

2WebKit layout engine http://www.webkit.org

3http://www.apple.com/safari/

4http://www.developer.nokia.com/Community/Wiki/S60WebKit

5http://us.blackberry.com/apps-software/devices/

6http://www.google.com/chrome

7Gecko layout engine https://developer.mozilla.org/en/Gecko

8http://www.mozilla.org/firefox/

9HTML 4.01 Specification http://www.w3.org/TR/html401/

10Cascading Style Sheets Level 2 Specification http://www.w3.org/TR/CSS2/

11A vocabulary and associated APIs for HTML and XHTML http://www.w3.org/TR/

html5/

12The WebSocket API http://www.w3.org/TR/websockets/

(25)

3. Requirements

The networking features should degrade gradually too. If the WebSockets are supported by the client browser, then the fastest interaction with the devices can be achieved. Other- wise, the techniques like AJAX¹³can be used to exchange the information with the server without reloading the page. However, in this case the networking latency may increase.

The web-based UI must stay usable even if nothing except the traditional page reloading can be used. In this case, the status of devices on the page need not to be refreshed to show the actual state. The web-based UI will then be used to limitedly support invoca- tion of default device actions. The features like goal editor need not to be supported, if support for client-side scripting is disabled.

3.4. UI server

The UI server should be horizontally scalable to ensure the server will perform fast enough to handle virtually unlimited number of clients. A standard web-technology involving multiple application servers and load-balancers may be used. By default there is one UI server handling all of the client devices. When the number of devices becomes to large, a new one can be added. Both servers will then be hidden by the common load balancer.

The load balancer acts like a single server, but forwards the requests to the application servers ensuring the load will be divided equally. To make this possible, the UI server must be stateless.

The possibility to supports thousands of client devices allows to manage a large building with lots of wall-mounted touch panels and lots of people inside, carrying mobile phones and using laptops to access the user interface. The device calibration information and the user profiles should be handled using isolated sessions. This allows the same user to access the UI from different devices simultaneously. Every page will then be tuned to each device independently.

The UI server is just a HTTP¹⁴ server generating pages for the clients. It must be extendible and efficient. It should be implemented on a modern, easily deployable and flexible platform, known to a broad range of developers. This ensures the prototype can be developed and extended further by 3rd party developers. This is important because the SM4ALL project is still being developed, the protocols and the specifications change often, the UI server should be easily modifiable to stay usable and integrated with the rest of SM4ALL services.

3.5. Internationalization

All messages, that the Web UI software can display, should reside in a language pack¹⁵. It should be possible to translate the Web UI to another language by providing the language pack for corresponding language. This is called interface internationalization and it should be implemented in the initial prototype shipped with this paper.

The objects provided by the central repository, for example, rooms, devices or goals, should be already translated to the target language. This is called content international-

13XMLHttpRequest http://www.w3.org/TR/XMLHttpRequest/

14Hypertext Transfer Protocol http://tools.ietf.org/html/rfc2616

15The prototype shipped with this paper uses gettext based language packs.

(26)

3.6. Functional requirements

ization. It is a good idea to provide with each objects the translations to each supported language. However, the repository is not currently required to support such level of internationalization. The prototype, shipped with this paper, does not need to support content internationalization too. Otherwise, it will induce unnecessary complexity because of the functionality that will be long time kept unused. The final implementation should support content internationalization, assuming the central repository will support it too.

3.6. Functional requirements

The functional requirements define the minimal set of features the software should support.

The requirements that are not implemented in the prototype shipped with this paper, will be further discussed in Chapter 6.

3.6.1. Screen calibration

The graphical user interface shipped with this paper is using grid-based layout to present the various objects to the user. There is a limited number of cells, that can fit on a screen without horizontal scrolling. The screen size of a client device can vary greatly. The personal preferences for the cell, the text size and the number of cells in a row can also vary greatly. The screen calibration allows the user to measure preferred grid properties to ensure the text in a cell stays readable and the cells are large enough to be conveniently viewed.

Because almost all other screens need the fully configured grid, the screen calibration must be the first procedure the user needs to finish when starting a new session¹⁶. The session keeps the measured parameters until it is invalidated. The session is invalidated, if the user signs out or does not access the Web UI for more than two weeks.

The following is a running list of screen calibration requirements:

• It should be possible to choose the maximum number of cells to be shown in a row.

This will be the grid width.

• It should be possible to adjust the text size. Larger text is more readable, but will take up more space in the cell, covering the widget graphics.

• The measured parameters should be stored in the user session. The next time the user opens Web UI, the parameters should be restored. The measured parameters can be lost only if the user’s session is ended.

• It should be possible to access the screen calibration page at any time, such that the user will be able to adjust own preferences if needed.

3.6.2. User profiles

The prototype shipped with this paper requires every user to be registered. It is possible to give the anonymous users the access to Web UI, but doing this will leave the demo

16The user sessions are described in Chapter 6.

(27)

3. Requirements

vulnerable to vandalism from the internet users. The future releases may be configured to allow anonymous users, if they access the user interface from the local network.

The user profile contains basic information about the user and his/her disabilities. The disabilities help the Web UI show only relevant information. The prototype shipped with this paper supports only three types of users:

• Administrators – they can manage everything from the administrator’s panel.

• Staff members – they have access to the administrator’s panel, but the assigned permissions restrict the set of allowed actions they may perform.

• Users – they can not access the administrator’s panel.

The administrator’s panel is called back-office, it makes the management of the web- based UI possible. The rest of the site is called front-office, it is the set of pages open to everyone. All users can access the front-office. Only the administrators and the staff members have the access to the back-office.

The following is a running list of the requirements related to user profiles:

• It should be possible to register new users. The prototype shipped with this paper gives only the superuser the permission to manage the user accounts.

• It should be possible to ban the registered user.

• The users should be able to view and modify their user profiles. This functionality requires the user to be signed in.

3.6.3. Catalog

The smart house can be represented by a set of rooms grouped by sections, where a section represents a building level or any other logical division of the house into the rooms. The devices and goals can be assigned to one or more rooms where they will be available.

The web-based UI should support the following set of features related to in-house navigation:

• The list of rooms grouped by the corresponding sections. The user should be able to select a room from the given list. The list should be available at any time.

• The bookmarks view is the default view, shown to the user when the room is selected. It should show the user’s bookmarks and optionally the preselected set of frequently used devices and goals.

• The goals view should become available when the room is selected.

• The devices view should become available when the room is selected.

The goals should be basically displayed in two states: ready to be executed and running state. The click on the goal icon can cancel its execution, if it is currently running.

Otherwise, the click on the goal starts its execution. These are the basic functions attached to the simple click on the goal icon. However, some goals can be more complex and show different states visualized by different icons or even draw some of the controls in place of or over the icon. Clicking on such a goal can open a page where all required parameters can be collected by further interacting with the user.

(28)

3.6. Functional requirements

3.6.4. Goal editor

The goal editor makes the management of the complex user goals possible. The editor should be able to load and store the goals. The following is a running list of the requirements related to the goals list view of the goal editor:

• It should be possible to show all of the user goals in a list. The system administrator should be able to see and modify the global goals. Other users can not see the global goals in the editor.

• It should be possible to delete the goal. The Web UI should first confirm the deletion, ensuring the user does not accidentally delete the goal. Deleted goal can not be restored.

• It should be possible to modify the existing goal.

• It should be possible to add a new goal.

The goal is created or edited in the editor view. The view should implement the following set of features:

• The goal properties should be editable via the user-friendly form. The form should be hidden by default when the goal body (script) is being edited. The name, icon, description, color and the assigned set of rooms should editable using the form. It should be also possible to tag the goal with common disabilities, ensuring the goal will not show up for the users having these disabilities. The disabilities can only be attached by the administrator when creating or modifying the global goals. The disabilities are ignored by the private goals.

• It should be possible to add new predicates to the goal. It should be possible to fill the predicates with the state variable expressions, created solely by the graphical dialogs, that may ask additional information (eg. service and variable pickers, expression editor).

• It should be possible to save the goal.

• It should be possible to cancel the edit session. The cancel operation should discard any changes and bring the user to the list of goals. The cancel action should be confirmed by the user in order to prevent loosing unsaved work on accidental hit on "Cancel" button.

(29)

(30)

4. Graphical design

The graphical design of a proper user interface is, by itself, a complex study, involving lots of focus-group testings. The main purpose of this paper is to create the technical design of a possible web-based graphical user interface. The graphical part is kept as simple as possible, just enough the show the basic idea.

The screen is divided into the three parts. The main menu resides on the top of the screen. Bellow the main menu is the notification area, that shows system messages when needed. The system message will stay for a brief period of time and then it will be hidden from the view. Figure 4.20 shows the message, when the goal is saved. Different colors are used for different kinds of messages, for example, yellow is used for success message and red is used for error messages.

Figure 4.3 shows the main menu. The Room, Goals and Devices items become available when a room is selected. These menu items refer to the pages showing the content related to the currently selected room. The Rooms menu item refers to the page, that lists all of the rooms. This menu item becomes available, when the user is signed in and his/her screen is calibrated. The Tools tab brings the user profile, goal editor and calibration pages under the same menu item. A sub-menu with links to corresponding pages will be shown when Tools item is opened. All of the pages will be discussed in the following sections.

4.1. Authentication

The prototype shipped with this paper can run in the Internet enabled environment and can be abused by the unethical people from around the world. To permit only the selected set of users, the initial prototype implements a simple form of authentication. The authentication screen is shown in Figure 4.1 The menu bar is disabled because all other views require the user being signed in. The simple authentication queries the username and the password from the user, which can be obtained from the system administrator.

At any time the user can sign out. By signing out the user’s session will be ended.

All previously collected information, including the screen calibration and optionally the usage statistics will be lost. The sign-out screen, shown in Figure 4.2, contains a link to the sign-in page, where the user can sign in again, possibly as a different user.

4.2. Calibration

As mentioned earlier, the screen of a mobile device should be calibrated before the web- based UI can be properly used. The calibration information is then stored in the user’s session, which is automatically started when the user is signed in.

(31)

4. Graphical design

Figure 4.1.: The sign-in view.

Figure 4.2.: The sign-out view.

The calibration view, shown in Figure 4.3, has a grid-width picker with cell size and text scale controls. The controls are self-explanatory, the

+

button increases the cell or font size, while the

-

has the opposite effect. Every time the cell size is changed, the grid-width picker redisplays the maximum number of cells that will fit in one row. The user can then pick up the proper grid-width by clicking on the right most cell. In the example, the cell size is set large, so only four cells can fit into the screen. The grid-width is chosen to be 3 cells, thus there will always be an empty space right to the screen. When the grid looks as desired, the user can hit the Ready button to save the changes.

(32)

4.3. Catalog

Figure 4.3.: The screen calibration view.

4.3. Catalog

A house can be logically divided into the sections, which can be further split into the rooms. A section can represent a building level or a larger room, that consist further of separate areas, for example, the kitchen and the living room. The division is purely logical, the intention is to organize possibly large number of devices and goals.

The user starts navigation through the user interface by choosing a room. If the client device is capable of tracking the user position, then switching to the current room can happen automatically. Otherwise, the user can choose the room from the list of rooms grouped by corresponding section. Figure 4.4 shows how the rooms are displayed. The empty space left to the room icon is by default filled with the room description, which is not specified for the rooms shown in the screenshot. The room name and icon are the links, a click will set the chosen room as the current one.

Once the room is chosen, the user can view the bookmarks, devices and goals assigned to the selected room. The list of bookmarks is shown as a default page when the room is selected. Figure 4.5 shows the list of goals assigned to the Living room of a simple house.

The same grid based view is used for devices and bookmarks. The number of cells in a row of the grid, the size of the cell and the size of the text inside the cell are the personal settings measured and stored in the session individually for each device via the calibration screen.

The widget, used to present the goal in the grid, shows a default icon assigned to the

(33)

4. Graphical design

Figure 4.4.: The list of rooms

Figure 4.5.: The list of goals

goal. This is the most simple way to represent the goal in the grid. More advanced widget could show different icons depending on the goal execution state. It is also possible to display some controls inside the widget, for example a button to cancel the executing goal.

(34)

4.4. User profile

The user profile shows the basic user information. Figure 4.6 shows the profile page of the administrator user. The permissions are shown only if the user can sign in to the back-office. The property shows the role of the user, possible values are administrator or staff member. The date joined is the date when the user was registered in the system, while the last login shows the date and time of the last session. The personal challenges section lists all of the user’s disabilities.

Figure 4.6.: The profile view page

The Edit button opens the profile edit page, shown in Figure 4.7. The user can change his/her name, e-mail address and the set of disabilities. The system administrator can add more disabilities. The elaborated set of disabilities make the fine grained item relevancy control possible.

4.5. Goal editor

The goal editor is the most advanced part of the client side software at the moment of initial prototype release. The editor makes it possible to construct the goal entirely using graphical constructs.

The goal editor consist of the two main screens. One of the screens is the list of goals, that shows all goals allowed to be edited by the current user. A normal user will see own goals only, while the system administrator will see the global goals too. The list of goals

(35)

4. Graphical design

Figure 4.7.: The profile edit page

is shown in Figure 4.8. The list shows name and description of the goal, together with its icon and location information.

The buttons offer possible actions related to the corresponding goals. Clicking on the Delete button will open up the confirmation screen, shown in Figure 4.9. If user is sure to delete the goal, then clicking on Yes, delete it will permanently delete the goal. Both buttons lead back to the list of goals.

The View button exists for debugging purpose only. It shows the goal script in two XML formats: internal format and original format. The internal format is easier to handle graphically because of the direct correspondence between the XML elements and the graphical blocks shown to the user. The format adds support for complex expressions, which are not limited to just a sum of two operands. This format is not supported by the current implementation of the composition layer /planner. The original format allows very simple expressions only and is supported by the current planner implementation.

The Add new goal and Edit buttons open the editor screen, which contains the goal properties form and the editor’s canvas. The form with all properties of the goal is shown in Figure 4.10. It is possible to change the appearance of the goal by setting the name, description, icon and the background color. All of the rooms, grouped by the corresponding sections, are listed under Rooms property, which defines where the goal will be available. The disabilities will be shown to the administrator user only. The

(36)

4.5. Goal editor

Figure 4.8.: The list of goals

Figure 4.9.: The screen confirming deletion of the goal

property lists the disabilities incompatible with this goal. For example, a goal controlling the room illumination might check the Visually challenged disability to become invisible to the users with limited sight.

Web based interface for a smart home