Leveraging Sensor Data And Context Information

Published Date: 02 Nov 2017

to improve mobile interaction performance

Desktop user interface design has evolved on the basis that users are stationary (i.e. - sitting at a desk) and can normally devote most of their visual and physical resource to the application with which they are interacting. The interfaces to desktop based applications are typically very graphical, often extremely detailed, and utilize the standard mouse and keyboard as interaction mechanisms. But the typical user is not facing a desktop machine in the relatively predictable office environment anymore. Rather, users have to deal with diverse devices (mobile or fixed) sporting diverse interfaces and used in diverse environments. However, many important pieces necessary to achieve the ubiquitous computing vision are not yet in place.

Compared to desktop computers, the use of mobile devices is more intimate because users often carry mobile device throughout their daily routine. Most notably, interaction paradigms with today’s devices fail to account for a major difference with the static desktop interaction model, so they present HCI design opportunities for a more intimate user experience. People also use mobile devices in many different and changing environments, so designers don’t have the luxury of forcing the user to "assume the position" to work with a device, as is the case with desktop computers. For example, the user must accept qualities of the environment such as light levels, sounds and conversations, and the proximity of people or other objects, all of which taken together comprise attributes of the context of interaction. But if mobile devices remain unaware of important aspects of the user’s context, then the devices cannot adapt the interaction to suit the current task or situation. Thus an inability to detect these important events and properties of the physical world can be viewed as missed opportunities, rather than the basis for leveraging deeper shared understanding between human and computer. Furthermore, the set of natural and effective gestures—the tokens that form the building blocks of the interaction design—may be very different for mobile devices than for desktop computers. Over the course of a day, users may pick up, put down, look at, walk around with, and put away (pocket/case) their mobile device many times; these are naturally occurring "gestures" that can and perhaps should become an integral part of interaction with the device.

Because the user may be simultaneously engaged in real world activities like walking along a busy street, talking to a colleague, or driving a car, and because typical sessions with the device may last seconds or minutes rather than hours, interactions also need to be minimally disruptive and minimally demanding of cognitive and visual attention.

One hypothesis that a number of ubiquitous computing researchers share is that enabling devices and applications to automatically adapt to changes in their surrounding physical and electronic environments will lead to an enhancement of the user experience. The information in the physical and electronic environments creates a context for the interaction between humans and computational services. Context is defined as any information that characterizes a situation related to the interaction between users, applications, and the surrounding environment.

The growing research activities within ubiquitous computing deals with the challenges of context-aware computing. Though the notion of context can entail very subtle and high-level interpretations of a situation, much of the effort within the ubiquitous computing community takes a bottom-up approach to context. The focus is mainly on understanding and handling context that can be sensed automatically in a physical environment and treated as implicit input to positively affect the behaviour of an application. Inclination of human finger on mobile screen imposes hindrance in proper usage of mobile applications. A potential example for this will be adaptation of improved interaction according to the user’s mobile grip. Another example will be inputting text by multimodalities like: gaze tracking and voice recognition. This application will be highly useful for people who cannot write. Speaking of another example, collecting the speech at different contexts and making use of that to refine the results for text input, this can be done by give higher priority to these words than other according to the context.

I believe that augmenting mobile devices with sensors has the potential to address some of these issues. There is an explosion of inexpensive but very capable sensors. While these sensors may enable new interaction modalities and new types of devices that can sense and adapt to the user’s environment, but they raise many unresolved research issues. What interaction techniques or services can benefit from this approach? What problems can arise? What are the implications for end-users?

I am looking forward to explore a range of interactive sensing techniques to gain experience with general issues and to explore issues of integrating techniques that may conflict with one another. Researchers have implement techniques such as voice memo recording by speaking into the device just as one would speak into a cell phone, switching between portrait and landscape display modes by holding the device in the desired orientation, automatically powering up when the user picks up the device, and scrolling the display using tilt. I suggest that there should be new points in the design space, such as using the contextual data to improve interaction, which will give less recognition errors, better word prediction, less touch error. This can be done by sensing and understanding the users’ activities when they are interacting with mobile devices.

2. Method

A literature review will be conducted, making use of various works done in the field of mobile interaction and sensors in mobile.

2.1 Research Questions

The research question will be refined after the literature review, from the following:

Q1: What are the challenges while interacting with mobile in different contexts?

Q2: What interaction techniques can benefit from contextual and sensor data?

Q3: What activities of users can be sensed when users are engaged with their mobile devices?

Q4: How useful are these contextual interaction at different contexts?

Q5: What are the implications for end-users?

Q6: What further consequences can be arose?

Q7: What are the impact of one solution on other cases?

2.2 Research Method

The research method consists of creating a prototype model and examining the accuracy of the model on people at the particular context. Various challenges of interaction with mobile at different contexts will be studied (Q1) and then different ways of solving the issue by the help of contextual and sensor data will be examined (Q2 and Q3). Finally, a prototype will be created to perform the experiment on people in that context. The data will be used to draw conclusions on how useful is that contextual interaction model is at the context (Q4). A usability testing followed by a survey will be done to find the implications of that model (Q5 and Q6).

My research methodology requires gathering relevant data about the present interaction issues at various contexts and analysing these to gain complete understanding. These issues will be studied in real life situation due to the practicality interaction with mobile, which will aim to find the main reasons behind the issues so that these can be solved by the appropriate usage of sensors and contextual data. This may be done by trying to reproduce the issue at different contexts. After the development of a prototype of such model, it will be evaluated and compared with the existing technology to analyse the impact of the work. Experiment will focus on the way of user’s interaction with mobile, which may be recorded for the better understanding of the scenario. Participants will also give feedback on the usefulness of the technologies in regards with improved interaction performance and challenges in working with them. Target participants will be people who uses there smartphones for their work as well.

2.3 Measures

Experiment will be conducted to examine the prototype with people at different contexts. Data will be collected when participants were given the prototype and original model to interact with. Audio-video recording of participants’ interaction with mobile complimented by questionnaires based on a standard format (e.g. Likert scale model) to obtain mainly qualitative responses from users. Demographic factors (age, educational experience, gender) also need to be considered.

Qualitative variables of the case study include: 

Structure, plan and control of the development process.

Study of issues of interaction with mobile at various possible contexts.

Perception of these issues at other contexts to know how people manage at different contexts.

Feedback from mobile users on approaches they believe to be useful.

Ability of the approaches to support users to elicit, analyse, define and manage the core issues.

Ability of the approaches to show presence of inefficiencies or opportunities. 

Users' influence on improved interaction designs.

2.4 Participants

Inclusion factors for the experiment (N=100) include the users of mobile device for various activities at different contexts. Participating user may be from any part of UK. This location was chosen due to their known importance and usage of mobile devices in various aspects of life in innovative ways. Participant in the case study will be selected using the data obtained in the survey, based on their representativeness of the issues.

2.5 Data analysis

Standard statistical packages (R) will be used to visualise quantitative data, perform correlation analysis and regression to establish relationships. For the qualitative data, a qualitative data analysis software package will be used to assist coding and visualise relationships from the interview and workshop data. The concepts of grounded theory and statistical analysis will also be applied to reduce data through coding, visualise data and document relationships in mind-maps to aid the analysis.

2.6 Outcomes

The intended final outcomes of the research will be:

Understanding of issues of interaction with mobile devices at various contexts.

Finding new of improved models for interaction with mobile devices by the help of contextual and sensor data.

Empirical evidence of the usefulness of these models.

Proposal of technology that can improve the interaction with mobile devices and adapt according to the context.

3. Significance

Smartphones have been a remarkable addition to our life providing ease of access which has become a necessity of our life. This wonderful invention allows users to work on these devices regardless of location and even when they are in motion. But each time they cannot devote all or any of their visual or physical resources to interaction with the mobile application. Additionally, such devices have limited screen size and traditional input and output capabilities are generally restricted. Consequently, we want to develop effective techniques on mobile technology which can embrace a model modification with respect to the improved interaction with such devices.

The outcomes of this study could also be applied in future application development. It is further hoped that the research will establish an innovative model to study implications of interaction with mobile devices at different contexts. This will help to make the mobile devices adaptable to various contexts without tempering the performance of the interaction. Speaking of an example, different ways of text input which gets adapted according to the contextual information.

4. Preparation

Presently I am pursuing the unique experience of using sensor data to find the context of users such as location, social environment, and current activities. This is what I am gaining while working for my current semester project (Ubhave) at University of Birmingham under the supervision of Dr. Mirco Musolesi.

I am aware of my lack of knowledge, particularly in the field of user’s interaction with mobile at different contexts. I have therefore started to work on a HCI research topic module on issues with user’s interaction with mobile at different contexts, at the University of Birmingham recently. The module focuses on the literature review of the area and finding the scope of work that can be done in that field. It gives me the opportunity to study about the area which will provide me with deep understanding about the current and past issues.

5. Proposed development

Phase One:

Comprehensive literature review to help refine the research questions and research methodology.

Rethinking the nature of the methodologies and consider the possibility to conduct action research to better observe the effects of the introduction and implementation of the methodologies.

Investigating issue with mobile interaction to gain deep understanding.

Finding different solutions to the problems.

Entering dialogue with researchers and practitioners in the field of HCI in mobile computing.

End of phase one:

Conducting survey and trying to reproduce the issues to confirm that our understanding is valid.

Writing of the progress report and refining the research plans for phase two.

Phase Two:

Evaluating all possible solutions.

Creating prototypes for the solution.

Conducting the experiment to evaluate the prototype.

Analysing the data according to the research plan.

Writing of the progress report and refining the research plans for phase three.

Phase Three:

Conducting a supplementary literature review.

Writing up of the thesis.

6. Conclusion

Improving user interaction with mobile is a big issue, where many researchers are contributing to improve the performance of user’s interaction with mobile. Leveraging the sensor and contextual data to make user’s interaction with mobile much easier, is likely to be proved as a useful way of handling this issue.