Paper Reading #24: Gesture Avatar, A Technique for Operating Mobile User Interfaces Using Gestures

Hao Lu is a computer science graduate student at the University of Washington.
Yang Li is a senior researcher at Google Research

This paper was presented at CHI 2011.

Summary
Hypothesis
The researcher's goal was to create a system that allows for the easy interaction with mobile interfaces.

Methods
To test their hypothesis, the authors set out to create a system called the Gesture Avatar system. To them, one of the main problems of current mobile interfaces is the small size of many interface elements. For example, small buttons that are hard to use or tiny links on a webpage that are hard to click.

To counter this problem, the Gesture Avatar system allows the user to draw a shape or a letter near the object the user wants to activate and the system let's the user control the object from this larger "gesture avatar".
For example, a slider might be hard to activate while on the move or it might just be too small. A user could draw a box close to the slider they want to control and then slide that box like the slider.
Another example is for selecting small texts or clicking small links on a webpage. By drawing the first letter of the link they're trying to activate, the user can then click on the letter and the system attempts to figure out what link is meant by the user.

To test their system, researchers used a Motorola Droid running the Android OS. They found 12 participants to test the system. They used two different tests. The first was a letter selection test. A small amount of letters would show up on the screen and the user would have to select the highlighted letter by drawing the letter gesture and select it. The next test was simply a target selection. They also had some participants walk and use the device and some were asked to walk on a treadmill and use the device.

Results


They used three different types of criteria to judge the Avatar Gesture system: time performance, error rates, and subjective preferences.

For all letter sizes, Gesture Avatar's performance time was about constant. It was significantly faster than an alternate system (Shift) on the 10 px letters however.

Error rates also remained low and about constant over the three target size tests. The error rates are SIGNIFICANTLY lower than the alternate Shift system.

In the subjective preferences, 10 out of 12 participants preferred Gesture Avatar over Shift.

Discussion
The Gesture Avatar system is certainly a novel system of controlling a device. One of the main problems with mobile interaction is the limited screen space of a device. Screen size cannot be increased without making the device larger. Thus, for cell phones, making a larger screen is not really a viable option when it comes to fostering better user interaction.

One problem that might arise from the system is the fact that they only tested the system using two hands. Often times, when I'm in a hurry walking down the street, I only used one hand on my mobile device. I would have liked to see what would happen if someone just used their thumb to draw the gestures as opposed to a whole new hand.

Paper Reading #23: User-Defined Motion Gestures for Mobile Interaction

Jaime Ruiz is a PhD student of HCI at the University of Waterloo
Yang Li is a senior researcher scientist at Google.
Edward Lank is an assistant professor of computer science at the University of Waterloo.

This paper was presented at CHI 2011

Summary


Hypothesis
Researchers set out to prove that certain motion gesture sets are more natural than others in mobile interaction.

Methods
In this research experiment, researchers essentially allowed participants to create the gestures to be investigated. The authors created a list of tasks of several different types. The tasks would either fall into the "action" or "navigation" category. These categories were then subdivided into smaller tasks.

Participants were given the full list of tasks to be completed as well as a smartphone. They were instructed to complete the simple task (like pretend to answer a call, or go to the homescreen) with the idea that the phone would know how to execute the task.

The phone they were using was equipped with specially designed software to recognize and keep track of gestures performed by the participant.

After the test, the user was asked to comment on the gestures they used and whether they were easy to perform or good match for it's use.

Results
Through the test, researchers discovered that many users preferred and performed the same gesture.

The authors found several trends in the description of motion gestures. They found that many of the gestures mimic normal use. For example, 17 out of 20 participants answered the phone by putting it to their ear. The gestures also use real-world metaphors.

Through all these results, the authors were able to analyze the various created gestures and create a taxonomy of these natural gestures.

Discussion

Research like this, in my opinion, should occur more often. By giving participants free-range of their actions, they can come up with whatever input or gestures are natural to them. After analyzing these gestures, we can figure out which are natural.

By creating more studies like this, we can find out what types of input are preferred in a system. For example, do most people prefer to execute gesture XYZ to access the menu or do they prefer to use gesture ABC instead.

Paper Reading #22: Mid-air Pan-and-Zoom on Wall-sized Displays

Mathieu Nancel is a PhD student of HCI at the University of Paris
Julie Wagner is a postgraduate research student at the University of Paris.
Emmanuel Pietriga is a full-time research assistant at the University of Paris.
Olivier Chapuis is a research scientist at University of Paris.
Wendy Mackay is the research director at INRIA. 


This paper was presented at CHI 2011.


Summary
Hypothesis
In this paper, researchers experimented with various ways to control a wall-sized display. They had several hypotheses regarding which ways would be preferred. They believed that two handed gestures will be faster and easier to use than one-handed gestures. They also believed that users will prefer circular gestures. They predicted that small gestures would be preferred.

Methods
In the project, researchers tested various different types of gestures. First, they used uni-manual (one handed input) and bi-manual (two handed input). Each of these manual gestures could be used in conjunction with 1D control, 2D surface, and 3D free space.
The 1D control was control in one plane using a linear scrollwheel or a circular scrollwheel.
The 2D surface was control using a PDA or smartphone.
The 3D free space allowed gestures to be used.
With the bi-manual input, one hand was used to show the focus of the gesture, i.e. where to focus the zoom or pan, etc.

In the experiment, the researchers tested three factors: Handedness (one handed, two handed), Gesture (circular, linear), and guidance (1D, 2D, 3D).

Each participant was given a target in a high zoom scale. They had to zoom out to display all targets and they had to pan the targets so they were all visible on the screen.

Results
In most cases, they found that two hands (bimanual gestures) were faster and had shorter movement times
They also found that 1D path control was much faster than the other input types.
Linear control was also found to be faster than circular.

In the qualitative results, participants seemed to prefer the one handed gestures. They also preferred the linear gestures to circular gestures. They found that circular gestures on a surface were too hard to do.

Discussion
The paper itself was fairly interesting although it was slightly difficult to identify with since I've never personally used a large wall-sized display. I wasn't able to really agree or disagree with their findings since I've never had to use gestures on such a large display.

This kind of research could potentially become very helpful in the future when wall-sized displays become more widely used, though. Another user qualitative might be helpful in the future as well. By letting users design their own gestures, they could choose the most natural gestures for the large wall.

Paper Reading #21: Human Model Evaluation in Interactive Supervised Learning

Rebecca Fiebrink is an assistant professor at Princeton in the school of Computer Science but also is associated with the music department.
Perry R. Cook is a professor emeritus of Princeton in the school of Computer Science and the Department of Music.
Daniel Trueman is an associate professor of music at Princeton University.

This paper was presented at CHI 2011.

Summary


Hypothesis
The researchers main hypothesis was that while using a machine learning system, humans could evaluate and adjust the machine learning generated model to better train the system.

Methods
To test their hypothesis, researchers created a system called The Wekinator. The Wekinator is a software system that allows human users to train a given machine learning given certain gesture or real time input. Based on the human input, the system attempts to figure out the best course of action for the human input. If the system is incorrect or it isn't certain enough, the human can tell the system what the correct action is and the human can train the system by performing the gesture or input over more iterations.

To test Wekinator and discover how humans used it, they created three different studies.

In Study A, researchers had members of the Music Composition department (PhD students and a faculty member) discuss Wekinator and allowed them to use it over a given amount of time. When they were done using it, the participants sat down with the researchers and talked about how they used Wekinator and they offered various suggestions. Most of the participants used Wekinator to create new sounds and new instruments through the gesture system.

In Study B, students ranging from 1st year to 4th year were tasked with creating two different types of interfaces for the machine learning system. The first was a simple interaction where based on an input (like a gesture or a movement on a trackpad), a certain sound would be produced. The next type was a "continuously controlled" instrument that would make different noises based on how the input is being entered.

In Study C, researchers worked with a cellist to build a system that could respond to the movements of a cello bow and report the movements of the bow correctly.

Results
Researchers found that in all cases, participants often time focused on iterative model-rebuilding by continuing to re-train the system and fixing the models as they needed to be adjusted with given input.

In all of the tests, direct evaluation was used more than cross-validation. Essentially, directly evaluating the model and modifying it was preferred over the cross-validation model.

The researchers also noted that cross-validation and direct evaluation was actual feedback to the participants so that they could know whether or not the system was properly interpreting their input.


Discussion

The usage of a system like Wekinator could have some very far-reaching benefits. At the moment, machine learning seems pretty bulky and hard to do if you're not an AI programmer or scientist. But Wekinator seems to make the task of machine learning easier to understand and to put into practice. In the second study, a lot of the participants didn't know much about machine learning until they were briefed on the subject. However, they were still able to put Wekinator into practice to create new types of musical instruments.

Wekinator (and systems like it) could open the door on many different input tasks. Gesture and motion control can be better fined tuned by quick training. Since multiple people might perform a gesture differently, using Wekinator, a system's model can quickly and efficiently be "tweaked" to fit a given person. This makes analog input (like speech recognition or gestures) more reliable and more accessible to a large group of people and technologies.

Paper Reading #20: The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only ANOVA Procedures

Since I completed Paper Reading #16, I'm choosing this blog as my blog to skip.

Paper Reading #19: Reflexivity in Digital Anthropology

Jennifer A. Rode is an Assistant Professor at Drexel University. She teaches in the school of Information.

This paper was presented at CHI 2011.

Summary


Hypothesis
Rode's main hypothesis and idea in this paper is to discuss the fact that ethnography are present and useful in the HCI field. She also discusses various types and aspects related to ethnography.

Methods
Through her paper, Rode talks about the needs and uses for what she calls "digital ethnography." A digital ethnography is simply an ethnography that is focused on technology in some fashion. One example that Rode gives in the paper is that of a researcher visiting super-churches and learning how each one employs technology.

To prove her points and to visit the various aspects of digital ethnography, Rode breaks down the various components. First, she covers the styles of ethnographic writing. Then she moves on to discussing ethnographic conventions before she looks at the framing ethnographic practices.

Results
First are the three styles of ethnographic writing.
The realist is interested in accurately displaying the subject. Rode writes that "the realist account strives for authenticity." The realist also goes to great lengths in order to show the reader the native's point of view.
The next kind of ethnographic writing is the confessional. In this kind of study, the writer comes up with a kind of theory on the subject, discusses it, then applies the theory to the ethnography and addresses it.
The final kind of ethnographic writing is the impressionistic. This is simply an attempt to "give an impression" of the subject. It is an attempt to capture the entire feel of the subject much like how the impressionist painters captured their subjects in a painting.

Next, she discussed the ethnographic conventions. The first discussed is the rapport that the ethnographer must build with their subject. If a researcher can't interact well with their subject, the subject is less likely to reveal information or to openly discuss a topic.
The next discussed was the participant-observation. Rode described this as simply "deep hanging out." By being around and observing the participant in their "native" environment, one can gain information.
The final ethnographic convention discussed was simply the use of theory. By coming up with theories on the subject, researchers can focus questions for focus their research.

Discussion

This paper was a good description of how ethnography applies to the digital world. This paper will be very helpful when we begin to work on the design project related to our class ethnography projects. It will help point out things to look for and ways to approach problems.

Paper Reading #18: Biofeedback Game Design, Using Direct and Indirect Physiological Control to Enhance Game Interaction

Lennart E. Nacke has a PhD in game development and is a assistant professor at the University of Ontario Institute of Technology.
Michael Kalyn is a graduate student of computer science at the University of Saskatchewan.
Calvin Lough is a student of computer science at the University of Saskatchewan.
Dr. Regan L. Mandryk is an assistant professor at the Univeristy of Saskatchewan.

This paper was presented at CHI 2011.

Summary
Hypothesis
The researchers' hypothesis in this paper was that a computer game could be controlled via both direct and indirect physiological inputs.

Methods
In order to test their hypothesis, they created a simple side-scrolling platforming game. In order to test the physiological aspects of the systems, they used 6 different sensors.

Gaze interaction: They tracked the pupil of the user's eyes
Electromyography: Detects the electrical activation of the subject's muscles.
Electrodermal activity: Detects galvanic skin response. Measures excitment.
Electrocardiography: senses the heart activity.
Respiration sensor: measured the breathing rate of the subject
Temperature sensor: directly controlled temperature sensor

To test the physiological inputs, they came up with two different sets of inputs as well as a control test.
In the control test, the game was simply played with an Xbox 360 controller.

In the study, they had 10 participants play all three tests: the two different sets of inputs and the controlled test.

Results


In the results, the researchers discovered that both input sets with the physiological inputs rated higher in fun than the control test. Participants also said that the physiological tests were more novel.

They also expressed favorite sensors. Participants expressed the fact that they preferred the physiological input sensors that they could directly control. The EKG sensor and GSR were the least favorite sensors since they were indirect.


Discussion

The usage of physiological input was proved to make games more fun than normal. In my opinion, this is pretty appealing. One of the main issues with controlling computer games is the input problem. By giving a wider ranger of inputs (besides just joysticks and buttons), it makes the game more interesting to play.

One thing that I found was interesting is the fact that the "novelty" of the system was so highly rated. I'd like to see if the novelty lasted over time. For example, the Nintendo Wii, when it first came out, was highly popular due to it's novelty. However, it's recently lost a lot of players over time.

Paper Reading #17: Privacy Risks Emerging from the Adoption of Innocuous Wearable Sensors in the Mobile Environment

Andrew Raij is an associate professor of electrical engineering at University of South Florida
Animikh Ghosh is a junior research associate at SETLabs.
Santosh Kumar is an associate professor at the University of Memphis
Mani Srivastava is an associate professor of electrical engineering at UCLA.

This paper was presented at CHI 2011.

Summary


Hypothesis
Researchers set out to prove, in this paper, that a users personal privacy can be violated by inferring data from mobile sensors.

Methods
Researchers developed a framework that allowed them to gain information from sensors and infer other data. By receiving raw data from a sensor, the information can be reconstructed to decipher behaviors and discover contexts.
An example they gave related to a woman who was jogging and has a seizure. Through sensor movement data, they could infer she was running and then that she was having seizure. They could also discover information about the context well (such as where she was jogging as well as time).

Due to this information being able to be inferred, researchers believed that this could lead to privacy threats (financial, psychological, and physical).

To confirm their thoughts, researchers set out to test three goals: discover how much people care about privacy concerns, use the framework to analyze behaviors, discover how identifications of the data effects concert.

To conduct this experiment, they used two groups. The first group had "no personal stake". They did not wear an AutoSense sensor. They were simply asked a questionnaire. The second group had a sensor and were also asked the same questionnaire.
The sensor group was asked to wear the AutoSense sensor during their awake hours for 3 days.

Results
At the end of the experiment, they were able to use data collected by AutoSense to analyze the sensor group's three days.

In the questionnaire, they were able to discover how concerned people were about given factors. The sensor group was much more concerned following the test. People said that they'd be concerned about people knowing their location at a given time.

The participants also expressed concern about who the results were shared with. They expressed much more concern if their identity was shared with the data.

Discussion

While parts of this paper were generally hard to follow, it was still an interesting concept. Many of our mobile devices are capable of sensing movement and audio. If our mobile devices started collecting information through the day about our movements, certain things might be extrapolated and thus might raise some privacy concerns.

One of the large problems in the coming age of technology is making sure this kind of sensor data does not get compromised or leak out. Especially if systems that analyze this data are created. This could become an entirely new branch of cybercrime.

Paper Reading #16: Classroom-Based Assistive Technology, Collective Use of Interactive Visual Schedules by Students with Autism

Meg Cramer is a graduate student at UC Irvine study Informatics.
Sen H. Hirano is a PhD student at UC Irvine also studying Informatics.
Monica Tentori is an assistant professor in Computer Science at the Universidad Autónoma de Baja California.
Michael T. Yeganyan is a researcher at UC Irvine.
Gillian R. Hayes is an assistant professor in Informatics at UC Irvine.

This paper was presented at CHI 2011.

Summary


Hypothesis
The researchers hypothesis was that, through their vSked interactive and collabortive system, they could assist and augment the teaching of students with autism.

Methods
The vSked system consists of portable touch devices for each student as well as a large touch display at the front of the classroom. This large display is connected to by the smaller portable devices. The smaller devices report the students progress through the day and their progression through a set of systems to the screen.

The vSked systems allows for the student to select a reward they want to work towards as they answer questions. The system can ask the student two different types of questions: either multiple choice or "voting" questions (where there is no right answer).

For the multiple choice questions, an icon for each answer is displayed for the student to tap. If the student gets it incorrect, the correct icon will wiggle on the screen.

After a student has been progressing through their questions, the teachers can award tokens that students can use towards their chosen reward.

To test the success of the system, the researchers used a combination of observation and interviews with the teachers and aids.

Results
They found that their system garnered independence as students were excited to work on the device. The students became more focused on the task at hand without needing a teacher to assist them in staying focused.
Students also stayed motivated due to the reward system based on a students correct answers.

The system also allowed for consistency through the student's educated in many different areas.
For example, the system allowed the students to keep track of their schedule and look at it whenever they needed to without being able to change it. They were also able to see changes that a teacher would make on the big screen, on their local smaller screen.

Finally, they found that the system built community. With each student having their own screen, sometimes students would want to see how another student was using the device. Students would also keep track of their peer's progress through the large screen at the front of the class room.

Discussion

Overall, the paper was a well documented, easy to understand experiment. This kind of technology used in the classroom, especially with students with autism, can be extremely powerful.

One interesting note that I'd like to know about in the future of this system is how it affected students in the long-term. Did students who used this system progress faster in their education than students who didn't use this system? Or did these students end up relying too heavily on the devices and thus this system hurt their education?
These are a few questions I'd like to learn about in the future.

Paper Reading #15: Madgets

Malte Weiss is a PhD student at RWTH Aachen University.
Florian Schwarz is a diploma thesis student at RWTH Aachen University.
Simon Jakubowski is a student assistant at RWTH Aachen University.
Jan Borchers is a professor of computer science at RWTH Aachen University.

This paper was presented at UIST 2010.

Summary


Hypothesis
The researchers attempted to prove the concept of a surface that would allow for interaction with dynamic physical interfaces on the surface through electromagnets.

Methods
The hardware set-up of the device is composed of a display panel, IR emitter and detector, electromagnet system, and fiberoptic cable.

With these system, physical devices (called Madgets in the paper) can be placed on the surface and either be manipulated by the user or by the system itself. Through this kind of system, dynamic interfaces can be created and recognized by the device.

Through complex calculations, the researchers solved various problems such as actuation, heat prevention, tracking, and more.

After setting up their system, they began to explore the possibilities of a system.

Since this paper was written as a proof-of-concept paper they did not bring in participants to interact with the surface.

Results
After the researchers created and designed their system, they discovered several interesting ways to interact with the device.

The first was the creation of general-purpose widgets. These are widgets like buttons, sliders and knobs. One example they gave was that a user could put a slider widget on a table while watching a video and control where the video was being played from. Then the device would move the slider's knob based on time left in the video.

The next was the usage of the dimension of height. Using the magnet surface, they could create buttons that would rest above the surface and could be depressed and activated.

They then explored force feedback. By using the electromagnets, knobs can be given "perceived friction" and can resist slightly when the user attempts to use them.

They even experimented with more novel madget ideas such as creating a motor through the interaction between the surface magnets and the magnets in the physical device. Also created was a madget that provided audio feedback through a bicycle bell.

Discussion


One difficult thing about this paper was the fact that they presented no user study whatsoever. I'd be very interested to see either a study on how users interacted with a given interface or how the users created their own interface with the use of Madgets to complete a given task.

Physical interaction is generally much more intuitive, natural, and more precise than interaction with touch devices. Therefore, a system like what was presented in this paper could be come very handy especially when one needs to make precise inputs.

Another beneficial use of Madgets would be the creation of custom interfaces based on user preference. If a user is trying to control something (like maybe a sound system) they might want a certain knob in one place and another slider somewhere else based on their usage. Madgets would allow this kind of custom usage.

Paper Reading #14: TeslaTouch, Electrovibration for Touch Surfaces

Olivier Bau is a research scientist at Disney Research. He has received a PhD in computer science. with a focus on Human-Computer Interaction.
Ivan Poupyrev is a senior research scientist at Disney Research.
Ali Israr is a research scientist at Disney Research. He has received a PhD in mechanical engineering.
Chris Harrison is a research scientist at Disney Research.

This paper was presented at UIST 2010.

Summary


Hypothesis


The researchers proposed a touch screen device with new tactile feedback so that a user can feel the object on the surface.

Methods


In order to complete their test, researchers designed what they called the TeslaTouch apparatus. By running a very slight current through an insulator, they could simulate a given surface based on the wave type of the current.
The entire system consisted of the surface, combined with the projector, and IR emitter and detector.

To better equip the device to work with humans, they experimented with waves of different properties and sizes and had users classify their view of the wave.

They then experimented with "psychophysics" of the device. They measured the detection threshold of the electrovibrations and the JND (Just-noticeable-differences) of the electrovibrations. By testing for these thresholds, they could determine which electrovibrations were noticeable and which were detected.

Finally, they discussed various types of interactions that TeslaTouch could be used with. These include simulations, GUI widgets, and various other rubbing interactions.

Results


Through their human factors test, they discovered the various hertz and voltage combinations that users would classify as sticky/waxy, bumpy/smooth, pleasant/unpleasant, friction/vibration. This test would aid in the future development of the device.

With the psychophysics test, they were also able to determine the thresholds previously mentioned

Discussion


Feedback is one of the most lacking features of touch screen devices. With this kind of simple tactile feedback, touch screen displays could become much more usable and create an even deeper layer of interaction.

Typing on a virtual keyboard would feel much more natural and would allow a user to rely on tactile feed as opposed to only visual feedback.

One thing the researchers could have included in this paper was a larger field study of this TeslaTouch apparatus. I would be very interested to see how normal people react to the device and to understand what kind of applications they could visualize.