HCI: Lesson 4

This article will now focus on the new functionnalities that append after the WIMP (windows, icons, menu, pointing) paradigm. These technologies are linked with the appearance of mobiles, but also AR and VR. We will study these different new solutions that were created during the last years.

Mobile and Ubiquitous Computing

Mobile computing is currently creating the majority of the internet traffic (203 minutes per day vs 128 for computers). HCI challenges differs from desktop computing. It is important to understand what do people do on their mobile to compute a better HCI. Here are the usages:

- computer: large display, rich input, mostly for working and gaming

- mobile: small display, simpler input, mostly for social media and streaming

We arrive at a period where we use both of them with the same percentage. So let understand some technologies linked to that.

Mobile Computing

Touch Technology

The most important technology that appeared in the mobile field was the touching screen. We use our finger, or a stylus to deform a polyester film and a top resistive circuit for them to touch a bottom resistive circuit. There is also capacitive touch, where electrods react to the finger and create a capacitive image. Another method is the FTIR touch: infra red lightings are spread into an object inside an acrylic surface. Our fingers change the reflection inside it.

All input devices are not created equal in their capabilities. Buxton presented a model of the different states interaction has with touch and non-touch interfaces. In this way, with a stylus or our finger, we can simulate a similar behaviour on tech interfaces.

Midas Touch

Using a pointer in WIMP interfaces enabled us to have to states (hover, click). However, using touch looses this hover ability. We have one potential solution: liftoff selection. We also see a video of VR where the player keep his fingers in the screen for the hover action, and release in order to click (or stand in one position for a longer time). This is one solution.

Fat Finger Problem

We are confronted to another problem. Fingers are large compared to stylus, so they can select many objects at the same time, or cover part of the screen.

One solution would be to use and touch the back of the screen: it avoids occlusion and selection troubles.

Also, in order to avoid the tiny selection that we must do with our fingers, we can use the Midas touch process (letting the finger on the screen and creating a shifted cursor, or considering the angle of the finger are solutions that are considered).

Why so unique ?

The usage of mobile is unique because of different reasons:

- small display (off screen visualization techniques, multi screens)

- large input device (fat finger correction, better understand what wants the user)

- additional sensors

- different context of usage

Ubiquitous Computing

The principle of ubiquitous computing is that computers are multiplying, they are around us and we won't know when we are using one. Interactions with these technologies will become automatic and implicit. 3 driving forces are important here:

- miniaturization (Moore's law drives the progress)

- networking (progress of protocols, network technologies)

- energy (use alternative like Energy Harvesting)

Tabs (phones, smart watches), pads (computers) or boards (TVs) are more and more used for this purpose. These devices allow also the appearance of new kinds of computing methods.

Devices mustn't be physic (projected keyboards for example), and they can we wearable (head-mounted displays, sensors on the body,...)

To conclude this part: technologies will be around us, not expensive, and we won't be able to know that we are using these devices.

AR/VR/MR/XR

First, let's have some definitions:

- Augmented Reality: combination of real and virtual, real-time interaction

- Virtual Reality: fully virtual world, real time interaction

- Mixed Reality: continuum between real and virtual environment

- Extended Reality: include everything mentioned before

We have different devices that are used for these purposes.

Projector-Camera System: projector system where a camera can interpret touches on what is displayed

CAVE: place with many screens in order to add immersion (we tried one in Evry)

Stationary VR: VR headset where we don't move

Mobile VR: same but we can move, put our smartphone into an headset

Handheld AR: like Pokemon go, on mobile device, or tablets, computers,...

Head-Mounted Displays

These displays are the best to access to AR and VR because we see the virtual elements without handing anything. It is composed of a display part (that imitates the natural viewing behaviour of human), a tracking part (locate the position of the user), an input part (allow to have a real time interaction) and a processing part (used for tracking and input).

Also, the important parameters are the field of view, the resolution and the refresh rate.

Now we will see the different kinds of head-mounted displays.

- Optical See-Thru HMDs: combining digital images with the real world using some form of optical combinations of the light from the real world and digital images (examples: HoloLens, Nreal, MagicLeap)

- Video See-Thru HMDs: combining digital images with a video of the real world using video compositor (example: Varjo XR-1 2, Vive Pro Eye)

- Occluded HMDs: the user only sees a virtual rendering which can be completely independent of the physical environment (example: Oculus Quest 2)

- Multiplexed HMDs: refers to glasses, wereables with informations shown on it

HMDs Tracking

We can find two types of tracking systems:

- active tracking: we use physical sensors and object to do the tracking. There are the mechanical arms with joints sensors (expensive but precise), magnetic systems (robust but sensible to interferences) and ultrasonic devices (measure the time of flight, cheap but not really precise).

- passive (optical) tracking: we combine cameras, computer vision and machine learning to do the tracking. We can use markers (standard computer vision can compute tracking thanks to them), or not (FAST corner detection, Keypoint Detection). We can also use SLAM for tracking and mapping if nothing in the scene changes.

HMDs Interactions

We can find different ways of interacting in AR and VR. We can navigate inside a scene, have interactions with object (selection and manipulation) and control the system (command and parameters).

- Navigation: Not a problem for AR, but more difficult for VR. The locomotion can be physical (walking in place, in the real world), virtual (by using joysticks, teleportation), but can cause some fatigue and sickness.

- Object interaction: We are able to modify the position, rotation of any object. There are common techniques (virtual hands, arm extensions, ray-casting) and common devices (using controllers, hand tracking). However, some problems are known (fatigue, lack of precision).

Another important problem is that we touch the void. So we have to simulate haptics.

- System Control: We can use commands in order to change the state of the system. They can be physical (hardware buttons on the headset, simple selections and volumes) or virtual (digital menus, flat 2D interfaces).

Perceptual Phenomena

The usage of HMDs leads to phenomena that must be considered. The immersion (feeling, sensation of being where the technology wants us to be) is really important, and can be verified thanks to questionaires (SUS, Wittmer and Singer for example). Motion sickness is also somthing that we have to consider (caused by a conflict between what we see and what we feel).