7.1 Camera and Facial Expressions
A simple webcam can be used as an interesting input device, especially by recognizing and tracking faces and facial expressions.
7.1.1 Challenges
Facial expression is a relevant interaction modality because it requires little effort to change facial expression and it can be performed in parallel to manual activities like moving the mouse or performing a gesture. On the downside, facial expressions can lead to social awkwardness when done in public. Moreover, one has to be careful to ensure that no unintentional actions are triggered.
7.1.2 Student Projects
Check out various student projects using camera input, and specifically facial expression, for interaction (there are several more):
- PICaFACE: Controlling camera features with facial expressions
- faceTYPE: Formatting text with face gestures
- Facial Call Skyping: Video conferencing using facial expressions
7.2 OpenCV for Processing
OpenCV is a powerful "computer vision" library developed by Intel. Computer vision is the science of extracting meaningful information from images or video.
Originally developed in C++, there is a port for Processing called OpenCV for Processing
For interaction, the following functionality may be interesting:
7.2.1 FaceDetection
Allows to identify a face in the frame. This becomes more unreliable when the person is not directly facing the camera.
7.2.2 BrightestPoint
Detects the brightest point in the frame. Can be used to interact with a small flashlight.
7.2.3 BackgroundSubtraction
Detects moving objects by comparing the current image frame with the previous one and subtracting pixels. Works with a static camera and constant light conditions.
7.3 Facial Expression as Input
FaceOSC is a tool that recognizes your face and puts a 3D mesh over it so that mouth shape, eyebrows and other features can be recognized. As the name implies the tool sends recognized key values via OSC to other tools (like Processing). Here's an example of what you can do:
Here's another example: the student project faceTYPE by Alice Strunkmann‐Meister and Rodrigo Blásquez at Augsburg University of Applied Sciences.
If you want to work in Processing you need to do the following steps:
- Install FaceOSC (pick the file for your OS); this is a separate software
- Install Processing
- In Processing, install the library oscP5 (under Sketch > import library)
- Go to the FaceOSC-Templates project and - in the processing folder - download e.g. FaceOSCReceiver
To start up everthing:
- Start FaceOSC: you will see a window with your camera screen and - if a face is present - a mesh over the face.
- In Processing, open and start FaceOSCReceiver: you will see a stylized face that mimics your facial movements
This is what Processing shows you:
Now you can think up all kinds of actions and features that you control with your eyebrows or mouth.
Here's a list of signals that you receive on the Processing side (from FaceOSCReceiver):
public void found(int i)
public void poseScale(float s)
public void posePosition(float x, float y)
public void poseOrientation(float x, float y, float z)
public void mouthWidthReceived(float w)
public void mouthHeightReceived(float h)
public void eyeLeftReceived(float f)
public void eyeRightReceived(float f)
public void eyebrowLeftReceived(float f)
public void eyebrowRightReceived(float f)
public void jawReceived(float f)
public void nostrilsReceived(float f)