Training

Each person using faceshift studio requires a custom profile. This profile represents a user specific expression model, i.e., a set of blendshapes that represent the different expressions a user can do (we use the term blendshape to represent a certain facial expression - the equivalent of morph targets). For example, it contains a blendshape Jaw Open which represents the way a specific person opens their mouth.

Currently, we are tracking a fixed set of blendshapes, which can then be mapped to the target rig. As of faceshift studio 2014.1, the default blendshape rig contains 51 blendshapes. Section Edit the Template Rig describes how you can change which blendshapes are tracked. If you would like to use the previous blendshape sets from previous versions of faceshift studio, please send a mail to support@faceshift.com.

In the training mode, the actor creates their profile by recording a set of predefined expressions and faceshift studio will automatically calculate all blendshapes for the user.

Tutorial - Training Overview

The Training module

Capturing a profile

In order to capture a set of training expressions, you need to do the following:

  • Select the expression to capture in the expression table on the left.
  • As a guide, we visualize the expression on the default head.
  • Look into the sensor, perform the required expression and keep it fixed; faceshift studio will detect your face and display it as a video image and depth scan (see figure training ready). In case the face is not detected, a message will appear (see figure no face detected).
  • Press and hold the Capture button, then turn the head slightly left and right without changing the expression so that faceshift studio can combine the data from the sensor to build one noise-free scan of your face (see figure scanning).
  • When finished, look into the camera again, release the Capture button and move on to the next expression.

Once the expressions have been captured, press Build and faceshift studio will build your actor profile. Save it using the Save button so that you may start a new session at any time without having to go through the training phase again. To load a trained expression model, click Load. Since faceshift studio 1.2, the profile file (*.fsu) also contains the scan data of all captured expressions.

Before using the profile, it is advisable to check that all expressions have been registered well. If an expression is not correctly fit, you can either redo the expression by pressing Rescan or add manual markers (see section Marker Set Editing).

If you are having trouble in performing particular expressions or getting a good scan for them, you are free to leave them unscanned. In that case, faceshift studio will create the blendshapes for you by applying a generic model to your neutral expression. However, the tracking results will be better using properly scanned expressions.

Note: When you are confident enough with this phase, you can speed up the training process by enabling “Automatically advance to next expression after scanning” in the “General” section of the Preferences menu.

The training pane, with the user ready to scan the neutral expression

Scanning of the neutral expression

Message shown when the face is not detected. Open the setup tab to check that the camera is set up properly, and the video stream and depth data are coming in at a good quality. Also, make sure to properly face the sensor.

Options

Manual Correspondences

You can help faceshift studio to compute a better fitting by setting manual correspondences between the model and the captured scan data. Using the buttons here you can change, load, and store the marker sets used for these correspondences. To edit the set of markers, click Edit Set; a dialog window pops up where you can add markers. Select markers on the model by clicking, and click again on the corresponding position on the scan to set a correspondence. For further details, see section Marker Set Editing.

Display Options

  • Show Wireframe: Displays the wireframe of the model to give you a better impression of the 3D shape.

  • Show model in scan: Overlays the model over the 3D scan data. This visualizes the alignment. If the alignment seems incorrect, you can try to manually align the model to the scan (see Manual Scan Alignment).

  • Show fitting quality: Overlays a heat map on the model to visualize where the model aligns well (blue) and where the distance between the model and the 3D point cloud is larger (red).

  • Display Texture: Shows the color textures of the different scanned images projected onto the model and/or the scan.

    When “Show Wireframe” and “Display Texture” are activated and the model has the focus, press the key “A” to toggle the visibility of the wireframe. This can be useful when trying to modify the eye and/or eyelid positions.

Manual Scan Alignment

If the model and the scan are misaligned after automatic fitting, you can correct the alignment manually. You can enter manual alignment mode by pressing the “T” key. You will then see the model displayed in green on top of the scan, similarly to when the Show model in scan display option is selected.

You can improve the alignment using the mouse. Drag with the left mouse button pressed to rotate the scan, drag with the right mouse button to translate the scan in the plane of the screen and use the scroll wheel to translate the scan in and out of the screen.

You can also press the “R” key to trigger a local automatic alignment. When you are finished, press “T” again to exit manual alignment mode. To reset the fit, perform “Delete Fit” and then “Fit” shown in the menu when right clicking on the expression.

On the left, a misaligned scan. On the right, the corrected alignment of this scan.

Eyes

The eye positions are detected during the fitting of the Neutral expression. At this point you are able to adjust the eye positions and their size if the detection was not accurate enough. When hovering the mouse over the eye regions, the different pickable parts will be highlighted. Clicking on the eyeballs or the eyelid parts will activate a coordinate system widget which allows you to change the position of the eyes or eyelids. Pick an arrow and drag it until the expected position is reached.

At this point, the other expressions need to be fitted again to use these new values. You can simply press Build to update your profile.

Eye PickingEyelid Picking

Eye Modifications

If the eye tracking doesn’t seem to deliver accurate results, we recommend to add eye expressions to the training template. To do that, simply press the Expressions button and select Add. In the rig viewer, give a name to your new expression (e.g. “EyeSquint”) and set the corresponding blendshapes to 1.0 (in this case EyeSquint_R and EyeSquint_L).

The recommended eye expressions consist of EyeOpen, EyeUp, EyeDown, EyeBlink and EyeSquint and you also have the possibility to adjust the eyelid positions if the fitting is not accurate enough for those.

Training the eye expressions can sometimes give worse results than without training them. This is especially a problem when they’re not trained well enough, e.g. training EyeOpen without having eyes more open than on the Neutral expression, which can cause artifacts in eye tracking (in this example, eyes won’t close properly anymore).

Important

We recommend to use eye expressions only when necessary if the eye tracking doesn’t seem to deliver accurate results. Always start by creating a profile without eye expressions and check the eye tracking first to decide if you should include eye expressions in your profile.

Note

Please note that if you fit the neutral expression again, the eye positions will be reset to their original positions. This will happen automatically if you move any of the markers on the neutral expression and then build the profile.

This tutorial shows you how to modify the position and size of the eyes:

Tutorial - Eyes

The Set of Training Expressions

Faceshift studio asks the user to perform the following expressions for training. It is also possible to train with fewer expressions, but this results in a decrease of the tracking quality. More expressions can be added to further improve the tracking by using the Expressions menu below the Training Expressions in faceshift studio.

Training Expressions in faceshift studio

Name Description
Neutral The actor’s neutral expression with closed mouth. For best results the actor should not fully clench the teeth but leave some space between the teeth instead. Note though that the lips should be closed.
Open The mouth should be opened widely without stretching the mouth corners.
Smile A wide smile with closed lips.
BrowsDown As for an angry expression, the brows should be put down as much as possible, while keeping the remainder of the face unchanged.
BrowsUp As for a surprised expression, the brows should be pulled up, but the remainder of the face should stay as still as possible.
Sneer Wrinkle the nose (as for a disgusted expression) and move the brows down, similar to the central part of the angry expression.
JawLeft Move the jaw to the left, without opening the lips.
JawRight Move the jaw to the right, without opening the lips.
JawFront Move the jaw to the front, without opening the lips.
MouthLeft Pull the mouth to the left.
MouthRight Pull the mouth to the right.
Dimple As in a smile, but without rasining the mouth corners.
ChinRaise Move the mouth towards the nose.
Kiss A kiss with the lips put together and pushed to the front.
Funnel Similar to kiss, but with an open mouth.
Frown As for an unhappy face with the mouth corners down.
M Rolling the lips in, as it happens when pronouncing an exaggerated “m”.
Puff Blowing out the cheeks.
Chew Open the jaw but keep the mouth closed.
MouthPress Press your lips.
Stretch Stretch your mouth corners as much as possible to the opposite sides. Different from dimple.
LipLowerDown Lower your lower lip without opening the jaw or moving the mouth corners.
LipUpperUp Raise your upper lip without opening the jaw or moving the mouth corners.

When scanning the expressions, observe the preview that tells you how it is supposed to look. For instance, Sneer is meant to be performed without showing the teeth. You may choose to train the expression by also pulling up the upper lip. In that case you must adjust the expression by clicking the Expressions button, selecting Edit, and changing the blendshape activations. For expressions where the mouth is closed, check the zipper checkbox, otherwise uncheck it.

Check out the tutorial video how to add additional expressions for training to improve the tracking model:

Tutorial - Training Expressions

Check out the tutorial video on you how you can export the training expressions and blendshapes models in various formats:

Tutorial - Training Export

Tracked Blendshapes

Since faceshift studio 2014.1, 51 blendshapes are used for tracking by default. In the main menu under Training, select Inspect Template Rig to have a look at all blendshapes used in the default template rig for tracking. Similarly, you can have a look at your own profile buy selecting Inspect Profile. By clicking on a specific blendshape name you are able to control its value and directly see its effect on the displayed mesh.

Rig viewer dialog

Edit the Template Rig

In the main menu under Training, you can select Edit Template Rig to add, remove, or modify the blendshapes used by the template. When the Split option is selected, the corresponding blendshape is separated into two independent blendshapes, generally for the left and right side.

For example, if the split option is activated for “MouthSmile”, “MouthSmile_L” (left side) and “MouthSmile_R” (right side) will be considered as two distinct blendshapes. It allows the tracking to be more accurate by handling asymmetric expressions.

Note that you can also disable shapes during tracking, for instance if you know that a certain shape will not be active in a particular take. See section Shapes.

Rig editor dialog

Marker Set Editing

Faceshift studio is a marker-less application. However, there are several cases where it is useful to simulate markers as they are used by facial motion capture. In faceshift studio, you will find them in three different contexts:

  • In the training mode, you can manually add markers to the scanned faces to help faceshift compute a better fitting.
  • Through network streaming, faceshift studio can send virtual markers to other applications as if they were captured by a marker-based motion capture system. The markers that are sent over the network can be configured in the preferences.
  • Likewise, virtual markers can be exported in C3D, BVH, or faceshift’s open file format.

Whatever the intended use, the marker set can be edited using the same dialog:

Marker editing dialog

In order to add and delete markers, right click the mesh and select the corresponding menu entry. If applicable, a symmetric counterpart will be added automatically.

Add a marker

The size of the marker spheres can be adjusted using the slider, which can be useful if you want to place markers close to each other. Using the Snap radio button, the markers can be forced to lie on a vertex or an edge. You can write the marker set to disk for later reuse using the Save button and load it again using the Load button.

Editing the markers in the Training module has the same functionalities: adding and removing markers, changing their positions, resizing the markers size using the slider or the mouse wheel while holding the Control key.

Editing markers in the Training Module

When you move a marker, its position will always be projected on the mesh’s surface. It can happen that this fitting is wrong and the marker doesn’t appear correctly on the surface. In this case, you can use the Shift key on the selected marker to see a coordinate system widget appear. You can then change the marker’s position in space. But as soon as you move the marker normally it will be projected to the mesh’s surface again.

Moving markers in space

Tutorial - Training Markers