Interfacing with Other Applications¶
We support a number of animation packages via different routes.
|Autodesk Maya||Autodesk Motion Builder||Autodesk Softimage/ FaceRobot||Autodesk 3DS Max||Blender||Maxon Cinema4D||Unity3D||Custom Tools|
|plugin||Plugin with integrated retargeting||Plugin generating a virtual device||open source plugin||Plugin with integrated retargeting|
|virtual markers||bvh and c3d format||c3d format||bvh format||c3d and bvh formats||bvh format||bvh, c3d, fsopen formats|
|blenshapes / morph targets||fbx format||fbx format, bvh format||fbx format||fbx format fsopen format||fbx, bvh, fsopen formats|
|tcp/ip stream||1||1||1||networking interface|
1 The plugins receive the data over a TCP/IP network stream, so it is also possible to stream to a different computer.
Export of Virtual Markers¶
One of the main advantages of faceshift studio is that it is marker-less. Nevertheless, there are many animation pipelines that work with marker-based motion capture solutions, and we therefore provide ways to export virtual markers. Faceshift studio supports marker export in c3d format, bvh format, and in our own Faceshift Open File Format (binary or ascii). The c3d format is compatible with Autodesk Softimage/FaceRobot, Autodesk MotionBuilder, and Blender. The bvh format is compatible with Blender and many other packages. The export options are the same for all four formats.
Export of Clips
To export a virtual marker set:
- Select a clip.
- Press the Export button.
- Choose the export format (c3d, bvh, or fs open format).
- Choose the output file.
- Configure the marker set output and other options in the panel on the right (see below).
- Save the file.
This tutorial shows you how to export virtual markers via c3d or bvh:
Tutorial - Marker
- Choose a marker set.
- none: do not export markers (only relevant for fs open format export)
- facerobot: an Autodesk Softimage/FaceRobot compatible marker set
- load...: load a marker file
- custom...: a custom marker set
- Press Edit to edit the marker set (see section Marker Set Editing).
- Choose reference: virtual markers are either exported as markers on the default rig (default face) or on the personalized rig of the user (actor face).
- Choose framerate: the sequence is resampled accordingly. Native does not perform resampling and uses the frames as they were captured during tracking, i.e., at an irregular frame rate.
- Choose unit: the markers are exported in the selected unit.
- Choose head pose output.
- define whether to export head pose (rotation and/or translation)
- the head pose may either be set in absolute coordinates (camera is at center), relative to the neutral pose set using the Orient Head button, or relative to the first frame of the sequence.
- Choose additional output.
- check Export Video to export the video to a multimedia file (faceshift studio will open a dialog asking for the filename, where you can also set the video compression settings and whether the audio should be included)
- check Export Video Frames to export the video to a JPEG or PNG image sequence (faceshift studio will ask for the filename and export a sequence of image files)
- check Export Audio to export the audio track to a wav file (faceshift studio will ask for the filename)
For c3d export, the head pose will be directly applied to the marker values. The eyes are automatically exported as marker positions, one marker for each eyeball center, and one marker for the pupil. The marker names are “ELCT” (eye left center), “ELPP” (eye left pupil), “ERCT” (eye right center), “ERPP” (eye right pupil).
The head pose and eye positions/gaze are exported as joints in the hierarchy. Additionally, the blendshape coefficients are also stored as an extra set of joints.
Export of Blendshapes¶
Blendshape coefficients can be exported in the faceshift text based format, as part of an FBX export, or as additional bones in the bvh output format.
When you export a clip in BVH format (see BVH Export), faceshift will add a set of bones in addition to the virtual markers, where each bone corresponds to a blendshape. This is reflected by the bone names. The blendshape activation between 0 and 1 will be mapped to a rotation between 0 and 90 degrees, as can be seen in this screenshot from Blender:
Blendshapes in Blender
The current blendshape list can be found in the Tracked Blendshapes section.
The FBX export dialog is very similar to the marker export dialog (see figure marker export dialog). Choose your target in the list, the head pose options (should it be relative to neutral, to the first frame or absolute?), the fbx version you want to export for and the format. Additionally, you can export the video data (as movie or image files) and the audio.
FBX Export Dialog
Note the Export All option in the main application menu. The same FBX export dialog appears but all your current clips will be exported automatically. If the clips were defined as takes, they will be exported in the take directory that you have defined, otherwise they will be exported in the performances directory of your workspace.
Tutorial - FBX Export
The video and audio content of a clip can be exported by selecting the corresponding items from the export menu. Moreover, the user can select to export a video containing the sequence of screenshots of the tracking display.
By selecting the corresponding entry in the export menu, the audio content of a clip can be exported into an uncompressed .wav file.
Video Export Dialog
The RGB video content of a clip can be saved into a multimedia file (.avi or .mov containers allowed). The video can be resampled to have a fixed framerate, or the frames can be encoded using their acquisition timestamps (native). The file can contain an uncompressed RGB stream, or an MJPEG (lossy) compression algorithm can be used to reduce the final video size. A slider allows you to find the desired compromise between video quality and size of the file.
Export Video (single frames)¶
The RGB stream can be exported as a sequence of .jpg images. A dialog asks the user for the name prefix, to which the frame number will be automatically appended. You can also select to resample the video to have a fixed framerate.
Export Video (screenshots)¶
This action lets you save a videoclip containing screenshots of the tracking, its content reflecting the choices made in the Display tab. The dialog that appears is similar to the one in the video export dialog above, letting you select the filename, the compression level, and whether audio should be included. In addition, the video size in pixels can be set, independently of the current window size.
This tutorial shows you how to use the different options of the video export:
Tutorial - Video Export
FBX Target Import¶
In addition to writing out different formats, we also allow you to import new characters to be animated into faceshift studio. You can also animate the character and then write the animation out again. Data exchange happen via the FBX file format.
Tutorial - FBX Import
In order to import your own character into faceshift studio, we have the following requirements and limitations:
- Blendshapes: faceshift studio currently only supports blendshapes to map the facial expressions of the actor face to your imported character. The blendshapes do not need to correspond one-to-one. Instead, you will need to define a combination of blendshape weights on your character for each blendshape of the actor face (see section Tracked Blendshapes) in faceshift studio.
- Head Pose: to map the head pose onto your character, your character needs to have a neck joint.
- Eye Gaze: to map the eye gaze onto your character, your character needs to have a joint for each eye.
In the Display tab you can select different targets for animation. faceshift studio provides a set of default characters, but you can also import your own character via the FBX file format. You will need to create a mapping from the faceshift studio actor face to your own characters. The mapping can be saved, and faceshift studio will automatically list all imported target characters where the created mapping file resides in the target directory of the workspace (see section General Preferences). To import a new character, click Import, or click Edit to modify a previously imported character.
In order to import your own character, you need to create a mapping between the faceshift studio actor face and your target character. Start by loading the FBX file of your character by clicking New Target. The left pane should now display your faceshift studio’s actor, while the right pane should display your imported character, potentially misaligned and at different scale. You can adapt the alignment and scale using the controls at the top left part of the window. Keep in mind that a global translational alignment between the actor’s face and the target character is automatically defined once you map the neck joint of the actor to the neck joint of your own character, so translational alignment should only be set after mapping the joints. You may define a name of your imported character (the FBX filename is set by default). In case textures are not displayed correctly, as the FBX file may contain invalid file paths, you can define a default texture directory where the textures of your target character reside.
Target Import Dialog
Head pose and eye gaze are defined by joints in the actor face. In order to map these to your own target character, your rig also needs to have the corresponding joints. Please select the appropriate joints in your rig for each of the actor face joints. Joints that are not mapped will be ignored, e.g., if you don’t map the neck joint, then head pose will not be mapped onto your target character.
Expressions are defined via blendshape mapping. For each of the blendshapes of faceshift studio, select and activate the corresponding blendshape(s) on your target character. One blendshape in faceshift studio may be mapped to multiple blendshapes in your target character and vice versa.
If the blendshape names of your character match the ones used by faceshift studio, the button Map Automatically will automatically assign them. Note that this is always done when you load a new target character that doesn’t have a mapping yet.
Note that for animation targets, we corrected the FBX importer in faceshift studio 2015.1 to use as blendshape identifiers the deformer name concatenated with the blendshape name. Before, parts of the channel name were included if they did not fit the deformer or blendshape name. This caused incorrect blendshape identifiers for models where the deformer and blendshape name did not match the channel names.
Your mapping may be stored to disk as a text file (.fst file) by clicking Save Mapping, and accordingly you can load a previous mapping by clicking Load Mapping. In order to have faceshift studio load a mapping by default, save the mapping file to the target directory in your workspace.
A networking module is provided for real-time streaming of the tracking data and for remote controlling of faceshift studio. The communication protocol uses the fs binary format via either TCP/IP or UDP. Remote control functionality is only supported in TCP/IP mode.
This video tutorial explains how to export the tracking data in various format via the network:
Tutorial - Network & fsopen
We offer a generic C++ parser for reading and writing the binary format. Download it from here.
In this section we describe the underlying format, if you want to write your own parser.
FS Binary Format¶
The fs binary format is a hierarchical block-based data structure (in little-endian encoding). A general binary block has the following structure:
|uint16||1||version number||1 (current version)|
|uint32||1||block payload size||N|
|N bytes||block specific playload, see below|
For all binary blocks, the basic data types are encoded in little-endian format. The data type string is encoded as follows:
String Data Type
|uint16||1||length of string||L|
|uint8||L||the ASCII characters|
Faceshift studio streams various data blocks over the network. The default stream only consists of the tracking parameters (tracking state) for each frame, the other data blocks may be requested via remote controls (see futher below).
|55355||Signal (Person detected or leaving)|
(Block ID 33433) The tracking state parameters are a set of sub-blocks that are contained within one single Tracking State block. You can therefore read a full frame by looking for the Block ID of the Tracking State (33433) and then read the full frame. Note that the tracking state is completely described by a single Tracking State data packet, i.e., data is not split into multiple packets. An exemplary binary data block is shown further below. The Tracking State block is a binary block containing multiple sub-blocks with individual tracking parameters:
33433: Tracking State
|uint16||1||the number of sub-blocks||N|
|BLOCK||N||(at most one sub-block of each type, see below)|
The Tracking State block contains five different binary sub-blocks:
Tracking State sub-blocks
The binary block structure for the various tracking parameters follows.
(Block ID 101) Frame Information - contains the timestamp and a flag telling whether tracking was successful for this frame.
101: Frame Information
|double||1||timestamp in ms|
|uint8||1||flag whether tracking successful||0/1|
(Block ID 102) Pose - the head pose as an absolute or relative transformation (depending on the setting). Relative means computed with respect to the head pose of the actor at the time when the “Calibrate Neutral Pose” button was hit.
|float||4||head pose - rotation in quaternion format (x,y,z,w)|
|float||3||head pose - translation (x,y,z)|
(Block ID 103) Blendshapes - the blendshape coefficients for the current frame.
|uint32||1||number of coefficients||N|
(Block ID 104) Eyes - the eye gaze for left and right eye in degrees.
|float||1||left eye theta (in degrees)||(-90,90) (neg: up, pos: down)|
|float||1||left eye phi (in degrees)||(-90,90) (neg: right, pos: left)|
|float||1||right eye theta (in degrees)||(-90,90)|
|float||1||right eye phi (in degrees)||(-90,90)|
(Block ID 105) Markers - the virtual marker positions as set within the application (depending on the setting in faceshift studio).
|uint16||1||number of markers||M|
|float||M*3||marker positions (x,y,z)|
(Block ID 33533) The M marker names currently streamed in the Tracking State are sent via network if a “Send marker names” packet is received from the client (see remote control commands below).
33533: Marker Names
|uint16||1||number of markers||M|
(Block ID 33633) The N current blendshape names of the coefficients streamed in the Tracking State are sent via network if a “Send blendshape names” packet is received from the client (see remote control commands below).
33633: Blendshape Names
|uint16||1||number of blendshapes||N|
(Block ID 33733) The current trackig rig is sent via network if a “Send rig” packet is received from the client (see remote control commands below). The rig includes the mesh topology and all blendshapes.
|uint32||1||number of quads||Q|
|uint32||1||number of triangles||T|
|uint32||1||number of vertices||V|
|float||V*3||the vertices of the neutral shape (x,y,z)|
|uint16||1||number of blendshapes||N|
|string||N||the blendshape names|
|uint16||1||number of blendshapes||N|
|float||V*3*N||the vertex offsets (x,y,z) of all blendshapes, in the same order as in Tracking State|
(Block ID 55355) Signals correspond to state changes of faceshift studio. Currently implemented signals are “person entered” and “person left”.
Faceshift studio supports remote control functionality in TCP/IP network mode. The following commands are accepted by faceshift studio (set message size of binary block structure to 0):
|44644||Send marker names|
|44744||Send blendshape names|
|44944||Stream head pose relative to neutral|
|44945||Stream head pose in absolute position (camera at zero)|
Faceshift Open File Format¶
Tracking data can be exported using the fs open format. When exporting in fs open format (binary or ascii), all tracking parameters and virtual markers are exported.
FS BINARY Format¶
The fs binary format is a hierarchical block-based data structure. The same format is also used for network streaming of the tracking data, therefore the exact format is documented in the network communication section (see section Faceshift Networking), along with links to the C++ classes we provide to simplify access.
An exported *.fsb binary file simply consists of one Blendshape Names block (ID 33633), followed by any number of Tracking State blocks (ID 33433), one for every frame of the clip. Since the format is also used for streaming, the file does not store the number of frames it contains.
FS ASCII Format¶
Faceshift studio supports output as ASCII files. The data is written in a block structure and each block starts with a specific keyword, marked in bold below:
- FS numblocks
- I timestamp (in ms) trackingSuccess (0 or 1)
- P qx qy qz qw (head rotation as quaternion) tx ty tz (head translation in mm) (the head pose coordinate system is defined in the application - absolute or relative)
- C numCoefficients v1 v2 ... vn (blendshape coefficients that are tracked in faceshift)
- E leftEyePitch leftEyeYaw rightEyePitch rightEyeYaw (in degrees)
- M numMarkers x1 y1 z1 ... xm ym zm (the unit depends on the setting in the application)
Two lines of the exemplary format without any markers:
FS ASCII Format
FS 5 I 33.3333 1 P -0.000162818 -0.00131751 0.000690874 0.999999 -0.244612 0.0544815 0.0306396 C 46 0.0997361 0.0997361 0.932142 0.932142 0.209866 0.205085 0 0.395189 0 0 0.220958 0 0 0 0.19631 0.345989 0.0517451 0 0 0.0506901 0.0107231 0.0588932 0 0 0.00105792 0 0 0 0.0153362 0 0 0 0.0262555 0.0680846 0.00323772 0 0 0 0 0.0654269 0 0 0 0 0 0 E -6.02331 6.37829 -5.88561 11.4575 M 0 FS 5 I 66.6667 1 P -7.38358e-06 -0.00308635 0.00197219 0.999993 -0.624882 0.0363846 0.043457 C 46 0.0965532 0.0965532 0.909633 0.909633 0.198588 0.19369 0 0.425377 0 0 0.251909 0 0 0 0.169044 0.295062 0.0402409 0 0 0.0649989 0.000186683 0.180477 0 0 1.84178e-05 0 0 0 0.0266345 0 0 0 0.000457093 0.116282 0.0708589 0 0 0 0 0.0424745 0 0 0 0 0 0 E -5.69853 7.27203 -5.55754 12.3389 M 0
Note that no block is contained for the blendshape names, the ASCII format simply assumes the current set of blendshapes, in order. The list of blendshapes can be found in the Tracked Blendshapes section.