Back to Page References

Plugin Overview

The OptiTrack Unity3D Plugin allows real-time streaming of live or recorded motion tracking data from Motive into Unity. Kinematic data from both rigid body and skeleton tracking can be streamed via the plugin, and received data can be used for animating objects and characters within a scene in Unity. The plugin contents are distributed in a unitypackage format and you can simply load this file into Unity projects. Once imported, the included C# scripts can be attached to Unity objects for instantiating a client and receiving the tracking data. Detailed steps will be explained in this guide.

Skeleton Retargetting

By integrating with Unity's animation system, Mecanim, the Unity3D plugin allows Motive to stream full body skeleton data without the need of separate retargeting middle software. The skeleton tracking data from Motive is streamed out as hierarchical bone segment orientations, and this data is fed into the Unity's Mecanim system which allows animating characters with different proportions.

Note: At the time of writing, Mecanim does not support explicit goals for inverse kinematics end-effectors when using real-time retargeting. In addition, you may observe a difference in the overall scale of the position data between the retargeted skeletal animations and streamed rigid bodies. These two limitations may lead to inconsistencies with actors interacting with rigid body props, and will hopefully be addressed in a future version of the integration.

Virtual Reality Experience

The plugin also features HMD tracking integration for virtual reality experiences. OptiTrack motion capture systems provide rigid body and full-body human motion tracking with outstanding tracking accuracy, and all of which can be accomplished with extremely low latency. Utilizing the tracking capability, an OptiTrack system can be used to provide fully immersive virtual reality experience by integrating a head mounted display (HMD) device. From this combined setup, even tiny movements can be precisely captured and reflected in the virtual space, and such accurate representation will deliver an immersive virtual experience.

For general instructions on developing VR application in Unity, refer to the Unity documentation: Unity Virtual Reality tutorial.

Supported HMD Models: Oculus DK2, Oculus CV1, and Gear VR.

Versions Requirements

Unity Version: 5.3+ (5.3.6p1+, 5.4.0p1, or above for HMD integration)
The version of the Oculus SDK should be compatible with Unity version. See: Unity-SDK version compatibility

Streaming tracking data into Unity

Motive Setup (Server)

Streaming Setup

Data Streaming settings in Motive.

Tracking data can be streamed in real-time (Live Mode) or from a recorded capture data (Edit Mode). From Motive, data broadcasting can be enabled, or disabled, from the Data Streaming pane, and in this panel, you can configure the connection settings and decide which data to include within the delivered data packet. For the best performance, disable streaming of unnecessary data types. This will reduce the size of data packets and simplify the parsing process on the client side. However, make sure the following data types are enabled for streaming:

Streaming Rigid Bodies or HMD

Stream Rigid Bodies → True

Streaming Skeletons

Stream Local Rigid Bodies → True
Stream Skeleton → True
Skeleton As Rigid Bodies → False

Streaming Over Wifi Network

Transmission Type → Unicast

Unity Setup (Client)

Unity plugin files.

Import the Plugin Package

While in the Unity project, double click on the plugin unitypackage file and import the plugin assets into the project. When the package has been successfully imported, the followign contents will be available within the project:

Plugin Contents
Folder	Content Description
Assets/OptiTrack	All of the Unity plugin contents are included in this folder.
Assets/OptiTrack/Scripts	This is the folder that you will mainly use. It contains plugin C# script components that can be imported into Unity objects for receiving streamed data.
Assets/OptiTrack/Editor	This folder contains editor scripts and contents for visualization of the rigid body markers.
Assets/OptiTrack/Plugins	This folder contains the plugin libraries and header files.
Assets/OptiTrack/Prefabs	This folder contains prefabs of sample client object, skeleton object, and HMD object.
Assets/OptiTrack/Scenes	This folder contains sample Unity scene that includes pre-configured client, rigid body, skeleton, and HMD objects.

Setting Up the Client Object

In order to receive tracking data from a server application (e.g. Motive), you need to create a client object. A client object can be any object within a scene in Unity, and it can be designated by attaching OptitrackStreamingClient.cs script. This script receives the tracking data from the connected server application (e.g. Motive) and makes the data available within the scene. You can either attach the client script onto an existing object or an empty object. Also, you can just import the sample from the Assets/Optitrack/Prefabs folder.

[Motive] In the Data Streaming pane, configure the desired connection settings and check the Broadcast Frame Data box.
[Unity] Under the Prefabs folder, import the client prefab object into the scene, or you can just attach OptitrackStreamingClient.cs script onto an existing object.
[Unity] In the Client object, configure the connection settings from the OptiTrack Streaming Client script so that the values (Connection Type, Local Address, Server Address, Server Command Port, Server Data Port) match the parameters under the Data Streaming pane in Motive.

Local Address: Local IP Address of the PC that the client application is running on.
Server Address: IP address of the PC that the server application is running on.

[Unity] Set the Bone Naming Convention type so that it matches the convention set in Motive.
[Unity] If you wish to receive tracking data from more than one server instances, you may create multiple objects with the client script attached.

Client object in Unity and the corresponding Motive data streaming network settings.

Position data in Unity

Position component of the streamed data will be received as reference positions in Unity. In other words, the streamed position will reflect object's relative displacement from its root object or the global origin if the object is placed at the root level of the hierarchy in Unity. For this reason, although it is not strictly necessary, you may find it helpful to organize your tracked objects as children of a common game object (e.g. the Client object) that can then be re-positioned in the Unity scene to effectively determine the location of the tracking volume origin. And doing so will make it easier to adjust the global origin of Motive within a scene in Unity.

[Unity Hierarchy Pane] Components set to children of the root Client object.

Animating Rigid Body

[Unity] On an object that you wish to animate, attach OpitrackRigidBody.cs script.
[Unity] In the Streaming Client entry, link the Client object which the client script is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
[Unity] For the Rigid Body ID entry, enter the User Data ID which is assigned to each rigid body asset in Motive.
[Motive] Make sure Motive is streaming, play the scene in Motive.
[Unity] Play the scene. The linked object will be animated according to the rigid body movement in Motive.

OptiTrack Rigid Body configuration along with the rigid body properties in Motive.

Animating Skeleton

[Unity] On Unity characters, attach OptitrackSkeletonAnimator.cs script as one of its components.
[Unity] For the Streaming Client entry, link the object which the client script is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
[Unity] Enter Skeleton Asset Name which is Assigned in Motive
[Unity] For the Destination Avatar entry, link to the character that the script is loaded to.
[Motive] From the Data Streaming pane, make sure Motive is streaming.
[Unity] Play the scene. When everything is set up properly, the linked avatar in Unity will be animated according to the streamed skeleton in Motive. The position of the actor will be in its reference position as explained above.

Optitrack Skeleton Animator script configuration from a character in Unity.

Integrating HMD (Oculus DK2, CV1)

[Motive] First of all, follow the HMD Tracking Setup section and prepare HMD rigid bodies in Motive.
Make sure the HMD software is running.
[Unity] In the Unity project, check the Virtual Reality Supported box from the player settings (Edit → Project Settings → Player), as shown in the image below.
[Unity] From the Prefabs folder (OptiTrack/Prefabs) of the imported plugin package, import the HMD prefab (HMD - OptiTrack) object into the scene.
[Unity] To this object, attach OptiTrackHmd.cs script.
[Unity] In the Streaming Client entry, specify the object which the streaming client is attached to. By default, it searches for an existing client instance, but this must be specified when there are more than one streaming client objects.
[Unity] For the Rigid Body ID entry, enter the User Data ID value given for the HMD rigid body in Motive.
[Unity] Play the scene. When the HMD is properly connected, the scene will be visible from the HMD.

VR Supported setting in Unity.

Integrating HMD (Android: Gear VR)

Internet Access settings for Android projects in Unity.

Gear VR devices can also be integrated and the tracking information can be streamed via wifi connections using a router with sufficient bandwidth. The required bandwidth will vary depending on many factors (e.g. router type, the number of tracked object, etc.). For more specific information on this setup, contact us. The following settings must be configured in addition to the above HMD settings for developing Gear VR experiences using the plugin. For more information on developing Gear VR applications in Unity, refer to Unity documentations.

Settings

[Unity] For developing Android applications in Unity, make sure the environment is set up for Android development: Getting Started with Android Development.
[Unity] Include the OSIG file (oculus signature) in the Project/Assets/Plugins/Android/assets/ directory. See: Oculus Signature File Generator.
[Unity] Check the Virtual Reality Supported box from the Player Settings (Edit → Project Settings → Player), as shown in the image below.
[Unity] Under Player Settings, set (Other Settings → Internet Access → Require).
[Motive] Under the Data Streaming pane, set the Network Type setting under the Advanced Settings to Unicast. Note that the plugin currently only supports the Unicast broadcasting for streaming onto Android, multicasting will be supported in the future releases.

Connecting via Wifi

1. [Android] Connect the smartphone to the internet router which the host PC (server) is connected to.

2. [Unity] Configure the Client object.

- Use Connection Type: Unicast.

- Set the Local IP address to (0.0.0.0). The local address is not required for unicast connections.

- For the Server Address, connect to the IP address that the host PC (server) is connected to. This could also be found from the Android OS when the smartphone is connected to the wifi router.

3. [Unity] Connect a smartphone into the PC, build and run the project. Make sure network permission is given to the Unity application.

4. Make sure Motive is streaming, and the Gear VR HMD tracking data will be streamed into the Unity application.

HMD Tracking Setup

Setup and optimize the motion capture volume as explained in the Getting Started guide or the Hardware Setup documentations. If you plan to install any obstacles (e.g. walls) within the capture volume, make sure they are non-reflective, and place and orient the cameras so that every corner is thoroughly captured by multiple cameras. For typical rigid body tracking, attach the rigid body markers as instructed from the Rigid Body Tracking page. However, when tracking an HMD, extra attention to placing the markers is necessary.

HMD Marker Setup

Attachment

When attaching retroreflective markers, make sure markers are securely attached and readily captured by the cameras. For attaching the markers, we recommend using our 20 mm wide and 30 mm tall M4 threaded plastic marker bases with Acrylic adhesives, available at the webstore, to attach the markers onto the HMD.

Placement

A markered HMD will be defined as a rigid body in Motive. When placing markers, make sure the placement asymmetry is respected in the arrangement within the HMD. Also, the marker arrangements between multiple HMDs must be incongruent. For more details, read about marker placement from the Rigid Body Tracking page. Also, for tracking the HMD, two landmark markers must be placed in the following locations:

Eye-level Side Markers (2)

Place two markers on left and right side of the HMD, these markers will serve two additional purposes. First, they will indicate yaw of the HMD, and they will be used to align the rigid body orientation with the orientation of the actual HMD component. Thus, a line interconnecting the two markers must be parallel to the frontal plane, or the display, of the HMD. Second, these markers will be used to locate the elevation of the eyes when creating the rigid body in Motive. In summary, the two landmark markers must be carefully placed considering the following:

The markers should align along eye-level of the user when the HMD is mounted.
Most importantly, place these markers in the exactly same location of the left and right side so that they form a precisely symmetrical arrangement.
Same dimension attachment bases must be used for both of the markers.

Sample marker placement: Front view. The two side markers are placed parallel to the HMD display
Sample marker placement: Top view.
Sample marker placement: Side view. The two side markers are aligned along the eye-level of the user.

HMD Rigid Body in Motive

The two landmark markers closely aligned to the x-axis of the global axes. When the rigid body is created, the orientation of the rigid body itself closely aligns with the global coordinate system.

Creating HMD Rigid Body

For HMD tracking, a modified rigid body needs to be defined in Motive. One important note here is that the rigid body orientation must closely align with the display for the best experience. To accomplish this, it is best to align both the HMD and the rigid body orientation along the global axes. This will ensure an appropriate orientation of the HMD rigid body to be exported.

Place the HMD upright on the floor
Click and switch to orthographic top view.
While monitoring the View pane, align the two landmark markers along the global x-axis in Motive.
Select the HMD markers and create a rigid body.

Creating a rigid body this way will roughly align its orientation with the orientation of the physical HMD component as well as the global coordinate axes in Motive. This will make it easier when precisely adjusting the pivot point position and orientation later. To display the rigid body orientation axes, set the Orientation to true in the rigid body display settings.

Rigid Body Settings

Rigid body properties for the HMD asset.

After creating a rigid body for the HMD, select the asset in the project pane, and access the corresponding rigid body properties. Adjust the rigid body properties as the following:

Deflection

Deflection setting is the tolerable distance, in millimeters, that a rigid body marker may deviate from its expected position before it is unlabeled and unassociated with the rigid body. The deflection is set to 4 mm by default. For HMD tracking applications, we recommended lowering this value to 3 or 2. This will reduce the amount of computation required for labeling, and overall latency may be reduced.

Tracking Algorithm

Use the level 3 tracking algorithm (or ray-based tracking). The tracking algorithm setting determines which protocol to use for solving the rigid body in Motive. This algorithm is recommended because it is less susceptible to marker occlusions and solves rigid bodies more robustly without introducing trajectory gaps. This is more suitable for tracking the HMD since even minor occlusions can acutely affect the experience.

User Data

The User Data value for each rigid body is used as an identification number to reference the rigid body in external applications. Log this value for each rigid body (including the HMD(s)) in the scene. This number will be used to associate the assets in UE4 using the plugin devices.

Min Marker Count

The minimum marker count required for the rigid body solve is set to 3 by default. To prevent the swapping of the rigid body definition, you may want to increase this value to 4. When tracking multiple HMDs, there could be limitations to a variety of unique marker arrangements that could be achieved. If this value is set to 3, a set of three markers on an HMD may be congruent to another set in a different HMD, and the rigid body definitions may be switched in Motive.

Rigid Body Pivot Point

As mentioned previously, the position and orientation of the HMD rigid body pivot point need to be carefully adjusted. This section provides recommended methods and steps to accomplish this. When adjusting the pivot point, the following criteria must be achieved:

The pivot point is placed on the bridge of the nose; specifically, right at the midpoint between two eyes.
Orientation axes of the rigid body should precisely align with the physical HMD component.

Pivot Point Position

For best virtual experiences, the pivot point of the HMD rigid body, in Motive, needs to be positioned on the midpoint between two eyes, of the user when the HMD is put on. To locate this, use the side and top-center landmark markers as references. For more information on adjusting rigid body pivot points, please read through the Rigid Body Tracking page.

1. Set the pivot point over the landmark marker. Use the Set Pivot Point to Selected Marker feature to assign the pivot point to the marker. This will set the elevation of the pivot point along the eye-level.

Top view: Rigid body pivot point is assigned to the left landmark marker.
Side view: Now both the landmark marker and the pivot point is positioned along the user's eye-level elevation.

2. Place the pivot point at the midpoint between the two markers. Enable Two Marker Distance visual aid from the perspective pane, and select the two landmark markers in Motive. This will provide a distance between two markers. Then, using this information, translate the pivot point laterally by half of the distance so that it is placed right on the midpoint between two markers.

Translating the pivot point by half of the measured distance to place it at the midpoint.

3. Translate the pivot point along the z-axis using the translation tool. For the most accurate position, you may need to physically measure the sagittal, z-axis, distance from the landmark marker to the root of nose, and apply the measured offset.

Pivot point translated along the z-axis
Perspective view of the adjusted pivot point position.

Orientation

Now that you have translated the pivot point, you need to make detailed adjustments to the orientation using the orientation transformation tool. For best results, align the two front markers along the x-axis grid and roughly center the rigid body along the z-axis grid. Then, check to make sure that each of the rigid body orientation axes is parallel to the grids lines in Motive. If there is any deviation, apply rotation to adjust the offset. If needed, transparency of the axes and the grids can be adjusted from the Application settings.

In Unreal Engine: the X-axis of the HMD rigid body must be directed forward.
In Unity: the Z-axis of the HMD rigid body must be directed forward.

Slight offset in the rigid body orientation.
After Adjusting, the rigid body orientation axes are precisely aligned with the global axes of Motive

Tip: Once you have the rigid body asset for the HMD configured, you can export the asset into a TRA file for future uses. Importing the TRA file (e.g. CV1.tra) will load the rigid body (HMD) asset and make it available for use; however, the marker placement must remain unchanged in order to re-load previously created rigid bodies.

OptiTrack Unity Plugin