Data Types

From NaturalPoint Product Documentation Ver 1.10
Jump to: navigation, search
Back to the Main PageBack to Motive Documentation


This page explains different types of capture data and details on the reconstruction pipeline, which converts 2D camera images into 3D data in both live and recorded captures. There are three data types in Motive: 2D, 3D, and JT. Understanding these types is essential in order to clearly perceive the data-processing workflow of converting 2D camera frames into 3D tracking data. Motive saves captured data into a Take file (TAK extension), which can be replayed in the Edit mode. After Take files have been captured, recorded data types can be accessed from the Project pane.


In Live mode, you can specify which data types to record. There are 2D/3D/JT buttons next to the recording trigger of the Timeline pane, and only the highlighted types will be saved in the Take file. As long as 2D data is recorded, other data types can be derived from the 2D data using post-processing pipelines.

Timeline pane: Recording 2D and 3D data.


In Edit Mode, available data types in an opened Take file will be displayed next to the file name in the Project pane and also from the top-left corner of the Perspective View pane. The accessed data type will be highlighted in white, and you can toggle between available types by clicking on the desired type in the Project pane.

Project pane: 3D data selected from a recorded Take.


Data Types[edit]


2D data

  • 2D object image of a single camera from the 2D camera preview.
  • Size and circularity information displayed from the 2D camera preview.
  • 2D data is the foundation of motion capture data. It mainly includes multiple 2D frames captured by each camera in a system.

    Images in recorded 2D data depends on the image processing mode, or video type, of each camera that was configured at the time of the capture. Cameras that were set to reference modes (raw or MJPEG grayscale images) record reference videos, and cameras that were set to tracking modes (object, precision, segment) record 2D object images which will be used in the reconstruction process. The 2D object images contain information on x and y centroid positions of the captured reflections as well as their corresponding sizes (in pixels) and roundness, as shown below.

    Using the 2D object data along with the camera calibration information, 3D data is computed. Extraneous reflections that fails to satisfy the 2D object filter settings (defined in Reconstruction settings) are filtered out, and only the remaining reflections are processed. The process of converting 2D centroid locations into 3D coordinates is called Reconstruction, which will be covered in the later section of this page.

    Captured 2D data can be reconstructed both in real-time and after the capture. In post-processing, recorded 2D data can be used to create a fresh set of 3D data by Reconstructing the capture again, and the Reconstruction parameters will be reapplied and any existing 3D data will be overwritten. Motive can also live-process (live-reconstruct) incoming 2D images on a frame-to-frame basis and stream the 3D tracking data into external pipelines with extremely low processing latency.

    • Contains 2D frames, or 2D object information captured by each camera in a system. 2D data can be monitored from the Camera Preview pane.
    • Recorded 2D data can be reconstructed and auto-labeled to derive the 3D data.
    • 3D tracking data is not computed yet. The tracking data can be exported only after reconstructing the 3D data.
    • In playback of recorded 2D data, 3D data will be Live-reconstructed into 3D data and reported in the 3D viewport.


    3D data

    Reconstructed 3D data shown in the Perspective View pane.

    3D data contains 3D coordinates of reconstructed markers, and from this, the 3D motion tracking is accomplished. Markers are reconstructed and displayed in the perspective view, and corresponding 3D trajectories can be monitored from the Editor window of the Timeline pane. Also, 3D tracking data can be exported into various file formats — CSV, C3D, FBX, and more.

    In recorded 3D data, marker labels can be assigned to reconstructed markers either through the auto-labeling process or by manually assigning it. For each trackable assets, Motive auto-labels related marker reconstructions and creates corresponding model. Accordingly, 3D data also includes position and orientation information of created assets. Lastly, in recorded 3D data, the edit tools can be used to interpolate the 3D marker trajectory gaps.

    • Reconstructed 3D marker positions.
    • Marker labels can be assigned.
    • Assets are modeled and the tracking information is available.
    • Edit tools can be used to fill the trajectory gaps.


    JT data

    JT data contains recorded orientation and translation information of assets in all of the captured frames. Each skeleton segment position and orientation is recorded through reverse kinematics of 3D marker locations and they are recorded for all of the captured frames.

    Note that the same orientation information is also available through 3D data, and JT data will be removed in future releases of Motive.

    Reconstruction[edit]



    In optical mocap systems, reconstruction is the core process of obtaining 3D coordinates from captured 2D object images. Centroid locations of detected circular reflections are processed through the Point Cloud reconstruction engine to triangulate the respective 3D marker position in a calibrated capture volume. From the Reconstruction pane, reconstruction settings and bounds can be adjusted to optimize the acquisition of 3D marker coordinates. If you wish to omit certain cameras from contributing to the 3D data reconstruction, uncheck the box on the left of each camera from the Cameras Pane.

    Read through the Reconstruction page for more details on the reconstruction settings.


    Real-time Reconstruction


    When in Live mode or in 2D data playback, you may wonder why some of the 3D information (3D orientations and positions) is available in Motive. This is because Motive reconstructs incoming frames in real-time. For each frame, captured or recorded 2D data is continuously reconstructed into 3D data, both in the Live and Edit mode. The parameters defined under the reconstruction settings are applied to the live reconstruction, and the solved data is used in 3D viewport as well as other real-time streaming pipelines. To save the 3D tracking information, 3D data must recorded in a Take or 2D data must be reconstructed in post-processing of the data.


    Enabling/Disabling Real-time Reconstruction

    If you wish to disable the live reconstruction, uncheck the Enable Point Cloud Reconstruction checkbox in the Reconstruction pane. When the Point Cloud reconstruction disabled, 3D information will no longer be available when accessing 2D data. You can also disable real-time reconstruction contributions for specific cameras by disabling real-time reconstructions setting for selected cameras on the Project pane context menu.


    • 3D tracking data being live-solved from 2D data.
    • Reconstruction pane: Disable/enable real-time reconstruction.


    Post-Processing Reconstruction and Auto-labeling

    Post-processing reconstruction and auto-labeling pipelines allow you to convert, or reconstruct, recorded 2D data into 3D data and auto-label the reconstructed markers using defined asset (rigid body and skeleton) definitions. These pipelines can be applied either on an entire Take frame range or within a selected range. When processing a single Take, you can also limit the post-processing pipelines to only specific ranges by selecting desired frames in the Timeline pane instead of processing the entire take. Also, entire frames of multiple Takes can be selected and processed altogether. The following sections detail on the Reconstruction and Auto-labeling operations. Standalone reconstruction and auto-label pipelines are featured in Motive 1.10 and above for better processing workflow.

    • Processing a single Take: Reconstruction and auto-labeling pipelines applied to an entire frame range (non-selected) or a selected frame range.
    • Processing multiple Takes: Reconstruction and auto-labeling pipeline applies to entire frames of the selected Takes.


    Reconstruct and Auto-label

    • Reconstructing and auto-labeling a Take.
    • Disabling cameras to omit them from reconstruction.

    When post-processing captured Take(s), new sets of 3D data can be obtained again from recorded 2D data by Reconstruct and Auto-labeling the Take(s). The reconstruct and auto-label feature simply combines the reconstruction and auto-labeling pipelines; both of which are further explained in the following sections. Recorded 2D data is processed through the reconstruction engine and markers on the associated assets are auto-labeled. If 3D data was not recorded within, or was deleted from, a Take file, this feature can be used to obtain the 3D data again. For Take(s) that already contain 3D data, the existing data will be overwritten and previous edits will be discarded.

    To reconstruct and auto-label Takes, select and right-click desired Takes from the Project pane and click reconstruct and auto-label, or simply click the Reconstruct and Auto-label ProjectPane Reconandautolabel.png icon.


    After an existing Take has been reconstructed and auto-labeled, the following changes will be applied:

    • Newly created assets are applied.
    • Marker for all active assets will be auto-labeled.
    • Adjusted reconstruction settings are applied.
    • Adjusted auto-labeling bounds are applied.
    • Previous edits on existing 3D data will be discarded.


    Note: Be careful when reconstructing a Take again either by Reconstruct or Reconstruct and Auto-label, because it will overwrite the 3D data and any post-processing edits on trajectories and marker labels will be discarded. Also, for Takes involving skeleton assets, the recorded skeleton marker labels, which were intact during the live capture, may be discarded, and reconstructed markers may not be auto-labeled again if the skeletons are never in well-trackable poses throughout the captured Take. This is another reason why you want to start a capture with a calibration pose (e.g. T-pose).



    Reconstruct

    Reconstructing a Take

    When a Take is reconstructed, 3D coordinates of recorded 2D objects are computed, and the 3D reconstructions in multiple frames are interrelated to create respective marker trajectories. The reconstruction parameters defined in the Reconstruction pane sets which 2D objects are reconstructed and which 3D reconstructions in different frames are combined into a trajectory. In post-processing, you may wish to modify the reconstruction settings and the 2D object filters (size and circularity) to optimize the acquisition of 3D markers. If the changes are made, the Take must be reconstructed again to obtain a fresh set of 3D data with the applied changes.

    There are reconstruction options (e.g. Reconstructed Markers Only, or Rigid Body Override) which allow Motive to acquire 3D points using untracked camera rays and associated asset definitions. These features can be useful for obtaining stable results in low camera count systems or for captures where markers are frequently occluded. When such features are enabled, the reconstruction pipeline will also utilize the active assets to better acquire occluded marker positions in the 3D data.

    In Motive 1.10 and above, a standalone reconstruct pipeline has been added, and it reconstructs recorded 2D data into 3D data without the auto-labeling process. If you reconstruct a Take already containing 3D data, it will overwrite the 3D data and all the post-processing edits will be discarded.


    Auto-label

    Auto-labeling a Take

    The auto-labeler labels 3D markers that are associated with active rigid body or skeleton assets (enabled from the Project pane) in captured Takes. To auto-label Take(s) in post-processing:

    1. Select Takes from the Project pane
    2. Right-click to bring up the context menu
    3. Click either auto-label or reconstruct and auto-label to process selected Takes. The combined reconstruct and auto-label will create a new set of 3D data and auto-label the markers from it.
    4. This will label all the markers that matches the corresponding asset definition.


    Each trackable asset definition stores respective arrangements of the involved markers, which was recorded when the asset was first created. During the auto-labeling process, Motive labels a set of reconstructed 3D points that resemble marker arrangements of the active assets, and then the corresponding rigid bodies and skeletons get modeled in the 3D view. Markers that are not associated with any of the assets are marked as Unlabeled unless they are manually labeled. After creating new assets in recorded capture, the Take must be auto-labeled again in order to assign respective marker labels.

    For Motive 1.10 and above, a separate auto-label pipeline has been added. To auto-label a take, the selected Take must contain 3D data. Unlike the combined reconstruct and auto-label pipeline, the standalone auto-label pipeline labels existing 3D data without reconstructing a new set of 3D data. If 3D data already contains labeled markers, the auto-labeler will respect the existing labels and will not re-label them. Only the unlabeled markers will be labeled via this pipeline, and any post-processing edits that were made will be preserved.


    Auto-labeler Settings

    The settings for the auto-labeling engine are defined in the Auto-labeler section of the Reconstruction pane. The auto-labeler parameters can be modified during post-processing pipelines, and they can be optimized for stable labeling of markers throughout the Take.


    • Rigid body marker ID’s in 2D data. the rigid body asset is live-solved.
    • In 3D data, rigid body markers are labeled and the corresponding asset is modeled.

    Refer to the images above. In 2D data, the markers are referenced with reconstruction ID numbers, whereas they are labeled in 3D data. When an asset is created in Motive, marker arrangements within the asset is calibrated and recorded. When a set of 3D markers are recognized through the auto-labeling process, they are annotated and modeled. When a new asset is created in the Edit mode, the asset may be visible only in the 2D data (live reconstruction) but not in 3D data. This is because the labeling changes have not been reflected in the 3D data yet. Takes must be auto-labeled again after making changes to the marker labels or the assets in the post-processing of the 3D data.


    Deleting Data[edit]


    • Delete 2D data dialog window.
    • Project pane context menu.

    Deleting 2D/Video/Audio data

    In Motive 1.10 and above, recorded 2D data, audio data, and reference videos can be deleted from a Take file. To do this, open the Project pane, right-click on a recorded Take, and click the Delete 2D Data from the context menu. Then, a dialogue window will pop-up, asking which types of data to delete. After removing the data, a backup file will be archived into a separate folder, just in case the original data need to be accessed again

    Deleting 2D data will significantly reduce the size of the Take file. You may want to delete recorded 2D data when there is already a final version of reconstructed 3D data recorded in a Take and the 2D data is no longer needed. However, be aware that deleting 2D data removes the most fundamental data from the Take file. After 2D data has been deleted, the action cannot be reverted, and without 2D data, 3D data cannot be reconstructed again.


    Deleting 3D Data

    • Project pane: Deleting 3D data from a recorded Take.

    Recorded 3D data can be deleted from the context menu in the Project pane. To delete 3D data, right-click on selected Takes and click Delete 3D data, and all reconstructed 3D information will be removed from the Take. When you delete the 3D data, all edits and labeling will be deleted as well. Again, a new 3D data can always be reacquired by reconstructing and auto-labeling the Take from 2D data.

    • Deleting 3D data for a single Take: When frame range is not selected, it will delete 3D data from the entire frame. When a frame range is selected from the Timeline Editor, this will delete 3D data in the selected ranges only.
    • Deleting 3D data for multiple Takes: Even when a frame range is selected from the timeline, it will unlabel all markers from all frame ranges of the selected Takes.


    Deleting Marker Labels

    Assigned marker labels can be deleted from the context menu in the Project pane. The Delete Marker Labels feature removes all marker labels from the 3D data of selected Takes. All markers will become unlabeled.

    • Deleting labels for a single Take: When no frame range is selected, it will unlabel all markers from all Takes. When a frame range is selected from the Timeline Editor, this will unlabel markers in the selected ranges only.
    • Deleting labels for multiple Takes: Even when a frame range is selected from the timeline, it will unlabel all markers from all frame ranges of the selected Takes.

    Recording and Deleting JT Data[edit]



    When there is a 3D skeleton data available in a Take, JT data can be recorded. From the Project pane context menu, select the Record Joint Angles to obtain JT data. And to delete the JT data, simply click Delete Joint Angle Data from the Project pane context menu.

    • DataTypes RecordJT.png
    • DataTypes DeleteJT.png



    Back: Data Recording

    Next: Labeling