# Hand Tracking

Hand Tracking Input

{% tabs %}
{% tab title="OpenXR (Recommended)" %}
{% hint style="info" %}
**Required XR Settings for Hand Tracking**\
![](https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FHEVAA8dXJCxu5CaHDLFV%2Fimage.png?alt=media\&token=c3e3a7ad-26f1-4d68-ae6c-0e7af2dd380a)

![](https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2F5Gaqi4QPthgYCSi1R00o%2Fimage.png?alt=media\&token=003e2d0f-8278-499a-8ef0-123eed069022)



After downloading the subpackage in **Auto Hand/Packages/OpenXR.package**

Example will be found in **Auto Hand/Examples/Scenes/OpenXR/Hand Tracking**
{% endhint %}

These are the required components for OpenXR hand tracking:

**`XR Hand Skeleton Driver`** Unity's script for getting transform data from OpenXR

**`XR Hand Tracking Events`** Unity's script for getting events from OpenXR

**`OpenXR Auto Hand Tracking`** Converts Unity hand tracking data to your Auto Hand skeleton as well as smoothing for hand tracking gesture and movement

**`OpenXR Auto Hand Tracking Grabber`** Manages if grabbing / holding conditions are being met



## OpenXR Auto Hand Tracking

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2Fb5vV9OzSJZ6rY5zdmEfE%2Fimage.png?alt=media&#x26;token=da0c4f99-0f45-498a-bc49-004520fe1a0d" alt=""><figcaption></figcaption></figure>

**`Hand`**: is a reference to which hand this script is driving

**`Up Axis`**: if your skeleton on your hand rig matches the OpenXR hand tracking skeleton, this axis should equal "Up". If your skeleton is different you figure out what your hands' up axis equivalent is on the OpenXR rig. On the robot hand example  the yellow up axis is equal to the OpenXR's left (opposite direction of the red arrow)

**`Forward Axis`**: same as previous value, but for your hand's blue forward arrow. In the example below the robot hand's forward axis matches the OpenXR hand's yellow up axis. So it is set to "Up"

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FrpTYx7LMVyvLHrE36iy1%2FHandTrackingTransforms.png?alt=media&#x26;token=e7746698-43b7-4976-b9e5-f69a3da4d417" alt=""><figcaption><p>This example shows why the "Up/Forward" Axis settings are "Left/Up" on the robot hand tracking</p></figcaption></figure>

**`Hand Offset`**: Position offset for the hand in local hand tracking space

**`Hand Rotation Offset`**: This offsets the hand relative to it's tracking target. In order for the hand tracking to work the hand model must have correct rotation offset to the tracking hand

**`Hand Pose Smoothing`**: Add a small smooth movement delay when matching the fingers to their target pose. This mainly helps the hand from moving jarringly when in using low confidence poses

**`Follow Position Smoothing Speed`**: This will smooth out the hands follow target position over some. The lower the value the slower smoother/slower the matching. This helps prevent jumping/jittering from lower quality hand tracking input.

**`Follow Rotation Smoothing Speed`**:  This will smooth out the hands follow target rotation over some. The lower the value the slower smoother/slower the matching. This helps prevent jumping/jittering from lower quality hand tracking input.





## OpenXR Auto Hand Tracking Grabbing

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2F5WkyOwrBzPdCCOwmWK3I%2Fimage.png?alt=media&#x26;token=f56e7416-6810-4fda-b8cb-9322200b3c65" alt=""><figcaption></figcaption></figure>



### **Touch Grab Settings**

**`handTracker`**: Reference to the hand tracker responsible for tracking hand movements and poses

**`releaseGrabDelay`**: The delay in seconds before a grab is released after the grab condition is no longer met

**`fingerTipRadiusMultiplier`**: Multiplier for the radius of the finger tip detection spheres used in touch grabbing

**`useFingerTouchGrabbing`**: Enables or disables grabbing objects by touching them with fingers

**`useFingerTouchReleasing`**: Enables or disables releasing objects by touching them with fingers

**`useTouchHoldingWithHeldPose`**: Enables or disables maintaining a held pose while holding an object with finger touch grabbing

### **Pose Grab Settings**

**`usePoseGrabbing`**: Enables or disables grabbing objects using predefined hand poses

**`minPoseGrabCloseness`**: The minimum closeness required for the hand pose to initiate a grab

**`maxPoseGrabCloseness`**: The maximum closeness allowed for the hand pose to initiate a grab

**`requiredDeltaPoseGrabCloseness`**: The required change in closeness to trigger a grab

### **Pose Release Settings**

**`usePoseRelease`**: Enables or disables releasing objects using predefined hand poses

**`minPoseReleaseOpenness`**: The minimum openness required for the hand pose to initiate a release

**`maxPoseReleaseOpenness`**: The maximum openness allowed for the hand pose to initiate a release

**`requiredDeltaPoseReleaseOpenness`**: The required change in openness to trigger a release

### **Pose Squeeze Settings**

**`usePoseSqueezing`**: Enables or disables squeezing objects using predefined hand poses

**`squeezeUnsqueezeDelay`**: The delay in seconds before a squeeze is unsqueezed after the squeeze condition is no longer met

**`squeezePoseSensitvityMultiplier`**: Multiplier for the sensitivity of the squeezing pose detection





## Raw OpenXR Hand Tracking Input Visuals

These "Right/Left Hand Tracking Visuals" show the default OpenXR hand tracking input before it's sent to the Auto Hand Tracking components.

If hand tracking quality seems low, you can check these visuals to see if it's the hand tracking input that isn't work well or if it's the Auto Hand Tracking that isn't working as expecting

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FAYq3SZLBPSQSOxvdldNC%2Fimage.png?alt=media&#x26;token=4ce9c0ee-8db5-46fe-8d91-5eaf832f3118" alt=""><figcaption><p>Disable these components to hide the ghost hands</p></figcaption></figure>
{% endtab %}

{% tab title="MetaXR" %}


## MetaXR Auto Hand Tracking

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FlR8QCQNYkWMAr6UHyDQd%2Fimage.png?alt=media&#x26;token=ae397b45-38d4-4ce3-9848-ddfa33079899" alt=""><figcaption></figcaption></figure>



**`Hand`**: is a reference to which hand this script is driving

**`Up Axis`**: if your skeleton on your hand rig matches the OpenXR hand tracking skeleton, this axis should equal "Up". If your skeleton is different you figure out what your hands' up axis equivalent is on the OpenXR rig. On the robot hand example  the yellow up axis is equal to the OpenXR's left (opposite direction of the red arrow)

**`Forward Axis`**: same as previous value, but for your hand's blue forward arrow. In the example below the robot hand's forward axis matches the OpenXR hand's yellow up axis. So it is set to "Up"

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FrpTYx7LMVyvLHrE36iy1%2FHandTrackingTransforms.png?alt=media&#x26;token=e7746698-43b7-4976-b9e5-f69a3da4d417" alt=""><figcaption><p>This example shows why the "Up/Forward" Axis settings are "Left/Up" on the robot hand tracking</p></figcaption></figure>

**`Hand Offset`**: Position offset for the hand in local hand tracking space

**`Hand Rotation Offset`**: This offsets the hand relative to it's tracking target. In order for the hand tracking to work the hand model must have correct rotation offset to the tracking hand

**`Hand Pose Smoothing`**: Add a small smooth movement delay when matching the fingers to their target pose. This mainly helps the hand from moving jarringly when in using low confidence poses

**`Follow Position Smoothing Speed`**: This will smooth out the hands follow target position over some. The lower the value the slower smoother/slower the matching. This helps prevent jumping/jittering from lower quality hand tracking input.

**`Follow Rotation Smoothing Speed`**:  This will smooth out the hands follow target rotation over some. The lower the value the slower smoother/slower the matching. This helps prevent jumping/jittering from lower quality hand tracking input.





## MetaXR Auto Hand Tracking Grabbing

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FPRe0HUczoWSFBC02uhMt%2Fimage.png?alt=media&#x26;token=b8c4e52a-9126-4d1d-91df-d13436a3a892" alt=""><figcaption></figcaption></figure>

### **Touch Grab Settings**

**`handTracker`**: Reference to the hand tracker responsible for tracking hand movements and poses

**`releaseGrabDelay`**: The delay in seconds before a grab is released after the grab condition is no longer met

**`fingerTipRadiusMultiplier`**: Multiplier for the radius of the finger tip detection spheres used in touch grabbing

**`useFingerTouchGrabbing`**: Enables or disables grabbing objects by touching them with fingers

**`useFingerTouchReleasing`**: Enables or disables releasing objects by touching them with fingers

**`useTouchHoldingWithHeldPose`**: Enables or disables maintaining a held pose while holding an object with finger touch grabbing

### **Pose Grab Settings**

**`usePoseGrabbing`**: Enables or disables grabbing objects using predefined hand poses

**`minPoseGrabCloseness`**: The minimum closeness required for the hand pose to initiate a grab

**`maxPoseGrabCloseness`**: The maximum closeness allowed for the hand pose to initiate a grab

**`requiredDeltaPoseGrabCloseness`**: The required change in closeness to trigger a grab

### **Pose Release Settings**

**`usePoseRelease`**: Enables or disables releasing objects using predefined hand poses

**`minPoseReleaseOpenness`**: The minimum openness required for the hand pose to initiate a release

**`maxPoseReleaseOpenness`**: The maximum openness allowed for the hand pose to initiate a release

**`requiredDeltaPoseReleaseOpenness`**: The required change in openness to trigger a release

### **Pose Squeeze Settings**

**`usePoseSqueezing`**: Enables or disables squeezing objects using predefined hand poses

**`squeezeUnsqueezeDelay`**: The delay in seconds before a squeeze is unsqueezed after the squeeze condition is no longer met

**`squeezePoseSensitvityMultiplier`**: Multiplier for the sensitivity of the squeezing pose detection







If you want to disable the visual hands the OVR includes disable these components found here

<figure><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2F9jJnkICMLxvKIqApVrQs%2Fimage.png?alt=media&#x26;token=9c6414aa-dd74-478e-bce7-cce94b212c5a" alt=""><figcaption></figcaption></figure>
{% endtab %}

{% tab title="SteamVR" %}
## IndexHandControllerLink

Grab Fingers Required will trigger grab when the included fingers are bent past the given amount, and release when any fingers are less than this value or when the grab action is triggered/untriggered

<div align="left"><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FV7vTrl4vT7fr9PBKlS2Y%2Fimage.png?alt=media&#x26;token=ae06680c-a424-41ef-baef-27cca28d2079" alt=""></div>
{% endtab %}

{% tab title="Oculus Integration (Deprecated)" %}
##

{% hint style="info" %}
The hand tracking Demo can be found within the **Oculus Integration** package. It will be found in Auto Hand/Examples/Scenes/Oculus Integration/Hand Tracking

If you can't find that scene follow the setup guide for [Oculus Integration](../setup#package-setup)
{% endhint %}

This component should go onto your hand along with the OVRHand and OVRSkeleton



Grab Fingers Required will trigger grab when the included fingers are bent past the given amount, and release when any fingers are less than this value



`Thumb, Index, Middle, Ring, Pinky` [finger ](hand/finger-component)components are found under the hand

`Free Fingers` will allow fingers to move while holding an open (as long as they don't break the grab finger requirements). For example lift the index finger up while holding a soda bottle

`Grab Action` will call the grab event when all the included fingers are bent past the given amount. Release event is called when grab action is no longer valid.

* In this case grab will occur when the index middle and thumb are bent past halfway or 0.5

`Squeeze Action` will trigger the squeeze event when all the included fingers are bent past the given amount.



### Custom Hand Tracking Event Component

Hand Tracking includes an event script which will call Press/Release

* Press is triggered if all the fingers in Finger Bend Past are bent past the bend value, and all the values in Finger Bend Below are below the bend value.
* Release is triggered if press was, but is no longer valid

<div align="left"><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FQ2co6BtjVxidRXKF9tjE%2Fimage.png?alt=media&#x26;token=1c64437e-1fe6-4eab-a523-eaf69b22db44" alt=""></div>



### Controller Swapping

The OVR Hand Controller Swapper will allow you to automatically swap between controllers and hand tracking. Connect the hand follow targets for hand tracking / controller tracking then connect any objects/scripts that should be disabled on swap

<div align="left"><img src="https://2959669391-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F5zKO0EvOjzUDeT2aiFk3%2Fuploads%2FZfO0qG2ngaMwyT6mIqji%2Fimage.png?alt=media&#x26;token=90e75191-462a-492c-b899-b110b8b46d7a" alt=""></div>
{% endtab %}
{% endtabs %}























