This video covers the basics of skeletal tracking using the Kinect sensor. You may find it easier to follow along by downloading the Kinect for Windows SDK Quickstarts samples and slides.
[h=1]Task: Setup skeleton tracking[/h]Create the Window_Loaded event
Go to the properties window (F4), select the MainWindow, select the Events tab, and double click on the Loaded event to create the Window_Loaded event
Initializing the runtime
Create a new variable outside of the Window_Loaded event to reference the Kinect runtime.
C#
Runtime nui = new Runtime();
Visual Basic
Dim nui As New Runtime()In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseSkeletalTracking to receive skeletal data and register for the SkeletonFrameReady event.
C#
nui.Initialize(RuntimeOptions.UseSkeletalTracking);nui.SkeletonFrameReady += new EventHandler(nui_SkeletonFrameReady);void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){}
Visual Basic
nui.Initialize(RuntimeOptions.UseSkeletalTracking)AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReadyPrivate Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs)End Sub[h=2]Running the application[/h]Add a breakpoint inside the SkeletonFrameReady event and run the application.
When the breakpoint fires, a skeleton position is now being tracked. You can inspect the SkeletonFrameReadyEventArgs to see that the SkeletonFrame returns a collection of six skeletons as shown below.
[h=1]Task: Add a basic hand tracked cursor[/h]In this example, we're going to use the position of the tracked skeleton's, head, left hand, and right hand to move ellipse controls.
[h=2]Designing your UI[/h]Starting from the project above, switch to MainWindow.xaml and make sure you can see the XAML code. We will add three three ellipses of varying color onto a Canvas control named MainCanvas as shown in the XAML below.
XAML
[h=3]Get the Tracked Skeleton[/h]In the SkeletonFrameReady event, we'll use a LINQ query to get the first tracked skeleton.
C#
void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){ SkeletonFrame allSkeletons = e.SkeletonFrame; //get the first tracked skeleton SkeletonData skeleton = (from s in allSkeletons.Skeletons where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault();}
Visual Basic
Private Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs) Dim allSkeletons As SkeletonFrame = e.SkeletonFrame 'get the first tracked skeleton Dim skeleton As SkeletonData = ( _ From s In allSkeletons.Skeletons _ Where s.TrackingState = SkeletonTrackingState.Tracked _ Select s).FirstOrDefault()End Sub[h=3]Getting a Joint position[/h]A Joint position returns X,Y,Z values as explained below
Given that the X and Y positions are between –1 and +1, we can use the Coding4Fun Kinect Toolkit ScaleTo method to scale the value to a maximum X and Y position as shown below.
C#
Joint HandRight = skeleton.Joints[JointID.HandRight].ScaleTo(640, 480);
Visual Basic
Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480)You can also constrain the maximum values for the skeletal joints to a range smaller than -1 to +1. For example, if you were designing an application where the user can reach around the screen to touch things, you may not want them to have to step left or right to reach the edges. By using the second version of the ScaleTo method, you can specify the range of the specified joint to someting smaller as shown:
C#
Joint HandRight = skeleton.Joints[JointID.HandRight].ScaleTo(640, 480, .5f, .5f);
Visual Basic
Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480, .5f, .5f)Essentially, this code will make the joint range of -0.5 to +0.5 map to the pixels at 0 to 640.
Also, as you can see above, you can get a particular Joint, like the HandRight Joint, by using the skeleton indexer for the Joints collection.
[h=3]Setting an Ellipse Position[/h]To move the ellipses in our MainWindow to the location of a Joint, we will use the method below that sets the Canvas.Left and Canvas.Top position to the X (Left) and Y (Top) value from the Joint parameter.
C#
private void SetEllipsePosition(FrameworkElement ellipse, Joint joint){ var scaledJoint = joint.ScaleTo(640, 480, .5f, .5f); Canvas.SetLeft(ellipse, scaledJoint.Position.X); Canvas.SetTop(ellipse, scaledJoint.Position.Y);}
Visual Basic
Private Sub SetEllipsePosition(ByVal ellipse As FrameworkElement, ByVal joint As Joint) Dim scaledJoint = joint.ScaleTo(640, 480,.5f,.5f) Canvas.SetLeft(ellipse, scaledJoint.Position.X) Canvas.SetTop(ellipse, scaledJoint.Position.Y)End Sub[h=3][/h][h=3]Putting it all together[/h]Finally, in the nui_SkeletonFrameReady event, call the SetEllipsePosition methods for the three joints as shown below:
C#
SetEllipsePosition(headEllipse, skeleton.Joints[JointID.Head]);SetEllipsePosition(leftEllipse, skeleton.Joints[JointID.HandLeft]);SetEllipsePosition(rightEllipse, skeleton.Joints[JointID.HandRight]);
Visual Basic
SetEllipsePosition(headEllipse, skeleton.Joints(JointID.Head))SetEllipsePosition(leftEllipse, skeleton.Joints(JointID.HandLeft))SetEllipsePosition(rightEllipse, skeleton.Joints(JointID.HandRight)) 
[h=1]Task: Using TransformSmoothing to for less skeletal jitter[/h]Using the same application as above, you may notice jitteriness in the hand positions as small changes between updates to the SkeletalFrameReady event change the location of the ellipses.
Using TransformSmoothParameters
To do this, set the TransformSmooth property of the SkeletonEngine to true. This will use a default set of smoothing parameters to reduce jitter in the skeletal tracking engine. If these defaults don't provide you with the result you're looking for, then you can build the set of TransformSmoothParameters yourself to meet the needs for your application.
You must set the TransformSmoothParameters after calling the nui.Initialize method.
private void Window_Loaded(object sender, RoutedEventArgs e){ //Initialize to do skeletal tracking nui.Initialize(RuntimeOptions.UseSkeletalTracking); //Must set to true and set after call to Initialize nui.SkeletonEngine.TransformSmooth = true; //Use to transform and reduce jitter var parameters = new TransformSmoothParameters { Smoothing = 0.75f, Correction = 0.0f, Prediction = 0.0f, JitterRadius = 0.05f, MaxDeviationRadius = 0.04f }; nui.SkeletonEngine.SmoothParameters = parameters;
Visual Basic
Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs) 'Initialize to do skeletal tracking nui.Initialize(RuntimeOptions.UseSkeletalTracking) 'Must set to true and set after call to Initialize nui.SkeletonEngine.TransformSmooth = True 'Use to transform and reduce jitter Dim parameters = New TransformSmoothParameters With {.Smoothing = 0.75f, .Correction = 0.0f, .Prediction = 0.0f, .JitterRadius = 0.05f, .MaxDeviationRadius = 0.04f} nui.SkeletonEngine.SmoothParameters = parameters 'add event to receive skeleton data AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReadyEnd Sub 
More...
- [00:31] Skeleton Tracking API
- [01:24] Understanding Skeleton Quality and Joint data
- [03:27] Setup skeleton tracking
- [03:44] Adding a basic hand tracked cursor
- [09:12] Using TransformSmoothing to remove “skeletal jitter”
[h=1]Task: Setup skeleton tracking[/h]Create the Window_Loaded event
Go to the properties window (F4), select the MainWindow, select the Events tab, and double click on the Loaded event to create the Window_Loaded event
Initializing the runtime
Create a new variable outside of the Window_Loaded event to reference the Kinect runtime.
C#
Runtime nui = new Runtime();
Visual Basic
Dim nui As New Runtime()In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseSkeletalTracking to receive skeletal data and register for the SkeletonFrameReady event.
C#
nui.Initialize(RuntimeOptions.UseSkeletalTracking);nui.SkeletonFrameReady += new EventHandler(nui_SkeletonFrameReady);void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){}
Visual Basic
nui.Initialize(RuntimeOptions.UseSkeletalTracking)AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReadyPrivate Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs)End Sub[h=2]Running the application[/h]Add a breakpoint inside the SkeletonFrameReady event and run the application.
Note – You will need to stand far enough away that the Kinect can see all or most of your skeleton for the SkeletonFrameReady event to fire.
When the breakpoint fires, a skeleton position is now being tracked. You can inspect the SkeletonFrameReadyEventArgs to see that the SkeletonFrame returns a collection of six skeletons as shown below.
[h=1]Task: Add a basic hand tracked cursor[/h]In this example, we're going to use the position of the tracked skeleton's, head, left hand, and right hand to move ellipse controls.
[h=2]Designing your UI[/h]Starting from the project above, switch to MainWindow.xaml and make sure you can see the XAML code. We will add three three ellipses of varying color onto a Canvas control named MainCanvas as shown in the XAML below.
XAML
[h=3]Get the Tracked Skeleton[/h]In the SkeletonFrameReady event, we'll use a LINQ query to get the first tracked skeleton.
C#
void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){ SkeletonFrame allSkeletons = e.SkeletonFrame; //get the first tracked skeleton SkeletonData skeleton = (from s in allSkeletons.Skeletons where s.TrackingState == SkeletonTrackingState.Tracked select s).FirstOrDefault();}
Visual Basic
Private Sub nui_SkeletonFrameReady(ByVal sender As Object, ByVal e As SkeletonFrameReadyEventArgs) Dim allSkeletons As SkeletonFrame = e.SkeletonFrame 'get the first tracked skeleton Dim skeleton As SkeletonData = ( _ From s In allSkeletons.Skeletons _ Where s.TrackingState = SkeletonTrackingState.Tracked _ Select s).FirstOrDefault()End Sub[h=3]Getting a Joint position[/h]A Joint position returns X,Y,Z values as explained below
- X = Horizontal position between –1 and +1
- Y = Vertical position between –1 and +1
- Z = Distance from Kinect measured in meters
Given that the X and Y positions are between –1 and +1, we can use the Coding4Fun Kinect Toolkit ScaleTo method to scale the value to a maximum X and Y position as shown below.
C#
Joint HandRight = skeleton.Joints[JointID.HandRight].ScaleTo(640, 480);
Visual Basic
Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480)You can also constrain the maximum values for the skeletal joints to a range smaller than -1 to +1. For example, if you were designing an application where the user can reach around the screen to touch things, you may not want them to have to step left or right to reach the edges. By using the second version of the ScaleTo method, you can specify the range of the specified joint to someting smaller as shown:
C#
Joint HandRight = skeleton.Joints[JointID.HandRight].ScaleTo(640, 480, .5f, .5f);
Visual Basic
Dim HandRight = skeleton.Joints(JointID.HandRight).ScaleTo(640, 480, .5f, .5f)Essentially, this code will make the joint range of -0.5 to +0.5 map to the pixels at 0 to 640.
Also, as you can see above, you can get a particular Joint, like the HandRight Joint, by using the skeleton indexer for the Joints collection.
[h=3]Setting an Ellipse Position[/h]To move the ellipses in our MainWindow to the location of a Joint, we will use the method below that sets the Canvas.Left and Canvas.Top position to the X (Left) and Y (Top) value from the Joint parameter.
C#
private void SetEllipsePosition(FrameworkElement ellipse, Joint joint){ var scaledJoint = joint.ScaleTo(640, 480, .5f, .5f); Canvas.SetLeft(ellipse, scaledJoint.Position.X); Canvas.SetTop(ellipse, scaledJoint.Position.Y);}
Visual Basic
Private Sub SetEllipsePosition(ByVal ellipse As FrameworkElement, ByVal joint As Joint) Dim scaledJoint = joint.ScaleTo(640, 480,.5f,.5f) Canvas.SetLeft(ellipse, scaledJoint.Position.X) Canvas.SetTop(ellipse, scaledJoint.Position.Y)End Sub[h=3][/h][h=3]Putting it all together[/h]Finally, in the nui_SkeletonFrameReady event, call the SetEllipsePosition methods for the three joints as shown below:
C#
SetEllipsePosition(headEllipse, skeleton.Joints[JointID.Head]);SetEllipsePosition(leftEllipse, skeleton.Joints[JointID.HandLeft]);SetEllipsePosition(rightEllipse, skeleton.Joints[JointID.HandRight]);
Visual Basic
SetEllipsePosition(headEllipse, skeleton.Joints(JointID.Head))SetEllipsePosition(leftEllipse, skeleton.Joints(JointID.HandLeft))SetEllipsePosition(rightEllipse, skeleton.Joints(JointID.HandRight))
[h=1]Task: Using TransformSmoothing to for less skeletal jitter[/h]Using the same application as above, you may notice jitteriness in the hand positions as small changes between updates to the SkeletalFrameReady event change the location of the ellipses.
Using TransformSmoothParameters
To do this, set the TransformSmooth property of the SkeletonEngine to true. This will use a default set of smoothing parameters to reduce jitter in the skeletal tracking engine. If these defaults don't provide you with the result you're looking for, then you can build the set of TransformSmoothParameters yourself to meet the needs for your application.
You must set the TransformSmoothParameters after calling the nui.Initialize method.
Note: Since every application is different, you will need to experiment with the parameters to understand what's right for your application.
C#
private void Window_Loaded(object sender, RoutedEventArgs e){ //Initialize to do skeletal tracking nui.Initialize(RuntimeOptions.UseSkeletalTracking); //Must set to true and set after call to Initialize nui.SkeletonEngine.TransformSmooth = true; //Use to transform and reduce jitter var parameters = new TransformSmoothParameters { Smoothing = 0.75f, Correction = 0.0f, Prediction = 0.0f, JitterRadius = 0.05f, MaxDeviationRadius = 0.04f }; nui.SkeletonEngine.SmoothParameters = parameters;
Visual Basic
Private Sub Window_Loaded(ByVal sender As Object, ByVal e As RoutedEventArgs) 'Initialize to do skeletal tracking nui.Initialize(RuntimeOptions.UseSkeletalTracking) 'Must set to true and set after call to Initialize nui.SkeletonEngine.TransformSmooth = True 'Use to transform and reduce jitter Dim parameters = New TransformSmoothParameters With {.Smoothing = 0.75f, .Correction = 0.0f, .Prediction = 0.0f, .JitterRadius = 0.05f, .MaxDeviationRadius = 0.04f} nui.SkeletonEngine.SmoothParameters = parameters 'add event to receive skeleton data AddHandler nui.SkeletonFrameReady, AddressOf nui_SkeletonFrameReadyEnd Sub
More...
