Kinect SDK and XNA - Tue, Dec 6, 2011

It seems many people are looking for the way to draw skeletons and store the RGB camera data into Texture2Ds. The KinectEngine class which is included in Neat game engine does all these stuff, but here’s the direct way to do these in XNA:

1. Initialization

First we have to initialize the Nui Runtime. Assuming we are going to use the first available Kinect device connected to our system, we can write:

public Runtime Nui = Runtime.Kinects[0];
Nui.Initialize(RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

The Runtime class is inside the Microsoft.Research.Kinect.Nui namespace. The second line tells the SDK that we are going to use both skeletal tracking and RGB (color) camera.

Now that we initialized the SDK, we have to tell it to do what we want when a video frame or a skeleton frame is ready:

Nui.VideoFrameReady += 
   new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
Nui.SkeletonFrameReady += 
   new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

2. RGB Stream

One of the most common things that XNA developers want to do with Kinect SDK, is to show the video captured by the RGB camera in their game. To do this, first you have to store the stream into a Texture2D object. Therefore, before we begin, we have to create our target texture:

public Texture2D KinectRGB =  new Texture2D(GraphicsDevice, xMax, yMax);

Now, we can perform the conversion when video frames become ready:

void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    PlanarImage image = e.ImageFrame.Image;

    int offset = 0;
    Color[] bitmap = new Color[xMax * yMax];
    for (int y = 0; y < yMax; y++)
        for (int x = 0; x < xMax; x++)
        {
            bitmap[y * xMax + x] = new Color(image.Bits[offset + 2], 
                image.Bits[offset + 1], image.Bits[offset], 255);
            offset += 4;
        }
    KinectRGB.SetData(bitmap);
}

The raw image data is stored in e.ImageFrame.Image. The usual method of filling XNA textures is by creating a Color array, fill it with pixel data and feed it to a texture using its SetData method. Color data is stored in BGR32 format, meaning that the value for blue channel is the first value stored, and the red value is the last. For more information about reading the camera data, watch this video.

To begin reading from the RGB camera, we have to open the video stream:

Nui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution640x480, ImageType.Color);

In this example, the value of xMax is 640 and yMax is 480.

3. Skeletal Tracking

Getting skeleton data from Kinect SDK is easy. This example stores the data into an array for further use in the game:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    var trackedSkeletons = from s in e.SkeletonFrame.Skeletons
           where s.TrackingState == SkeletonTrackingState.Tracked
           select s;
    trackedSkeletonsCount = trackedSkeletons.Count();
    for (int i = 0; i < trackedSkeletonsCount; i++)
        Skeletons[i] = trackedSkeletons.ElementAt(i);
}

To draw a skeleton, you just have to get each joint’s position and draw a line between adjacent joints.

shot0006

This helper function draws a skeleton using Neat engine’s LineBrush class.

public void DrawSkeleton(SpriteBatch spriteBatch, LineBrush lb, Vector2 position, 
Vector2 size, Color color, int skeletonId = 0)
{
    if (Skeletons.Length <= skeletonId || Skeletons[skeletonId] == null)
    {
        //Skeleton not found. Draw an X
        lb.Draw(spriteBatch, position, position + size, color);
        lb.Draw(spriteBatch, new LineSegment(position.X+size.X, position.Y, 
            position.X, position.Y + size.Y), color);
        return;
    }

    //Right Hand
    lb.Draw(spriteBatch, ToVector2(JointID.HandRight, size, skeletonId), 
        ToVector2(JointID.WristRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.WristRight, size, skeletonId), 
        ToVector2(JointID.ElbowRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ElbowRight, size, skeletonId), 
        ToVector2(JointID.ShoulderRight, size, skeletonId), color, position);

    //Head & Shoulders
    lb.Draw(spriteBatch, ToVector2(JointID.ShoulderRight, size, skeletonId), 
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Head, size, skeletonId), 
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ShoulderCenter, size, skeletonId), 
        ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position);
            
    //Left Hand
    lb.Draw(spriteBatch, ToVector2(JointID.HandLeft, size, skeletonId), 
        ToVector2(JointID.WristLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.WristLeft, size, skeletonId), 
        ToVector2(JointID.ElbowLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.ElbowLeft, size, skeletonId), 
        ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position);

    //Hips & Spine
    lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId), 
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId), 
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId), 
        ToVector2(JointID.HipCenter, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId), 
        ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position);

    //Left foot
    lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId), 
        ToVector2(JointID.KneeLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.KneeLeft, size, skeletonId), 
        ToVector2(JointID.AnkleLeft, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.AnkleLeft, size, skeletonId), 
        ToVector2(JointID.FootLeft, size, skeletonId), color, position);

    //Right foot
    lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId), 
        ToVector2(JointID.KneeRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.KneeRight, size, skeletonId), 
        ToVector2(JointID.AnkleRight, size, skeletonId), color, position);
    lb.Draw(spriteBatch, ToVector2(JointID.AnkleRight, size, skeletonId), 
        ToVector2(JointID.FootRight, size, skeletonId), color, position);
}

Be sure to read the source code of Neat’s KinectEngine class for more information.

Back to Home