Programming the Kinect

I recently went back to programming to Kinect for a little off-base fun.  I wanted to see how hard it would be to move the Kinect up and down and then capture a picture.  Forgoing TDD, I created a WPF project that referenced the Kinect SDK.

I then added some buttons to control the up/down of the Kinect and a button to take a picture of the streaming image. I threw in 2 image controls – one to show the streaming image and one to show the still picture that was taken.

Here is the designer for those of you who are XAML-Impaired:

I then wired up the code behind to stream the images:

public MainWindow()
{
    InitializeComponent();          
    kinectSensor.DepthStream.Enable();
    kinectSensor.ColorStream.Enable();
    kinectSensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
    kinectSensor.Start();
}

void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
    ColorImageFrame currentFrame = e.OpenColorImageFrame();
    if (currentFrame != null)
    {
        byte[] pixelData = new byte[currentFrame.PixelDataLength];
        currentFrame.CopyPixelDataTo(pixelData);
        BitmapSource bitMapSource = BitmapImage.Create(currentFrame.Width,
            currentFrame.Height,96,96, PixelFormats.Bgr32, null,
            pixelData, currentFrame.Width * currentFrame.BytesPerPixel);
        this.streamingVideoImage.Source = bitMapSource;
    }

}

And here is the output:

I then added in the ability to take a picture:

void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
        //Same code as before
 
        this.streamingVideoImage.Source = bitMapSource;
        if (takePicture)
        {
            this.takePictureImage.Source = bitMapSource;
            takePicture = false;
        }
    }
}

 

And here is the output:

Feelin’ good, I then went to add the ability to move the Kinect up and down.  I read this article with the point of emphasis that you shouldn’t run the motor often or continuously.  I thought of moving the sensor of groups of 5 degrees at a time like so:

private void upButton_Click(object sender, RoutedEventArgs e)
{
    
    if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
    {
        kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
    }
}

With a complimentary function for down.  Sure enough, it worked.

The only thing I don’t like is that the picture freezes between the adjustment.  In addition, if you follow the MSDN article where you should throw is a Thread.Sleep(1000) after each change, the effect is kinda hokey.  I then thought about putting in an Image like this while the camera is adjusting and the thread sleeps – just so the user knows that the camera is adjusting.  I whipped up a function like this:

private void downButton_Click(object sender, RoutedEventArgs e)
{
    
    if (kinectSensor.ElevationAngle > kinectSensor.MinElevationAngle + 5)
    {
        kinectSensor.ElevationAngle = kinectSensor.ElevationAngle - 5;
    }
    XOutTheStreamingImage();
    Thread.Sleep(1000);
}

private void XOutTheStreamingImage()
{
    this.InvalidateVisual();
    BitmapImage xImage = new BitmapImage();
    xImage.BeginInit();
    xImage.UriSource = new Uri(@"C:\Users\Jamie\Documents\Visual Studio 2010\Projects\Tff.KinectExample_Solution\Tff.KinectExample\X.PNG");
    xImage.EndInit();
    this.streamingVideoImage.Source = xImage;
    this.InvalidateVisual();
}

One of the problem is – it doesn’t work.  A larger problem is – I don’t know why.  The image never shows – though the thread is sleeping.  I then put the XOutTheStreamingImage before the sensor change.  Nope.  I then detached the event handler and then reattached it:

private void upButton_Click(object sender, RoutedEventArgs e)
{
    kinectSensor.AllFramesReady -= new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
    XOutTheStreamingImage();
    if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
    {
        kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
    }
    Thread.Sleep(1000);
    kinectSensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
}

Nope.  I then detached it and left it detached.  It then kinda worked.  If I hit up/down a couple of times, I got it to work:

This kind of random behavior smells like a multi-threading issue.  If the AllFramesReady event was on a different thread, then the X image would shows for 1/6 of a second.  If the elevation angle was also on a different thread – then the sleep wouldn’t matter.  I fired up IL Spy and sure enough, check the locks:

And after searching for “Thread”, I found this:

Sure enough, the frame is processed a a different thread. And check out the Initialize method:

so it looks like I can’t put my XImage  into the elevationChange event and the frameReady event.  Instead, I can unhook the event handler and show the image:

kinectSensor.AllFramesReady -= new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
XOutTheStreamingImage();
if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
{
    kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
}

Sure enough – that works – but it only flashes for a second.  I then tried and displayed the X for a second after the screen is invalidated:

 

private void XOutTheStreamingImage()
{
    this.InvalidateVisual();
    BitmapImage xImage = new BitmapImage();
    xImage.BeginInit();
    xImage.UriSource = new Uri(@"C:\Users\Jamie\Documents\Visual Studio 2010\Projects\Tff.KinectExample_Solution\Tff.KinectExample\X.PNG");
    xImage.EndInit();
    this.streamingVideoImage.Source = xImage;
    this.InvalidateVisual();
    Thread.Sleep(1000);
}

However, that doesn’t work.  The thread sleeps BEFORE the screen refreshes, so I am stuck with the last image from the Kinect.  I am thinking I am stuck – there is no way to:

• Stop the Kiect from capturing images
• Update the screen
• Move the Kinect Angle
• Start the Kiect to capture images

The way the API is set up…

I think I need my thinking chair…