I recently went back to programming to Kinect for a little off-base fun. I wanted to see how hard it would be to move the Kinect up and down and then capture a picture. Forgoing TDD, I created a WPF project that referenced the Kinect SDK.
I then added some buttons to control the up/down of the Kinect and a button to take a picture of the streaming image. I threw in 2 image controls – one to show the streaming image and one to show the still picture that was taken.
Here is the designer for those of you who are XAML-Impaired:
I then wired up the code behind to stream the images:
public MainWindow()
{
InitializeComponent();
kinectSensor.DepthStream.Enable();
kinectSensor.ColorStream.Enable();
kinectSensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
kinectSensor.Start();
}
void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
ColorImageFrame currentFrame = e.OpenColorImageFrame();
if (currentFrame != null)
{
byte[] pixelData = new byte[currentFrame.PixelDataLength];
currentFrame.CopyPixelDataTo(pixelData);
BitmapSource bitMapSource = BitmapImage.Create(currentFrame.Width,
currentFrame.Height,96,96, PixelFormats.Bgr32, null,
pixelData, currentFrame.Width * currentFrame.BytesPerPixel);
this.streamingVideoImage.Source = bitMapSource;
}
}
And here is the output:
I then added in the ability to take a picture:
void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
//Same code as before
this.streamingVideoImage.Source = bitMapSource;
if (takePicture)
{
this.takePictureImage.Source = bitMapSource;
takePicture = false;
}
}
}
And here is the output:
Feelin’ good, I then went to add the ability to move the Kinect up and down. I read this article with the point of emphasis that you shouldn’t run the motor often or continuously. I thought of moving the sensor of groups of 5 degrees at a time like so:
private void upButton_Click(object sender, RoutedEventArgs e)
{
if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
{
kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
}
}
With a complimentary function for down. Sure enough, it worked.
The only thing I don’t like is that the picture freezes between the adjustment. In addition, if you follow the MSDN article where you should throw is a Thread.Sleep(1000) after each change, the effect is kinda hokey. I then thought about putting in an Image like this 
while the camera is adjusting and the thread sleeps – just so the user knows that the camera is adjusting. I whipped up a function like this:
private void downButton_Click(object sender, RoutedEventArgs e)
{
if (kinectSensor.ElevationAngle > kinectSensor.MinElevationAngle + 5)
{
kinectSensor.ElevationAngle = kinectSensor.ElevationAngle - 5;
}
XOutTheStreamingImage();
Thread.Sleep(1000);
}
private void XOutTheStreamingImage()
{
this.InvalidateVisual();
BitmapImage xImage = new BitmapImage();
xImage.BeginInit();
xImage.UriSource = new Uri(@"C:\Users\Jamie\Documents\Visual Studio 2010\Projects\Tff.KinectExample_Solution\Tff.KinectExample\X.PNG");
xImage.EndInit();
this.streamingVideoImage.Source = xImage;
this.InvalidateVisual();
}
One of the problem is – it doesn’t work. A larger problem is – I don’t know why. The image never shows – though the thread is sleeping. I then put the XOutTheStreamingImage before the sensor change. Nope. I then detached the event handler and then reattached it:
private void upButton_Click(object sender, RoutedEventArgs e)
{
kinectSensor.AllFramesReady -= new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
XOutTheStreamingImage();
if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
{
kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
}
Thread.Sleep(1000);
kinectSensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
}
Nope. I then detached it and left it detached. It then kinda worked. If I hit up/down a couple of times, I got it to work:
This kind of random behavior smells like a multi-threading issue. If the AllFramesReady event was on a different thread, then the X image would shows for 1/6 of a second. If the elevation angle was also on a different thread – then the sleep wouldn’t matter. I fired up IL Spy and sure enough, check the locks:
And after searching for “Thread”, I found this:
Sure enough, the frame is processed a a different thread. And check out the Initialize method:
so it looks like I can’t put my XImage into the elevationChange event and the frameReady event. Instead, I can unhook the event handler and show the image:
kinectSensor.AllFramesReady -= new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
XOutTheStreamingImage();
if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
{
kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
}
Sure enough – that works – but it only flashes for a second. I then tried and displayed the X for a second after the screen is invalidated:
private void XOutTheStreamingImage()
{
this.InvalidateVisual();
BitmapImage xImage = new BitmapImage();
xImage.BeginInit();
xImage.UriSource = new Uri(@"C:\Users\Jamie\Documents\Visual Studio 2010\Projects\Tff.KinectExample_Solution\Tff.KinectExample\X.PNG");
xImage.EndInit();
this.streamingVideoImage.Source = xImage;
this.InvalidateVisual();
Thread.Sleep(1000);
}
However, that doesn’t work. The thread sleeps BEFORE the screen refreshes, so I am stuck with the last image from the Kinect. I am thinking I am stuck – there is no way to:
- Stop the Kiect from capturing images
- Update the screen
- Move the Kinect Angle
- Start the Kiect to capture images
The way the API is set up…
I think I need my thinking chair…
Jamie Dixon's Home 