Terminator Program: With The Kinect 2

I got my hands on a Kinect2 last week so I decided to re-write the Terminator program using the Kinect2 api.  Microsoft made some major changes to the domain api (no more skeleton frame, now using a body) but the underlying logic is still the same.  Therefore, it was reasonably easy to port the code.  There is plenty of places in the V2 api that are not documented yet but because I did some work in the V1 api, I could still get things done.  For example, the V2 api documentation and code samples use event handlers to work with any new frame that arrives from the Kinect.  This lead to some pretty laggy code.  However, by using polling on a second thread, I was able to get the performance to where it needs to be.  Also, a minor annoyance is that you have to use Win8 with the Kinect 2.

So here is the Terminator application, Gen 2.  The UI is still just a series of UI controls:

 1 <Window x:Class="ChickenSoftware.Terminator.Gen2.UI.MainWindow"
 2         xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
 3         xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
 4         Title="MainWindow" Height="700" Width="650" Loaded="Window_Loaded">
 5     <Canvas Width="650" Height="700">
 6         <Image x:Name="kinectColorImage" Width="640" Height="480" />
 7         <Canvas x:Name="bodyCanvas" Width="640" Height="480" />
 8         <Button x:Name="takePhotoButton" Canvas.Left="10" 
 9                 Canvas.Top="485" Height="40" Width="125" Click="takePhotoButton_Click">Take Photo</Button>
10         <TextBox x:Name="facialRecognitionTextBox" Canvas.Left="10" Canvas.Top="540" Width="125" Height="40" FontSize="8" />
11         <Image x:Name="currentImage" Canvas.Left="165" Canvas.Top="485" Height="120" Width="170" />
12         <Image x:Name="compareImage" Canvas.Left="410" Canvas.Top="485" Height="120" Width="170" />
13     </Canvas>
14 </Window>
15 

In the code behind, I set up some class-level variables.  The only real difference is that the photo is moving from 640/480 to 1920/1080:

1         KinectSensor _kinectSensor = null;
2         Boolean _isKinectDisplayActive = false;
3         Boolean _isTakingPicture = false;
4         WriteableBitmap _videoBitmap = null;
5         Int32 _width = 1920;
6         Int32 _height = 1080;

When the page is loaded, a new thread is spun up that handles rendering the Kinect data:

1         private void Window_Loaded(object sender, RoutedEventArgs e)
2         {
3             SetUpKinect();
4             _isKinectDisplayActive = true;
5             Thread videoThread = new Thread(new ThreadStart(DisplayKinectData));
6             videoThread.Start();
7         }

Setting up the Kinect is a bit different (KinectSensor.GetDefault()) but intuitive:

1         internal void SetUpKinect()
2         {
3             _videoBitmap = new WriteableBitmap(1920, 1080, 96, 96, PixelFormats.Bgr32, null);
4             _kinectSensor = KinectSensor.GetDefault();
5             _kinectSensor.Open();
6         }

With the big change in the DisplayKinectData method

 1         internal void DisplayKinectData()
 2         {
 3             var colorFrameSource = _kinectSensor.ColorFrameSource;
 4             var colorFrameReader = colorFrameSource.OpenReader();
 5             var bodyFrameSource = _kinectSensor.BodyFrameSource;
 6             var bodyFrameReader = bodyFrameSource.OpenReader();
 7 
 8             while (_isKinectDisplayActive)
 9             {
10                 using (var colorFrame = colorFrameReader.AcquireLatestFrame())
11                 {
12                     if (colorFrame == null) continue;
13                     using (var bodyFrame = bodyFrameReader.AcquireLatestFrame())
14                     {
15                         if (bodyFrame == null) continue;
16                         //Color
17                         var colorFrameDescription = colorFrame.ColorFrameSource.CreateFrameDescription(ColorImageFormat.Bgra);
18                         var bytesPerPixel = colorFrameDescription.BytesPerPixel;
19                         var frameSize = colorFrameDescription.Width * colorFrameDescription.Height * bytesPerPixel;
20                         var colorData = new byte[frameSize];
21                         if (colorFrame.RawColorImageFormat == ColorImageFormat.Bgra)
22                         {
23                             colorFrame.CopyRawFrameDataToArray(colorData);
24                         }
25                         else
26                         {
27                             colorFrame.CopyConvertedFrameDataToArray(colorData, ColorImageFormat.Bgra);
28                         }
29                         //Body
30                         var bodies = new Body[bodyFrame.BodyCount];
31                         bodyFrame.GetAndRefreshBodyData(bodies);
32                         var trackedBody = bodies.FirstOrDefault(b => b.IsTracked);
33 
34                         //Update
35                         if (_isTakingPicture)
36                         {
37                             Dispatcher.Invoke(new Action(() => AnalyzePhoto(colorData)));
38                         }
39                         else
40                         {
41                             if (trackedBody == null)
42                             {
43                                 Dispatcher.Invoke(new Action(() => UpdateDisplay(colorData)));
44                             }
45                             else
46                             {
47                                 Dispatcher.Invoke(new Action(() => UpdateDisplay(colorData, trackedBody)));
48                             }
49                         }
50                     }
51                 }
52             }
53         }
54 

I am using a frameReader and frameSource for both the color (the video image) and the body (the old skeleton).  The method to get the frame has changed –> I am using AquireLatestFrame().  It is nice that we are still using byte[] to hold the data.

With the data in the byte[] arrays, the display is updated.  There are two UpdateDisplay methods:

 1         internal void UpdateDisplay(byte[] colorData)
 2         {
 3             var rectangle = new Int32Rect(0, 0, _width, _height);
 4             _videoBitmap.WritePixels(rectangle, colorData, _width * 4, 0);
 5             kinectColorImage.Source = _videoBitmap;
 6         }
 7 
 8         internal void UpdateDisplay(byte[] colorData, Body body)
 9         {
10             UpdateDisplay(colorData);
11             var drawingGroup = new DrawingGroup();
12             using (var drawingContext = drawingGroup.Open())
13             {
14                 var headPosition = body.Joints[JointType.Head].Position;
15                 if (headPosition.Z < 0)
16                 {
17                     headPosition.Z = 0.1f;
18                 }
19                 var adjustedHeadPosition = _kinectSensor.CoordinateMapper.MapCameraPointToDepthSpace(headPosition);
20                 bodyCanvas.Children.Clear();
21                 Rectangle headTarget = new Rectangle();
22                 headTarget.Fill = new SolidColorBrush(Colors.Red);
23                 headTarget.Width = 10;
24                 headTarget.Height = 10;
25                 Canvas.SetLeft(headTarget, adjustedHeadPosition.X + 75);
26                 Canvas.SetTop(headTarget, adjustedHeadPosition.Y);
27                 bodyCanvas.Children.Add(headTarget);
28             }
29         }

This is pretty much like V1 where the video byte[] is being written to a WritableBitmap and the body is being drawn on the canvas.  Note that like V1, the coordinates of the body need to be adjusted to the color frame.  The API has a series of overloads that makes it easy to do the translation.

With the display working, I added in taking the photo, sending it to Azure blob storage, and having Sky Biometry analyze the results.  This code is identical to V1 with the connection strings for Azure and Sky Biometry broken out into their own methods and the sensitive values placed into the app.config:

1         internal void AnalyzePhoto(byte[] colorData)
2         {
3             var bitmapSource = BitmapSource.Create(_width, _height, 96, 96, PixelFormats.Bgr32, null, colorData, _width * 4);
4             JpegBitmapEncoder encoder = new JpegBitmapEncoder();
5             encoder.Frames.Add(BitmapFrame.Create(bitmapSource));
6             var photoImage = UploadPhotoImage(encoder);
7             CompareImages(photoImage);
8             _isTakingPicture = false;
9         }

 1         internal PhotoImage UploadPhotoImage(JpegBitmapEncoder encoder)
 2         {
 3             using(MemoryStream memoryStream = new MemoryStream())
 4             {
 5                 encoder.Save(memoryStream);
 6                 var photoImage = new PhotoImage(Guid.NewGuid(), memoryStream.ToArray());
 7 
 8                 var customerUniqueId = new Guid(ConfigurationManager.AppSettings["customerUniqueId"]);
 9                 var connectionString = GetAzureConnectionString();
10 
11                 IPhotoImageProvider provider = new AzureStoragePhotoImageProvider(customerUniqueId, connectionString);
12                 provider.InsertPhotoImage(photoImage);
13                 memoryStream.Close();
14                 return photoImage;
15             }
16         }

 1         internal void CompareImages(PhotoImage photoImage)
 2         {
 3             String skyBiometryUri = ConfigurationManager.AppSettings["skyBiometryUri"];
 4             String uid = ConfigurationManager.AppSettings["skyBiometryUid"];
 5             String apiKey = ConfigurationManager.AppSettings["skyBiometryApiKey"];
 6             String apiSecret = ConfigurationManager.AppSettings["skyBiometryApiSecret"];
 7             var imageComparer = new SkyBiometryImageComparer(skyBiometryUri, uid, apiKey, apiSecret);
 8 
 9             String basePhotoUri = GetBasePhotoUri();
10             String targetPhotoUri = GetTargetPhotoUri(photoImage);
11             currentImage.Source = new BitmapImage(new Uri(targetPhotoUri));
12             compareImage.Source = new BitmapImage(new Uri(basePhotoUri));
13 
14             var matchValue = imageComparer.CalculateFacialRecognitionConfidence(basePhotoUri, targetPhotoUri);
15             facialRecognitionTextBox.Text = "Match Value Confience is: " + matchValue.Confidence.ToString();
16         }

With the code in place, I can the run the Terminator Gen 2:

I think I am doing the Sky Biometry recognition incorrectly so I will look at that later.  In any event, working with the Kinect V2 was fairly easy because it was close enough to the V1 that the concepts could translate.  I look forward to adding the targeting system this weekend!!!

Programming the Kinect

I recently went back to programming to Kinect for a little off-base fun.  I wanted to see how hard it would be to move the Kinect up and down and then capture a picture.  Forgoing TDD, I created a WPF project that referenced the Kinect SDK.

I then added some buttons to control the up/down of the Kinect and a button to take a picture of the streaming image. I threw in 2 image controls – one to show the streaming image and one to show the still picture that was taken.

Here is the designer for those of you who are XAML-Impaired:

I then wired up the code behind to stream the images:

public MainWindow()
{
    InitializeComponent();          
    kinectSensor.DepthStream.Enable();
    kinectSensor.ColorStream.Enable();
    kinectSensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
    kinectSensor.Start();
}

void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
    ColorImageFrame currentFrame = e.OpenColorImageFrame();
    if (currentFrame != null)
    {
        byte[] pixelData = new byte[currentFrame.PixelDataLength];
        currentFrame.CopyPixelDataTo(pixelData);
        BitmapSource bitMapSource = BitmapImage.Create(currentFrame.Width,
            currentFrame.Height,96,96, PixelFormats.Bgr32, null,
            pixelData, currentFrame.Width * currentFrame.BytesPerPixel);
        this.streamingVideoImage.Source = bitMapSource;
    }

}

And here is the output:

I then added in the ability to take a picture:

void kinectSensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
{
        //Same code as before
 
        this.streamingVideoImage.Source = bitMapSource;
        if (takePicture)
        {
            this.takePictureImage.Source = bitMapSource;
            takePicture = false;
        }
    }
}

And here is the output:

Feelin’ good, I then went to add the ability to move the Kinect up and down.  I read this article with the point of emphasis that you shouldn’t run the motor often or continuously.  I thought of moving the sensor of groups of 5 degrees at a time like so:

private void upButton_Click(object sender, RoutedEventArgs e)
{
    
    if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
    {
        kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
    }
}

With a complimentary function for down.  Sure enough, it worked.

The only thing I don’t like is that the picture freezes between the adjustment.  In addition, if you follow the MSDN article where you should throw is a Thread.Sleep(1000) after each change, the effect is kinda hokey.  I then thought about putting in an Image like this while the camera is adjusting and the thread sleeps – just so the user knows that the camera is adjusting.  I whipped up a function like this:

private void downButton_Click(object sender, RoutedEventArgs e)
{
    
    if (kinectSensor.ElevationAngle > kinectSensor.MinElevationAngle + 5)
    {
        kinectSensor.ElevationAngle = kinectSensor.ElevationAngle - 5;
    }
    XOutTheStreamingImage();
    Thread.Sleep(1000);
}

private void XOutTheStreamingImage()
{
    this.InvalidateVisual();
    BitmapImage xImage = new BitmapImage();
    xImage.BeginInit();
    xImage.UriSource = new Uri(@"C:\Users\Jamie\Documents\Visual Studio 2010\Projects\Tff.KinectExample_Solution\Tff.KinectExample\X.PNG");
    xImage.EndInit();
    this.streamingVideoImage.Source = xImage;
    this.InvalidateVisual();
}

One of the problem is – it doesn’t work.  A larger problem is – I don’t know why.  The image never shows – though the thread is sleeping.  I then put the XOutTheStreamingImage before the sensor change.  Nope.  I then detached the event handler and then reattached it:

private void upButton_Click(object sender, RoutedEventArgs e)
{
    kinectSensor.AllFramesReady -= new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
    XOutTheStreamingImage();
    if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
    {
        kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
    }
    Thread.Sleep(1000);
    kinectSensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
}

Nope.  I then detached it and left it detached.  It then kinda worked.  If I hit up/down a couple of times, I got it to work:

This kind of random behavior smells like a multi-threading issue.  If the AllFramesReady event was on a different thread, then the X image would shows for 1/6 of a second.  If the elevation angle was also on a different thread – then the sleep wouldn’t matter.  I fired up IL Spy and sure enough, check the locks:

And after searching for “Thread”, I found this:

Sure enough, the frame is processed a a different thread. And check out the Initialize method:

so it looks like I can’t put my XImage  into the elevationChange event and the frameReady event.  Instead, I can unhook the event handler and show the image:

kinectSensor.AllFramesReady -= new EventHandler<AllFramesReadyEventArgs>(kinectSensor_AllFramesReady);
XOutTheStreamingImage();
if (kinectSensor.ElevationAngle < kinectSensor.MaxElevationAngle - 5)
{
    kinectSensor.ElevationAngle = kinectSensor.ElevationAngle + 5;
}

Sure enough – that works – but it only flashes for a second.  I then tried and displayed the X for a second after the screen is invalidated:

private void XOutTheStreamingImage()
{
    this.InvalidateVisual();
    BitmapImage xImage = new BitmapImage();
    xImage.BeginInit();
    xImage.UriSource = new Uri(@"C:\Users\Jamie\Documents\Visual Studio 2010\Projects\Tff.KinectExample_Solution\Tff.KinectExample\X.PNG");
    xImage.EndInit();
    this.streamingVideoImage.Source = xImage;
    this.InvalidateVisual();
    Thread.Sleep(1000);
}

However, that doesn’t work.  The thread sleeps BEFORE the screen refreshes, so I am stuck with the last image from the Kinect.  I am thinking I am stuck – there is no way to:

• Stop the Kiect from capturing images
• Update the screen
• Move the Kinect Angle
• Start the Kiect to capture images

The way the API is set up…

I think I need my thinking chair…

Kinect SDK

I purchased a Kinect last week so that I could start messing around with its API.

1) There are two versions of the Kinect.  The XBOX 360 one and the Windows one.  The only difference between the two that I could gather is that the Windows one pre-loads the SDK and allows you to distribute your software commercially.  Since I am just a hobbyist, I went with the XBOX one for $100 cheaper.

2) The Kinect for the XBOX360 requires an additional power cord for it to connect to your computer,  You don’t need to buy it though as it comes included.  I made that mistake (and compounded it by buying from the Microsoft store at a premium)

3) There are a couple different SDKs floating around out there.  There is the 1.0 SDK and the 1.5 SDK.  You will want to use the 1.5 one (because newer is always better) and there is a HUGE difference in the APIs between the two versions to the point that anything you wrote in 1.0 is useless.

4) I started digging into programming the Kinect with this book.  After reading the SDK samples and documentation, it really isn’t necessary.  The SDK is really well documented and is probably the best place to start to learn about the technology.

5) Once I dove into programming the Kinect, I realize that that this is no small task.  For C#, the amount of code you need to write and the complexity is higher than any other Microsoft technology I have seen.  You will need to know about bit shifts, the low-level details of graphical classes, and advanced data structures.  For example, here is an example from the Kinect Explorer solution:

// Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
// that displays different players in different colors
private void ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream)
{
    int tooNearDepth = depthStream.TooNearDepth;
    int tooFarDepth = depthStream.TooFarDepth;
    int unknownDepth = depthStream.UnknownDepth;

    // Test that the buffer lengths are appropriately correlated, which allows us to use only one
    // value as the loop condition.
    if ((depthFrame.Length * 4) != this.depthFrame32.Length)
    {
        throw new InvalidOperationException();
    }

    for (int i16 = 0, i32 = 0; i32 < this.depthFrame32.Length; i16++, i32 += 4)
    {
        int player = depthFrame[i16] & DepthImageFrame.PlayerIndexBitmask;
        int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;
        
        if (player == 0 && realDepth == tooNearDepth)
        {
            // white 
            this.depthFrame32[i32 + RedIndex] = 255;
            this.depthFrame32[i32 + GreenIndex] = 255;
            this.depthFrame32[i32 + BlueIndex] = 255;
        }
        else if (player == 0 && realDepth == tooFarDepth)
        {
            // dark purple
            this.depthFrame32[i32 + RedIndex] = 66;
            this.depthFrame32[i32 + GreenIndex] = 0;
            this.depthFrame32[i32 + BlueIndex] = 66;
        }
        else if (player == 0 && realDepth == unknownDepth)
        {
            // dark brown
            this.depthFrame32[i32 + RedIndex] = 66;
            this.depthFrame32[i32 + GreenIndex] = 66;
            this.depthFrame32[i32 + BlueIndex] = 33;
        }
        else
        {
            // transform 13-bit depth information into an 8-bit intensity appropriate
            // for display (we disregard information in most significant bit)
            byte intensity = (byte)(~(realDepth >> 4));

            // tint the intensity by dividing by per-player values
            this.depthFrame32[i32 + RedIndex] = (byte)(intensity >> IntensityShiftByPlayerR[player]);
            this.depthFrame32[i32 + GreenIndex] = (byte)(intensity >> IntensityShiftByPlayerG[player]);
            this.depthFrame32[i32 + BlueIndex] = (byte)(intensity >> IntensityShiftByPlayerB[player]);
        }
    }
}

My goal is to have enough to work with to present at TriNug’s code camp in November.  That might be a stretch…