Consuming Sky Biometry’s Image Recognition API

I was looking at Sky Biometry a couple of days ago to do some facial recognition.  I am not sure how many other companies out there do this, but working with their API from F# was a great experience. 

Sky Biometry uses both Json and Xml for API calls.  They also have an example covering library in C#.  I decided to use F# and use the json type provider to explore their api and the quality of their recognition algorithms.  They have a good documentation page and it didn’t take very long to get my account up and running and then making API calls.

I thought it would be fun to use an example from the first Terminator movie when Ah-nold went around looking for Sarah Connor.  I picked two images from the movie series.  The first is this image of a photograph of Sarah Connor has taken at the end of the first movie.  I know that the terminator didn’t have this photograph in the movie, but working off-script, pretend that the Terminator has that photograph.  This second image is from the first movie when she is in a bar.  So if the Terminator was powered by Sky Biometry and found Sarah in the bar, how close would it match her to the first photo?


The first thing I did was to fire up a FSharp project in Visual Studio and start scripting the API calls that I would need to do facial recognition. 


With all of the calls working in the REPL, I then moved the code into my module.  I declared the types at the top and then crated a class that could be consumed by external projects.


You will notice that the type providers are using a local copy of the json to infer the type in the module.  I did run into the problem where the type provider using the web call was not capturing the full graph in the type definition, so I took the json and made it local.  This led to an interesting problem because a FSharp project out of the box in Visual Studio does not support adding folders.  To get around that, I went to my file system and added a folder


I then created files in the folder that correspond to the type providers


Next I unloaded the FS project and edited it


and in the project, I added those files to the ItemGroup that brings in all of the files from disk

  1. <ItemGroup>
  2.   <Compile Include="SkyBiometryImageComparer.fs" />
  3.   <None Include="Script.fsx" />
  4.   <None Include="packages.config" />
  5.   <None Include="app.config" />
  6.   <None Include="SkyBiometryImageJson/AddTags.json" />
  7.   <None Include="SkyBiometryImageJson/FaceDetection.json" />
  8.   <None Include="SkyBiometryImageJson/FaceRecognition.json" />
  9.   <None Include="SkyBiometryImageJson/FaceTraining.json" />
  10.   <None Include="SkyBiometryImageJson/RemoveTags.json" />
  11. </ItemGroup>

Once the project is re-loaded, the folder and the files show up in Solution Explorer.



In each of the .json files, I pasted in the results from the service call that I did in the REPL.  For example,


So then I swapped them out in the module and now I get the full graph


With that out of the way, I implemented a one to one match of supporting methods to the API calls:

  1. member this.DetectFace(imageUri:string)=
  2.     let stringBuilder = new StringBuilder()
  3.     stringBuilder.Append(skybiometryUri) |> ignore
  4.     stringBuilder.Append("/fc/faces/detect.json?urls=") |> ignore
  5.     stringBuilder.Append(imageUri) |> ignore
  6.     stringBuilder.Append("&api_key=") |> ignore
  7.     stringBuilder.Append(apiKey) |> ignore
  8.     stringBuilder.Append("&api_secret=") |> ignore
  9.     stringBuilder.Append(apiSecret) |> ignore
  10.     let faceDetection =  skybiometryFaceDetection.Load(stringBuilder.ToString())
  11.     faceDetection.Photos.[0].Tags.[0].Tid
  13. member this.SaveTag(tid:string)=
  14.     let stringBuilder = new StringBuilder()
  15.     stringBuilder.Append(skybiometryUri) |> ignore
  16.     stringBuilder.Append("/fc/tags/save.json?uid=") |> ignore
  17.     stringBuilder.Append(uid) |> ignore
  18.     stringBuilder.Append("&tids=") |> ignore
  19.     stringBuilder.Append(tid) |> ignore
  20.     stringBuilder.Append("&api_key=") |> ignore
  21.     stringBuilder.Append(apiKey) |> ignore
  22.     stringBuilder.Append("&api_secret=") |> ignore
  23.     stringBuilder.Append(apiSecret) |> ignore
  24.     let tags = skybiometryAddTags.Load(stringBuilder.ToString())
  25.     tags.Status
  27. member this.TrainFace()=
  28.     let stringBuilder = new StringBuilder()
  29.     stringBuilder.Append(skybiometryUri) |> ignore
  30.     stringBuilder.Append("/fc/faces/train.json?uids=") |> ignore
  31.     stringBuilder.Append(uid) |> ignore
  32.     stringBuilder.Append("&api_key=") |> ignore
  33.     stringBuilder.Append(apiKey) |> ignore
  34.     stringBuilder.Append("&api_secret=") |> ignore
  35.     stringBuilder.Append(apiSecret) |> ignore
  36.     let training = skybiometryFaceTraining.Load(stringBuilder.ToString())
  37.     training.Status
  39. member this.RecognizeFace(imageUri:string)=
  40.     let stringBuilder = new StringBuilder()
  41.     stringBuilder.Append(skybiometryUri) |> ignore
  42.     stringBuilder.Append("/fc/faces/recognize.json?uids=") |> ignore
  43.     stringBuilder.Append(uid) |> ignore
  44.     stringBuilder.Append("&urls=") |> ignore
  45.     stringBuilder.Append(imageUri) |> ignore
  46.     stringBuilder.Append("&api_key=") |> ignore
  47.     stringBuilder.Append(apiKey) |> ignore
  48.     stringBuilder.Append("&api_secret=") |> ignore
  49.     stringBuilder.Append(apiSecret) |> ignore
  50.     let recognition = skybiometryFaceRecognition.Load(stringBuilder.ToString())
  51.     if recognition.Photos.[0].Tags |> Seq.length > 0 then
  52.         recognition.Photos.[0].Tags.[0].Attributes.Face.Confidence
  53.     else
  54.         0
  56. member this.RemoveTag(tid:string) =
  57.     let stringBuilder = new StringBuilder()
  58.     stringBuilder.Append(skybiometryUri) |> ignore
  59.     stringBuilder.Append("/fc/tags/remove.json?tids=") |> ignore
  60.     stringBuilder.Append(tid) |> ignore
  61.     stringBuilder.Append("&api_key=") |> ignore
  62.     stringBuilder.Append(apiKey) |> ignore
  63.     stringBuilder.Append("&api_secret=") |> ignore
  64.     stringBuilder.Append(apiSecret) |> ignore
  65.     let tagRemoval = skybiometryRemoveTags.Load(stringBuilder.ToString())
  66.     tagRemoval.Status

The StringBuilder makes the code pretty verbose, but I understand that it is the most efficient way to aggregate strings so I went with it. Also note that this is happy path programming with no error checking and I assume that the json coming back is well formed.

In any event, with the supporting methods done, it was just an exercise of calling each one in turn:

  1. member this.CalculateFacialRecognitionConfidence(baseUri:string, comparisionUri:string) =
  2.     let tid = this.DetectFace(baseUri)
  3.     if this.SaveTag(tid) = "success" then
  4.         if this.TrainFace() = "success" then
  5.             let confidence = this.RecognizeFace(comparisionUri)
  6.             this.RemoveTag(tid) |> ignore
  7.             confidence
  8.         else
  9.             0
  10.     else
  11.         0

And hopping over to a C# unit test project, I can call the FSharp and run some tests.  I created a test for each of the supporting methods and then three happy path tests for the CalculateFacialRecognitionConfidence: comparing the same image should be 100, comparing 2 completely unrelated images should be 0, and then our exercise of identifying Sarah Connor should be better than 50/50.  Here is an example of the 100% match use case:


And here is the actual test of finding Sarah Connor.:

  1.     var imageComparer = new SkyBiometryImageComparer(skyBiometryUri, uid, apiKey, apiSecret);
  2.     String basePhotoUri = ";;
  3.     String targetPhotoUri = ";;
  4.     Int32 confidence = imageComparer.CalculateFacialRecognitionConfidence(basePhotoUri, targetPhotoUri);
  6.     bool expected = true;
  7.     bool actual = confidence > 50;
  8.     Assert.AreEqual(expected, actual);
  9. }

It runs green (the actual value is 58% for identifying Sarah Connor). 


My guess is that that Terminator would start shooting… first at Sarah Connor and then at C#.  The C# project that Sky Biometry provides has 2 classes of 1600 and 1420 lines of code to get functional equivalence of the 93 lines of F# code that I wrote (and a vast majority of that code is dealing with the string builder).