Consuming Sky Biometry’s Image Recognition API

June 24, 2014 2 Comments

I was looking at Sky Biometry a couple of days ago to do some facial recognition.  I am not sure how many other companies out there do this, but working with their API from F# was a great experience. 

Sky Biometry uses both Json and Xml for API calls.  They also have an example covering library in C#.  I decided to use F# and use the json type provider to explore their api and the quality of their recognition algorithms.  They have a good documentation page and it didn’t take very long to get my account up and running and then making API calls.

I thought it would be fun to use an example from the first Terminator movie when Ah-nold went around looking for Sarah Connor.  I picked two images from the movie series.  The first is this image of a photograph of Sarah Connor has taken at the end of the first movie.  I know that the terminator didn’t have this photograph in the movie, but working off-script, pretend that the Terminator has that photograph.  This second image is from the first movie when she is in a bar.  So if the Terminator was powered by Sky Biometry and found Sarah in the bar, how close would it match her to the first photo?

image

The first thing I did was to fire up a FSharp project in Visual Studio and start scripting the API calls that I would need to do facial recognition. 

image

With all of the calls working in the REPL, I then moved the code into my module.  I declared the types at the top and then crated a class that could be consumed by external projects.

image

You will notice that the type providers are using a local copy of the json to infer the type in the module.  I did run into the problem where the type provider using the web call was not capturing the full graph in the type definition, so I took the json and made it local.  This led to an interesting problem because a FSharp project out of the box in Visual Studio does not support adding folders.  To get around that, I went to my file system and added a folder

image

I then created files in the folder that correspond to the type providers

image

Next I unloaded the FS project and edited it

imageimage

and in the project, I added those files to the ItemGroup that brings in all of the files from disk

  1. <ItemGroup>
  2.   <Compile Include="SkyBiometryImageComparer.fs" />
  3.   <None Include="Script.fsx" />
  4.   <None Include="packages.config" />
  5.   <None Include="app.config" />
  6.   <None Include="SkyBiometryImageJson/AddTags.json" />
  7.   <None Include="SkyBiometryImageJson/FaceDetection.json" />
  8.   <None Include="SkyBiometryImageJson/FaceRecognition.json" />
  9.   <None Include="SkyBiometryImageJson/FaceTraining.json" />
  10.   <None Include="SkyBiometryImageJson/RemoveTags.json" />
  11. </ItemGroup>

Once the project is re-loaded, the folder and the files show up in Solution Explorer.

image

 

In each of the .json files, I pasted in the results from the service call that I did in the REPL.  For example,

image

So then I swapped them out in the module and now I get the full graph

image

With that out of the way, I implemented a one to one match of supporting methods to the API calls:

  1. member this.DetectFace(imageUri:string)=
  2.     let stringBuilder = new StringBuilder()
  3.     stringBuilder.Append(skybiometryUri) |> ignore
  4.     stringBuilder.Append("/fc/faces/detect.json?urls=") |> ignore
  5.     stringBuilder.Append(imageUri) |> ignore
  6.     stringBuilder.Append("&api_key=") |> ignore
  7.     stringBuilder.Append(apiKey) |> ignore
  8.     stringBuilder.Append("&api_secret=") |> ignore
  9.     stringBuilder.Append(apiSecret) |> ignore
  10.     let faceDetection =  skybiometryFaceDetection.Load(stringBuilder.ToString())
  11.     faceDetection.Photos.[0].Tags.[0].Tid
  12.  
  13. member this.SaveTag(tid:string)=
  14.     let stringBuilder = new StringBuilder()
  15.     stringBuilder.Append(skybiometryUri) |> ignore
  16.     stringBuilder.Append("/fc/tags/save.json?uid=") |> ignore
  17.     stringBuilder.Append(uid) |> ignore
  18.     stringBuilder.Append("&tids=") |> ignore
  19.     stringBuilder.Append(tid) |> ignore
  20.     stringBuilder.Append("&api_key=") |> ignore
  21.     stringBuilder.Append(apiKey) |> ignore
  22.     stringBuilder.Append("&api_secret=") |> ignore
  23.     stringBuilder.Append(apiSecret) |> ignore
  24.     let tags = skybiometryAddTags.Load(stringBuilder.ToString())
  25.     tags.Status
  26.  
  27. member this.TrainFace()=
  28.     let stringBuilder = new StringBuilder()
  29.     stringBuilder.Append(skybiometryUri) |> ignore
  30.     stringBuilder.Append("/fc/faces/train.json?uids=") |> ignore
  31.     stringBuilder.Append(uid) |> ignore
  32.     stringBuilder.Append("&api_key=") |> ignore
  33.     stringBuilder.Append(apiKey) |> ignore
  34.     stringBuilder.Append("&api_secret=") |> ignore
  35.     stringBuilder.Append(apiSecret) |> ignore
  36.     let training = skybiometryFaceTraining.Load(stringBuilder.ToString())
  37.     training.Status
  38.  
  39. member this.RecognizeFace(imageUri:string)=
  40.     let stringBuilder = new StringBuilder()
  41.     stringBuilder.Append(skybiometryUri) |> ignore
  42.     stringBuilder.Append("/fc/faces/recognize.json?uids=") |> ignore
  43.     stringBuilder.Append(uid) |> ignore
  44.     stringBuilder.Append("&urls=") |> ignore
  45.     stringBuilder.Append(imageUri) |> ignore
  46.     stringBuilder.Append("&api_key=") |> ignore
  47.     stringBuilder.Append(apiKey) |> ignore
  48.     stringBuilder.Append("&api_secret=") |> ignore
  49.     stringBuilder.Append(apiSecret) |> ignore
  50.     let recognition = skybiometryFaceRecognition.Load(stringBuilder.ToString())
  51.     if recognition.Photos.[0].Tags |> Seq.length > 0 then
  52.         recognition.Photos.[0].Tags.[0].Attributes.Face.Confidence
  53.     else
  54.         0
  55.  
  56. member this.RemoveTag(tid:string) =
  57.     let stringBuilder = new StringBuilder()
  58.     stringBuilder.Append(skybiometryUri) |> ignore
  59.     stringBuilder.Append("/fc/tags/remove.json?tids=") |> ignore
  60.     stringBuilder.Append(tid) |> ignore
  61.     stringBuilder.Append("&api_key=") |> ignore
  62.     stringBuilder.Append(apiKey) |> ignore
  63.     stringBuilder.Append("&api_secret=") |> ignore
  64.     stringBuilder.Append(apiSecret) |> ignore
  65.     let tagRemoval = skybiometryRemoveTags.Load(stringBuilder.ToString())
  66.     tagRemoval.Status

The StringBuilder makes the code pretty verbose, but I understand that it is the most efficient way to aggregate strings so I went with it. Also note that this is happy path programming with no error checking and I assume that the json coming back is well formed.

In any event, with the supporting methods done, it was just an exercise of calling each one in turn:

  1. member this.CalculateFacialRecognitionConfidence(baseUri:string, comparisionUri:string) =
  2.     let tid = this.DetectFace(baseUri)
  3.     if this.SaveTag(tid) = "success" then
  4.         if this.TrainFace() = "success" then
  5.             let confidence = this.RecognizeFace(comparisionUri)
  6.             this.RemoveTag(tid) |> ignore
  7.             confidence
  8.         else
  9.             0
  10.     else
  11.         0

And hopping over to a C# unit test project, I can call the FSharp and run some tests.  I created a test for each of the supporting methods and then three happy path tests for the CalculateFacialRecognitionConfidence: comparing the same image should be 100, comparing 2 completely unrelated images should be 0, and then our exercise of identifying Sarah Connor should be better than 50/50.  Here is an example of the 100% match use case:

image

And here is the actual test of finding Sarah Connor.:

  1.     var imageComparer = new SkyBiometryImageComparer(skyBiometryUri, uid, apiKey, apiSecret);
  2.     String basePhotoUri = "http://img2.wikia.nocookie.net/__cb20080226002749/terminator/images/thumb/7/75/T2_sarah_polaroid.jpg/300px-T2_sarah_polaroid.jpg&quot;;
  3.     String targetPhotoUri = "http://emandrews2.files.wordpress.com/2013/04/sarah-connor-scared.jpg&quot;;
  4.     Int32 confidence = imageComparer.CalculateFacialRecognitionConfidence(basePhotoUri, targetPhotoUri);
  5.  
  6.     bool expected = true;
  7.     bool actual = confidence > 50;
  8.     Assert.AreEqual(expected, actual);
  9. }

It runs green (the actual value is 58% for identifying Sarah Connor). 

image

My guess is that that Terminator would start shooting… first at Sarah Connor and then at C#.  The C# project that Sky Biometry provides has 2 classes of 1600 and 1420 lines of code to get functional equivalence of the 93 lines of F# code that I wrote (and a vast majority of that code is dealing with the string builder).

 

Filed under F#, Image Recognition

TRINUG F# Analytics SIG Prep

June 17, 2014 1 Comment

I am putting together the finishing touches of the lab that the F#/data analytics group will do on Wednesday. We are working through Johan Astborg’s awesome book F# For Quantitative Finance.  Since the group is pretty much all C# with some bi-curious F#ers, I created a solution that mixes both C# and F#.  I put F# in the places where the language really shines: the data layer and the analytical layer.  I put C# in the places where C# has popular support: UI and unit tests.

image

In this lab, I am introducing script files and the REPL for the first time as a way to ‘prove out’ concepts.  Once the idea is sufficiently hashed out in the REPL, the code is moved into a source file and corresponding unit tests are created.  For example, in the data layer, we use the csv type provider to hit up the yahoo data and save and retrieve it from disk:

  1. #r "C:\Users\Jamie\Desktop\NewCo.OptionsTradingProgram.Solution\packages\FSharp.Data.2.0.8\lib\portable-net40+sl5+wp8+win8\FSharp.Data.dll"
  2. #r "C:\Users\Jamie\Desktop\NewCo.OptionsTradingProgram.Solution\packages\Newtonsoft.Json.6.0.3\lib\\net45\Newtonsoft.Json.dll"
  3.  
  4. open System
  5. open FSharp.Data
  6. open Newtonsoft.Json
  7. open System.IO
  8.  
  9. type csvProvider = CsvProvider<"http://ichart.finance.yahoo.com/table.csv?s=MSFT&quot;>
  10.  
  11. type YahooStockProvider() =
  12.     member this.GetData(stockSymbol: string) =
  13.             csvProvider.Load("http://ichart.finance.yahoo.com/table.csv?s=&quot; + stockSymbol).Rows
  14.     
  15. type FileSystemStockProvider() =
  16.     member this.PutData(filePath:string, stockData) =
  17.         let serializedData = stockData
  18.                                 |> Seq.map(fun row -> JsonConvert.SerializeObject(row))
  19.         File.WriteAllLines(filePath,serializedData)
  20.  
  21.     member this.GetData(filePath:string) =
  22.         let serializedData = File.ReadAllLines(filePath)
  23.         serializedData
  24.             |> Seq.map(fun row -> JsonConvert.DeserializeObject<(DateTime*float*float*float*float*int*float)>(row))

In the analytics module, we spend a good amount of time implementing mathematical calculations and getting comfortable with some of the differences between how imperative C# might implement a solution and how F# would

  1. open System.Collections.Generic
  2.  
  3. let testData = [1.0 .. 6.0]
  4. let sum = testData |> Seq.sum
  5. let average = testData |> Seq.average
  6. let min = testData |> Seq.min
  7. let max = testData |> Seq.max
  8. let evens = testData |> Seq.filter(fun number -> number % 2. = 0.)
  9. let addOne = testData |> Seq.map(fun number -> number + 1.)
  10.  
  11.  
  12. //http://www.mathsisfun.com/data/standard-deviation.html
  13. let variance (source:IEnumerable<double>) =
  14.     let mean = Seq.average source
  15.     let deltas = Seq.map(fun x -> pown(x-mean) 2) source
  16.     Seq.average deltas
  17.  
  18. let standardDeviation(values:IEnumerable<double>) =
  19.     sqrt(variance(values))
  20.  
  21. let movingAverage(values:IEnumerable<double>, windowSize:int)=
  22.     values
  23.         |> Seq.windowed (windowSize)
  24.         |> Seq.map(fun window -> window |> Seq.average)
  25.  
  26. let movingStandardDeviation(values:IEnumerable<double>, windowSize:int)=
  27.     values
  28.         |> Seq.windowed (windowSize)
  29.         |> Seq.map(fun window -> window |> standardDeviation)
  30.  
  31. let bollingerBands (values:IEnumerable<double>, windowSize:int)=
  32.     let movingAverage = movingAverage(values,windowSize)
  33.     let movingStandardDeviation = movingStandardDeviation(values,windowSize)
  34.     let movingStandardDeviation' = movingStandardDeviation |> Seq.map(fun window -> window * 2.)
  35.     Seq.zip movingAverage movingStandardDeviation'

Finally, we have a WPF application that does some basic charting and graphing.  The point of this lab is not about building a snazzy UI.  The UI (and in fact, the data layer) are just plug-ins to the heart of the application – the analytical module.  Also, a majority of the TRINUG’ content focuses on the UI so there is plenty of that already.   In any event, here is the code behind in C#.

  1. private void GoButton_Click(object sender, RoutedEventArgs e)
  2. {
  3.     FileSystemStockProvider provider = new FileSystemStockProvider(@"C:\Data\TEST.txt");
  4.     var stockPrices = provider.GetData().Take(20);
  5.     this.StockPriceDataGrid.ItemsSource = stockPrices;
  6.  
  7.     var adjustedClosePrices = from stockPrice in stockPrices
  8.             select stockPrice.Item7;
  9.  
  10.     var dates = from stockPrice in stockPrices.Skip(2)
  11.                              select new { stockPrice.Item1 };
  12.  
  13.     var calculations = new Calculations();
  14.     var movingAverage = calculations.MovingAverage(adjustedClosePrices, 3);
  15.     var movingAverages = dates.Zip(movingAverage, (d, p) => new { date=d.Item1, price=p});
  16.  
  17.     var bollingerBands = calculations.BollingerBands(adjustedClosePrices, 3);
  18.     var upperBandBands = dates.Zip(bollingerBands, (d, bb) => new { date = d.Item1, upperBand = bb.Item1 + (bb.Item2 * 2) });
  19.     var lowerBandBands = dates.Zip(bollingerBands, (d, bb) => new { date = d.Item1, lowerBand = bb.Item1 + (bb.Item2 * 2) * -1 });
  20.  
  21.     this.stockPriceLineGraph.DependentValuePath = "price";
  22.     this.stockPriceLineGraph.IndependentValuePath = "date";
  23.     this.stockPriceLineGraph.ItemsSource = movingAverages;
  24.  
  25.     this.stockPriceLineGraph2.DependentValuePath = "upperBand";
  26.     this.stockPriceLineGraph2.IndependentValuePath = "date";
  27.     this.stockPriceLineGraph2.ItemsSource = upperBandBands;
  28.  
  29.     this.stockPriceLineGraph3.DependentValuePath = "lowerBand";
  30.     this.stockPriceLineGraph3.IndependentValuePath = "date";
  31.     this.stockPriceLineGraph3.ItemsSource = lowerBandBands;
  32. }

 

I am toying of not using Linq and doing this all via imperative code, which would really drive home the point and power of a functional approach.  Here is the UI when running:

image

This was a alot fun to do and I am looking forward to the next lab, which is implementing the remainder of Astborg’s book.

 

 

Filed under F#

F# For Quantitative Finance

June 10, 2014 3 Comments

I picked up Johan Astborg’s F# for Quantitative Finance a couple of days ago and I thought it was great.  It was a practical hands-on way of working with FSharp.  It is so good, I think I am going to use it as a basis for a couple Triangle .NET User group F# SIGs.  One of the great things about the book are the code samples –they expose features of the F# language in a way that allows beginners to understand.  So in the trade-off between clarity and cleverness, Astborg errs on the side of clarity.   The only thing I think I would change would be to remove Chapter 2.  Chapter 2 suffers the same problem that many “intro to X computer language” books have –> pages of language features  with small independent samples.  This kind of book does not help many (if any) people learn the language easily, and it reads about as well as the MSDN component docs.

I did notice that Astborg did use the typical scientific computing approach to variable naming.  For example, on page 141 he creates a function like this:

  1. member this.black_scholes call_put_flag s x t r v =
  2.     let cnd x =
  3.         let a1 = 0.3198253
  4.         let a2 = -0.35653782
  5.         let a3 = 1.781477937
  6.         let a4 = -1.821255978
  7.         let a5 = 1.330274429
  8.         let pi = 3.141592654
  9.         let l = abs(x)
  10.         let k = 1.0/(1.0 + 0.2316419)
  11.         let w = (1.0)
  12.         if x < 0.0 then 1.0 – w else w
  13.  
  14.     let d1=(log(s/x) + (r*v*v*0.5)*t)/(v*sqrt(t))
  15.     let d2=d1-v*sqrt(t)
  16.     match call_put_flag with
  17.         | Put -> x*exp(-r*t)*cnd(-d2)-s*cnd(-d1)
  18.         | Call ->s*cnd(d1)-x*exp(-r*t)*cnd(d2)

Concentrating only on the function’s arguments, what is s,x,t,r, and v?  It is not immediately apparent and you have to read the comments above the function to discover that s is the stockprice, etc…  I think there is a better way and I wonder if FxCop (if it ever comes to F#) would agree with me..  For example, this is a slightly better version

  1. member this.black_scholes call_put_flag stockPrice strikePriceOfPotion timeToExpierationInYears riskFreeInterestRate volatility =
  2.     let cnd x =

The problem is that the number of arguments makes the naming unwieldy.  A third, and much preferred option is to pass in a data structure:

  1. type PutCallFlag = Put | Call
  2.  
  3. type black_scholes_input_data =
  4.     {stockPrice:float;
  5.     strikePriceOfPotion:float;
  6.     timeToExpierationInYears:int;
  7.     riskFreeInterestRate:float;
  8.     volatility:float}
  9.  
  10. type StockAnalyzer() =
  11.     member this.black_scholes (call_put_flag:PutCallFlag,inputData:black_scholes_input_data) =
  12.         let cnd x =
  13.             let a1 = 0.3198253
  14.             let a2 = -0.35653782
  15.             let a3 = 1.781477937
  16.             let a4 = -1.821255978
  17.             let a5 = 1.330274429
  18.             let pi = 3.141592654
  19.             let l = abs(x)
  20.             let k = 1.0/(1.0 + 0.2316419)
  21.             let w = (1.0)
  22.             if x < 0.0 then 1.0 – w else w
  23.  
  24.         let d1=(log(inputData.stockPrice/inputData.strikePriceOfPotion)

I doubt there is a performance hit and the code becomes much more readable.  It also forces the really long functions to be broken up and each piece becomes more testable.I  f there is not a corresponding domain for the input_arguments, then that naming will have to do.

 

Filed under Book Review, F#

Using Subsets for Association Rule Learning

June 3, 2014 1 Comment

I finished up writing the association rule program from MSDN in F# last week.  One of the things bothering me about the way I implemented the algorithms is that I hard-coded the combinations (antecedent and consequent) from the item-sets:

  1. static member GetCombinationsForDouble(itemSet: int[]) =
  2.     let combinations =  new List<int[]*int[]*int[]>()
  3.     combinations.Add(itemSet, [|itemSet.[0]|],[|itemSet.[1]|])
  4.     combinations
  5.  
  6. static member GetCombinationsForTriple(itemSet: int[]) =
  7.     let combinations =  new List<int[]*int[]*int[]>()
  8.     combinations.Add(itemSet, [|itemSet.[0]|],[|itemSet.[1];itemSet.[2]|])
  9.     combinations.Add(itemSet, [|itemSet.[1]|],[|itemSet.[0];itemSet.[2]|])
  10.     combinations.Add(itemSet, [|itemSet.[2]|],[|itemSet.[0];itemSet.[1]|])
  11.     combinations.Add(itemSet, [|itemSet.[0];itemSet.[1]|],[|itemSet.[2]|])
  12.     combinations.Add(itemSet, [|itemSet.[0];itemSet.[2]|],[|itemSet.[1]|])
  13.     combinations.Add(itemSet, [|itemSet.[1];itemSet.[2]|],[|itemSet.[0]|])
  14.     combinations

I thought it would be a fun exercise to make a function that returns the combinations for an N number of itemSets.  My first several attempts failed because  I started off with the wrong vocabulary.  I spent several days trying to determine how to create all of the combinations and/or permutations from the itemSet.  It then hit me that I would be looking at getting all subsets and what do you know, there are some excellent examples out there.

So if I was going to use the yield and yield-bang method of calculating the subsets in my class, I first needed to remove the rec and just let the class call itself.

  1. static member Subsets s =
  2.     set [
  3.         yield s
  4.         for e in s do
  5.             yield! AssociationRuleProgram2.Subsets (Set.remove e s) ]

I then needed a way of translating the itemSet which is a an int array into a set and back again.  Fortunately, the set module has ofArray and toArray functions so I wrote my code exactly the way I just described the problem:

  1. static member GetAntcentAndConsequent(itemSet: int[]) =
  2.     let combinations =  new List<int[]*int[]*int[]>()
  3.     let itemSet' = Set.ofArray itemSet
  4.     let subSets = AssociationRuleProgram2.Subsets itemSet'
  5.     let subSets' = Set.toArray subSets
  6.     let subSets'' = Array.map(fun s-> Set.toArray s)
  7.     let subSets''' = Array.map(fun s -> Seq.toArray s, AssociationRuleProgram2.GetAntcentAndConsequent s)

 

Note that I had to call toArray twice because the Subsets returns a Set<Set<Int>>.

In any event, I then needed a way of spitting the itemSet into antecedents and consequents (called combinations) based on the current subset.  I toyed around with a couple different ways of solving the problem when I stumbled upon a way that makes alot of sense to me.  I changed the itemset from an array of int to an array of tuple<int*bool>.  If the subset is in the itemSet, then the bool flag is true, if not it is false.  Then, I would apply an Seq.filter to the array and separate it out into antecedents and consequents.

  1. static member GetCombination array subArray =
  2.     let array' = array |> Seq.map(fun i -> i, subArray |> Array.exists(fun j -> i = j))
  3.     let antecedent = array' |> Seq.filter(fun (i,j) -> j = true) |> Seq.toArray
  4.     let consquent = array' |> Seq.filter(fun (i,j) -> j = false) |> Seq.toArray
  5.     let antecedent' = antecedent|> Seq.map(fun (i,j) -> i)
  6.     let consquent' = consquent|> Seq.map(fun (i,j) -> i)
  7.     Seq.toArray antecedent', Seq.toArray consquent'

The major downside of this approach is that I am using Array.exists for my filter flag so if there is more than one of the same value in the itemset, it does not work.  However, the original example had each itemset being unique so think I am OK.

So with these tow methods, I now have a way of dealing with N number of itemsets.  Interestingly, the amount of code (even with my verbose F#) is significantly less than the C# equivalent and closer to how I think I think.

 

 

 

Filed under Analytics, F#