March | 2014 | Jamie Dixon's Home

JavaScript Signature Capture Panel

March 25, 2014 Leave a comment

I am attempting to teach myself some more JavaScript. To that end I decided to replicate some of the projects I did in WPF/C# in HTML5/JavaScript. One of the 1st ‘hello world’ projects I did in WPF was creating a signature panel – so it seemed like a good place to start. The original blog post is here. The original WPF project took advantage of the InkCanvas class. Below is a code snippet of the how the events were captured in the original project:

private void inkSignature_MouseDown(object sender, MouseButtonEventArgs e)
{
    IsCapturing = true;
    glyph = new Glyph();
 
}
 
private void inkSignature_MouseUp(object sender, MouseButtonEventArgs e)
{
    IsCapturing = false;
    _signature.Glyphs.Add(glyph);
    startPoint = new Point();
    endPoint = new Point();
 
}
 
private void inkSignature_MouseMove(object sender, MouseEventArgs e)
{
    if (IsCapturing)
    {
        if (startPoint.X == 0 && startPoint.Y == 0 && endPoint.X == 0 && endPoint.Y == 0)
        {
            endPoint = new Point(e.GetPosition(this).X, e.GetPosition(this).Y);
        }
        else
        {
            startPoint = endPoint;
            endPoint = new Point(e.GetPosition(this).X, e.GetPosition(this).Y);
            Line line = new Line(startPoint, endPoint);
            glyph.Lines.Add(line);
        }
 
    }
 
}

To have the same effect in the browser, I swapped out the InkCanvas with the Canvas tag.

<canvas id="myCanvas" width="578" height="200" style="border:solid"></canvas>
<br />
<button id="resultButton" onclick="showSignature()"></button>

I then stubbed out the ‘mousedown’, ‘mouseup’, and ‘mousemove’ events to see if I was hooked up to them correctly and they were firing as expected:

<body>
 
    <script>
        canvas.addEventListener('mousemove', function (event) {
        }, false);
 
        canvas.addEventListener('mousedown', function (event) {
            alert("mousedown");
        }, false);
 
        canvas.addEventListener('mouseup', function (event) {
            alert("mouseup");
        }, false);
    </script>
 
</body>

I then thought about how to implement the InkCanvas code in JavaScript so I added some variables that all of the examples of StackOverflow use:

var canvas = document.getElementById('myCanvas');
var context = canvas.getContext('2d');

I then needed the function to calculate the mouse position relative to the signature panel (versus the screen). This was also pretty common on StackOverflow:

function getMousePosition(canvas, event) {
    var rectangle = canvas.getBoundingClientRect();
    return {
        x: event.clientX – rectangle.left,
        y: event.clientY – rectangle.top
    };
};

Finally, I could implement the WPF-equivalent logic. First was the variables to maintain state:

var isCapturing = false;
var startX = 0;
var startY = 0;
var endX = 0;
var endY = 0;
var signature = [];
var glyph = [];

And then the 3 event handlers:

canvas.addEventListener('mousemove', function (event) {
    if (isCapturing) {
        var mousePosition = getMousePosition(canvas, event);
 
        if (startX === 0 && startY === 0 && endX === 0 && endY === 0) {
            endX = mousePosition.x;
            endY = mousePosition.y;
        }
        else {
            startX = endX;
            startY = endY;
            endX = mousePosition.x;
            endY = mousePosition.y;
 
            context.beginPath();
            context.moveTo(startX, startY);
            context.lineTo(endX, endY);
            context.stroke()
 
            glyph.push(startX, startY, endX, endY);
        }
    }
}, false);
 
canvas.addEventListener('mousedown', function (event) {
    isCapturing = true;
    glyph = [];
}, false);
 
canvas.addEventListener('mouseup', function (event) {
    isCapturing = false;
    signature.push(glyph);
    var startX = 0;
    var startY = 0;
    var endX = 0;
    var endY = 0;
}, false);

When I ran it, I <almost> got it right:

The problem is that the mouseup event was not resetting the starting value of the next point to 0, so the signature was coming out as 1 long line. After sleeping on it (my pattern is write bugs at night, fix them in the morning), I realized I just had to reset the start and end coordinates on mouseup and then inspect in the mousemove. Here is the complete final code:

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml"&gt;
<head>
    <title></title>
</head>
<body>
    <canvas id="myCanvas" width="578" height="200" style="border:solid"></canvas>
    <br />
    <button id="resultButton" onclick="showSignature()"></button>
 
 
    <script>
        function showSignature() {
            alert(signature.length);
        };
    </script>
 
    <script>
        var canvas = document.getElementById('myCanvas');
        var context = canvas.getContext('2d');
        var isCapturing = false;
        var startX = 0;
        var startY = 0;
        var endX = 0;
        var endY = 0;
        var signature = [];
        var glyph = [];
 
        function getMousePosition(canvas, event) {
            var rectangle = canvas.getBoundingClientRect();
            return {
                x: event.clientX – rectangle.left,
                y: event.clientY – rectangle.top
            };
        };
 
        canvas.addEventListener('mousemove', function (event) {
            if (isCapturing) {
                var mousePosition = getMousePosition(canvas, event);
 
                if (endX === 0 && endY === 0) {
                    endX = mousePosition.x;
                    endY = mousePosition.y;
                }
                else {
                    startX = endX;
                    startY = endY;
                    endX = mousePosition.x;
                    endY = mousePosition.y;
 
                    context.beginPath();
                    context.moveTo(startX, startY);
                    context.lineTo(endX, endY);
                    context.stroke()
 
                    glyph.push(startX, startY, endX, endY);
                }
            }
        }, false);
 
        canvas.addEventListener('mousedown', function (event) {
            isCapturing = true;
            glyph = [];
 
            var mousePosition = getMousePosition(canvas, event);
            var startX = mousePosition.x;
            var startY = mousePosition.y;
        }, false);
 
        canvas.addEventListener('mouseup', function (event) {
            isCapturing = false;
            signature.push(glyph);
 
            startX = 0;
            startY = 0;
            endX = 0;
            endY = 0;
        }, false);
    </script>
 
</body>
</html>

And here it is in action:

Now all I have to do is put the points into the same data structures that I used in the WPF project: Signature –> Glyphs[] –> Lines[] –> Line.StartPoint && Line.EndPoint.

Filed under JavaScript

Apriori Algorithm and F# Using Elevator Inspection Data

March 18, 2014 1 Comment

Now that I have the elevator dataset in a workable state, I wanted to see what I could see with the data. I was reading Machine Learning In Action and the authors suggested that an Apriori Algorithm as a way to quantify associations among data points. I read both Harrington’s code and Wikipedia’s description and I found both the be impenetrable – the former because their code was unreadable and the later because the mathematical formulas depended on a level of algebra that I don’t have.

Fortunately, I found a C# project on Codeproject that had both an excellent example/introduction and C# code. I used the examples on the website to formulate my F# implementation.

The first thing I did was create a class that matched the 1st grid in the example

namespace ChickenSoftware.ElevatorChicken.Analysis
 
open System.Collections.Generic
 
type Transaction = {TID: string; Items: List<string> }
 
type Apriori(database: List<Transaction>, support: float, confidence: float) = 
    member this.Database = database
    member this.Support = support
    member this.Confidence = confidence

Note that because F# is immutable by default, the properties are read-only. I then created a unit test project that makes sure the constructor works without exceptions. The data matches the example:

public AprioriTests()
{
    var database = new List<Transaction>();
    database.Add(new Transaction("100", new List<string>() { "A", "C", "D" }));
    database.Add(new Transaction("200", new List<string>() { "B", "C", "E" }));
    database.Add(new Transaction("300", new List<string>() { "A", "B", "C", "E" }));
    database.Add(new Transaction("400", new List<string>() { "B", "E" }));
 
    _apriori = new Apriori(database, .5, .80);
 
}
 
[TestMethod]
public void ConstructorUsingValidArguments_ReturnsExpected()
{
    Assert.IsNotNull(_apriori);
}

I then need a function to count up all of the items in the Itemsets. I refused to use loops, so I first started using Seq.Fold, but I was having zero luck because I was trying to fold a Seq of List. I then started experimenting with other functions when I found Seq.Collect – which was perfect. So I created a function like this:

member this.GetC1() =
    database
 
member this.GetL1() =
    let numberOfTransactions = this.GetC1().Count
 
    this.GetC1()
        |> Seq.collect(fun d -> d.Items)
        |> Seq.countBy(fun i -> i)
        |> Seq.map(fun (t,i) -> t, i, float i/ float numberOfTransactions)
        |> Seq.filter(fun (t,i,p) -> p >= support)
        |> Seq.map(fun (t,i,p) -> t,i)
        |> Seq.sort
        |> Seq.toList

Note that the numberOfTransactions is for the database, not the individual items in the List<Item>. And the results match the example:

So this is great. My next stop was to build a list of pair combinations of the remaining values

The trick is that is not a Cartesian join of the original sets – it is only the surviving sets that are needed. My first attempt looked like:

let C1 = database
 
let L1 = C1
        |> Seq.map(fun t -> t.Items)
        |> Seq.collect(fun i -> i)
        |> Seq.countBy(fun i -> i)
        |> Seq.map(fun (t,i) -> t, i, float i/ float numberOftransactions)
        |> Seq.filter(fun (t,i,p) -> p >= support)
        |> Seq.toArray
let C2A = L1 
            |> Seq.map(fun (x,y,z) -> x)
            |> Seq.toArray
let C2B = L1 
            |> Seq.map(fun (x,y,z) -> x)
            |> Seq.toArray
let C2 = C2A |> Seq.collect(fun x -> C2B |> Seq.map(fun y -> x+y))
C2   

With the output like this:

I was running out of Saturday morning so I went over to stack overflow and got a couple of responses. I was on the right track with the concat, but I didn’t think about the List.Filter(), which would prune my list. With this in mind, I copied Mark’s code and got what I was looking for

member this.GetC2() =
    let l1Itemset = this.GetL1() 
                    |> Seq.map(fun (i,s) -> i)
 
    let itemset = 
        l1Itemset
            |> Seq.map(fun x -> l1Itemset |> Seq.map(fun y -> (x,y)))
            |> Seq.concat
            |> Seq.filter(fun (x,y) -> x < y)
            |> Seq.sort
            |> Seq.toList         
    
    let listContainsItem(l:List<string>, a,b) =
            l.Contains(a) && l.Contains(b)
    
    let someFunctionINeedToRename(l1:List<string>, l2)=
            l2 |> Seq.map(fun (x,y) -> listContainsItem(l1,x,y))
 
    let itemsetMatches = this.GetC1()
                            |> Seq.map(fun t -> t.Items)
                            |> Seq.map(fun i -> someFunctionINeedToRename(i,itemset))
 
    let itemSupport = itemsetMatches
                            |> Seq.map(Seq.map(fun i -> if i then 1 else 0))
                            |> Seq.reduce(Seq.map2(+))
 
    itemSupport
        |> Seq.zip(itemset)
        |> Seq.toList

So now I have C2 filling correctly:

Taking the results, I needed to get L2.

That was much simpler that getting C2 –> here is the code:

member this.GetL2() = 
    let numberOfTransactions = this.GetC1().Count
    
    this.GetC2()
            |> Seq.map(fun (i,n) -> i,n,float n/float numberOfTransactions)
            |> Seq.filter(fun (i,n,p) -> p >= support)
            |> Seq.map(fun (t,i,p) -> t,i)
            |> Seq.sort
            |> Seq.toList    

And when I run it – it matches this example exactly:

Finally, I added in a C# and L3. This code is identical to the C2/L2 code with one exception: mapping a triple and not a tuple: The C2 code maps like this

let itemset = 
    l1Itemset
        |> Seq.map(fun x -> l1Itemset |> Seq.map(fun y -> (x,y)))
        |> Seq.concat
        |> Seq.filter(fun (x,y) -> x < y)
        |> Seq.sort
        |> Seq.toList     

and the C3 code looks like this (took me 15 minutes to figure out line 3 below):

let itemset = 
    l2Itemset
        |> Seq.map(fun x -> l2Itemset |> Seq.map(fun y-> l2Itemset |> Seq.map(fun z->(fst x,fst y,snd z))))
        |> Seq.concat
        |> Seq.collect(fun d -> d)
        |> Seq.filter(fun (x,y,z) -> x < y && y < z)
        |> Seq.distinct
        |> Seq.sort
        |> Seq.toList    

With the C3 and L3 matching the example also:

I was now ready to put in the elevator data into the analysis. I think I am getting better at F# because I did the mapping, filtering, and transformation of the data from the server without looking at any other material and it look only 15 minutes.

type public ElevatorBuilder() = 
    let connectionString = ConfigurationManager.ConnectionStrings.["localData2"].ConnectionString;
 
    member public this.GetElevatorTransactions() =
        let transactions = this.GetElevators() 
                              |> Seq.map(fun e ->this.ConvertElevatorToTransaction(e))
        let transactionsList = new System.Collections.Generic.List<Transaction>(transactions)
        transactionsList
 
    member public this.ConvertElevatorToTransaction(i: string, t:string, c:string, s:string) =
        let items = new System.Collections.Generic.List<String>()
        items.Add(t)
        items.Add(c)
        items.Add(s)
        let transaction = {TID=i; Items=items}
        transaction
 
    member public this.GetElevators () =
        SqlConnection.GetDataContext(connectionString).ElevatorData201402
            |> Seq.map(fun e -> e.ID, e.EquipType,e.Capacity,e.Speed)
            |> Seq.filter(fun (i,et,c,s) -> not(String.IsNullOrEmpty(et)))
            |> Seq.filter(fun (i,et,c,s) -> c.HasValue)
            |> Seq.filter(fun (i,et,c,s) -> s.HasValue)
            |> Seq.map(fun (i,t,c,s) -> i, this.CatagorizeEquipmentType(t),c,s)
            |> Seq.map(fun (i,t,c,s) -> i,t,this.CatagorizeCapacity(c.Value),s)
            |> Seq.map(fun (i,t,c,s) -> i,t,c,this.CatagorizeSpeed(s.Value))
            |> Seq.map(fun (i,t,c,s) -> i.ToString(),t,c,s)

The longest part was aggregating the free-form text of the Equipment Type field (here is partial snip, you get the idea…)

member public this.CatagorizeEquipmentType(et: string) =
    match et.Trim() with 
        | "OTIS" -> "OTIS"
        | "OTIS (1-2)" -> "OTIS"
        | "OTIS (2-1)" -> "OTIS"
        | "OTIS hydro" -> "OTIS"
        | "OTIS, HYD" -> "OTIS"
        | "OTIS/ ASHEVILLE " -> "OTIS"
        | "OTIS/ MOUNTAIN " -> "OTIS"
        | "OTIS/#1" -> "OTIS"
        | "OTIS/#19 " -> "OTIS"

Assigning categories for speed and capacity was a snap using F#

member public this.CatagorizeCapacity(c: int) =
    let lowerBound = (c/25 * 25) + 1
    let upperBound = lowerBound + 24
    lowerBound.ToString() + "-" + upperBound.ToString()        
 
member public this.CatagorizeSpeed(s: int) =
    let lowerBound = (s/50 * 50) + 1
    let upperBound = lowerBound + 49
    lowerBound.ToString() + "-" + upperBound.ToString()    

With this in hand, I created a Console app that takes the 27K records and pushes them though the apriori algorithm:

private static void RunElevatorAnalysis()
{
    Stopwatch stopwatch = new Stopwatch();
    stopwatch.Start();
    ElevatorBuilder builder = new ElevatorBuilder();
    var transactions = builder.GetElevatorTransactions();
    stopwatch.Stop();
    Console.WriteLine("Building " + transactions.Count + " transactions took: " + stopwatch.Elapsed.TotalSeconds);
    var apriori = new Apriori(transactions, .1, .75);
    var c2 = apriori.GetC2();
    stopwatch.Reset();
    stopwatch.Start();
    var l1 = apriori.GetL1();
    Console.WriteLine("Getting L1 took: " + stopwatch.Elapsed.TotalSeconds);
    var l2 = apriori.GetL2();
    Console.WriteLine("Getting L2 took: " + stopwatch.Elapsed.TotalSeconds);
    var l3 = apriori.GetL3();
    Console.WriteLine("Getting L3 took: " + stopwatch.Elapsed.TotalSeconds);
    stopwatch.Stop();
    Console.WriteLine("–L1");
    foreach (var t in l1)
    {
        Console.WriteLine(t.Item1 + ":" + t.Item2);
    }
    Console.WriteLine("–L2");
    foreach (var t in l2)
    {
        Console.WriteLine(t.Item1 + ":" + t.Item2);
    }
    Console.WriteLine("–L3");
    foreach (var t in l3)
    {
        Console.WriteLine(t.Item1 + ":" + t.Item2);
    }
}

I then made an offering to the F# Gods and hit F5:

Doh! The gods were not pleased. I then went back to my initial filtering function and added a Seq.Take(25000) and the results:

So there a couple of things to draw from this exercise.

1) Apriori Algorithm is the wrong classification technique for this dataset. I had to bring the support way down (10%) to even get any readings. Also, there is too much dispersion of the values. This kind of algorithm is much better with N number of a smaller set of data values versus a fixed number of large values.

2) Even so, how cool is this? Compare the files just to make the C#/OO work versus with F#

And the Total LOC is 539 for C# versus 120 for F# – and the F# can be optimized using a better way to create search and itemsets. Hard-coding each level was a hack I did to get thing working and give me an understanding of how AA works. I bet this can be consolidated to well under 75 lines without sacrificing readability

3) I think the StackOverflow exception is because I am doing a Cartesian join and then paring the result. Using one of the other techniques suggested on SO will give much better results.

I any event, what a fun project! I can’t wait to optimize this and perhaps throw a different algorithm at the dataset in the coming weeks.

Filed under Analytics, F#, Open Data

Elevator App: Part 1 – Data Layer Using F#

March 11, 2014 2 Comments

At Open Data Day, fellow TRINUGER Elaine Cahill told me about a website where you can get all of the elevator inspection data for the state. It is found here. She went ahead and put the Wake County data onto Socrata. I wanted to look at the entire state so I went to the report page like so:

Unfortunately, when you try and pull down the entire state, you cause a server exception:

So I split the download in half. I then Imported it into Access and then SSISed it into Azure Sql. I then created a project to server the data and I decided to use F# type providers as a replacement for Entity Framework for my ORM. I could either use the SqlEntity TP or the SqlDataConnection TP to access the Sql Database on Azure. Both do not work out of the box.

SqlDataConnection

I could not get SqlDataConnection to work at all. When I hooked it up to a standard connection string in the config file, I got:

So when I copy and paste the connection string into the TP directly, it does make the connection to Azure, but then it comes back with this exception:

Without looking at the source. my guess is that the TP has hard-coded a reference to ‘syscomments’ and alas, Azure does not have that table.

SqlEntity

I then headed over to the SlqEntityTP to see if I could have better luck. Fortunately, the SqlEntity does work with both an Azure connection string in the .config file and can make a connection to an Azure database.

The problem I ran into was when I wanted to expose the SqlConnection the the WebAPI project that I wrote in C#. You can not mark SqlEntityTPs as public:

Note that the SqlDataConnection can be marked as public. <sigh>. I marked the SqlEntityTP as internal and then created a POCO to map between the SqlEntity type and a type that can be consumed by the outside world:

type public Elevator ={
        ID: int
        County: string
        StateId: string
        Type: string
        Operation: string
        Owner: string
        O_Address1: string
        O_Address2: string
        O_City: string
        O_State: string
        O_Zip: string
        User: string
        U_Address1: string
        U_Address2: string
        U_City: string
        U_State: string
        U_Zip: string
        U_Lat: double
        U_Long: double
        Installed: DateTime
        Complied: DateTime
        Capacity: int
        CertStatus: int
        EquipType: string
        Drive: string
        Volts: string
        Speed: int
        FloorTo: string
        FloorFrom: string
        Landing: string
        Entrances: string
        Ropes: string
        RopeSize: string
    }
 
type public DataRepository() = 
    let connectionString = ConfigurationManager.ConnectionStrings.["azureData"].ConnectionString;
 
    member public this.GetElevators () =
        SqlConnection.GetDataContext(connectionString).ElevatorData201402
        |> Seq.map(fun x -> this.GetElevatorFromElevatorData(x))
 
    member public this.GetElevator (id: int) =
        SqlConnection.GetDataContext(connectionString).ElevatorData201402
        |> Seq.where(fun x -> x.ID = id)
        |> Seq.map(fun x -> this.GetElevatorFromElevatorData(x))
        |> Seq.head
 
    member internal this.GetElevatorFromElevatorData(elevatorData: SqlConnection.ServiceTypes.ElevatorData201402) =
        let elevator = {ID= elevatorData.ID;
            County=elevatorData.County;
            StateId=elevatorData.StateID;
            Type=elevatorData.Type;
            Operation=elevatorData.Operation;
            Owner=elevatorData.Owner;
            O_Address1=elevatorData.O_Address1;
            O_Address2=elevatorData.O_Address2;
            O_City=elevatorData.O_City;
            O_State=elevatorData.O_St;
            O_Zip=elevatorData.O_Zip;
            User=elevatorData.User;
            U_Address1=elevatorData.U_Address1;
            U_Address2=elevatorData.U_Address2;
            U_City=elevatorData.U_City;
            U_State=elevatorData.U_St;
            U_Zip=elevatorData.U_Zip;
            U_Lat=elevatorData.U_lat;
            U_Long=elevatorData.U_long;
            Installed=elevatorData.Installed.Value;
            Complied=elevatorData.Complied.Value;
            Capacity=elevatorData.Capacity.Value;
            CertStatus=elevatorData.CertStatus.Value;
            EquipType=elevatorData.EquipType;
            Drive=elevatorData.Drive;
            Volts=elevatorData.Volts;
            Speed=int elevatorData.Speed;
            FloorTo=elevatorData.FloorTo;
            FloorFrom=elevatorData.FloorFrom;
            Landing=elevatorData.Landing;
            Entrances=elevatorData.Entrances;
            Ropes=elevatorData.Ropes;
            RopeSize=elevatorData.RopeSize
        }
        elevator

I am not happy about writing any of this code. I have 84 lines of code for a single class. I might have well used the code code gen of EF. I could have taken the performance hit and used System.Reflection to map field of the same names (I have done that on other projects) , but that also feels like a hack. In any event, I then added a reference to my F# project in my C# WebAPI project. I did have to add a reference to FSharp.Core in the C# project (which further vexed me), but then I created a couple of GET methods to expose the data:

public class ElevatorController : ApiController
{
    // GET api/Elevator
    public IEnumerable<Elevator> Get()
    {
        DataRepository repository = new DataRepository();
        return repository.GetElevators();
    }
 
    // GET api/Elevator/5
    public Elevator Get(int id)
    {
        DataRepository repository = new DataRepository();
        return repository.GetElevator(id);
    }
 
}

When I viewed the JSON from a handy browser, it looks like, well, junk:

So now I have to get rid of that random characters (x0040 suffix)– yet a 3rd POCO, this one in C#:

public class ElevatorController : ApiController
{
    // GET api/Elevator
    public IEnumerable<CS.Elevator> Get()
    {
        List<CS.Elevator> elevators = new List<CS.Elevator>();
        FS.DataRepository repository = new FS.DataRepository();
        var fsElevators = repository.GetElevators();
        foreach (var fsElevator in fsElevators)
        {
            elevators.Add(GetElevatorFromFSharpElevator(fsElevator));
        }
        return elevators;
    }
 
    // GET api/Elevator/5
    public CS.Elevator Get(int id)
    {
        FS.DataRepository repository = new FS.DataRepository();
        return GetElevatorFromFSharpElevator(repository.GetElevator(id));
    }
 
    internal CS.Elevator GetElevatorFromFSharpElevator(FS.Elevator fsElevator)
    {
        CS.Elevator elevator = new CS.Elevator();
        elevator.ID = fsElevator.ID;
        elevator.County = fsElevator.County;
        elevator.StateId = fsElevator.StateId;
        elevator.Type = fsElevator.Type;
        elevator.Operation = fsElevator.Operation;
        elevator.Owner = fsElevator.Owner;
        elevator.O_Address1 = fsElevator.O_Address1;
        elevator.O_Address2 = fsElevator.O_Address2;
        elevator.O_City = fsElevator.O_City;
        elevator.O_State = fsElevator.O_State;
        elevator.O_Zip = fsElevator.O_Zip;
        elevator.User = fsElevator.User;
        elevator.U_Address1 = fsElevator.U_Address1;
        elevator.U_Address2 = fsElevator.U_Address2;
        elevator.U_City = fsElevator.U_City;
        elevator.U_State = fsElevator.U_State;
        elevator.U_Zip = fsElevator.U_Zip;
        elevator.Installed = fsElevator.Installed;
        elevator.Complied = fsElevator.Complied;
        elevator.Capacity = fsElevator.Capacity;
        elevator.CertStatus = fsElevator.CertStatus;
        elevator.EquipType = fsElevator.EquipType;
        elevator.Drive = fsElevator.Drive;
        elevator.Volts = fsElevator.Volts;
        elevator.Speed = fsElevator.Speed;
        elevator.FloorTo = fsElevator.FloorTo;
        elevator.FloorFrom = fsElevator.FloorFrom;
        elevator.Landing = fsElevator.Landing;
        elevator.Entrances = fsElevator.Entrances;
        elevator.Ropes = fsElevator.Ropes;
        elevator.RopeSize = fsElevator.RopeSize;
        return elevator;
    }
 
}

So that gives me that I want…

As a side note, I learned the hard way that the only way to force the SqlEntityTP to update based on a schema change in the DB is to change the connection string in the .config file.

Finally, when I published the WebAPI project to Azure, I got an exception.

<Error><Message>An error has occurred.</Message><ExceptionMessage>Could not load file or assembly 'FSharp.Core, Version=4.3.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.</ExceptionMessage><ExceptionType>System.IO.FileNotFoundException</ExceptionType><StackTrace> at System.Web.Http.ApiController.<InvokeActionWithExceptionFilters>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__0.MoveNext()</StackTrace

Turns out you need to not only add a reference to the F# project and FSharp.Core, you have to deploy the .dlls to Azure also. Thanks to hocho on SO for that one.

In conclusion, I love the promise of TPs. I want nothing more than to throw away all of the EF code-gen, .tt files, seeding for code-first nonsense, etc… and replace it with a single line TP. I have done this on a local project, but when I did it with an Azure, things were harder than they should be. Since it is easier to throw hand grenades than catch them, I made a list of the things I want to help the open source FSharp.Data project accomplish in the coming months:

1) SqlDatabaseConnection working with Azure Sql Storage

2) MSAccessConnection needed

3) ActiveDirectoryConnection needed

4) Json and WsdlService ability to handle proxies

5) SqlEntityConnection exposing classes publicly

Regardless of what the open-source community does, MSFT will still have to make a better commitment to F# on Azure, IMHO…

Filed under F#, Open Data

Elevator App: Part 0

March 4, 2014 Leave a comment

When I was young, I used to have window races with my siblings in the car. In those days, windows would be manual with a crank like this:

Each kid would get a door and on “go”, we would crank, crank, crank the window all the way up and all the way down. These were great time passers on a long trip – until dad noticed that it was getting alternately windy and calm in the car and put an end to it. With the advent of electric motors for the windows in the car, the races became much closer – we honestly thought that harder you pushed on the button would cause the window move up and down faster. It was good fun until dad caught me pouring Crisco down the back left window of the Buick Skylark. Greasing the wheels, if you will.

Fast forward 15 years when I was just out of school at my 1^st job in San Francisco. One of the things we did when we didn’t have any money (often) on a Friday night was to ride elevators. Find a nice building, walk in like you knew what you were doing, and ride to the top and back down. Hotels were the best – ideally you could get free food in the bar for happy hour too – it was dinner theatre. Sometimes, if there were two elevators in the bank, we would turn it into a competition – pick your elevator wisely. If someone got on your elevator while you were racing, you were out of luck. This added a good deal of tension to the race. You were in the lead on the way back down but then someone stopped you on floor 2 to go down to G. “UGHHHHH. Take the stairs fat-ass!”

(BTW: how great would this be for a plot of a movie? The main characters pick different building to race until one night, they picked the wrong one….)

The biggest problem with elevator races (outside of getting caught and spending a night in jail), was that if the building only had 1 elevator, you really couldn’t race. Even if you had a stopwatch, you really didn’t know if the person made it to the top. Also, the chance of other people who were waiting stopping your elevator was now 100%. Plus, do you really trust your competition with the stop-watch? Maybe your friends, but not mine…

Fast forward 20 more years and I started working with open data. One of the data sets that TRINUG wanted to look at was elevator inspection data. When I went to the website where you can do reporting and there was a column called…. speed. Holy smokes, we can see the speed of the different elevators in town! I then thought about a phone app that could measure the speed of the elevator versus the reported speed. Also, we finally have a solution to the 1-elevator race problem! I then thought about how to tie this into coming up to speed with the latest javascript technologies (I am trying to get my MCSD in Web too). Thus, I am going to create an elevator speed app using Angular and Phonegap. Let’s see how it goes…

Filed under Certifications, Phone App

Restaurant Classifier: Async For Faster Performance?

March 4, 2014 2 Comments

Going back to my restaurant classifier using F# from last week, I decided to speed things up some. Each request to the Yellow Pages API takes 1 second, so with the 5,682 records, I am looking at a little over 1.5 hours to pull down the data when running serial.

I first thought about making my methods async so I changed the API call method to async and used the Http.AsyncRequest method like so (line 10 below):

member this.GetCatagoriesAsync(restaurantName: string, restaurantAddress: string) =
         async{
             if(String.IsNullOrEmpty(restaurantName)) then
                 failwith("restaurantName cannot be null or empty.")
             if(String.IsNullOrEmpty(restaurantAddress)) then
                 failwith("restaurantAddress cannot be null or empty.")
             let cleanedName = this.CleanName(restaurantName)
             let cleanedAddress = this.CleanAddress(restaurantAddress);
             let uri = "http://pubapi.atti.com/search-api/search/devapi/search?term=&quot;+cleanedName+"&searchloc="+cleanedAddress+"&format=json&key=qj5l8pphj5"
             let! response = FSharp.Net.Http.AsyncRequest(uri, headers=["user-agent", "None"])
             let ypResult = ypProvider.Parse(response)
             try
                 return ypResult.SearchResult.SearchListings.SearchListing.[0].Categories
             with
                 | ex -> return String.Empty
         } 

I then made the covering function async also (line 11 below)

member this.IsRestaurantInCatagoryAsync(restaurantName: string, restaurantAddress: string, restaurantCatagory: string) =
    async {
        if(String.IsNullOrEmpty(restaurantName)) then
            failwith("restaurantName cannot be null or empty.")
        if(String.IsNullOrEmpty(restaurantAddress)) then
            failwith("restaurantAddress cannot be null or empty.")
        if(String.IsNullOrEmpty(restaurantCatagory)) then
            failwith("restaurantCatagory cannot be null or empty.")
 
        System.Threading.Thread.Sleep(new System.TimeSpan(0,0,1))
        let! catagories = this.GetCatagoriesAsync(restaurantName, restaurantAddress)
        if(String.IsNullOrEmpty(catagories)) then return false
        else return this.IsCatagoryInCatagories(catagories,restaurantCatagory)
    }

The problem is that invoking the covering function via an anonymous method did not work easily.

After screwing around with the synax a bit, I went over to stack overflow where I found out two things:

There is not an easy way to do it (I was hoping for a Seq.FilterAsyc method)
Thomas Petricek is above my pay-grade.

In any event, I decided to drop the async and just look at parallelism. Turns out that there is a Parallel Seq class called PSeq, it is just not in the FSharp core library yet. I created a PSeq file in my project, moved it to the top and dropped the code in. I then changed the method call to use PSeq to invoke the serial methods:

member public this.GetChineseRestaurants () = 
    let catagoryRepository = new RestaurantCatagoryRepository()
    let catagory = "Chinese"
    this.GetRestaurants()
            |> PSeq.filter(fun (name, address) -> catagoryRepository.IsRestaurantInCatagory(name, address,catagory))
            |> Seq.toList    

When I first invoked it and looked at Fiddler (OT: did anyone notice that Fiddler’s new logo looks alot like a FSharp one? Probably just a coincidence), it was clear that things were running in parallel and that performance would improve. I have two cores on this workstation so my time be cut in half.

With the parallel method in my back pocket, I decided to see the ultimate result of the restaurant classification. I created a quick console app

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("Start");
 
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        RestaurantBuilder builder = new RestaurantBuilder();
        var restaurants = builder.GetChineseRestaurants();
        
        foreach (var restaurant in restaurants)
        {
            Console.WriteLine(restaurant.Item1 + ":" + restaurant.Item2);
        }
        
        stopwatch.Stop();
        Console.WriteLine("Number of Chinese Restaurants: " + restaurants.Count());
        Console.WriteLine(stopwatch.Elapsed.ToString());
        Console.WriteLine("End");
        Console.ReadKey();
    }
}

I then ran the search on YP.com using my 4 core laptop and got the following results:

Compared to my original classifier based on name:

So the results make sense. The YP serial search would take at least 94.7 minutes, the YP parallel search took 41 minutes, and the in-memory name search took 3 seconds. The YP search(s) found restaurants that the name did not (Wang’s Kitchen, Crazy Fire Mongolian Grill, etc…) – 275 to 221, or 24% more restaurants.

I think that the next step is to look at the classifier and see how many restaurants are in both datasets and why the ones that are not in the YP one – where they are (did they even pay to be in the Yellow Pages?). Perhaps there is another YP category that can be considered. Also, it would be interesting to see of the restaurants that are in the name search and in the Yellow Pages that were not classified as Chinese – the false positive rate. Finally, I did see some 500s in Fiddler that had “read time out” so there is room for improvement to account for the transient faults…

Filed under Async, F#, Parallelism

Jamie Dixon's Home

JavaScript Signature Capture Panel

Apriori Algorithm and F# Using Elevator Inspection Data

Elevator App: Part 1 – Data Layer Using F#

Elevator App: Part 0

Restaurant Classifier: Async For Faster Performance?

Categories

Recent Posts

Archives

Blogroll

Meta