Taking a Hiatus From Blogging

I have blogged every Tuesday for the last six years.  Thank you to all of the people who took the time to read my posts and give comments.  I hope you benefitted from the content (if not the rookie formatting and sometimes creative spelling/grammar).  Recently, I signed a deal with  a publisher to write a book that has a a pretty aggressive deadline.  Because of the time commitment that writing a book takes,  I am taking a hiatus from blogging.  If all goes to plan, I will resume blogging in mid-2016.


Using Mocking Frameworks To Help With Unit Testing UI Controls

One of the more common reasons that developers tell us of why they don’t unit test is “All of my application is visual controls with code behind. Refactoring all of that code to a .dll that can be united tested will take more time than it is worth.” While it is true that unit testing is easier if your application lives “in code” as a separate assembly, you can still use unit testing in a UI-heavy code base. By judiciously using a mocking framework, you can speed up the process even more.

Consider this form from a WinForm application written in VB.NET.


The grid view has the following code behind :

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click For counter = 0 To DataGridView1.RowCount - 1 If (DataGridView1.Rows(counter).Cells(11).FormattedValue) Then If (DataGridView1.Rows(counter).Cells(10).Value <> "") Then TextBox1.Text = FormatCurrency(TextBox1.Text - DataGridView1.Rows(counter).Cells(10).Value, 2) End If End If Next End Sub


The business logic is intermingled with the visual controls (TextBox1, DataGridView1, etc…). Is there a way to easily unit test this code? The answer is yes. Step one is to add a unit test project to the solution. Step two is to break the sub method into a function method. Once the methods have an input and an output, you can put a unit test on it. For example:

Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click For counter = 0 To DataGridView1.RowCount - 1 Dim initialValue = TextBox1.Text Dim cell0 = DataGridView1.Rows(counter).Cells(11) Dim cell1 = DataGridView1.Rows(counter).Cells(10) TextBox1.Text = GetCalculatedValue(initialValue, cell0, cell1) Next End Sub Public Function GetCalculatedValue(initalValue As String, cell0 As DataGridViewCell, cell1 As DataGridViewCell) As String Dim returnValue As String = initalValue If (cell0.FormattedValue) Then If (cell1.Value <> "") Then initalValue = FormatCurrency(TextBox1.Text - cell1.Value, 2) End If End If Return initalValue End Function

And we can add a unit test like this:


[TestMethod] public void GetCalculatedValue_ReturnsExpected() { Form1 instance = new Form1(); String baseValue = "$10.00"; DataGridView gridView = new DataGridView(); gridView.Columns.Add("TEST1", "TEST1"); gridView.Columns.Add("TEST2", "TEST2"); gridView.Rows.Add(new DataGridViewRow()); gridView.Rows[0].Cells[0].Value = "$1.00"; gridView.Rows[0].Cells[1].Value = "$2.00"; DataGridViewCell cell0 = gridView.Rows[0].Cells[0]; DataGridViewCell cell1 = gridView.Rows[0].Cells[1]; var actual = instance.GetCalculatedValue(baseValue, cell0, cell1); var expected = "$8.00"; Assert.AreEqual(expected, actual); }


Although this test runs green, it is suboptimal because we have to standup lots of objects (DataGridView, Columns, DataGridRow) just to get to the class we are interested in, in this case DataGridViewCell. Instead of generating all of that superfluous code, there is a better way to set the state of only the class we want – enter Mocking frameworks. Mocking frameworks give us the ability to focus only on the subjects under test (SUT) while ignoring everything else.

But there is a catch. There are 2 types of mocking frameworks: ones that generate their code based on inspecting the types and ones that generate their code based on the compiled IL. The former group includes RhinoMocks and Moq . If you try and add Moq to this unit test project and generate a DataGridViewCell like this:

[TestMethod] public void GetCalculatedValue_ReturnsExpected() { Form1 instance = new Form1(); String baseValue = "$10.00"; var mock0 = new Mock<DataGridViewCell>(); mock0.SetupGet(dataGridViewCell => dataGridViewCell.Value).Returns("$1.00"); var mock1 = new Mock<DataGridViewCell>(); mock1.SetupGet(dataGridViewCell => dataGridViewCell.Value).Returns("$2.00"); var actual = instance.GetCalculatedValue(baseValue, mock0.Object, mock1.Object); var expected = "$8.00"; Assert.AreEqual(expected, actual); }

You will get an exception



Since we don’t control DataGridViewCell’s code, there is no way to change those properties to overidable/nonvirtual. As a general rule, you only use RhinoMocks/Moq on classes that you can control.

The other type of mocking framework (based on IL) can solve this problem. There are 2 commercial frameworks (JustMock, TypeMock) but they cost $399/year (as of this writing). There is a 3rd framework we can use and it is built into Visual Studio 2012+. It is called the Microsoft Fakes Framework. By adding this to your test project,


you can craft your unit test like so:

[TestMethod] public void GetCalculatedValue_ReturnsExpected() { Form1 instance = new Form1(); String baseValue = "$10.00"; using (ShimsContext.Create()) { var cell0 = new ShimDataGridViewCell(new StubDataGridViewCell()); cell0.FormattedValueGet = () => { return "$1.00"; }; var cell1 = new ShimDataGridViewCell(new StubDataGridViewCell()); cell1.ValueGet = () => { return "$2.00"; }; var actual = instance.GetCalculatedValue(baseValue, cell0, cell1); var expected = "$8.00"; Assert.AreEqual(expected, actual); } }

and get green. The downside of using Microsoft Fakes is that you need to re-generate the fakes if the code changes. This makes it ideal for faking external libraries that don’t change much (like ADO.NET) but not assemblies that are under active development.

EjectABed Version 2 – Now Using the Raspberry Pi (Part 2)

With the connection from Twitter to the PI working well, I decided to hook up the bed top the PI.  The Bed is controlled via a server attached to a bellow that forces air to the screw drive.  You can read about how we figured that one out here.
My initial thought was that it would be easy as the Netduino implementation to control the servo was all of 5 lines of code.  The Netduino has built-in PWM ports and the api has a PWM class:
1 uint period = 20000; 2 uint duration = SERVO_NEUTRAL; 3 _servo = new PWM(PWMChannels.PWM_PIN_D5, period, duration, PWM.ScaleFactor.Microseconds, false); 4 _servo.Start(); 5 _servoReady = true;

However, when I went to look for a PWM port, there wasn’t one!  Ugh!  I want over to Stack Overflow to confirm with this question and sure enough, no PWM.  The only example for servo control that the Windows 10 code samples have are using the GPIO to activate a servo forwards and backwards, but that will not work because I need to hold the bellow in a specific place for the air to push correctly.  The Windows IoT team suggested that I use the AdaFruit PWM shield for the control


So I ordered a couple and then my son soldered the pins in

20150904_201350 20150904_201104

I then hooked up the shield to the servo and the PI
and went to look for some PI code to control the pwms.  Another problem, there isn’t any!  I went over to the Raspberry Pi forums and it turns out, they are waiting for MSFT to finish that piece.  Ugh, I decided to take the path of least resistance and I removed that PWM shield and added back in the Netduino


Now I have the ability to control the servo from the PI.  I would have rather cut out the Netduino completely, but the limitations of Win10 on Raspberry Pi won’t allow me to do that.  Oh well, it is still a good entry and it was a lot of fun to work on.

EjectABed Version 2 – Now Using the Raspberry Pi (Part 1)

I recently entered a hackster.io competition that centered around using Windows 10 on the Raspberry Pi.  I entered the ejectabed and it was accepted to the semi-final round.  My thought was to take the existing ejectabed controller from a Netduino and move it to a Raspberry Pi.  While doing that, I could open the ejectabed from my local area network to the internet so anyone could eject Sloan.
My 1st step was hook my Raspberry Pi up to my home network and deploy from Visual Studio to it.  Turns out, it was pretty straightforward.
I took a old Asus Portable Wireless Router and plugged it into my home workstation.  I then configured the router to act as an Access Point so that it would pass though all traffic from the router to which my developer workstation is attached.  I then attached the router to the PI and powered it though the PI’s USB port.  I then plugged the PI’s HDMI out to a spare monitor of mine.


With all of the hardware plugged in, I headed over to Windows On Devices and followed the instructions on how to set up a Raspberry PI.  After installing the correct software on my developer workstation, flashing the SD card with win10, plugging the SD card into the PI, turning the PI on, and then remoting into the PI via powershell, I could see the PI on my local workstation via the Windows IoT Core Watcher and the PI showing its friendly welcome screen via HDMI.



I then headed over to Visual Studio and copy/pasted the equisite “Hello IoT World” Blinky project to the Pi and watched the light go on and off.


With that out of the way, I decided to look at controlling the light via Twitter and Azure.  The thought was to have the PI monitor a message queue on Azure and whenever there was a message, blink on or off (simulating the ejectabed being activated).  To that end, I went into Azure and created a basic storage account.  One of the nice things about Azure is that you get a queue out of the box when you create a storage account:


One of the not so nice things about Azure is that there is no way to control said Queue via their UI.  You have to create, push, and pull from the queue in code.  I went back to visual studio and added in the Azure Storage Nuget package


I then created a method to monitor the queue
1 internal async Task<Boolean> IsMessageOnQueue() 2 { 3 var storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=ejectabed;AccountKey=xxx"; 4 var storageAccount = CloudStorageAccount.Parse(storageConnectionString); 5 var client = storageAccount.CreateCloudQueueClient(); 6 var queue = client.GetQueueReference("sloan"); 7 var queueExists = await queue.ExistsAsync(); 8 if (!queueExists) 9 { 10 GpioStatus.Text = "Queue does not exist or is unreachable."; 11 return false; 12 } 13 var message = await queue.GetMessageAsync(); 14 if (message != null) 15 { 16 await queue.DeleteMessageAsync(message); 17 return true; 18 } 19 GpioStatus.Text = "No message for the EjectABed."; 20 return false; 21 } 22

Then if there is a message, the PI would run the ejection sequence (in this case blink the light)
1 internal void RunEjectionSequence() 2 { 3 bedCommand.Eject(); 4 bedTimer = new DispatcherTimer(); 5 bedTimer.Interval = TimeSpan.FromSeconds(ejectionLength); 6 bedTimer.Tick += LightTimer_Tick; 7 bedTimer.Start(); 8 }


I deployed the code to the PI without a problem.  I then created a Basic console application to push messages to the queue that the PI could drain
1 class Program 2 { 3 static String storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=ejectabed;AccountKey=xxx"; 4 5 static void Main(string[] args) 6 { 7 Console.WriteLine("Start"); 8 Console.WriteLine("Press The 'E' Key To Eject. Press 'Q' to quit..."); 9 10 var keyInfo = ConsoleKey.S; 11 do 12 { 13 keyInfo = Console.ReadKey().Key; 14 if (keyInfo == ConsoleKey.E) 15 { 16 CreateQueue(); 17 WriteToQueue(); 18 //ReadFromQueue(); 19 } 20 21 } while (keyInfo != ConsoleKey.Q); 22 23 Console.WriteLine("End"); 24 Console.ReadKey(); 25 } 26 27 private static void CreateQueue() 28 { 29 var storageAccount = CloudStorageAccount.Parse(storageConnectionString); 30 var client = storageAccount.CreateCloudQueueClient(); 31 var queue = client.GetQueueReference("sloan"); 32 queue.CreateIfNotExists(); 33 Console.WriteLine("Created Queue"); 34 } 35 36 private static void WriteToQueue() 37 { 38 var storageAccount = CloudStorageAccount.Parse(storageConnectionString); 39 var client = storageAccount.CreateCloudQueueClient(); 40 var queue = client.GetQueueReference("sloan"); 41 var message = new CloudQueueMessage("Eject!"); 42 queue.AddMessage(message); 43 Console.WriteLine("Wrote To Queue"); 44 } 45 46 47 private static void ReadFromQueue() 48 { 49 var storageAccount = CloudStorageAccount.Parse(storageConnectionString); 50 var client = storageAccount.CreateCloudQueueClient(); 51 var queue = client.GetQueueReference("sloan"); 52 var queueExists = queue.Exists(); 53 if (!queueExists) 54 Console.WriteLine("Queue does not exist"); 55 var message = queue.GetMessage(); 56 if (message != null) 57 { 58 queue.DeleteMessage(message); 59 Console.WriteLine("Message Found and Deleted"); 60 } 61 else 62 { 63 Console.WriteLine("No messages"); 64 } 65 } 66

I could then Write to the queue and the PI would read and react.  You can see it in action here:


With the queue up and running, I was ready to add in the ability for someone to Tweet to the queue.  I created a cloud service project and pointed to a new project that will monitor Twitter and then push to the queue:



The Twitter project uses the TweetInvi nuget package and is a worker project.  It makes a call to Twitter every 15 seconds and if there is a tweet to “ejectabed” with a person’s name, it will write to the queue (right now, only Sloan’s name is available)
1 type TwitterWorker() = 2 inherit RoleEntryPoint() 3 4 let storageConnectionString = RoleEnvironment.GetConfigurationSettingValue("storageConnectionString") 5 6 let createQueue(queueName) = 7 let storageAccount = CloudStorageAccount.Parse(storageConnectionString) 8 let client = storageAccount.CreateCloudQueueClient() 9 let queue = client.GetQueueReference(queueName); 10 queue.CreateIfNotExists() |> ignore 11 12 let writeToQueue(queueName) = 13 let storageAccount = CloudStorageAccount.Parse(storageConnectionString) 14 let client = storageAccount.CreateCloudQueueClient() 15 let queue = client.GetQueueReference(queueName) 16 let message = new CloudQueueMessage("Eject!") 17 queue.AddMessage(message) |> ignore 18 19 let writeTweetToQueue(queueName) = 20 createQueue(queueName) 21 writeToQueue(queueName) 22 23 let getKeywordFromTweet(tweet: ITweet) = 24 let keyword = "sloan" 25 let hasKeyword = tweet.Text.Contains(keyword) 26 let isFavourited = tweet.FavouriteCount > 0 27 match hasKeyword, isFavourited with 28 | true,false -> Some (keyword,tweet) 29 | _,_ -> None 30 31 32 override this.Run() = 33 while(true) do 34 let consumerKey = RoleEnvironment.GetConfigurationSettingValue("consumerKey") 35 let consumerSecret = RoleEnvironment.GetConfigurationSettingValue("consumerSecret") 36 let accessToken = RoleEnvironment.GetConfigurationSettingValue("accessToken") 37 let accessTokenSecret = RoleEnvironment.GetConfigurationSettingValue("accessTokenSecret") 38 39 let creds = Credentials.TwitterCredentials(consumerKey, consumerSecret, accessToken, accessTokenSecret) 40 Tweetinvi.Auth.SetCredentials(creds) 41 let matchingTweets = Tweetinvi.Search.SearchTweets("@ejectabed") 42 let matchingTweets' = matchingTweets |> Seq.map(fun t -> getKeywordFromTweet(t)) 43 |> Seq.filter(fun t -> t.IsSome) 44 |> Seq.map (fun t -> t.Value) 45 matchingTweets' |> Seq.iter(fun (k,t) -> writeTweetToQueue(k)) 46 matchingTweets' |> Seq.iter(fun (k,t) -> t.Favourite()) 47 48 Thread.Sleep(15000) 49 50 override this.OnStart() = 51 ServicePointManager.DefaultConnectionLimit <- 12 52 base.OnStart()

Deploying to Azure was a snap
And now when I Tweet,
the PI reacts.  Since Twitter does not allow the same Tweet to be sent again, I deleted it every time I wanted to send a new message to the queue.

Facebook Api Using F#

A common requirement for modern user-facing applications is to interface with Facebook.  Unfortunately, Facebook does not make it easy on developers –> in fact it is one of the harder apis that I have seen.  However, there is a covering sdk that you can use, along with some hoop jumping, to get it working.  The problem is one of assumptions.  The .NET sdk assumes that you want to build a Windows Store or Phone app and it is human to facebook connections.  Once you get past those assumptions, you can do pretty well.

The first thing you need to do is set up a Facebook account.



Then register as a developer and create an application


In Visual Studio, Nuget in the facebook sdk


Then, in the REPL add the following code to get the auth token

1 #r "../packages/Facebook.7.0.6/lib/net45/Facebook.dll" 2 #r "../packages/Newtonsoft.Json.7.0.1/lib/net45/Newtonsoft.Json.dll" 3 4 open Facebook 5 open Newtonsoft.Json 6 7 type Credentials = {client_id:string; client_secret:string; grant_type:string;scope:string} 8 let credentials = {client_id="123456"; 9 client_secret="123456"; 10 grant_type="client_credentials"; 11 scope="manage_pages,publish_stream,read_stream,publish_checkins,offline_access"} 12 13 14 let client = FacebookClient() 15 let tokenJson = client.Get("oauth/access_token",credentials) 16 type Token = {access_token:string} 17 let token = JsonConvert.DeserializeObject<Token>(tokenJson.ToString());

Which gives


Once you get the token, you can make a request to user and post to the page

1 let client' = FacebookClient(token.access_token) 2 client'.Get("me") 3 4 let pageId = "me" 5 type FacecbookPost = {title:string; message:string} 6 let post = {title="Test Title"; message = "Test Message"} 7 client'.Post(pageId + "/feed", post) 8

I was getting this message though


So then the fun part.  Apparently, you need to submit your application to the facebook team to be approved to be used.  So now I have to submit icons and a description on how this application will be used before I can make a POST.  <sigh>

Thanks to Gene Belitski for his help on my question on Stack Overflow

Wake County Voter Analysis Using FSharp, AzureML, and R

One of the real strengths of FSharp its ability to plow through and transform data in a very intuitive way,  I was recently looking at Wake Country Voter Data found here to do some basic voter analysis.  My first thought was to download the data into R Studio.  Easy?  Not really.  The data is available as a ginormous Excel spreadsheet of database of about 154 MB in size.  I wanted to slim the dataset down and make it a .csv for easy import into R but using Excel to export the data as a .csv kept screwing up the formatting and importing it directly into R Studio from Excel resulting in out of memory crashes.  Also, the results of the different election dates were not consistent –> sometimes null, sometimes not.   I managed to get the data into R Studio without a crash and wrote a function of either voted “1” or not “0” for each election

1 #V = voted in-person on Election Day 2 #A = voted absentee by mail or early voting (through May 2006) 3 #M = voted absentee by mail (November 2006 - present) 4 5 #O = voted One-Stop early voting (November 2006 - present) 6 #T = voted at a transfer precinct on Election Day 7 #P = voted a provisional ballot 8 #L = Legacy data (prior to 2006) 9 #D = Did not show 10 11 votedIndicated <- function(votedCode) { 12 switch(votedCode, 13 "V" = 1, 14 "A" = 1, 15 "M" = 1, 16 "O" = 1, 17 "T" = 1, 18 "P" = 1, 19 "L" = 1, 20 "D" = 0) 21 } 22

However, every time I tried to run it, the IDE would crash with an out of memory issue. 

 Stepping back, I decided to transform the data in Visual Studio using FSharp. I created a sample from the ginormous excel spreadsheet and then imported the data using a type provider.  No memory crashes!

1 #r "../packages/ExcelProvider.0.1.2/lib/net40/ExcelProvider.dll" 2 open FSharp.ExcelProvider 3 4 [<Literal>] 5 let samplePath = "../../Data/vrdb-Sample.xlsx" 6 7 open System.IO 8 let baseDirectory = __SOURCE_DIRECTORY__ 9 let baseDirectory' = Directory.GetParent(baseDirectory) 10 let baseDirectory'' = Directory.GetParent(baseDirectory'.FullName) 11 let inputFilePath = @"Data\vrdb.xlsx" 12 let fullInputPath = Path.Combine(baseDirectory''.FullName, inputFilePath) 13 14 type WakeCountyVoterContext = ExcelFile<samplePath> 15 let context = new WakeCountyVoterContext(fullInputPath) 16 let row = context.Data |> Seq.head

I then applied a similar function for voted or not and then exported the data as a .csv

1 let voted (voteCode:obj) = 2 match voteCode = null with 3 | true -> "0" 4 | false -> "1" 5 6 open System 7 let header = "Id,Race,Party,Gender,Age,20080506,20080624,20081104,20091006,20091103,20100504,20100622,20101102,20111011,20111108,20120508,20120717,20121106,20130312,20131008,20131105,20140506,20140715,20141104" 8 9 let createOutputRow (row:WakeCountyVoterContext.Row) = 10 String.Format("{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11},{12},{13},{14},{15},{16},{17},{18},{19},{20},{21},{22},{23}", 11 row.voter_reg_num, 12 row.race_lbl, 13 row.party_lbl, 14 row.gender_lbl, 15 row.eoy_age, 16 voted(row.``05/06/2008``), 17 voted(row.``06/24/2008``), 18 voted(row.``11/04/2008``), 19 voted(row.``10/06/2009``), 20 voted(row.``11/03/2009``), 21 voted(row.``05/04/2010``), 22 voted(row.``06/22/2010``), 23 voted(row.``11/02/2010``), 24 voted(row.``10/11/2011``), 25 voted(row.``11/08/2011``), 26 voted(row.``05/08/2012``), 27 voted(row.``07/17/2012``), 28 voted(row.``11/06/2012``), 29 voted(row.``03/12/2013``), 30 voted(row.``10/08/2013``), 31 voted(row.``11/05/2013``), 32 voted(row.``05/06/2014``), 33 voted(row.``07/15/2014``), 34 voted(row.``11/04/2014``) 35 ) 36 37 let outputFilePath = @"Data\vrdb.csv" 38 39 let data = context.Data |> Seq.map(fun row -> createOutputRow(row)) 40 let fullOutputPath = Path.Combine(baseDirectory''.FullName, outputFilePath) 41 42 let file = new StreamWriter(fullOutputPath,true) 43 44 file.WriteLine(header) 45 context.Data |> Seq.map(fun row -> createOutputRow(row)) 46 |> Seq.iter(fun r -> file.WriteLine(r)) 47

The really great thing is that I could write and then dispose of each line so I could do it without any crashes.  Once the data was into a a .csv (10% the size of Excel), I could then import it into R Studio without a problem.  It is a common lesson but really shows that using the right tool for the job saves tons of headaches.

I knew from a previous analysis of voter data that the #1 determinate of a person from wake county voting in a off-cycle election was their age:




So then in R, I created a decision tree for just age to see what the split was:

1 library(rpart) 2 temp <- rpart(all.voters$X20131008 ~ all.voters$Age) 3 plot(temp) 4 text(temp)

Thanks to Placidia for answering my question on stats.stackoverflow


So basically politicians should be targeting people 50 years or older or perhaps emphasizing issues that appeal to the over 50 crowd.





Kaggle and R

Following up on last week’s post on doing a Kaggle competition, I then decided to see if I could explore the data more in R on my local desktop.  The competition is about analyzing a large group of house claims to give them a risk score.

I started the R studio to take a look at the initial data:

1 train <- read.csv("../Data/train.csv") 2 head(train) 3 summary(train) 4 5 plot(train$Hazard)


A couple of things popped out.  All of the X variables look to be categorical.  Even the result “Hazard” is an integer with most of the values falling between 1 and 9.

With that in mind, I decided to split the dataset into two sections: the majority and the minority.

1 train.low <- subset(train, Hazard < 9) 2 train.high <- subset(train, Hazard >= 9) 3 4 plot(train.low$Hazard) 5 plot(train.high$Hazard)

With the under as:


And the over 9 is like this


But I want to look at the Hazard score from a distribution point of view:

1 hazard.frame <- as.data.frame(table(train$Hazard)) 2 colnames(hazard.frame) <- c("hazard","freq") 3 hist(hazard.frame$freq) 4 plot(x=hazard.frame$hazard, y=hazard.frame$freq) 5 plot(x=hazard.frame$hazard, log(y=hazard.frame$freq)) 6

The hist shows the left skew




and the log plot really shows the distribution


So there is clearly a diminishing return going on.   As of this writing, the leader is at 40%, which is about 20,400 of the 51,000 entries.   So if you could identify all of the ones correctly, you should get 37% of the way there.  To test it out, I submitted to Kaggle only ones:


LOL, so they must take away for incorrect answers as it is same as “all 0” benchmark.  So going back, I know that if I can predict the ones correctly and make a reasonable guess at the rest, I might be OK.   I went back and tuned my model some to get me out of the bottom 25% and then let it be.  I assume that there is something obvious/industry standard that I am missing because there are so many people between my position and the top 25%.

Kaggle and AzureML

If you are not familiar with Kaggle, it is probably the de-facto standard for data science competitions.  The competitions can be hosted by a private company with cash prizes or it can be a general competition with bragging rights on the line.  The Titanic Kaggle competition is one of the more popular “hello world” data science projects that is a must-try for aspiring data scientists.

Recently, Kaggle hosted a competition sponsored by Liberty Mutual to help predict the insurance risk of houses.  I decided to see how well AzureML could stack up against the best data scientists that Kaggle could offer.

My first step was to get the mechanics down (I am a big believer in getting dev ops done first).  I imported the train and test datasets from Kaggle into AzureML.  I visualized the data and was struck that all of the vectors were categorical, even the Y variable (“Hazard”) –> it is an int with a range between 1 and 70.


I created a quick categorical model and ran it.  Note I did a 60/40 train/test split of the data


Once I had a trained model, I hit the “Set Up Web Service” button.


I then went into that “web service” and changed the input from a web service input to the test dataset that Kaggle provided.  I then outputted the data to azure blob storage.  I also added a transform to only export the data that Kaggle wants to evaluate the results: ID and Hazard:


Once the data was in blob storage, I could download it to my desktop and then upload it to Kaggle to get an evaluation and a ranking.


With the mechanics out of the way, I decided to try a series of out of box models to see what gives the best result.  Since the result was categorical, I stuck to the classification models and this is what I found:



The OOB Two Class Bayes Point Machine is good for 1,278 place, out of about 1,200 competitors.

Stepping back, the hazard is definitely left-skewed so perhaps I need two models.  If I can predict if the hazard is between low and high group, I should be able to be right with most of the predictions and then let the fewer outlier predictions use a different model.  To test that hypotheses, I went back to AzureML and added a filter module for Hazard < 9


The problem is that the AUC dropped 3%.  So it looks like the outliers are not really skewing the analysis.  The next thought is that perhaps AzureML can help me identify the x variables that have the greatest predictive power.  I dragged in a Filter Based Feature Selection module and ran that with the model

image image

The results are kinda interesting.  There is a significant drop-off after these top 9 columns


So I recreated the model with only these top 9 X variables


And the AUC moved to .60, so I am not doing better.

I then thought of treating the Hazard score not as a factor but as a continuous variable.   I rejiggered the experiment to use a boosted decision tree regression


So then sending that over to Kaggle, I moved up.  I then rounded the decimal but that did this:


So Kaggle rounds to an int anyway.  Interestingly, I am at 32% and the leader is at 39%. 

I then used all of the OOB models for regression in AzureML and got the following results:


Submitting the Poisson Regression, I got this:


I then realized that I could mike my model <slightly> more accurate by not including the 60/40 split when doing the predictive effort.  Rather, I would put all 100% of the training data to the model:


Which moved me up another 10 spots…


So that is a good way to stop with the out of the box modeling in AzureML. 

There are a couple of notes here

1) Kaggle knows how to run a competition.  I love how easy it is to set up a team, submit an entry, and get immediate feedback.

2) AzureML OOB is a good place to start and explore different ideas.  However, it is obvious that stacked against more traditional teams, it does not do well

3) Speaking of which.  You are allowed to submit 5 entries a day and the competition lasts 90 days or so.  With 450 entries, I am imagine a scenario where a person can spend their time gaming their submissions.  There are 51,000 entries so and the leading entry (as of this writing) is around 39% so there are 20,000 correct answers.  That is about 200 correct answers a day or 40 each submission.

F#, REPL Driven Development, and Scrum

Last week,  I did a book review of sorts on Scrum: The Art of Doing Twice The Work In Half The Time.  When I was reading the text, an interesting thought hit me several times.  As a pragmatic practitioner of Test Driven Development (TDD), which often goes hand in hand with Agile and Scrum ideas, I often wonder if I am doing something the best way.  I remember distinctly Robert C Martin talking about being your new CTO with the goal of all of the code working correctly all of the time and that he didn’t care if you used TDD, but he doesn’t know a better way.


I was thinking how lately I have been practicing REPL-driven development using F#.  If you are not familiar, REPL stands for “READ-EVALUATE-PRINT-LOOP” and has been the primary methodology of functional programmers and data scientists for years.  In RDD, I quickly prove out my idea tin the REPL to see if I make sense.  it is strictly happy path programming.  Once I think I have a good idea or a solution to whatever problem I am working on, I lift the code into a compiled assembly.  The data elements I used in the REPL then get ported over into my unit tests.  I typically use C# unit tests so that I can confirm that my FSharp code will interop nicely with any VB.NET/C# projects in the solution.  I then layer on code coverage to make sure I have covered all happy paths and then throw some fail cases at the code.

Thinking of this methodology, I think it is closer to Scrum than traditional TDD for a couple of reasons:

Fail Fast and Fix early.  You cannot prove out ideas any faster than in the REPL except for maybe a dry board.  Curly-braces and OO-centered languages like Java and C# are great for certain jobs, but the require much more ceremony and code for code’s sake.  As Sutherland points out, context-switching is a killer.  The less you have to worry about code (classes, moqs, etc..) the faster and better you will be at solving your problem.

Working Too Hard Makes More Work. One of the most startling things about using F# on real projects is that there is just not very much code.  I finished and looked around to see what I missed.  My unit tests were passing, code coverage was high, and there just wasn’t much code.  It was quite unsettling.  I now realize that lots of C#/Java code needs to be generated for real programming projects (exception handling, class hierarchies, design patterns, etc…).  But as the Dartmouth Basic Manual once said “typing is not a substitute for thinking”, all of this code begets more code.  It is a cycle of work that creates more work that F# does not have.

Duplication/Boilerplates/Templates. Complete and Total Waste So this one is pretty self-explanatory.  Many people  (myself included) think that Visual Studio needs better F# templates.  However, once you get good at writing F# code, you really don’t need them.  Maybe it is good that there aren’t many more?  In any event, you don’t use templates and boiler plates in the REPL…


The Wright Brothers and Scrum

I recently read two books that, on the cover have nothing to do with each other, but actually have very much similar lessons.  The first is David MacCullugh’s The Wright Brothers and Jeff Sutherland’s Scrum: The art of doing twice the work in half the time.


Although the human-interest side of the story was kind of interesting to me, what really stood out was how the Wright brothers got their machine in the air.  If you are not familiar with the details of how they constructed the Wright Flyer, there are some pretty interesting points:

1) The Wright brothers were a small agile team that comprised of no fewer than two (the brothers themselves) and no more than seven.  I did not realize how important Charlie Taylor and William Tate was to different pieces of the project.

2) The Wright brothers spent the 1st part of their journey doing research, combing all of the scientific literature, and engaging with the current though leaders of the day in a very much open-source style where they would freely share knowledge but retain the final product for themselves.  This was in direct contrast to other teams that operated in silos and secrecy.

3) The Wright Brothers believed in doing one thing at a time well.  They realized that there were two major problems with heavier than air flight –> thrust  and balance.  They separated these two concerns and tackled the balance problem first.  Once they figured out how to make a glider stable in flight, they then tacked how to add a motor to it.

4) The Wright Brothers made hundreds of small incremental changes with each change able to stand for itself.  For example, they went out to Kitty Hawk in the summer of 1900, 1901, and 1902 with gliders before going out the forth time in 1903 with their airplane.  Each time, the designs got bigger and closer to the final goal.

5) The Wright brothers were willing to challenge conventional and commonly accepted “facts” when their evidence did not support it.  The Wright brothers relied heavily on the calculations of Lilienthal and Chanute to measure lift and drag.  After several failed experiments, the Wright brothers ditched those and went with their own, painstakingly researched, measurement tables.

3) The Wright Brothers were in direct competition with Samuel Langley’s airplane.  In contrast to the Wright Brother’s agile approach, Langley had a large team that operated in absolutely secrecy while taking massive (at the time) amounts of public funds.  When Langley finally rolled out his “final product” it failed miserably every time.

So what does the Wright Brother’s methodology have to do with Scrum?  Everything. If you look art the core tenants of Sutherland’s book,  most of them can be found in how the Wrights conquered the air.  I went though the end of each chapter of Scrum and pulled out some of the take-away points that direct match to how Orville and Wilber did things:


The Wright brothers were doing scrum a full hundred years before it became a thing. As amazing what they created, how they did it is really remarkable.  Interestingly for me, the “It’s the journey, not the destination” just rang home.  As I write this blog, it is Friday night and I am on my front porch.  A neighbor stopped by to say “hello”.  When  told her I was working on a blog post related to my profession, she said “Oh, I am sorry you are not doing anything fun tonight.”  And I said “But this is fun.”  and internally I was thinking “I wonder why so many people think work is not fun?  Why are some many people socialized that way?  I hope my kids don’t wind up like that.”