December | 2013 | Jamie Dixon's Home

Correlation Between Recruit Rankings and Final Standings in Big Ten Football

December 31, 2013 1 Comment

Following up on my last post about screen scraping college football in F#, I took the next step and analyzed the data that I scraped. I am a big believer in Domain Specific Language so ‘Rankings’ means the ranking assigned by Rivals about how well a school recruits players. ‘Standings’ means the final position in the Big Ten after the games have been played. Ranking is for recruiting and standings is for actually playing the games.

Going back to the code, the 1st thing I did was to separate the Standings call from the search for a given school – so that the XmlDocument is loaded once and then searched several times versus loading it for each search. This improved performance dramatically:

static member getAnnualConferenceStandings(year:int)=
    let url = "http://espn.go.com/college-football/conferences/standings/_/id/5/year/&quot;+year.ToString()+"/big-ten-conference";         
    let request = WebRequest.Create(Uri(url)) 
    use response = request.GetResponse() 
    use stream = response.GetResponseStream() 
    use reader = new IO.StreamReader(stream) 
    let htmlString = reader.ReadToEnd()
    let divMarkerStartPosition = htmlString.IndexOf("my-teams-table");
    let tableStartPosition = htmlString.IndexOf("<table",divMarkerStartPosition);
    let tableEndPosition = htmlString.IndexOf("</table",tableStartPosition);
    let data = htmlString.Substring(tableStartPosition, tableEndPosition- tableStartPosition+8)
    let xmlDocument = new XmlDocument();
    xmlDocument.LoadXml(data);
    xmlDocument        
 
static member getSchoolStanding(xmlDocument: XmlDocument,school) =
    let keyNode = xmlDocument.GetElementsByTagName("td")
                        |> Seq.cast<XmlNode>
                        |> Seq.find (fun node -> node.InnerText = school)
    let valueNode = keyNode.NextSibling
    let returnValue = (keyNode.InnerText, valueNode.InnerText)
    returnValue
 
static member getConferenceStandings(year:int) =
    let xmlDocument = RankingProvider.getAnnualConferenceStandings(year)
    Seq.map(fun school -> RankingProvider.getSchoolStanding(xmlDocument,school)) RankingProvider.schools
        |> Seq.sortBy snd
        |> Seq.toList
        |> List.rev
        |> Seq.mapi(fun index (school,ranking) -> school, index+1)
        |> Seq.sortBy fst
        |> Seq.toList

Thanks for Valera Kolupaev for showing me how to use mapi to create a tuple from the list of schools and what rank they were in the list in getConferenceStandings().

I then went to the rankings call and added a way to parse down only the schools I am interested in. That way I can compare individual schools, groups of schools, or the entire conference:

static member getConferenceRankings(year) =
    RankingProvider.schools
            |> Seq.map(fun schoolName -> RankingProvider.getSchoolInSequence(year, schoolName))
            |> Seq.toList
    
 
static member getSchoolInSequence(year, schoolName) =
    RankingProvider.getRecrutRankings(year)
                    |> Seq.find(fun (school,rank) -> school = schoolName)

After these two refactorings, my unit tests still ran green so I was ready to do the analysis.

I went out to my project of a couple of weeks ago for correlation and copied in the module. The Correlation function takes in two lists of doubles. The first list would be a school’s ranking and the second would be the standings:

static member getCorrelationBetweenRankingsAndStandings(year, rankings, standings ) =
    let ranks = Seq.map(fun (school,rank) -> rank) rankings
    let stands = Seq.map(fun (school,standing) -> standing) standings
    Calculations.Correlation(ranks,stands)
 
static member getCorrelation(year:int) =
    let rankings = RankingProvider.getConferenceRankings year
                    |> Seq.map(fun (school,rank) -> school,Convert.ToDouble(rank))
    let standings = RankingProvider.getConferenceStandings(year+RankingProvider.yearDifferenceBetwenRankingsAndStandings)
                    |> Seq.map(fun (school, standing) -> school, Convert.ToDouble(standing))
    let correlation = RankingProvider.getCorrelationBetweenRankingsAndStandings(year,rankings, standings)
    (year, correlation)

A couple of things to note:

1) This function assumes that both the rankings and the standings are the same length and are in order by school name. A production application would check this as part of standard argument validation.

2) I used Convert.ToDouble() to change the Int32 of the ranking to Double of the correlation function. Having these .NET assemblies available at key points in the application really moved things along.

In any event, all that was left was to list the Big Ten schools to analyze, the number of years to analyze, and the year difference between the recruit rankings and the standings from the games they played in.

As a first step, I did all original big ten schools with 7 years of recruiting and a 1,2,3,4 years difference (2002 ranking compared to 2003, 2004,2005,2006 standings ,etc…):

The average is .3303/.2650/.5138/.6065

And so yeah – there is a really strong correlation between a recruit ranking and the outcome on the field. Also, the most impact the class has seems to be senior year – which makes sense. I don’t have a hypothesis on why it drops sophomore year – perhaps the ‘impact freshmen’ leave after 1 year?

Also of interest, the correlation does not seem to follow a normal distribution. If you only look at the schools that have an emphasis on academics, the correlation drops significantly – to a negative correlation!

The average is .1485/-.1446/-.2817/-.0381

So another great reason to create the new big ten – sometimes there is a really good recruit class does not do well on the field and other times a poorly-ranked recruiting class does well on the field. This kind of unpredictability is both exciting and probably much more likely to bring in the casual fans.

Based on this analysis, here is what is going to happen in the Big Ten next year:

Michigan State and Ohio State will be the leaders
Michigan and Penn State are in the best position beat Michigan State and Ohio State

But you didn’t need a statistical analysis to tell you that. The one key surprise that this analysis tells you is that

Nebraska will have a significant improvement in the standings in 2014
Indiana will have a significant improvement in the standings in 2015 and 2016

As a final note, I got this after doing a bunch of requests to Yahoo:

So I wonder if I hit the page too many times and my IP was flagged a as a bot? I waited a day for the server to reset to finish my analysis. Perhaps this is a case where I should get the data when the getting is good and take their pages and bring them locally?

Filed under Analytics, F#

Screen Scraping College Football Statistics

December 24, 2013 3 Comments

As a follow-up to my post of the correlation of Academic Ranking and Football Rankings in the Big Ten, I thought I would look that the relationship between two different kinds of Football Rankings: the recruiting ranking assigned by Rivals and the actual results on the field. To that end, I went to collect the data programmically because I am doing a time-series analysis and I didn’t want to do data-entry.

My first stop was to find a free service that exposes this data on the web. No luck – either the data was a service that cost money or the data was presented as a web page. Since I have never screen-scraped using F# (and I am cheap), I chose option #2.

My first data point was the recruiting ranking found here. When I inspected the source of the page, I caught a break – the data is actually stored as Json on the page.

So firing up Visual Studio, I created a solution with 1 F# project and 2 C# projects:

I then wrote a unit test to check that something is being returned:

[TestMethod]
public void getRecrutRankings_RetunsExpected()
{
    var rankings = RankingProvider.getRecrutRankings("2012");
    Assert.AreNotEqual(0, rankings.Length);
}

I then went over the F#. I created the RankingProvider type and then add a function that pulls in the rankings for a given year:

static member getRecrutRankings(year) =
    let url = "http://sports.yahoo.com/footballrecruiting/football/recruiting/teamrank/&quot;+year+"/BIG10/all";
    let request = WebRequest.Create(Uri(url)) 
    use response = request.GetResponse() 
    use stream = response.GetResponseStream() 
    use reader = new IO.StreamReader(stream) 
    let htmlString = reader.ReadToEnd()
    let startPosition = htmlString.IndexOf("var rankingsTableData =")
    let headerLength = 23
    let endPosition = htmlString.IndexOf(";",startPosition)
    let data = htmlString.Substring(startPosition+headerLength,endPosition-startPosition-headerLength).Trim()
    let results = JsonConvert.DeserializeObject(data)
    let castedResults = results :?> Newtonsoft.Json.Linq.JArray
                                            |> Seq.map(fun x -> (x.Value("name").ToString(), Int32.Parse(x.Value("rank").ToString())))
                                            |> Seq.toList

A couple of things to note.

Lines 2 through 12 are language-agnostic. You would write the exact same code in C#/VB.NET with a slightly different syntax.
Line 13 is where things get interesting. I used the :?> operator to cast the Json to a typed structure. :?> wins as the weirdest symbol I have ever used in computer programming. I guess I haven’t been programming long enough?
Lines 14 and 15 is where you can see why F# is better than C#. I created a function that takes the Json and pushes it into a tuple. With no iteration, the code is both easier to read and less likely to have bugs

Hoping to press my luck, I went over the the other page (the one that holds the standings from the actual games) to see if they used Json. No dice – so back to mid-2000s screen scraping. I created a function that loads the table into an XML document and then searches for a given school.

static member getConferenceStanding(year, school) =
    let url = "http://espn.go.com/college-football/conferences/standings/_/id/5/year/&quot;+year+"/big-ten-conference";         
    let request = WebRequest.Create(Uri(url)) 
    use response = request.GetResponse() 
    use stream = response.GetResponseStream() 
    use reader = new IO.StreamReader(stream) 
    let htmlString = reader.ReadToEnd()
    let divMarkerStartPosition = htmlString.IndexOf("my-teams-table");
    let tableStartPosition = htmlString.IndexOf("<table",divMarkerStartPosition);
    let tableEndPosition = htmlString.IndexOf("</table",tableStartPosition);
    let data = htmlString.Substring(tableStartPosition, tableEndPosition- tableStartPosition+8)
    let xmlDocument = new XmlDocument();
    xmlDocument.LoadXml(data);
    let keyNode = xmlDocument.GetElementsByTagName("td")
                    |> Seq.cast<XmlNode>
                    |> Seq.find (fun node -> node.InnerText = school)
    let valueNode = keyNode.NextSibling
    (keyNode.InnerText, valueNode.InnerText)

A couple of things to note:

Lines 2-7 are identical to the prior function so they should be combined into a single function that can be independently testable.
Lines 8-13 are language-agnostic. You would write the exact same code in C#/VB.NET with a slightly different syntax.
Lines 14-18 is where F# really shines. Like the prior function, by using functional programming techniques in F#, I saved myself time, avoid bugs, and made the code much more intuitive.
I am making a web call for each function call– this should be optimized so the call is made once and the xmlDocument is passed in. This would also make the function much more testable (even without a mocking framework)

Next up, I needed to call this function for each of the Big Ten Schools:

static member getConferenceStandings(year)=
    let schools =[|"Nebraska";"Michigan";"Northwestern";"Michigan State";"Iowa";
        "Minnesota";"Ohio State";"Penn State";"Wisconsin"; "Purdue"; "Indiana"; "Illinois"|]
    Seq.map(fun school -> RankingProvider.getConferenceStanding(year,school)) schools
        |> Seq.sortBy snd
        |> Seq.toList
        |> List.rev

This is purely F# and is a pure joy to write (and look the least amount of time). Note that the sort is on the second element of the tuple and that the list is reversed because the second element is the wins-losses so F# is sorting ascending on the number of wins. Since Seq does not have a rev function, I turned it into a List, which does have the rev function

Some might ask “Why didn’t you use type-providers?” My answer is “I tried, but I couldn’t get them to work.” For example, here is the code that I used for the type provider when parsing the xmlDocument:

xmlDocument.LoadXml(data);
let document = XmlProvider<xmlDocument>

The problem is that the type provider expects a uri (and I can’t find an overload to pass in the document). It looks like type providers are more designed for providers that are ready to, well, provide (Web Services, Databases, etc..) versus jerry-rigged data (like screen scraping).

In any event, with these two functions, ready, I went to the UI project and decided to see how the teams did in 2012 on the field compared to how the teams did in recruiting 2 years before:

static void Main(string[] args)
{
    Console.WriteLine("Start");
 
    Console.WriteLine("——-Rankings");
    var rankings = RankingProvider.getRecrutRankings("2010");
    foreach (var school in rankings)
    {
        Console.WriteLine(school.Item1 + ":" + school.Item2);
    }
 
    Console.WriteLine("——-Standings");
    var standings = RankingProvider.getConferenceStandings("2012");
    foreach (var school in standings)
    {
        Console.WriteLine(school.Item1 + ":" + school.Item2);
    }
 
    Console.WriteLine("End");
    Console.ReadKey();
}

And the results:

I have no idea if a 2-year lag between recruiting and rankings is the right number – perhaps an analysis of the correct lag will be done. After all, between red-shirt freshmen, transfer rules, and attrition, there are plenty of variables the determine when a recruiting class has the biggest impact. Also, the standings are a blend of recruiting classes and since I am not evaluating individual players, I can’t go to that level of detail. 2 years out seems reasonable, but as Bluto famiously once said

static member getBlutoQuote() =
    "Seven years of college down the drain.";

the average might be different. In any event, I now have the data I want so the next step is to analyze it to see if there is any correlation. At first glance, there might be something – the top 4 schools for recruiting all finished in the top 4 in the standings – but the bottom 4 is more muddled with only Illinois doing poorly in both recruiting and the standings.

More to come…

Filed under F#, Statistics

F# and Monopoly Simulation Redux

December 17, 2013 2 Comments

Now that I am 4 months into my F# adventure, I thought I would revisit the monopoly simulation that I wrote in August. There are some pretty big differences

1) I am not using the ‘if…then’ construct at all –> rather I am using pattern matching. For example, consider the original communityChest function:

let communityChest x y =
    if y = 1 then
        0
    else if y = 2 then
        10
     else
        x

and the most recent one:

let communityChest (tile, randomNumber) =
    match randomNumber with
        | 1 -> 0
        | 2 -> 10
        | _ -> tile

“Big deal”, you are saying to yourself (or at least I did). But the power of pattern matching is put on display with the revised chance. The code is much more readable and understandable.

Original:

let chance x y =
    if y = 1 then
        0
    else if y = 2 then
        10
    else if y = 3 then
        11
    else if y = 4 then
        39
    else if y = 5 then
        x – 3
    else if y = 6 then
        5
    else if y = 7 then
        24
    else if y = 8 then
        if x < 5 then
            5
        else if x < 15 then
            15
        else if x < 25 then
            25
        else if x < 35 then
            35
        else
            5
    else if y = 9 then
        if x < 12 then
            12
        else if x < 28 then
            28
        else
            12
    else
        x

Revised:

let goToNearestRailroad tile =
    match tile with
        | 36|2 -> 5
        | 7 -> 15
        | 17|22 -> 25
        | 33 -> 35
        | _ -> failwith "not on chance"
 
let goToNearestUtility tile =
    match tile with
        | 36|2|7 -> 12
        | 12|22|33-> 28
        | _ -> failwith "not on chance"
 
let chance (tile, randomNumber) =
    match randomNumber with
        | 1 -> 0
        | 2 -> 10
        | 3 -> 11
        | 4 -> 39
        | 5 -> tile – 3
        | 6 -> 5
        | 7 -> 24
        | 8 -> goToNearestRailroad tile
        | 9 -> goToNearestUtility tile
        | _ -> tile

As a side note, I ditched the x and y values because they are unreadable. When I went back to the code after 3 months, I spent way too long trying to figure out what the heck ‘x’ was. I know that scientific code uses cryptic values, but clean code does not. I changed them and the code became much better.

I then took a look at the move() function. The original:

let move x y z =
    if x + y > 39 then
        x + y – 40
    else if x + y = 30 then
        10
    else if x + y = 2 then
        communityChest 2 z
    else if x + y = 7 then
        chance 7 z
    else if x + y = 17 then
        communityChest 17 z
    else if x + y = 22 then
        chance 22 z
    else if x + y = 33 then
        communityChest 33 z
    else if x + y = 36 then
        chance 36 z
    else
        x + y  

and the revised:

let getBoardMove (currentTile, dieTotal) =
    let initialTile = currentTile + dieTotal
      matchinitialTile with
          | 2 ->communityChest (2, random.Next())
          | 7 ->chance (7, random.Next())
          | 17 ->communityChest (17, random.Next())
          | 22 ->chance (22, random.Next())
        | 30 -> 10
          | 33 ->communityChest (2, random.Next())
          | 36 ->chance (7, random.Next())
        | 40|41|42|43|44|45|46|47|48|49|50|51 -> initialTile – 40
        | _ -> initialTile   

I am not happy with line 11 above – but apparently there is not a way in F# to do this ‘>40’ or even ‘[40 .. 51]’ in the left hand side of the pattern match.

So far, the biggest changes were to make the values more understandable and to get rid of the if…then statements and replace them with pattern matching. Both these techniques make the code more readable and understandable. The next big change came with the actual game play itself. The original version:

let simulation =
    let mutable startingTile = 0
    let mutable endingTile = 0
    let mutable doublesCount = 0
    let mutable inJail = false
    let mutable jailRolls = 0
    for diceRoll in 1 .. 10000 do
        let dieOneValue = random.Next(1,7)
        let dieTwoValue = random.Next(1,7)
        let cardDraw = random.Next(1,17)
        let numberOfMoves = dieOneValue + dieTwoValue
        
        if dieOneValue = dieTwoValue then
            doublesCount <- doublesCount + 1
        else
            doublesCount <- 0
 
        if inJail = true then
            if doublesCount > 1 then
                inJail <- false
                jailRolls <- 0
                endingTile <- move 10 numberOfMoves cardDraw
            else
                if jailRolls = 3 then
                    inJail <- false
                    jailRolls <- 0
                    endingTile <- move 10 numberOfMoves cardDraw
                else
                    inJail <- true
                    jailRolls <- jailRolls + 1
        else
            if doublesCount = 3 then
                inJail <- true
                endingTile <- 10
            else
                endingTile <- move startingTile numberOfMoves cardDraw
         
        printfn "die1: %A + die2: %A = %A FROM %A TO %A"
            dieOneValue dieTwoValue numberOfMoves startingTile endingTile
 
        startingTile <- endingTile
        tiles.[endingTile] <- tiles.[endingTile] + 1

You will notice that the word ‘’mutable” shows up six times. Using the word mutable in F# is a code smell so I refactored it out like so:

let rec rollDice (currentTile, rollCount, doublesCount, inJail, jailRollCount)=
    let dieOneValue = random.Next(1,7)
    let dieTwoValue = random.Next(1,7)
    let dieTotal = dieOneValue + dieTwoValue
    let newRollCount = rollCount + 1
    
    let newDoublesCount = 
        if dieOneValue = dieTwoValue then doublesCount + 1
        else 0
 
    let newTile = getTileMove(currentTile,dieTotal,newDoublesCount,inJail,jailRollCount)
    
    let newInJail = 
        if newTile = 10 then true
        else false
 
    let newJailRollCount =
        if newInJail = inJail then jailRollCount + 1
        else 0
 
    let targetTuple = scorecard.[newTile]
    let newTuple = (fst targetTuple, snd targetTuple + 1)
    scorecard.[newTile] <- newTuple
 
            if rollCount < 10000 then
        rollDice (newTile, newRollCount, newDoublesCount, newInJail, newJailRollCount)
    else
        scorecard

No “mutable” (thanks to recursion) and only 1 assignment. I also wanted to get rid of that one ‘<-‘ and Thomas Petrick was kind enough to demonstrate the correct way to do this on stack overflow. Finally, I had to throw in a supporting function to make the decision logic account for rolling doubles that may put you in jail or may get you out of jail depending on prior state (were you in jail when you rolled doubles, were you out of jail when you rolled doubles for the 3rd time, etc…). I spent way too much time monkeying around with a series of nest if…then statements when it hit me that I should be using tuples and pattern matching:

let getTileMove (currentTile, dieTotal, doublesCount, inJail, jailRollCount) =
    match (inJail,jailRollCount, doublesCount) with
        | (true,3,_) -> getBoardMove(10,dieTotal)
        | (true,_,_) -> 10
        | (false,_,3) -> 10
        | (false,_,_) -> getBoardMove(10,dieTotal)

So here if the real power of F# on display. I can think of hundreds of applications that I have seen in C#/VB.NET that have a high cyclomatic complexity and hidden bugs that have reared their head at the most inopportune time because of complex business logic using a series of case..switch and/or if..then. statements. Even by putting step into its own function only helps partially because the code is still there –> it is just labeled better.

By using tupled pattern matching, all of that complexity goes away and we have a succinct series of statements that actually reflect how the brain thinks about the problem. By using F#, there are fewer lines of code (and therefore fewer unit tests to maintain) and you can write code that better represents how the wetware is approaching the problem.

Filed under F#

The Big Ten and F#

December 10, 2013 3 Comments

I was talking to fellow TRINUGer David Green about football schools a couple of weeks ago. He went to Northwestern and I went to Michigan and we were discussing the relative merits of universities doing well in football. Assuming Goro was counting, on one hand, it is great to have a sport that can bring in tons of money to the school to fund non-football sports and activities, on the second hand it keeps alumni interested in their school, on the 3rd hand it can give locals a source of pride in the school, and on the last hand it can take the focus away from the other parts of the academic institution.

I then was talking to a professor at Ohio State University – she cares absolutely zero about the football team. I made the comment that the smartest kids in Ohio don’t go to OSU. They will go and root for their gladiators on Saturday but when it comes down to their academic and subsequent professional success, they look elsewhere. She agreed.

Putting those two conversations together, it put OSU and MSU’s continued success in the Big Ten in context – as the inevitable bellyaching that those teams get the short stick when compared to the SEC. For example, OSU and MSU both would be undefeated in the Ivy League in 2013– does that mean they should be considered in the same conversation as Alabama and Auburn for the national championship? I think the biggest problem that OSU and MSU have is that they are in the Big Ten – which historically has been about geography, academic success, and athletic competition (in that order).

Looking at the Big Ten Schools, I pulled their most recent academic ranking for US News and World Report and their BCS Ranking. I then went over to MathIsFun to get the recipe for correlation:

I then went over to Visual Studio and created a solution like so:

Learning from my last project, I created my unit test first to verify that the calculation is correct:

[TestMethod]
public void FindCorrelationUsingStandardInput_ReturnsExpectedValue()
{
    Double[] tempatures = new Double[12] { 14.2, 16.4, 11.9, 15.2, 18.5, 22.1, 19.4, 25.1, 23.4, 18.1, 22.6, 17.2 };
    Double[] sales = new Double[12] { 215, 325, 185, 332, 406, 522, 412, 614, 544, 421, 445, 408 };
 
    Double expected = .9575;
    Double actual = Calculations.Correlation(tempatures, sales);
    Assert.AreEqual(expected, actual);
}

I then hopped over to my working code and started coding:

type Calculations() =
    static member Correlation(x:IEnumerable<double>, y:IEnumerable<double>) =
        let meanX = Seq.average x
        let meanY = Seq.average y
        
        let a = Seq.map(fun x -> x-meanX) x
        let b = Seq.map(fun y -> y-meanY) y
 
        let ab = Seq.zip a b
        let abProduct = Seq.map(fun (a,b) -> a * b) ab
 
        let aSquare = Seq.map(fun a -> a * a) a
        let bSquare = Seq.map(fun b -> b * b) b
        
        let abSum = Seq.sum abProduct
        let aSquareSum = Seq.sum aSquare
        let bSquareSum = Seq.sum bSquare
 
        let sums = aSquareSum * bSquareSum
        let squareRootOfSums = sqrt(sums)
 
        abSum/squareRootOfSums

What I noticed is that those intermediate variables make the code much more wordy than they need to be – so a mathematician might think that the code is too verbose– but a developer might appreciate that each step is laid out. In fact, I would argue that a better component design would be to break out each of the steps into their own function that can be independently testable (and perhaps reused by other functions):

[TestMethod]
public void GetMeanUsingStandardInputReturnsExpectedValue()
{
    Double[] tempatures = new Double[12] { 14.2, 16.4, 11.9, 15.2, 18.5, 22.1, 19.4, 25.1, 23.4, 18.1, 22.6, 17.2 };
    Double expected = 18.675;
    Double actual = Calculations.Mean(tempatures);
    Assert.AreEqual(expected, actual);
}
 
[TestMethod]
public void GetBothMeansProductUsingStandardInputReturnsExpectedValue()
{
    Double[] tempatures = new Double[12] { 14.2, 16.4, 11.9, 15.2, 18.5, 22.1, 19.4, 25.1, 23.4, 18.1, 22.6, 17.2 };
    Double[] sales = new Double[12] { 215, 325, 185, 332, 406, 522, 412, 614, 544, 421, 445, 408 };
 
    Double expected = 5325;
    Double actual = Calculations.MeanProduct(tempatures);
    Assert.AreEqual(expected, actual);
}
 
[TestMethod]
public void GetMeanSquareUsingStandardInputReturnsExpectedValue()
{
    Double[] tempatures = new Double[12] { 14.2, 16.4, 11.9, 15.2, 18.5, 22.1, 19.4, 25.1, 23.4, 18.1, 22.6, 17.2 };
    Double[] sales = new Double[12] { 215, 325, 185, 332, 406, 522, 412, 614, 544, 421, 445, 408 };
 
    Double expected = 177;
    Double actual = Calculations.MeanSquared(tempatures);
    Assert.AreEqual(expected, actual);
}

I’ll leave that implementation for another day as it is already getting late. In any event, I ran the unit test and I got red (pink, really):

The spreadsheet rounded and my calculation does not. I adjusted the unit test appropriately:

[TestMethod]
public void FindCorrelationUsingStandardInput_ReturnsExpectedValue()
{
    Double[] tempatures = new Double[12] { 14.2, 16.4, 11.9, 15.2, 18.5, 22.1, 19.4, 25.1, 23.4, 18.1, 22.6, 17.2 };
    Double[] sales = new Double[12] { 215, 325, 185, 332, 406, 522, 412, 614, 544, 421, 445, 408 };
 
    Double correlation = Calculations.Correlation(tempatures, sales);
    Double expected = .9575;
 
    Double actual = Math.Round(correlation, 4);
    Assert.AreEqual(expected, actual);
}

And now I am green:

So going back to the original question, I took the current Big Ten Schools and put their academic rankings and football rankings side by side:

I then made a revised Big Ten that had a much higher academic ranking based on schools that play in a power football conference but still maintain high academics.

Note that I left Penn State out of both of these lists b/c they have a NaN for their football ranking – but they certainly have a high enough academic score to be part of the revised Big Ten.

And then when I put those values through the correlation function via a Console UI:

static void Main(string[] args)
{
    Console.WriteLine("Start");
 
    Double[] academicRanking = new Double[12] { 12,28,41,41,52,62,68,69,73,73,75,101 };
    Double[] footballRanking = new Double[12] { 65,41,82,19,7,61,105,36,4,34,63,37 };
 
    Double originalCorrelation = Calculations.Correlation(academicRanking, footballRanking);
    Console.WriteLine("Original BigTen Correlation {0}", originalCorrelation);
 
    academicRanking = new Double[10] { 7,12,17,18,23,23,28,30,41,41 };
    footballRanking = new Double[10] { 24, 65, 32, 26, 94, 84, 41, 58, 82, 19 };
    Double revisedCorrelation = Calculations.Correlation(academicRanking, footballRanking);
    Console.WriteLine("Revised BigTen Correlation {0}", revisedCorrelation);
 
    
    Console.WriteLine("End");
    Console.ReadKey();
}

I get:

And just looking at the data seems to support this. There is a negative correlation between academics and football success in the current Big Ten – Higher the academics = lower the football ranking and vice versa. In the revised Big Ten, there is positive correlation of the same magnitude – higher academics and higher (relative) football rankings. Put another way, the new Big Ten has a much stronger academic ranking and pretty much the same football ranking.

Looking at a map, this new conference is like a doughnut with Ohio, West Virginia, and Kentucky in the middle. Perhaps they can have a football championship sponsored by Krispie Kreeme? In any event, OSU and MSU are much closer academically and football-wise to the Alabamas and Auburns than the Northwesterns and Michigans of the world. In terms of geographic proximity, Columbus, Ohio is closer to Tuscalosa, AL than Lincoln, NB. So perhaps the OSU and MSU fans would be better served in a conference that is more aligned with their University’s priorities? If they went undefeated or even 1 loss, they would still be in the national championship discussion.

Filed under Analytics, F#

F# > C# when doing math

December 10, 2013 5 Comments

My friend/coworker Rob Seder sent me this code project link and said it might be an interesting exercise to duplicate what he had done in F#. Interesting indeed! Challenge accepted!

I first created a solution like so:

I then copied the Variance calculation from the post to the C# implementation:

public class Calculations
{
    public static Double Variance(IEnumerable<Double> source)
    {
        int n = 0;
        double mean = 0;
        double M2 = 0;
 
        foreach (double x in source)
        {
            n = n + 1;
            double delta = x – mean;
            mean = mean + delta / n;
            M2 += delta * (x – mean);
        }
        return M2 / (n – 1);
    }
}

I then created a couple of unit tests for the method and made sure that the results ran green:

[TestClass]
public class CSharpCalculationsTests
{
    [TestMethod]
    public void VarianceOfSameNumberReturnsZero()
    {
        Collection<Double> source = new Collection<double>();
        source.Add(2.0);
        source.Add(2.0);
        source.Add(2.0);
 
        double expected = 0;
        double actual = Calculations.Variance(source);
        Assert.AreEqual(expected, actual);
    }
 
    [TestMethod]
    public void VarianceOfOneAwayNumbersReturnsOne()
    {
        Collection<Double> source = new Collection<double>();
        source.Add(1.0);
        source.Add(2.0);
        source.Add(3.0);
 
        double expected = 1;
        double actual = Calculations.Variance(source);
        Assert.AreEqual(expected, actual);
    }    
}

I then spun up the same unit tests to test the F# implementation and then went over to the F# project. My first attempt started along the lines like this:

namespace Tff.BasicStats.FSharp
 
open System
open System.Collections.Generic
 
type Calculations() = 
    static member Variance (source:IEnumerable<double>) =
        let mean = Seq.average(source)
        let deltas = Seq.map(fun x -> x-mean) source
        let deltasSum = Seq.sum deltas
        let deltasLength = Seq.length deltas
        deltasSum/(double)deltasLength

I then realized that I was writing procedural code in F# – I was not taking advantage of the power that the expressiveness that the language provides. I also realized that looking at the C# code to understand how to calculate Variance was useless – I was getting lost in the loop and the poorly-named variables. I went over to Wikipedia’s definition to see if that could help me understand Variance better but I got lost in all of the formulas. I then binged Variance on Google and one of the 1st links is MathIsFun with this explanation. This was more like it! Cool dog pictures and a stupid simple recipe for calculating Variance. The steps are:

I hopped over to Visual Studio and wrote a one-for-one line of code to match the recipe:

namespace Tff.BasicStats.FSharp
 
open System
open System.Collections.Generic
 
type Calculations() = 
    static member Variance (source:IEnumerable<double>) =
        let mean = Seq.average source
        let deltas = Seq.map(fun x -> sqrt(x-mean)) source
        Seq.average deltas

I ran the unit tests but they were running red! I was getting a NaN.

Hearing my cursing, my 7th grade son came over and said – “Dad, that is wrong. You don’t use the square root on the (x-mean), you square it. Also, you can’t take the square root of a negative number and any item in that list that is less than the average will return that ” Let me repeat that – a 7th grader with no coding experience but who knows about Variance from his math class just read the code and found the problem.

I then changed the code to square the value like so:

namespace Tff.BasicStats.FSharp
 
open System
open System.Collections.Generic
 
type Calculations() = 
    static member Variance (source:IEnumerable<double>) =
        let mean = Seq.average source
        let deltas = Seq.map(fun x -> pown(x-mean) 2) source
        Seq.average deltas

And now my unit test… runs…. Red!

Not understanding why, I turned to the REPL (F# Interactive Window). I first entered my test set:

I then entered the calculation from each line against the test set:

Staring at the resulting array, it hit me that perhaps the original unit test’s expected value was wrong! I went over to TutorVista and entered in my array. Would you believe it?

The calculation on the code project site is incorrect! The correct way to do the unit test is:

[TestMethod]
public void VarianceOfOneAwayNumbersReturnsOne()
{
    Collection<Double> source = new Collection<double>();
    source.Add(1.0);
    source.Add(2.0);
    source.Add(3.0);
 
    //double expected = 6666666667;
    double expected = 2f / 3f; 
    double actual = Calculations.Variance(source);
    Assert.AreEqual(expected, actual);
}    

(Note that expected was the easiest way I could come up with .6 repeating without getting all crazy on the formatting). Now both my unit tests run green and one of the C# ones runs red.

I have no interest in trying to figure out how to fix that C# code – I care less about how to solve my problem and more about just solving the problem. The real power of F# really is on display here. The coolest parts of this exercise were:

One-for-one correspondence between the steps to solve a problem and the code
The code is much more readable to non developers
By concentrating on how to solve the problem in C#, the original developer lost sight of what he was trying to accomplish. F# focuses you on the result, not the code.
Unit tests can be wrong – if you let your code’s result drive the expected and not a external source.

Filed under F#, Statistics

F# and SignalR Stock Ticker: Part 2

December 3, 2013 1 Comment

Following up on my prior post found here about using F# to write the Stock Ticker example found on SignalR’s website, I went to implement the heart of the application – the stock ticker class.

The original C# class suffers from a violation of command/query separation and also does a couple of things. Breaking out the code functionality, the class creates a list of random stocks in the constructor.

Then there is a timer that loops and periodically updates the current stock price.

Finally, it broadcasts the new stock price to any connected clients.

Because the class depends on the clients for its creation and lifetime, it implements the singleton pattern – you access the class via its Instance property. This is a very common pattern:

//Singleton instance
private readonly static Lazy<StockTicker> _instance = 
    new Lazy<StockTicker>(() => 
        new StockTicker(GlobalHost.ConnectionManager.GetHubContext<StockTickerHub>().Clients));

public static StockTicker Instance
{
    get
    {
        return _instance.Value;
    }
}

Attacking the class from a F# point of view, I first addressed the singleton pattern. I checked out the Singleton pattern in Liu’s F# for C# Developers. The sentience that caught my eye was “An F# value is immutable by default, and this guarantees there is only on instance.” (p149) Liu then goes and builds an example using a private class and shows how to reference it via a Instance method. My take-away from the example is that you don’t need a Singleton pattern in F# – because everything is Singleton by default. Another way to look at it is that a Singleton pattern is a well-accepted workaround the limitations that mutability brings when using C#.

I then jumped over to the updating stock prices – after all, how can you send out a list of new stock prices if you can’t mutate the list or the individual stocks within the list? Quite easily, in fact.

The first thing I did was to create a StockTicker class that takes in a SignalR HubContext and a list of stocks.

type StockTicker(clients: IHubConnectionContext, stocks: IEnumerable<Stock>) = class

I then added the logic to update the list and stocks.

let rangePercent = 0.002
let updateInterval = TimeSpan.FromMilliseconds(250.)
let updateStockPrice stock:Stock = 
    let updateOrNotRandom = new Random()
    let r = updateOrNotRandom.NextDouble();
    match r with
        | r when r <= 1. -> stock
        | _ ->
 
            let random = new Random(int(Math.Floor(stock.Price)))
            let percentChange = random.NextDouble() * rangePercent
            let pos = random.NextDouble() > 0.51
            let change = Math.Round(stock.Price * decimal(percentChange),2)
            let newPrice = stock.Price + change
            new Stock(stock.Symbol, stock.DayOpen, newPrice)
let updatedStocks = stocks
                        |> Seq.map(fun stock -> updateStockPrice(stock))

Looking at the code, the word “update” in the prior sentence is wrong. I am not updating anything. I am replacing the list and the stocks with the new price (if determined). Who needs a singleton? F# doesn’t.

I then attempted to notify the clients like so:

member x.Clients = clients
member x.Stocks = stocks
member x.BroadcastStockPrice (stock: Stock) =
    x.Clients.All.updateStockPrice(stock)

But I got a red squiggly line of approbation (RSLA) on the updateStockPrice method. The compiler is complaining that

Error 1 The field, constructor or member ‘updateStockPrice’ is not defined

And reading the SignalR explanation here:

The updateStockPrice method that you are calling in BroadcastStockPrice doesn’t exist yet; you’ll add it later when you write code that runs on the client. You can refer to updateStockPrice here because Clients.All is dynamic, which means the expression will be evaluated at runtime

So how does F# accommodate the dynamic nature of Clients.All? I don’t know so off to StackOverflow I go….

In any event, I can then wire up a method that broadcasts the new stock prices like so:

member x.BroadcastAllPrices =
    x.Clients.All.updateAllStockPrices(updatedStocks)

And then write a method that calls this broadcast method every quarter second:

 member x.Start =
     async {
            while true do
             do! Async.Sleep updateInterval
             x.BroadcastAllPrices
     } |> Async.StartImmediate

Note that I tried to figure out the Timer class and subsequent Event for the timer, but I couldn’t. I stumbled upon this post to ditch the timer in favor of the code above and since it works, I am all for it. Figuring out events in F# is a battle for another day…

Filed under F#, SignalR

Jamie Dixon's Home

Correlation Between Recruit Rankings and Final Standings in Big Ten Football

Screen Scraping College Football Statistics

F# and Monopoly Simulation Redux

The Big Ten and F#

F# > C# when doing math

F# and SignalR Stock Ticker: Part 2

Categories

Recent Posts

Archives

Blogroll

Meta