Consuming Twitter With F#

I set up a meetup for TRINUG’s F#/data analytics SIG to center around consuming and analyzing Tweets.  Since Twitter is just JSON, I assumed it would be easy enough to search Tweets for a given subjects in a given time period.  How wrong I was.  I spent several hours research different ways to consume Twitter to varying degrees of success.  My 1st stop was to investigate some of the more common libraries that C# developers use to consume Twitter.  Here is my survey of some of the more popular ones:

Twitterizer: No longer maintained

  1. // Install-Package twitterizer -Version 2.4.2
  2. // Update-Package Newtonsoft.Json -Reinstall
  3. open Twitterizer
  4.  
  5. type public TwitterProvider() =
  6.     member this.GetTweetsForDateRange(ticker:string, startDate: DateTime, endDate: DateTime) =
  7.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  8.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  9.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  10.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  11.         
  12.         let tokens = new OAuthTokens()
  13.         tokens.set_ConsumerKey(consumerKey)
  14.         tokens.set_ConsumerSecret(consumerSecret)
  15.         tokens.set_AccessToken(accessToken)
  16.         tokens.set_AccessTokenSecret(accessTokenSecret)
  17.  
  18.         let searchOptions = new SearchOptions()
  19.         searchOptions.SinceDate <- startDate
  20.         searchOptions.UntilDate <- endDate
  21.         let results = TwitterSearch.Search(tokens, ticker,searchOptions)
  22.         results.ResponseObject
  23.                     |> Seq.map(fun r -> r.CreatedDate, r.Text)

TweetSharp: No longer maintained

  1. open TweetSharp
  2.  
  3. type public TwitterProvider() =
  4.     member this.GetTweetsForDateRange(ticker:string, startDate: DateTime, endDate: DateTime) =
  5.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  6.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  7.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  8.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  9.         
  10.         let service = new TwitterService(consumerKey, consumerSecret)
  11.         service.AuthenticateWith(accessToken, accessTokenSecret)
  12.  
  13.         let searchOptions = new SearchOptions()
  14.         searchOptions.Q <- "IBM%20since%3A2014-03-01&src=typd"
  15.         service.Search(searchOptions).Statuses
  16.                                         |> Seq.map(fun s -> s.CreatedDate, s.Text)

Note that I did try and add a date range the way the Twitter API instructs, but it still came back with only 20 tweets.

LinqToTwitter: Active but nave to use Linq syntax.  Ugh!

Twitterinvi: Active but does not have date range functionality

  1. open System
  2. open System.Configuration
  3. open Tweetinvi
  4.  
  5. type public TwitterProvider() =
  6.     member this.GetTodaysTweets(ticker: string) =
  7.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  8.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  9.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  10.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  11.  
  12.         TwitterCredentials.SetCredentials(accessToken, accessTokenSecret, consumerKey, consumerSecret)
  13.         let tweets = Search.SearchTweets(ticker);
  14.         tweets |> Seq.map(fun t -> t.CreatedAt, t.RetweetCount)
  15.  
  16.     member this.GetTweetsForDateRange(ticker: string, startDate: DateTime)=
  17.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  18.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  19.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  20.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  21.  
  22.         TwitterCredentials.SetCredentials(accessToken, accessTokenSecret, consumerKey, consumerSecret)
  23.         let searchParameter = Search.GenerateSearchTweetParameter(ticker)
  24.         searchParameter.Until <- startDate;
  25.         let tweets = Search.SearchTweets(searchParameter);
  26.         tweets |> Seq.map(fun t -> t.CreatedAt, t.RetweetCount)

So without an out of the box API to use, I thought about using a Json Type Provider the way Lincoln Atkinson did.  The problem is that is example is for V1 of Twitter and V 1.1 uses Oauth.  If you run his code, you get

image

I then thought about a 3rd party API that captures Tweets.  I ran across gnip ($500!) and Topsy (no longer accepting new licenses b/c Apple bought them) so I am back to square one.

So finally I thought about rolling my own (with OAuth being the hard part) but I am quickly running out of time to get ready for the SIG and I don’t want to spend the time on only this part. 

Why isn’t there a Twitter type provider?  I’ll add it to the list….

Using the Twitter API

As part of my alerting project, I wanted to implement a Twitter push. I first googled on Bing and got this post. I made a quick implementation of my IAlert Interface like so:

public void Send()
{

    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(_twitterUri);
    request.Credentials = new NetworkCredential(_twitterUserId, _twitterPassword);
    request.Timeout = 5000;
    request.Method = "POST";
    request.ContentType = "application/x-www-form-urlencoded";
    using (Stream requestStream = request.GetRequestStream())
    {
        using (StreamWriter streamWriter = new StreamWriter(requestStream))
        {
            streamWriter.Write(_message);
        }
    }
    WebResponse response = request.GetResponse();
}

However, I get a 401

image

it looks like it is an old implementation of the Twitter API.  I then went to this page and it looks like I can’t do basic HTTP authentication with Twitter – I have to use OAuth.  Twitter’s help page recommended using the C# Twitterizer library.  I went ahead and and installed it from NuGet

 

image

I then re-wrote the send method to use the Twitterizer:

public void Send()
{
    OAuthTokens tokens = new OAuthTokens();
    tokens.ConsumerKey = _consumerKey;
    tokens.ConsumerSecret = _consumerSecret;
    tokens.AccessToken = _accessToken;
    tokens.AccessTokenSecret = _accessTokenSecret;

    TwitterResponse<TwitterStatus> tweetResponse = TwitterStatus.Update(tokens, _message);
}

 

Alas, when I re-ran my unit tests

image

Ugh – I guess I need to add a reference to my test project.  Sure enough, that did the trick:

image

Now I am getting this message back from Twitter

image

I then when to my account settings for the app on Twitter and changed it from Read only to full control:

image

Hit save, regenerated the keys, and still got the same message

I then stumbled onto this.  I then restarted, changed the text of my integration test and voila (that’s French for “Whoop-de-do")

image