Consuming Twitter With F#

April 8, 2014 7 Comments

I set up a meetup for TRINUG’s F#/data analytics SIG to center around consuming and analyzing Tweets.  Since Twitter is just JSON, I assumed it would be easy enough to search Tweets for a given subjects in a given time period.  How wrong I was.  I spent several hours research different ways to consume Twitter to varying degrees of success.  My 1st stop was to investigate some of the more common libraries that C# developers use to consume Twitter.  Here is my survey of some of the more popular ones:

Twitterizer: No longer maintained

  1. // Install-Package twitterizer -Version 2.4.2
  2. // Update-Package Newtonsoft.Json -Reinstall
  3. open Twitterizer
  4.  
  5. type public TwitterProvider() =
  6.     member this.GetTweetsForDateRange(ticker:string, startDate: DateTime, endDate: DateTime) =
  7.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  8.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  9.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  10.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  11.         
  12.         let tokens = new OAuthTokens()
  13.         tokens.set_ConsumerKey(consumerKey)
  14.         tokens.set_ConsumerSecret(consumerSecret)
  15.         tokens.set_AccessToken(accessToken)
  16.         tokens.set_AccessTokenSecret(accessTokenSecret)
  17.  
  18.         let searchOptions = new SearchOptions()
  19.         searchOptions.SinceDate <- startDate
  20.         searchOptions.UntilDate <- endDate
  21.         let results = TwitterSearch.Search(tokens, ticker,searchOptions)
  22.         results.ResponseObject
  23.                     |> Seq.map(fun r -> r.CreatedDate, r.Text)

TweetSharp: No longer maintained

  1. open TweetSharp
  2.  
  3. type public TwitterProvider() =
  4.     member this.GetTweetsForDateRange(ticker:string, startDate: DateTime, endDate: DateTime) =
  5.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  6.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  7.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  8.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  9.         
  10.         let service = new TwitterService(consumerKey, consumerSecret)
  11.         service.AuthenticateWith(accessToken, accessTokenSecret)
  12.  
  13.         let searchOptions = new SearchOptions()
  14.         searchOptions.Q <- "IBM%20since%3A2014-03-01&src=typd"
  15.         service.Search(searchOptions).Statuses
  16.                                         |> Seq.map(fun s -> s.CreatedDate, s.Text)

Note that I did try and add a date range the way the Twitter API instructs, but it still came back with only 20 tweets.

LinqToTwitter: Active but nave to use Linq syntax.  Ugh!

Twitterinvi: Active but does not have date range functionality

  1. open System
  2. open System.Configuration
  3. open Tweetinvi
  4.  
  5. type public TwitterProvider() =
  6.     member this.GetTodaysTweets(ticker: string) =
  7.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  8.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  9.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  10.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  11.  
  12.         TwitterCredentials.SetCredentials(accessToken, accessTokenSecret, consumerKey, consumerSecret)
  13.         let tweets = Search.SearchTweets(ticker);
  14.         tweets |> Seq.map(fun t -> t.CreatedAt, t.RetweetCount)
  15.  
  16.     member this.GetTweetsForDateRange(ticker: string, startDate: DateTime)=
  17.         let consumerKey = ConfigurationManager.AppSettings.["consumerKey"]
  18.         let consumerSecret = ConfigurationManager.AppSettings.["consumerSecret"]
  19.         let accessToken = ConfigurationManager.AppSettings.["accessToken"]
  20.         let accessTokenSecret = ConfigurationManager.AppSettings.["accessTokenSecret"]
  21.  
  22.         TwitterCredentials.SetCredentials(accessToken, accessTokenSecret, consumerKey, consumerSecret)
  23.         let searchParameter = Search.GenerateSearchTweetParameter(ticker)
  24.         searchParameter.Until <- startDate;
  25.         let tweets = Search.SearchTweets(searchParameter);
  26.         tweets |> Seq.map(fun t -> t.CreatedAt, t.RetweetCount)

So without an out of the box API to use, I thought about using a Json Type Provider the way Lincoln Atkinson did.  The problem is that is example is for V1 of Twitter and V 1.1 uses Oauth.  If you run his code, you get

image

I then thought about a 3rd party API that captures Tweets.  I ran across gnip ($500!) and Topsy (no longer accepting new licenses b/c Apple bought them) so I am back to square one.

So finally I thought about rolling my own (with OAuth being the hard part) but I am quickly running out of time to get ready for the SIG and I don’t want to spend the time on only this part. 

Why isn’t there a Twitter type provider?  I’ll add it to the list….

Filed under F#, Twitter

7 Responses to Consuming Twitter With F#

  1. Dave Thomas (@7sharp9) says:
    April 8, 2014 at 11:34 am

    Could do with a variable type provider so you could construct one at run time with a custom expression, although that would make anything other than desktops troublesome. Could you not use query expressions with Linq2Twitter?

    Reply
  2. Dave Thomas (@7sharp9) says:
    April 8, 2014 at 11:37 am

    Stuff like this would be interesting too:
    type FSharpFeed = TwitterProvider
    let fsFeed = FSharpFeed()
    fsFeed.Listen |> Observable.filter myRealtimeFilter

    Reply

  4. Pingback: F# Weekly #15, 2014 | Sergey Tihon's Blog

  6. Pingback: 4/19/2014 Article Links | The Puntastic Programmer

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s

%d bloggers like this: