Azure | Jamie Dixon's Home

EjectABed Version 2 – Now Using the Raspberry Pi (Part 1)

September 8, 2015 1 Comment

I recently entered a hackster.io competition that centered around using Windows 10 on the Raspberry Pi. I entered the ejectabed and it was accepted to the semi-final round. My thought was to take the existing ejectabed controller from a Netduino and move it to a Raspberry Pi. While doing that, I could open the ejectabed from my local area network to the internet so anyone could eject Sloan.

My 1st step was hook my Raspberry Pi up to my home network and deploy from Visual Studio to it. Turns out, it was pretty straightforward.

I took a old Asus Portable Wireless Router and plugged it into my home workstation. I then configured the router to act as an Access Point so that it would pass though all traffic from the router to which my developer workstation is attached. I then attached the router to the PI and powered it though the PI’s USB port. I then plugged the PI’s HDMI out to a spare monitor of mine.

With all of the hardware plugged in, I headed over to Windows On Devices and followed the instructions on how to set up a Raspberry PI. After installing the correct software on my developer workstation, flashing the SD card with win10, plugging the SD card into the PI, turning the PI on, and then remoting into the PI via powershell, I could see the PI on my local workstation via the Windows IoT Core Watcher and the PI showing its friendly welcome screen via HDMI.

I then headed over to Visual Studio and copy/pasted the equisite “Hello IoT World” Blinky project to the Pi and watched the light go on and off.

With that out of the way, I decided to look at controlling the light via Twitter and Azure. The thought was to have the PI monitor a message queue on Azure and whenever there was a message, blink on or off (simulating the ejectabed being activated). To that end, I went into Azure and created a basic storage account. One of the nice things about Azure is that you get a queue out of the box when you create a storage account:

One of the not so nice things about Azure is that there is no way to control said Queue via their UI. You have to create, push, and pull from the queue in code. I went back to visual studio and added in the Azure Storage Nuget package

I then created a method to monitor the queue

 1         internal async Task<Boolean> IsMessageOnQueue()
 2         {
 3             var storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=ejectabed;AccountKey=xxx";
 4             var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
 5             var client = storageAccount.CreateCloudQueueClient();
 6             var queue = client.GetQueueReference("sloan");
 7             var queueExists = await queue.ExistsAsync();
 8             if (!queueExists)
 9             { 
10                 GpioStatus.Text = "Queue does not exist or is unreachable.";
11                 return false;
12             }
13             var message = await queue.GetMessageAsync(); 
14             if (message != null)
15             {
16                 await queue.DeleteMessageAsync(message);
17                 return true;
18             }
19             GpioStatus.Text = "No message for the EjectABed.";
20             return false;
21         }
22

Then if there is a message, the PI would run the ejection sequence (in this case blink the light)

1         internal void RunEjectionSequence()
2         {
3             bedCommand.Eject();
4             bedTimer = new DispatcherTimer();
5             bedTimer.Interval = TimeSpan.FromSeconds(ejectionLength);
6             bedTimer.Tick += LightTimer_Tick;
7             bedTimer.Start();
8         }

I deployed the code to the PI without a problem. I then created a Basic console application to push messages to the queue that the PI could drain

 1     class Program
 2     {
 3         static String storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=ejectabed;AccountKey=xxx";
 4 
 5         static void Main(string[] args)
 6         {
 7             Console.WriteLine("Start");
 8             Console.WriteLine("Press The 'E' Key To Eject.  Press 'Q' to quit...");
 9 
10             var keyInfo = ConsoleKey.S;
11             do
12             {
13                 keyInfo = Console.ReadKey().Key;
14                 if (keyInfo == ConsoleKey.E)
15                 {
16                     CreateQueue();
17                     WriteToQueue();
18                     //ReadFromQueue();
19                 }
20 
21             } while (keyInfo != ConsoleKey.Q);
22 
23             Console.WriteLine("End");
24             Console.ReadKey();
25         }
26 
27         private static void CreateQueue()
28         {
29             var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
30             var client = storageAccount.CreateCloudQueueClient();
31             var queue = client.GetQueueReference("sloan");
32             queue.CreateIfNotExists();
33             Console.WriteLine("Created Queue");
34         }
35 
36         private static void WriteToQueue()
37         {
38             var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
39             var client = storageAccount.CreateCloudQueueClient();
40             var queue = client.GetQueueReference("sloan");
41             var message = new CloudQueueMessage("Eject!");
42             queue.AddMessage(message);
43             Console.WriteLine("Wrote To Queue");
44         }
45 
46 
47         private static void ReadFromQueue()
48         {
49             var storageAccount = CloudStorageAccount.Parse(storageConnectionString);
50             var client = storageAccount.CreateCloudQueueClient();
51             var queue = client.GetQueueReference("sloan");
52             var queueExists = queue.Exists();
53             if (!queueExists)
54                 Console.WriteLine("Queue does not exist");
55             var message = queue.GetMessage();
56             if (message != null)
57             {
58                 queue.DeleteMessage(message);
59                 Console.WriteLine("Message Found and Deleted");
60             }
61             else
62             {
63                 Console.WriteLine("No messages");
64             }
65         }
66

I could then Write to the queue and the PI would read and react. You can see it in action here:

With the queue up and running, I was ready to add in the ability for someone to Tweet to the queue. I created a cloud service project and pointed to a new project that will monitor Twitter and then push to the queue:

The Twitter project uses the TweetInvi nuget package and is a worker project. It makes a call to Twitter every 15 seconds and if there is a tweet to “ejectabed” with a person’s name, it will write to the queue (right now, only Sloan’s name is available)

 1 type TwitterWorker() =
 2     inherit RoleEntryPoint() 
 3 
 4     let storageConnectionString = RoleEnvironment.GetConfigurationSettingValue("storageConnectionString")
 5 
 6     let createQueue(queueName) =
 7         let storageAccount = CloudStorageAccount.Parse(storageConnectionString)
 8         let client = storageAccount.CreateCloudQueueClient()
 9         let queue = client.GetQueueReference(queueName);
10         queue.CreateIfNotExists() |> ignore
11 
12     let writeToQueue(queueName) =
13         let storageAccount = CloudStorageAccount.Parse(storageConnectionString)
14         let client = storageAccount.CreateCloudQueueClient()
15         let queue = client.GetQueueReference(queueName)
16         let message = new CloudQueueMessage("Eject!")
17         queue.AddMessage(message) |> ignore
18 
19     let writeTweetToQueue(queueName) =
20         createQueue(queueName)
21         writeToQueue(queueName)
22 
23     let getKeywordFromTweet(tweet: ITweet) = 
24         let keyword = "sloan"
25         let hasKeyword = tweet.Text.Contains(keyword)
26         let isFavourited = tweet.FavouriteCount > 0
27         match hasKeyword, isFavourited  with
28         | true,false -> Some (keyword,tweet)
29         | _,_ -> None
30         
31 
32     override this.Run() =
33         while(true) do
34             let consumerKey = RoleEnvironment.GetConfigurationSettingValue("consumerKey")
35             let consumerSecret = RoleEnvironment.GetConfigurationSettingValue("consumerSecret")
36             let accessToken = RoleEnvironment.GetConfigurationSettingValue("accessToken")
37             let accessTokenSecret = RoleEnvironment.GetConfigurationSettingValue("accessTokenSecret")
38 
39             let creds = Credentials.TwitterCredentials(consumerKey, consumerSecret, accessToken, accessTokenSecret)
40             Tweetinvi.Auth.SetCredentials(creds)
41             let matchingTweets = Tweetinvi.Search.SearchTweets("@ejectabed")
42             let matchingTweets' =  matchingTweets |> Seq.map(fun t -> getKeywordFromTweet(t))
43                                                   |> Seq.filter(fun t -> t.IsSome)
44                                                   |> Seq.map (fun t -> t.Value)
45             matchingTweets' |> Seq.iter(fun (k,t) -> writeTweetToQueue(k))        
46             matchingTweets' |> Seq.iter(fun (k,t) -> t.Favourite())        
47 
48             Thread.Sleep(15000)
49 
50     override this.OnStart() = 
51         ServicePointManager.DefaultConnectionLimit <- 12
52         base.OnStart()

Deploying to Azure was a snap

And now when I Tweet,

the PI reacts. Since Twitter does not allow the same Tweet to be sent again, I deleted it every time I wanted to send a new message to the queue.

Filed under Azure, F#, IoT, Raspberry Pi

Aggregation of WCPSS Tax Records with School Assignment

March 17, 2015 Leave a comment

So the next part of my WCPSS hit parade, I need a way of combing the screen scrape that I did from the Wake County Tax Records as described here and the screen scrape of the Wake County Public School Assignments as found here. Getting data from the DocumentDb is straight foreword as long as you don’t ask too much from the query syntax.

I created two functions that pull the tax record and the school assignment via the index number:

 1 let getAssignment (id:int) =
 2     let collection = client.CreateDocumentCollectionQuery(database.CollectionsLink).Where(fun dc -> dc.Id = "houseassignment").ToArray().FirstOrDefault()
 3     let documentLink = collection.SelfLink
 4     let queryString = "SELECT * FROM houseassignment WHERE houseassignment.houseIndex = " + id.ToString()
 5     let query = client.CreateDocumentQuery(documentLink,queryString)
 6     match query |> Seq.length with
 7     | 0 -> None
 8     | _ -> 
 9         let assignmentValue = query |> Seq.head
10         let assignment = HouseAssignment.Parse(assignmentValue.ToString())
11         Some assignment
12 
13 let getValuation (id:int) =
14     let collection = client.CreateDocumentCollectionQuery(database.CollectionsLink).Where(fun dc -> dc.Id = "taxinformation").ToArray().FirstOrDefault()
15     let documentLink = collection.SelfLink
16     let queryString = "SELECT * FROM taxinformation WHERE taxinformation.index = 1"
17     let query = client.CreateDocumentQuery(documentLink,queryString)
18     match query |> Seq.length with
19     | 0 -> None
20     | _ -> 
21         let valuationValue = query |> Seq.head
22         let valuation = HouseValuation.Parse(valuationValue.ToString())
23         Some valuation

Note option types are being used because there any many index values where there is not a corresponding record. Also, there might a situation where the assignment has a record but the valuation does not and vice-versa so I created a function to only put the records together where there both records:

1 let assignSchoolTaxBase (id:int) =
2     let assignment = getAssignment(id)
3     let valuation = getValuation(id)
4     match assignment.IsSome,valuation.IsSome with
5     | true, true -> assignment.Value.Schools 
6                     |> Seq.map(fun s -> s, valuation.Value.AssessedValue)
7                     |> Some
8     | _ -> None

And running this on the first record, we are getting expected.

Also, running it on an index where there there is not a record, we are also getting expected

With the matching working, we need a way of bring all of the school arrays together and then aggregating the tax valuation. I decided to take a step by step approach to this, even though there might be a more terse way to write it.

1 #time
2 indexes |> Seq.map(fun i -> assignSchoolTaxBase(i))
3         |> Seq.filter(fun s -> s.IsSome)
4         |> Seq.collect(fun s -> s.Value)   
5         |> Seq.groupBy(fun (s,av) -> s)
6         |> Seq.map(fun (s,ss) -> s,ss |> Seq.sumBy(fun (s,av)-> av))
7         |> Seq.toArray

When I run it on the 1st 10 records, the values come back as expected

So the last step is to run it on all 350,000 indexes (let indexes = [|1..350000|]). The problem is that after a long period of time, things were not returning. So this is where the power of Azure comes in –> there is no problem so large I can’t thow more cores at it. I went to management portal and increased the VM to 8 cores

I then went into the code base and added pseq for the database calls (which I assume was taking the longest time):

 1 #time
 2 let indexes = [|1..350000|]
 3 let assignedValues = indexes |> PSeq.map(fun i -> assignSchoolTaxBase(i)) |> Seq.toArray
 4 
 5 let filePath = @"C:\Git\WakeCountySchoolScores\SchoolValuation.csv"
 6 
 7 assignedValues
 8         |> Seq.filter(fun s -> s.IsSome)
 9         |> Seq.collect(fun s -> s.Value)   
10         |> Seq.groupBy(fun (s,av) -> s)
11         |> Seq.map(fun (s,ss) -> s,ss |> Seq.sumBy(fun (s,av)-> av))
12         |> Seq.map(fun (s,v) -> s + "," + v.ToString() + Environment.NewLine)
13         |> Seq.iter(fun (s) -> File.AppendAllText(filePath, s))

and after 2 hours:

Filed under Azure, DocumentDb, F#

Moving Files Between Azure Blob Storage Using F#

December 23, 2014 1 Comment

Dear Future Jamie:

In case you forget (again) about how to move files from one container to another on Azure Blob Storage, here is the code:

 1 //http://azure.microsoft.com/en-us/documentation/articles/storage-dotnet-how-to-use-blobs/#configure-access
 2 
 3 #r "../packages/WindowsAzure.Storage.4.3.0/lib/net40/Microsoft.WindowsAzure.Storage.dll"
 4 
 5 open Microsoft.WindowsAzure.Storage
 6 open Microsoft.WindowsAzure.Storage.Auth
 7 open Microsoft.WindowsAzure.Storage.Blob
 8 open System.IO
 9 
10 let connectionString = "youconnectionStringHere"
11 let storageAccount = CloudStorageAccount.Parse(connectionString)
12 let blobClient = storageAccount.CreateCloudBlobClient()
13 
14 let sourceContainer = blobClient.GetContainerReference("source")
15 let targetContainer = blobClient.GetContainerReference("target")
16 
17 let copyBlob (sourceBlob:CloudBlockBlob) =
18     sourceBlob.FetchAttributes()
19     let blobName = sourceBlob.Name
20     let arrayLength = int sourceBlob.Properties.Length
21     let byteArray = Array.zeroCreate(arrayLength)
22     sourceBlob.DownloadToByteArray(byteArray,0) |> ignore
23 
24     let targetBlob = targetContainer.GetBlockBlobReference(blobName)
25     targetBlob.UploadFromByteArray(byteArray,0,arrayLength)
26 
27 let sourceBlobs = sourceContainer.ListBlobs()
28 sourceBlobs |> Seq.map(fun b -> b :?> CloudBlockBlob)
29             |> Seq.iter(fun b -> copyBlob b)
30 
31 
32 let result = targetContainer.ListBlobs()
33 Seq.length result

Love,

Jamie from Dec 2014

PS: You really should exercise more….

Filed under Azure, F#

Jamie Dixon's Home