Machine Learning | Jamie Dixon's Home

Neural Network Part 3: Perceptrons

August 5, 2014 1 Comment

I went back to my code for building a Perceptron and I made some changes. I realized that although McCaffrey combines the code together, there are actually two actions for the perceptron: training and predicting. I created a diagram to help me keep the functions that I need for each in mind:

I also skeletoned out some data structures that I think I need:

With the base diagrams out of the way, I created different data structures that were tailored to each action. These are a bit different than the diagrams –> I didn’t go back and update the diagrams because the code is where you would look to see how the system works:

type observation = {xValues:float List}
type weightedObservation = {xws:(float*float) List}
type confirmedObservation = {observation:observation;yExpected:float}
type weightedConfirmedObservation = {weightedObservation:weightedObservation;yExpected:float}
type neuronInput = {weightedObservation:weightedObservation;bias:float}

type cycleTrainingInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float}
type adjustmentInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float;yActual:float}
type adjustmentOutput = {weights:float List; bias:float}
type rotationTrainingInput = {confirmedObservations:confirmedObservation List;weights:float List;bias:float;alpha:float}
type trainInput = {confirmedObservations:confirmedObservation List; weightSeedValue:float;biasSeedValue:float;alpha:float; maxEpoches:int}

type cyclePredictionInput = {weightedObservation:weightedObservation;bias:float}
type rotationPredictionInput = {observations:observation List;weights:float List;bias:float}
type predictInput = {observations:observation List;weights:float List;bias:float}

Note that I am composing data structures with the base being an observation. The observation is a list of different xValues for a given, well, observation. The weighted observation is the XValue paired with the perceptron weights. The confirmedObservation is for training –> given an observation, what was the actual output?

With the data structures out of the way, I went to the Perceptron and added in the basic functions for creating seed values:

    member this.initializeWeights(xValues, randomSeedValue) =
        let lo = -0.01
        let hi = 0.01
        let xWeight = (hi-lo) * randomSeedValue + lo
        xValues |> Seq.map(fun w -> xWeight)

    member this.initializeBias(randomSeedValue) = 
        let lo = -0.01
        let hi = 0.01
        (hi-lo) * randomSeedValue + lo

Since I was doing TDD, here are the unit tests I used for these functions:

        [TestMethod]
        public void initializeWeightsUsingHalfSeedValue_ReturnsExpected()
        {
            var weights = _perceptron.initializeWeights(_observation.xValues, .5);
            var weightsList = new List<double>(weights);

            var expected = 0.0;
            var actual = weightsList[0];
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeWeightsUsingLessThanHalfSeedValue_ReturnsExpected()
        {
            var weights = _perceptron.initializeWeights(_observation.xValues, .4699021627);
            var weightsList = new List<double>(weights);

            var expected = -0.00060;
            var actual = Math.Round(weightsList[0],5);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeBiasesUsingHalfSeedValue_ReturnsExpected()
        {
            var expected = 0.0;
            var actual = _perceptron.initializeBias(.5);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeBiasesUsingLessThanHalfSeedValue_ReturnsExpected()
        {
            var expected = -0.00060;
            var bias = _perceptron.initializeBias(.4699021627);
            var actual = Math.Round(bias, 5);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeBiasesUsingGreaterThanHalfSeedValue_ReturnsExpected()
        {
            var expected = 0.00364;
            var bias = _perceptron.initializeBias(.6820621978);
            var actual = Math.Round(bias,5);
            Assert.AreEqual(expected, actual);
        }

I then created a base neuron and activation function that would work for both training and predicting:

    member this.runNeuron(input:neuronInput) = 
        let xws = input.weightedObservation.xws
        let output = xws 
                    |> Seq.map(fun (xValue,xWeight) -> xValue*xWeight) 
                    |> Seq.sumBy(fun x -> x)
        output + input.bias

    member this.runActivation(input) = 
        if input < 0.0 then -1.0 else 1.0

        [TestMethod]
        public void runNeuronUsingNormalInput_ReturnsExpected()
        {
            var expected = -0.0219;
            var perceptronOutput = _perceptron.runNeuron(_neuronInput);
            var actual = Math.Round(perceptronOutput, 4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runActivationUsingNormalInput_ReturnsExpected()
        {
            var expected = -1;
            var actual = _perceptron.runActivation(-0.0219);
            Assert.AreEqual(expected, actual);
        }

I then created the functions for training –> specifically to return adjusted weights and biases based on the result of the activation function

    member this.calculateWeightAdjustment(xValue, xWeight, alpha, delta) =
        match delta > 0.0, xValue >= 0.0 with
            | true,true -> xWeight - (alpha * abs(delta) * xValue)
            | false,true -> xWeight + (alpha * abs(delta) * xValue)
            | true,false -> xWeight - (alpha * abs(delta) * xValue)
            | false,false -> xWeight + (alpha * abs(delta) * xValue)

    member this.calculateBiasAdjustment(bias, alpha, delta) =
        match delta > 0.0 with
            | true -> bias - (alpha * abs(delta))
            | false -> bias + (alpha * abs(delta))

    member this.runAdjustment (input:adjustmentInput) =
        match input.weightedConfirmedObservation.yExpected = input.yActual with
            | true -> 
                let weights = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (x,w) -> w)
                let weights' = new List<float>(weights)
                {adjustmentOutput.weights=weights';adjustmentOutput.bias=input.bias}
            | false ->
                let delta = input.yActual - input.weightedConfirmedObservation.yExpected
                let weights' =  input.weightedConfirmedObservation.weightedObservation.xws 
                                    |> Seq.map(fun (xValue, xWeight) -> 
                                        this.calculateWeightAdjustment(xValue,xWeight,input.alpha,delta))
                                    |> Seq.toList
                let weights'' = new List<float>(weights')                                         
                let bias' = this.calculateBiasAdjustment(input.bias,input.alpha,delta)
                {adjustmentOutput.weights=weights'';adjustmentOutput.bias=bias'}

        [TestMethod]
        public void calculateWeightAdjustmentUsingPositiveDelta_ReturnsExpected()
        {
            var xValue = 1.5;
            var xWeight = .00060;
            var delta = 2;

            var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta);
            var actual = Math.Round(weightAdjustment, 4);
            var expected = -.0024;
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void calculateWeightAdjustmentUsingNegativeDelta_ReturnsExpected()
        {
            var xValue = 1.5;
            var xWeight = .00060;
            var delta = -2;

            var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta);
            var actual = Math.Round(weightAdjustment, 5);
            var expected = .0036;
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void calculateBiasAdjustmentUsingPositiveDelta_ReturnsExpected()
        {
            var bias = 0.00364;
            var delta = 2;
            var expected = .00164;
            var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void calculateBiasAdjustmentUsingNegativeDelta_ReturnsExpected()
        {
            var bias = 0.00364;
            var delta = -2;
            var expected = .00564;
            var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runAdjustmentUsingMatchingData_ReturnsExpected()
        {

            var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, -1.0);
            var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
            var expected = .0065;
            var actual = Math.Round(adjustedWeights.weights[0],4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runAdjustmentUsingNegativeData_ReturnsExpected()
        {
            weightedConfirmedObservation weightedConfirmedObservation = 
                new NeuralNetworks.weightedConfirmedObservation(_weightedObservation, 1.0);
            var adjustmentInput = new adjustmentInput(weightedConfirmedObservation, _bias, _alpha, -1.0);
            var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
            var expected = .0125;
            var actual = Math.Round(adjustedWeights.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runAdjustmentUsingPositiveData_ReturnsExpected()
        {
            var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, 1.0);
            var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
            var expected = .0005;
            var actual = Math.Round(adjustedWeights.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

With these functions ready, I could run a training cycle for a given observation

    member this.runTrainingCycle (cycleTrainingInput:cycleTrainingInput) =
        let neuronTrainingInput = 
            {neuronInput.weightedObservation=cycleTrainingInput.weightedConfirmedObservation.weightedObservation;
            neuronInput.bias=cycleTrainingInput.bias}
        let neuronResult = this.runNeuron(neuronTrainingInput)
        let activationResult = this.runActivation(neuronResult)
        let adjustmentInput = {weightedConfirmedObservation=cycleTrainingInput.weightedConfirmedObservation;
                                bias=cycleTrainingInput.bias;alpha=cycleTrainingInput.alpha;
                                yActual=activationResult}
        this.runAdjustment(adjustmentInput)

        [TestMethod]
        public void runTrainingCycleUsingNegativeData_ReturnsExpected()
        {
            var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha);
            var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput);
            var expected = .0125;
            var actual = Math.Round(adjustmentOutput.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runTrainingCycleUsingPositiveData_ReturnsExpected()
        {
            var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha);
            var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput);
            var expected = .0065;
            var actual = Math.Round(adjustmentOutput.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

And then I could run a cycle for each of the observations in the training set, a rotation. I am not happy that I am mutating the weights and biases here, though I am not sure how to fix that. I looked for a Seq.Scan function where the results of a function applied to the 1st element of a Seq is used in the input of the next –> all I could see were examples of threading a collector of int (like Seq.mapi). This will be something I will ask the functional ninjas when I see them again.

    member this.runTrainingRotation(rotationTrainingInput: rotationTrainingInput)=
        let mutable weights = rotationTrainingInput.weights
        let mutable bias = rotationTrainingInput.bias
        let alpha = rotationTrainingInput.alpha
        for i=0 to rotationTrainingInput.confirmedObservations.Count-1 do
            let currentConfirmedObservation = rotationTrainingInput.confirmedObservations.[i]
            let xws = Seq.zip currentConfirmedObservation.observation.xValues weights
            let xws' = new List<(float*float)>(xws)
            let weightedObservation = {xws=xws'}
            let weightedTrainingObservation = {weightedObservation=weightedObservation;yExpected=currentConfirmedObservation.yExpected}
            let cycleTrainingInput = {
                                cycleTrainingInput.weightedConfirmedObservation=weightedTrainingObservation;
                                cycleTrainingInput.bias=bias;
                                cycleTrainingInput.alpha=alpha}
            let cycleOutput = this.runTrainingCycle(cycleTrainingInput)
            weights <- cycleOutput.weights
            bias <- cycleOutput.bias
        {adjustmentOutput.weights=weights; adjustmentOutput.bias=bias}

        [TestMethod]
        public void runTrainingRotationUsingNegativeData_ReturnsExpected()
        {

            var xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(4.0);
            var observation = new observation(xValues);
            var yExpected = -1.0;
            var confirmedObservation0 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(1.5);
            xValues.Add(2.0);
            yExpected = -1.0;
            var confirmedObservation1 = new confirmedObservation(observation, yExpected);

            var trainingObservations = new List<confirmedObservation>();
            trainingObservations.Add(confirmedObservation0);
            trainingObservations.Add(confirmedObservation1);
            
            var weights = new List<double>();
            weights.Add(.0065);
            weights.Add(.0123);

            var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha);
            var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput);

            var expected = -0.09606;
            var actual = Math.Round(trainingRotationOutput.bias, 5);
            Assert.AreEqual(expected, actual);

        }

        [TestMethod]
        public void runTrainingRotationUsingPositiveData_ReturnsExpected()
        {
            var xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(4.0);
            var observation = new observation(xValues);
            var yExpected = 1.0;
            var confirmedObservation0 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(1.5);
            xValues.Add(2.0);
            yExpected = 1.0;
            var confirmedObservation1 = new confirmedObservation(observation, yExpected);

            var trainingObservations = new List<confirmedObservation>();
            trainingObservations.Add(confirmedObservation0);
            trainingObservations.Add(confirmedObservation1);


            var weights = new List<double>();
            weights.Add(.0065);
            weights.Add(.0123);

            var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha);
            var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput);

            var expected = -.09206;
            var actual = Math.Round(trainingRotationOutput.bias, 5);
            Assert.AreEqual(expected, actual);

        }

With the rotation done, I could write the train function which runs rotations for N number of times to tune the weights and biases:

    member this.train(trainInput:trainInput) = 
        let currentObservation = trainInput.confirmedObservations.[0].observation
        let weights = this.initializeWeights(currentObservation.xValues,trainInput.weightSeedValue)
        let weights' = new List<float>(weights)
        let mutable bias = this.initializeBias(trainInput.biasSeedValue)
        let alpha = trainInput.alpha
        for i=0 to trainInput.maxEpoches do
            let rotationTrainingInput={rotationTrainingInput.confirmedObservations=trainInput.confirmedObservations;
                rotationTrainingInput.weights = weights'; 
                rotationTrainingInput.bias=bias;
                rotationTrainingInput.alpha=trainInput.alpha}
            this.runTrainingRotation(rotationTrainingInput) |> ignore
        {adjustmentOutput.weights=weights'; adjustmentOutput.bias=bias}

        [TestMethod]
        public void trainUsingTestData_RetunsExpected()
        {
            var xValues = new List<double>();
            xValues.Add(1.5);
            xValues.Add(2.0);
            var observation = new observation(xValues);
            var yExpected = -1.0;
            var confirmedObservation0 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(2.0);
            xValues.Add(3.5);
            observation = new observation(xValues);
            yExpected = -1.0;
            var confirmedObservation1 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(5.0);
            observation = new observation(xValues);
            yExpected = -1.0;
            var confirmedObservation2 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(3.5);
            xValues.Add(2.5);
            observation = new observation(xValues);
            yExpected = -1.0;
            var confirmedObservation3 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(4.5);
            xValues.Add(5.0);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation4 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(5.0);
            xValues.Add(7.5);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation5 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(5.5);
            xValues.Add(8.0);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation6 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(6.0);
            xValues.Add(6.0);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation7 = new confirmedObservation(observation, yExpected);


            var trainingObservations = new List<confirmedObservation>();
            trainingObservations.Add(confirmedObservation0);
            trainingObservations.Add(confirmedObservation1);
            trainingObservations.Add(confirmedObservation2);
            trainingObservations.Add(confirmedObservation3);
            trainingObservations.Add(confirmedObservation4);
            trainingObservations.Add(confirmedObservation5);
            trainingObservations.Add(confirmedObservation6);
            trainingObservations.Add(confirmedObservation7);

            var random = new Random();
            var weightSeedValue = random.NextDouble();
            var biasSeedValue = random.NextDouble();
            var alpha = .001;
            var maxEpoches = 100;

            var trainInput = new trainInput(trainingObservations, weightSeedValue, biasSeedValue, alpha, maxEpoches);
            var trainOutput = _perceptron.train(trainInput);

            Assert.IsNotNull(trainOutput);

        }

With the training out of the way, I could concentrate on the prediction. The prediction was much easier because there are no adjustments and the rotation is run once. The data structures are also simpler because I don’t have to pass in the knownY values. I also only have 1 covering (all be it long) unit test that looks that the results of the prediction.

    member this.runPredictionCycle (cyclePredictionInput:cyclePredictionInput) = 
        let neuronInput = 
            {neuronInput.weightedObservation=cyclePredictionInput.weightedObservation;
            neuronInput.bias=cyclePredictionInput.bias}
        let neuronResult = this.runNeuron(neuronInput)
        this.runActivation(neuronResult)

    member this.runPredictionRotation (rotationPredictionInput:rotationPredictionInput) = 
        let output = new List<List<float>*float>();
        let weights = rotationPredictionInput.weights
        for i=0 to rotationPredictionInput.observations.Count-1 do
            let currentObservation = rotationPredictionInput.observations.[i];
            let xws = Seq.zip currentObservation.xValues weights
            let xws' = new List<(float*float)>(xws)
            let weightedObservation = {xws=xws'}
            let cyclePredictionInput = {
                            cyclePredictionInput.weightedObservation = weightedObservation;
                            cyclePredictionInput.bias = rotationPredictionInput.bias}
            let cycleOutput = this.runPredictionCycle(cyclePredictionInput)
            output.Add(currentObservation.xValues, cycleOutput)
        output

    member this.predict(predictInput:predictInput) = 
        let rotationPredictionInput = {
                        rotationPredictionInput.observations = predictInput.observations;
                        rotationPredictionInput.weights = predictInput.weights;
                        rotationPredictionInput.bias = predictInput.bias }
        this.runPredictionRotation(rotationPredictionInput)

        [TestMethod]
        public void predictUsingTestData_ReturnsExpected()
        {
            var xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(4.0);
            var observation0 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(0.0);
            xValues.Add(1.0);
            var observation1 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(2.0);
            xValues.Add(5.0);
            var observation2 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(5.0);
            xValues.Add(6.0);
            var observation3 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(9.0);
            xValues.Add(9.0);
            var observation4 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(4.0);
            xValues.Add(6.0);
            var observation5 = new observation(xValues);

            var observations = new List<observation>();
            observations.Add(observation0);
            observations.Add(observation1);
            observations.Add(observation2);
            observations.Add(observation3);
            observations.Add(observation4);
            observations.Add(observation5);


            var weights = new List<double>();
            weights.Add(.0065);
            weights.Add(.0123);

            var bias = -0.0906;

            var predictInput = new predictInput(observations, weights, bias);

            var predictOutput = _perceptron.predict(predictInput);
            Assert.IsNotNull(predictOutput);
        }

When I run all of the unit tests the all run green:

With the Perceptron created, I can now go back and change the code and figure out:

1) Why my weights across the XValues are the same (wrong!)

2) How to implement a more idomatic/recursive way of running rotations so I can remove the mutation

With my unit tests running green, I know I am covered in case I make a mistake

Filed under F#, Machine Learning

Neural Network Part 2: Perceptrons

July 22, 2014 1 Comment

I started working though the second chapter of McCaffrey’s book Neural Networks Using C# Succinctly to see if I could write the examples using F#.

McCaffrey’s code is tough to read though because of its emphasis on loops and global mutable variables. I read though his description and this is how <I think> the Perceptron should be constructed.

The inputs are a series of independent variables (in this case age and income) and the output is a single dependent variable (in this case party affiliation). The values have been encoded and normalized like in this post here.

An example of the input (from page 31 of his book) is:

Or in a more abstract manner:

In terms of data structures, individual inputs (each row) is placed into an array of floats and the output is a single float

I call this single set of inputs an “observation” (my words, not McCaffrey).

Looking at McCaffrey’s example for a perceptron Input-Output,

all of the variables you need are not included. Here is what you need:

Where A0 and B0 are the same as X0 and X1 respectively in his diagram. Also, McCaffrey uses the word “Perceptron” to mean two different concepts: the entire system as a whole and the individual calculation for a given list of X and Bias. I am a big believer of domain ubiquitous languages so I am calling the individual calculation a neuron.

Once you run these values through the neuron for the 1st observation, you might have to alter the Weights and Bias based on the (Y)result. Therefore, the data structure coming out of the Neuron is

These values are feed into the adjustment function to alter the weights and bias with the output as

I am calling this process of taking the a single observation, the xWeights, , and the bias and turning them into a series of weights and bais as a “cycle” (my words, not McCaffrey)

The output of a cycle is then fed with the next observation and the cycle repeats for as many observations as there are fed into the system.

I am calling the process of running a cycle for each observation in the input dataset a rotation (my words, not McCaffrey) and that the perceptron runs rotations for an x number of times to train itself.

Finally, the Perceptron takes a new set of observations where the Y is not known and runs a Rotation once to predict what the Y will be.

So with that mental image in place, the coding became much easier. Basically, there was a 1 to 1 correspondence of F# functions to each step laid out. I started with an individual cycle

type cycleInput = {xValues:float list;yExpected:float;mutable weights:float list;mutable bias:float;alpha:float}
 
let runNeuron (input:cycleInput) = 
    let valuesAndWeights = input.xValues |> List.zip input.weights
    let output = valuesAndWeights 
                    |> List.map(fun (xValue, xWeight) -> xValue*xWeight) 
                    |> List.sumBy(fun x -> x)
    output + input.bias
 
let runActivation input = 
    if input < 0.0 then -1.0 else 1.0

I used record types all over the place in this code just so I could keep things straight in my head. McCaffrey uses ambiguously-named arrays and global variables. Although this makes my code a bit more wordy (esp for functional people), I think the increased readability is worth the trade-off.

In any event, with the Neuron and Activation calc out of the way, I created the functions that adjust the weights and bias:

let calculateWeightAdjustment(xValue, xWeight, alpha, delta) =
    match delta > 0.0, xValue >= 0.0 with
        | true,true -> xWeight – (alpha * delta * xValue)
        | false,true -> xWeight + (alpha * delta * xValue)
        | true,false -> xWeight – (alpha * delta * xValue)
        | false,false -> xWeight + (alpha * delta * xValue)
 
let calculateBiasAdjustment(bias, alpha, delta) =
    match delta > 0.0 with
        | true -> bias – (alpha * delta)
        | false -> bias + (alpha * delta)

This code is significantly different than the for, nested if that McCaffrey uses.

I maintain using this kind of pattern matching makes the intention much easier to comprehend. I also split out the adjustment of the weights and the adjustment of the bias into individual functions.

With these functions ready, I created an input and output record type and implemented the adjustment function

let runAdjustment (input:adjustmentInput) =
    match input.yExpected = input.yActual with
        | true -> {weights=input.weights;bias=input.bias;yActual=input.yActual}
        | false ->
            let delta = input.yActual – input.yExpected
            let valuesAndWeights = input.xValues |> List.zip input.weights
            let weights' =  valuesAndWeights |> List.map(fun (xValue, xWeight) -> calculateWeightAdjustment(xValue,xWeight,input.alpha,delta))
            let bias' = calculateBiasAdjustment(input.bias,input.alpha,delta)
            {weights=weights';bias=bias';yActual=input.yActual}

There is not a corresponding method in McCaffrey’s code, rather he just does some Array.copy and mutates the global variables in the Update method. I am not a fan of side-effect programming so I created a function that explicitly does the modification.

And to wrap up the individual cycle:

let runCycle (cycleInput:cycleInput) =
    let neuronResult = runNeuron(cycleInput)
    let activationResult = runActivation(neuronResult)
    let adjustmentInput = {xValues=cycleInput.xValues;weights=cycleInput.weights;yExpected=cycleInput.yExpected;
                            bias=cycleInput.bias;alpha=cycleInput.alpha;yActual=activationResult}
    runAdjustment(adjustmentInput)

Up next is to run the cycle for each of the observations (called a rotation)

type observation = {xValues:float list;yExpected:float}
type rotationInput = {observations: observation list;mutable weights:float list;mutable bias:float;alpha:float}
type trainingRotationOutput = {weights:float list; bias:float}
type predictionRotationOutput = {observation: observation;yActual:float}
 
let runTrainingRotation(rotationInput: rotationInput)=
    for i=0 to rotationInput.observations.Length do
        let observation = rotationInput.observations.[i]
        let neuronInput = {cycleInput.xValues=observation.xValues;cycleInput.yExpected=observation.yExpected;cycleInput.weights=rotationInput.weights;
                            cycleInput.bias=rotationInput.bias;cycleInput.alpha=rotationInput.alpha}
        let cycleOutput = runCycle(neuronInput)
        rotationInput.weights <- cycleOutput.weights
        rotationInput.bias <- cycleOutput.bias
    {weights=rotationInput.weights; bias=rotationInput.bias}

Again, note the liberal use of records to keep the inputs and outputs clear. I also created a prediction rotation that is designed to be run only once that does not alter the weights and bias.

let runPredictionRotation(rotationInput: rotationInput)=
    let output = new System.Collections.Generic.List<predictionRotationOutput>()
    for i=0 to rotationInput.observations.Length do
        let observation = rotationInput.observations.[i]
        let neuronInput = {cycleInput.xValues=observation.xValues;cycleInput.yExpected=observation.yExpected;cycleInput.weights=rotationInput.weights;
                            cycleInput.bias=rotationInput.bias;cycleInput.alpha=rotationInput.alpha}
        let cycleOutput = runCycle(neuronInput)
        let predictionRotationOutput = {observation=observation;yActual=cycleOutput.yActual}
        output.Add(predictionRotationOutput)   
    output

With the rotations done, the last step was to create the Perceptron to train and then predict:

type perceptronInput = {observations: observation list;weights:float list;bias:float}
type perceptronOutput = {weights:float list; bias:float}
 
let initializeWeights(xValues, randomSeedValue) =
    let lo = -0.01
    let hi = 0.01
    let xWeight = (hi-lo) * randomSeedValue + lo
    xValues |> List.map(fun w -> xWeight)
 
let initializeBias(randomSeedValue) = 
    let lo = -0.01
    let hi = 0.01
    (hi-lo) * randomSeedValue + lo
 
let runTraining(perceptronInput: perceptronInput, maxEpoches:int) =
    let random = System.Random()
    let alpha = 0.001
    let baseObservation = perceptronInput.observations.[0]
    let mutable weights = initializeWeights(baseObservation.xValues,random.NextDouble())       
    let mutable bias = initializeBias(random.NextDouble())
    let rotationList = [0..maxEpoches]
    for i=0 to maxEpoches do
        let rotationInput = {observations=perceptronInput.observations;weights=weights;bias=bias;alpha=alpha}
        let rotationOutput = runTrainingRotation(rotationInput)
        weights <- rotationOutput.weights
        bias <- rotationOutput.bias
    {weights=weights;bias=bias}
 
let runPrediction(perceptronInput: perceptronInput, weights: float list, bias: float) =
    let random = System.Random()
    let alpha = 0.001
    let rotationInput = {observations=perceptronInput.observations;weights=weights;bias=bias;alpha=alpha}
    runPredictionRotation(rotationInput)

Before I go too much further, I have a big code smell. I am iterating and using the mutable keyword. I am not sure how to take the results of a function that is applied to the 1st element in a sequence and then input that into the second. I need to do that with the weights and bias data structures –> each time it is used in a expression, it need to change and feed into the next expression. I think the answer is the List.Reduce, so I am going to pick this up after looking at that in more detail. I also need to implement the shuffle method so that that cycles are not called in the same order across rotations….

Filed under F#, Machine Learning

Infer.NET

August 6, 2013 Leave a comment

I was on vacation last week so I decided to have some fun with Infer.Net. As someone interested in F#/Machine learning, Infer.Net seemed like a cool side project. I was not disappointed. I downloaded the most recent bits (though no NuGet) and wired up a basic problem of determining the probability of 2 even-sided coins to both come up heads:

Console.WriteLine("Start");
 
Variable<bool> firstCoin = Variable.Bernoulli(0.5);
Variable<bool> secondCoin = Variable.Bernoulli(0.5);
Variable<bool> bothHeads = firstCoin & secondCoin;
 
InferenceEngine engine = new InferenceEngine();
Console.WriteLine("Probability both coins are heads: " + engine.Infer(bothHeads));
 
Console.WriteLine("End");
Console.ReadKey();

Sure enough:

I can’t wait to dig into the other tutorials and then apply the inference to some of the real data sets I have collected…

Filed under Machine Learning

Machine Learning For Hackers: Chapter 1, Part 2

June 11, 2013 Leave a comment

I then wanted to show the data graphically. I Added a ASP.NET Web Form project to my solution and added a Chart to the page. That Chart points to an objectDataSource that consumes the UFO Library method:

<asp:Chart ID="Chart1" runat="server" DataSourceID="ObjectDataSource1">
    <series>
        <asp:Series Name="Series1" XValueMember="Item4" YValueMembers="Item1">
        </asp:Series>
    </series>
    <chartareas>
        <asp:ChartArea Name="ChartArea1">
        </asp:ChartArea>
    </chartareas>
</asp:Chart>
<asp:ObjectDataSource ID="ObjectDataSource1" runat="server" 
    SelectMethod="GetDetailData" 
    TypeName="Tff.MachineLearningWithFSharp.Chapter01.UFOLibrary">
</asp:ObjectDataSource>

When I ran it, I got this:

So that is cool that the Chart control can access the data and show ‘something’. I then read this tutorial about how to show every state on the X Axis

Unfortunately, I was diving down too deep into the weeds of charting controls, which is really not where I want to be. I then decided to build a function that aggregates the data

member this.GetSummaryData() =
    let subset =
        this.GetDetailData()
        |> Seq.map(fun (a,b,c,d,e,f,g) -> a,d)
        |> Seq.map(fun (a,b) -> 
            a.Year,
            b)
 
    let summary =
        subset 
        |> Seq.groupBy fst 
        |> Seq.map (fun (a, b) -> (a, b 
            |> Seq.countBy snd))

Sure enough, when I look on my console app

I then decided to switch it so that the state would come up first and each year of UFO sightings would be shown (basically switching the Seq.Map)

|> Seq.map(fun (a,b) -> 
    b,
    a.Year)

And now:

So then I added another method that only returns a state’s aggregate data:

member this.GetSummaryData(stateCode: string) =
    let stateOnly =
        this.GetSummaryData()
        |> Seq.filter(fun (a,_) -> a = stateCode)
    stateOnly

And I changed the ASP.NET UI to show that state:

<div class="content-wrapper">
    <asp:DropDownList ID="StatesDropList" runat="server" Width="169px" Height="55px">
        <asp:ListItem Value="AZ"></asp:ListItem>
        <asp:ListItem>MD</asp:ListItem>
        <asp:ListItem>CA</asp:ListItem>
        <asp:ListItem>NC</asp:ListItem>
 
    </asp:DropDownList>
    <br />
    <br />
    <asp:Chart ID="Chart1" runat="server" DataSourceID="ObjectDataSource1" Width="586px">
        <series>
            <asp:Series Name="Series1" XValueMember="item1" YValueMembers="item2">
            </asp:Series>
        </series>
        <chartareas>
            <asp:ChartArea Name="ChartArea1">
                <AxisX IsLabelAutoFit="False">
                    <LabelStyle Interval="Auto" IsStaggered="True" />
                </AxisX>
            </asp:ChartArea>
        </chartareas>
    </asp:Chart>
    <br />
    <asp:ObjectDataSource ID="ObjectDataSource1" runat="server" 
        SelectMethod="GetSummaryData" 
        TypeName="Tff.MachineLearningWithFSharp.Chapter01.UFOLibrary">
        <SelectParameters>
            <asp:ControlParameter ControlID="StatesDropList" DefaultValue="NC" Name="stateCode" PropertyName="SelectedValue" Type="String" />
        </SelectParameters>
    </asp:ObjectDataSource>
</div>

The problem is this:

I need to flatten the subTuple so that only native types are sent to the ODS.

Filed under F#, Machine Learning

Machine Learning for Hackers: Using F#

May 21, 2013 1 Comment

I decided I wanted to learn more about F# so my Road Alert project. I started by watching this great video. After reviewing it a couple of times, I realized that I could try and do chapter 1 of Machine Learning for Hackers using F#.

Since I already had the data from this blog post, I just had to follow Luca’s example. I wrote the following code in an F# project in Visual Studio 2012.

open System.IO
type UFOLibrary() = 
    member this.GetDetailData() =
        let path = "C:\Users\Jamie\Documents\Visual Studio 2012\Projects\MachineLearningWithFSharp_Solution\Tff.MachineLearningWithFSharp.Chapter01\ufo_awesome.txt"
        let fileStream = new FileStream(path,FileMode.Open,FileAccess.Read)
        let streamReader = new StreamReader(fileStream)
        let contents = streamReader.ReadToEnd()
        let usStates = [|"AL";"AK";"AZ";"AR";"CA";"CO";"CT";"DE";"DC";"FL";"GA";"HI";"ID";"IL";"IN";"IA";
                         "KS";"KY";"LA";"ME";"MD";"MA";"MI";"MN";"MS";"MO";"MT";"NE";"NV";"NH";"NJ";"NM";
                         "NY";"NC";"ND";"OH";"OK";"OR";"PA";"RI";"SC";"SD";"TN";"TX";"UT";"VT";"VA";"WA";
                          "WV";"WI";"WY"|]
        let cleanContents =
            contents.Split([|'\n'|])
            |> Seq.map(fun line -> line.Split([|'\t'|]))
            Seq.head()

I then added a C# console project to the solution and added the following code:

static void Main(string[] args)
{
    Console.WriteLine("Start");
    UFOLibrary ufoLibrary = new UFOLibrary();
 
    foreach (String currentString in ufoLibrary.GetDetailData())
    {
        Console.WriteLine(currentString);
    }
    Console.WriteLine("End");
    Console.ReadKey();
}

Sure enough, when I hit F5

How cool is it to call F# code from a C# project and it just works? I feel a whole new world of possibilites just opened to me.

I then went back to the book and saw that they used the head function in R that returns the top 10 rows of data. The F# head only returns the top 1 so I had to make the following change to my F# to duplicate the effect:

let cleanContents =
    contents.Split([|'\n'|])
    |> Seq.map(fun line -> line.Split([|'\t'|]))
    |> Seq.take(10)

I then had to remove the defective rows that had malformed data. To do this, I went back to the F# code and changed it to this

let cleanContents =
    contents.Split([|'\n'|])
    |> Seq.map(fun line -> line.Split([|'\t'|]))

I then went back to the Console app to change it like this:

Console.WriteLine("Start");
UFOLibrary ufoLibrary = new UFOLibrary();
IEnumerable<String> rows = ufoLibrary.GetDetailData();
Console.WriteLine(String.Format("Number of rows: {0}", rows.Count()));
Console.WriteLine("End");
Console.ReadKey();

And I see this when I hit F5

So now I have a baseline of 61,394 rows.

My 1st step is to removed rows that do not have 6 columns. To do that, I changed my code to this:

Console.WriteLine("Start");
UFOLibrary ufoLibrary = new UFOLibrary();
IEnumerable<String> rows = ufoLibrary.GetDetailData();
Console.WriteLine(String.Format("Number of rows: {0}", rows.Count()));
Console.WriteLine("End");
Console.ReadKey();

and when I hit F5, I can see that the number of records has dropped:

I then want to removed the bad date fields the way they did it in the book – all dates have to be 8 characters in length, no more, no less.

Going back to the F# code, I added this line

|> Seq.filter(fun values -> values.[0].Length = 8)

and sure enough, fewer records in my dataset:

And finally applying the same logic to the second column – which is also a date

|> Seq.filter(fun values -> values.[1].Length = 8)

Which raises eyebrows, I assume there would be some malformed data in the 2ndcolumn independent of the 1st column, but I guess not.

I then wanted to convert the 1st two columns from strings into DateTimes. Going back to Luca’s examples, I did this:

|> Seq.map(fun values -> 
    System.DateTime.Parse(values.[0]),
    System.DateTime.Parse(values.[1]),
    values.[2],
    values.[2],
    values.[3],
    values.[4],
    values.[5])

Interestingly, I then went back to my Console application and got this

Error 1 Cannot implicitly convert type ‘System.Collections.Generic.IEnumerable<System.Tuple<System.DateTime,System.DateTime,string,string,string,string>>’ to ‘System.Collections.Generic.IEnumerable<string[]>’. An explicit conversion exists (are you missing a cast?)

So I then did this:

   1: var rows = ufoLibrary.GetData();

so I can compile again. When I ran it, I got his exception:

So it looks like R can handle YYYYMMDD while F# DateTime.Parse() can not. So I went back to The different ways to parse in .NET I changed the parsing to this:

System.DateTime.ParseExact(values.[0],"yyyymmdd",System.Globalization.CultureInfo.InvariantCulture),
System.DateTime.ParseExact(values.[1],"yyyymmdd",System.Globalization.CultureInfo.InvariantCulture),

When I ran it, I got this:

Which I am not sure is progress. so then it hit me that the data in the strings might be out of bounds – for example a month of “13”. So I added the following filters to the dataset:

|> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(0,4)) > 1900)
|> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(0,4)) > 1900)
|> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(0,4)) < 2100)
|> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(0,4)) < 2100)
|> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(4,2)) > 0)
|> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(4,2)) > 0)
|> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(4,2)) <= 12)
|> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(4,2)) <= 12)      
|> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(6,2)) > 0)
|> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(6,2)) > 0)
|> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(6,2)) <= 31)
|> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(6,2)) <= 31)

Sure enough, now when I run it:

Which matches what the book’s R example.

I then wanted to match what the book does in terms of cleaning the city,state field (column). We are only interested in data from the united states that follows the “City,State” pattern. The R examples does some conditional logic to clean this data, up, which I didn’t want to do in F#.

So I added this filter than split the City,State column and checked that the state value is only 2 characters in length R uses the “Clean” keyword to remove white space, F# uses “Trim()”

|> Seq.filter(fun values -> values.[2].Split(',').[1].Trim().Length = 2)

Next, the book limits the location values to only the Unites States. To do that, it creates a list of values of all 50 postal codes (lower case) to then compare the state portion of the location field. To that end, I added a string array like so:

let usStates = [|"AL";"AK";"AZ";"AR";"CA";"CO";"CT";"DE";"DC";"FL";"GA";"HI";"ID";"IL";"IN";"IA";
                 "KS";"KY";"LA";"ME";"MD";"MA";"MI";"MN";"MS";"MO";"MT";"NE";"NV";"NH";"NJ";"NM";
                 "NY";"NC";"ND";"OH";"OK";"OR";"PA";"RI";"SC";"SD";"TN";"TX";"UT";"VT";"VA";"WA";
                  "WV";"WI";"WY"|]

I then add this filter (took me about 45 minutes to figure out):

|> Seq.filter(fun values -> Seq.exists(fun elem -> elem = values.[2].Split(',').[1].Trim().ToUpperInvariant()) usStates)

So now I am 1/2 way done with Chapter 1 – the data has now been cleaned and is ready to be analyzed. Here is the code that I have so far:

member this.GetDetailData() =
    let path = "C:\Users\Jamie\Documents\Visual Studio 2012\Projects\MachineLearningWithFSharp_Solution\Tff.MachineLearningWithFSharp.Chapter01\ufo_awesome.txt"
    let fileStream = new FileStream(path,FileMode.Open,FileAccess.Read)
    let streamReader = new StreamReader(fileStream)
    let contents = streamReader.ReadToEnd()
    let usStates = [|"AL";"AK";"AZ";"AR";"CA";"CO";"CT";"DE";"DC";"FL";"GA";"HI";"ID";"IL";"IN";"IA";
                     "KS";"KY";"LA";"ME";"MD";"MA";"MI";"MN";"MS";"MO";"MT";"NE";"NV";"NH";"NJ";"NM";
                     "NY";"NC";"ND";"OH";"OK";"OR";"PA";"RI";"SC";"SD";"TN";"TX";"UT";"VT";"VA";"WA";
                      "WV";"WI";"WY"|]
    let cleanContents =
        contents.Split([|'\n'|])
        |> Seq.map(fun line -> line.Split([|'\t'|]))
        |> Seq.filter(fun values -> values |> Seq.length = 6)
        |> Seq.filter(fun values -> values.[0].Length = 8)
        |> Seq.filter(fun values -> values.[1].Length = 8)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(0,4)) > 1900)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(0,4)) > 1900)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(0,4)) < 2100)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(0,4)) < 2100)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(4,2)) > 0)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(4,2)) > 0)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(4,2)) <= 12)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(4,2)) <= 12)      
        |> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(6,2)) > 0)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(6,2)) > 0)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[0].Substring(6,2)) <= 31)
        |> Seq.filter(fun values -> System.Int32.Parse(values.[1].Substring(6,2)) <= 31)
        |> Seq.filter(fun values -> values.[2].Split(',').[1].Trim().Length = 2)
        |> Seq.filter(fun values -> Seq.exists(fun elem -> elem = values.[2].Split(',').[1].Trim().ToUpperInvariant()) usStates)
        |> Seq.map(fun values -> 
            System.DateTime.ParseExact(values.[0],"yyyymmdd",System.Globalization.CultureInfo.InvariantCulture),
            System.DateTime.ParseExact(values.[1],"yyyymmdd",System.Globalization.CultureInfo.InvariantCulture),
            values.[2].Split(',').[0].Trim(),
            values.[2].Split(',').[1].Trim().ToUpperInvariant(),
            values.[3],
            values.[4],
            values.[5])
    cleanContents

I now want to finish up the chapter where the analysis happens. R uses some built-in plotting libraries (ggplot). Following Luca’s example of this

I went to the flying frogs libraries and, alas, there is no longer a free edition.

So I am bit stuck. I’ll continue to work on it for next week’s blog…

Filed under F#, Machine Learning

Machine Learning For Hackers: Using MSFT technologies

April 16, 2013 Leave a comment

So I picked up Machine Learning for Hackers and started going though the 1st couple of chapters last night. I love the premise that developers can learn something about machine learning using some easily-understood examples. Why I didn’t love was that it used R. After reading the 1st chapter, I said to myself, “self, everything they do in this chapter you can do in MSFT office.” Now I get that this is supposed to be a simple example to get the reader up and running with R, but I thought, “Hey, I wonder how much of this book can I do using the MSFT stack?”

Chapter 1 was all about cleaning up data to get it ready for analysis. This brought me back to my 1st job out of school (marketing analyst) when I spent more time collecting and cleaning the data than actually analyzing the data.

My 1st stop was to download the sample UFO dataset from the publisher site and save it locally with a .txt extension. I then imported the data into Microsoft access: Note that I assigned column names and made sure that the Long Description is memo:

Note that there were

With that base data imported, I then added an additional field to the table called valid. The book follows the process of analyzing each column’s data and removing rows that are not analyzable. I learned the hard way many years ago that you should never permanently remove data because you need to be able to trace the data set back to its initial state for audit purposes. I like the ability to make queries upon queries in Access to layer up data validation steps.

For example, in the book, the first validation is to parse the DateOccured field into year/month/columns – the problem is that some columns don’t have valid data. To this 2 step process, the first is to aggregate the values in the column and see what the valid values look like:

Sure enough, there are over 250 rows of data that cannot be used. I created a table of the aggregate dates and the inner joined that table back to the base datatable. Rows with invalid data was flagged as Invalid:

I then divided the Location field into City, State:

And then created table for valid states – the same way I did for the valid DateOccured. I had to remove about 10% of the dataset – because of both malformed data and the fact that the dataset is world-wide and the book’s example is only interested in the USA. The fact that 90% of the world’s UFO sightings is in America probably says something, though I am not sure what.

In any event, then exported the data into Excel, threw a pivot table on the data and added some conditional formatting:

Note that some malformed data slipped in (“NT”, “NE”, etc…) but I was too lazy to go back and clean it up.

Note that this is total sightings, not per-capita so you would expect that the states with larger populations to have more sightings (assuming that UFOs are targeting the USA evenly). I think the color presentation is more effective and really draw your eye to CA and WA much more than the 50 histogram display that is found in the book.

I then filtered the data to only greater than 1990 like the book did and changed the pivot table to filter on state. Here is California:

The color really shows how the number of sightings are increasing in CA and seem to be more consistent in WA. I do wonder what happened in 1995 in WA to cause that spike?

The next chapter is about regression analysis, so I assume that I can use the built-in functions of Excel for that too…

Filed under Book Review, Machine Learning, MS Office

Jamie Dixon's Home

Neural Network Part 3: Perceptrons

Neural Network Part 2: Perceptrons

Infer.NET

Machine Learning For Hackers: Chapter 1, Part 2

Machine Learning for Hackers: Using F#

Machine Learning For Hackers: Using MSFT technologies

Categories

Recent Posts

Archives

Blogroll

Meta