Controlling Servos Using Netdunio and Phidgets →

Neural Network Part 3: Perceptrons

August 5, 2014 1 Comment

I went back to my code for building a Perceptron and I made some changes. I realized that although McCaffrey combines the code together, there are actually two actions for the perceptron: training and predicting. I created a diagram to help me keep the functions that I need for each in mind:

I also skeletoned out some data structures that I think I need:

With the base diagrams out of the way, I created different data structures that were tailored to each action. These are a bit different than the diagrams –> I didn’t go back and update the diagrams because the code is where you would look to see how the system works:

type observation = {xValues:float List}
type weightedObservation = {xws:(float*float) List}
type confirmedObservation = {observation:observation;yExpected:float}
type weightedConfirmedObservation = {weightedObservation:weightedObservation;yExpected:float}
type neuronInput = {weightedObservation:weightedObservation;bias:float}

type cycleTrainingInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float}
type adjustmentInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float;yActual:float}
type adjustmentOutput = {weights:float List; bias:float}
type rotationTrainingInput = {confirmedObservations:confirmedObservation List;weights:float List;bias:float;alpha:float}
type trainInput = {confirmedObservations:confirmedObservation List; weightSeedValue:float;biasSeedValue:float;alpha:float; maxEpoches:int}

type cyclePredictionInput = {weightedObservation:weightedObservation;bias:float}
type rotationPredictionInput = {observations:observation List;weights:float List;bias:float}
type predictInput = {observations:observation List;weights:float List;bias:float}

Note that I am composing data structures with the base being an observation. The observation is a list of different xValues for a given, well, observation. The weighted observation is the XValue paired with the perceptron weights. The confirmedObservation is for training –> given an observation, what was the actual output?

With the data structures out of the way, I went to the Perceptron and added in the basic functions for creating seed values:

    member this.initializeWeights(xValues, randomSeedValue) =
        let lo = -0.01
        let hi = 0.01
        let xWeight = (hi-lo) * randomSeedValue + lo
        xValues |> Seq.map(fun w -> xWeight)

    member this.initializeBias(randomSeedValue) = 
        let lo = -0.01
        let hi = 0.01
        (hi-lo) * randomSeedValue + lo

Since I was doing TDD, here are the unit tests I used for these functions:

        [TestMethod]
        public void initializeWeightsUsingHalfSeedValue_ReturnsExpected()
        {
            var weights = _perceptron.initializeWeights(_observation.xValues, .5);
            var weightsList = new List<double>(weights);

            var expected = 0.0;
            var actual = weightsList[0];
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeWeightsUsingLessThanHalfSeedValue_ReturnsExpected()
        {
            var weights = _perceptron.initializeWeights(_observation.xValues, .4699021627);
            var weightsList = new List<double>(weights);

            var expected = -0.00060;
            var actual = Math.Round(weightsList[0],5);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeBiasesUsingHalfSeedValue_ReturnsExpected()
        {
            var expected = 0.0;
            var actual = _perceptron.initializeBias(.5);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeBiasesUsingLessThanHalfSeedValue_ReturnsExpected()
        {
            var expected = -0.00060;
            var bias = _perceptron.initializeBias(.4699021627);
            var actual = Math.Round(bias, 5);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void initializeBiasesUsingGreaterThanHalfSeedValue_ReturnsExpected()
        {
            var expected = 0.00364;
            var bias = _perceptron.initializeBias(.6820621978);
            var actual = Math.Round(bias,5);
            Assert.AreEqual(expected, actual);
        }

I then created a base neuron and activation function that would work for both training and predicting:

    member this.runNeuron(input:neuronInput) = 
        let xws = input.weightedObservation.xws
        let output = xws 
                    |> Seq.map(fun (xValue,xWeight) -> xValue*xWeight) 
                    |> Seq.sumBy(fun x -> x)
        output + input.bias

    member this.runActivation(input) = 
        if input < 0.0 then -1.0 else 1.0

        [TestMethod]
        public void runNeuronUsingNormalInput_ReturnsExpected()
        {
            var expected = -0.0219;
            var perceptronOutput = _perceptron.runNeuron(_neuronInput);
            var actual = Math.Round(perceptronOutput, 4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runActivationUsingNormalInput_ReturnsExpected()
        {
            var expected = -1;
            var actual = _perceptron.runActivation(-0.0219);
            Assert.AreEqual(expected, actual);
        }

I then created the functions for training –> specifically to return adjusted weights and biases based on the result of the activation function

    member this.calculateWeightAdjustment(xValue, xWeight, alpha, delta) =
        match delta > 0.0, xValue >= 0.0 with
            | true,true -> xWeight - (alpha * abs(delta) * xValue)
            | false,true -> xWeight + (alpha * abs(delta) * xValue)
            | true,false -> xWeight - (alpha * abs(delta) * xValue)
            | false,false -> xWeight + (alpha * abs(delta) * xValue)

    member this.calculateBiasAdjustment(bias, alpha, delta) =
        match delta > 0.0 with
            | true -> bias - (alpha * abs(delta))
            | false -> bias + (alpha * abs(delta))

    member this.runAdjustment (input:adjustmentInput) =
        match input.weightedConfirmedObservation.yExpected = input.yActual with
            | true -> 
                let weights = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (x,w) -> w)
                let weights' = new List<float>(weights)
                {adjustmentOutput.weights=weights';adjustmentOutput.bias=input.bias}
            | false ->
                let delta = input.yActual - input.weightedConfirmedObservation.yExpected
                let weights' =  input.weightedConfirmedObservation.weightedObservation.xws 
                                    |> Seq.map(fun (xValue, xWeight) -> 
                                        this.calculateWeightAdjustment(xValue,xWeight,input.alpha,delta))
                                    |> Seq.toList
                let weights'' = new List<float>(weights')                                         
                let bias' = this.calculateBiasAdjustment(input.bias,input.alpha,delta)
                {adjustmentOutput.weights=weights'';adjustmentOutput.bias=bias'}

        [TestMethod]
        public void calculateWeightAdjustmentUsingPositiveDelta_ReturnsExpected()
        {
            var xValue = 1.5;
            var xWeight = .00060;
            var delta = 2;

            var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta);
            var actual = Math.Round(weightAdjustment, 4);
            var expected = -.0024;
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void calculateWeightAdjustmentUsingNegativeDelta_ReturnsExpected()
        {
            var xValue = 1.5;
            var xWeight = .00060;
            var delta = -2;

            var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta);
            var actual = Math.Round(weightAdjustment, 5);
            var expected = .0036;
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void calculateBiasAdjustmentUsingPositiveDelta_ReturnsExpected()
        {
            var bias = 0.00364;
            var delta = 2;
            var expected = .00164;
            var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void calculateBiasAdjustmentUsingNegativeDelta_ReturnsExpected()
        {
            var bias = 0.00364;
            var delta = -2;
            var expected = .00564;
            var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runAdjustmentUsingMatchingData_ReturnsExpected()
        {

            var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, -1.0);
            var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
            var expected = .0065;
            var actual = Math.Round(adjustedWeights.weights[0],4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runAdjustmentUsingNegativeData_ReturnsExpected()
        {
            weightedConfirmedObservation weightedConfirmedObservation = 
                new NeuralNetworks.weightedConfirmedObservation(_weightedObservation, 1.0);
            var adjustmentInput = new adjustmentInput(weightedConfirmedObservation, _bias, _alpha, -1.0);
            var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
            var expected = .0125;
            var actual = Math.Round(adjustedWeights.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runAdjustmentUsingPositiveData_ReturnsExpected()
        {
            var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, 1.0);
            var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
            var expected = .0005;
            var actual = Math.Round(adjustedWeights.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

With these functions ready, I could run a training cycle for a given observation

    member this.runTrainingCycle (cycleTrainingInput:cycleTrainingInput) =
        let neuronTrainingInput = 
            {neuronInput.weightedObservation=cycleTrainingInput.weightedConfirmedObservation.weightedObservation;
            neuronInput.bias=cycleTrainingInput.bias}
        let neuronResult = this.runNeuron(neuronTrainingInput)
        let activationResult = this.runActivation(neuronResult)
        let adjustmentInput = {weightedConfirmedObservation=cycleTrainingInput.weightedConfirmedObservation;
                                bias=cycleTrainingInput.bias;alpha=cycleTrainingInput.alpha;
                                yActual=activationResult}
        this.runAdjustment(adjustmentInput)

        [TestMethod]
        public void runTrainingCycleUsingNegativeData_ReturnsExpected()
        {
            var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha);
            var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput);
            var expected = .0125;
            var actual = Math.Round(adjustmentOutput.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

        [TestMethod]
        public void runTrainingCycleUsingPositiveData_ReturnsExpected()
        {
            var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha);
            var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput);
            var expected = .0065;
            var actual = Math.Round(adjustmentOutput.weights[0], 4);
            Assert.AreEqual(expected, actual);
        }

And then I could run a cycle for each of the observations in the training set, a rotation. I am not happy that I am mutating the weights and biases here, though I am not sure how to fix that. I looked for a Seq.Scan function where the results of a function applied to the 1st element of a Seq is used in the input of the next –> all I could see were examples of threading a collector of int (like Seq.mapi). This will be something I will ask the functional ninjas when I see them again.

    member this.runTrainingRotation(rotationTrainingInput: rotationTrainingInput)=
        let mutable weights = rotationTrainingInput.weights
        let mutable bias = rotationTrainingInput.bias
        let alpha = rotationTrainingInput.alpha
        for i=0 to rotationTrainingInput.confirmedObservations.Count-1 do
            let currentConfirmedObservation = rotationTrainingInput.confirmedObservations.[i]
            let xws = Seq.zip currentConfirmedObservation.observation.xValues weights
            let xws' = new List<(float*float)>(xws)
            let weightedObservation = {xws=xws'}
            let weightedTrainingObservation = {weightedObservation=weightedObservation;yExpected=currentConfirmedObservation.yExpected}
            let cycleTrainingInput = {
                                cycleTrainingInput.weightedConfirmedObservation=weightedTrainingObservation;
                                cycleTrainingInput.bias=bias;
                                cycleTrainingInput.alpha=alpha}
            let cycleOutput = this.runTrainingCycle(cycleTrainingInput)
            weights <- cycleOutput.weights
            bias <- cycleOutput.bias
        {adjustmentOutput.weights=weights; adjustmentOutput.bias=bias}

        [TestMethod]
        public void runTrainingRotationUsingNegativeData_ReturnsExpected()
        {

            var xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(4.0);
            var observation = new observation(xValues);
            var yExpected = -1.0;
            var confirmedObservation0 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(1.5);
            xValues.Add(2.0);
            yExpected = -1.0;
            var confirmedObservation1 = new confirmedObservation(observation, yExpected);

            var trainingObservations = new List<confirmedObservation>();
            trainingObservations.Add(confirmedObservation0);
            trainingObservations.Add(confirmedObservation1);
            
            var weights = new List<double>();
            weights.Add(.0065);
            weights.Add(.0123);

            var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha);
            var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput);

            var expected = -0.09606;
            var actual = Math.Round(trainingRotationOutput.bias, 5);
            Assert.AreEqual(expected, actual);

        }

        [TestMethod]
        public void runTrainingRotationUsingPositiveData_ReturnsExpected()
        {
            var xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(4.0);
            var observation = new observation(xValues);
            var yExpected = 1.0;
            var confirmedObservation0 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(1.5);
            xValues.Add(2.0);
            yExpected = 1.0;
            var confirmedObservation1 = new confirmedObservation(observation, yExpected);

            var trainingObservations = new List<confirmedObservation>();
            trainingObservations.Add(confirmedObservation0);
            trainingObservations.Add(confirmedObservation1);


            var weights = new List<double>();
            weights.Add(.0065);
            weights.Add(.0123);

            var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha);
            var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput);

            var expected = -.09206;
            var actual = Math.Round(trainingRotationOutput.bias, 5);
            Assert.AreEqual(expected, actual);

        }

With the rotation done, I could write the train function which runs rotations for N number of times to tune the weights and biases:

    member this.train(trainInput:trainInput) = 
        let currentObservation = trainInput.confirmedObservations.[0].observation
        let weights = this.initializeWeights(currentObservation.xValues,trainInput.weightSeedValue)
        let weights' = new List<float>(weights)
        let mutable bias = this.initializeBias(trainInput.biasSeedValue)
        let alpha = trainInput.alpha
        for i=0 to trainInput.maxEpoches do
            let rotationTrainingInput={rotationTrainingInput.confirmedObservations=trainInput.confirmedObservations;
                rotationTrainingInput.weights = weights'; 
                rotationTrainingInput.bias=bias;
                rotationTrainingInput.alpha=trainInput.alpha}
            this.runTrainingRotation(rotationTrainingInput) |> ignore
        {adjustmentOutput.weights=weights'; adjustmentOutput.bias=bias}

        [TestMethod]
        public void trainUsingTestData_RetunsExpected()
        {
            var xValues = new List<double>();
            xValues.Add(1.5);
            xValues.Add(2.0);
            var observation = new observation(xValues);
            var yExpected = -1.0;
            var confirmedObservation0 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(2.0);
            xValues.Add(3.5);
            observation = new observation(xValues);
            yExpected = -1.0;
            var confirmedObservation1 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(5.0);
            observation = new observation(xValues);
            yExpected = -1.0;
            var confirmedObservation2 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(3.5);
            xValues.Add(2.5);
            observation = new observation(xValues);
            yExpected = -1.0;
            var confirmedObservation3 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(4.5);
            xValues.Add(5.0);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation4 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(5.0);
            xValues.Add(7.5);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation5 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(5.5);
            xValues.Add(8.0);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation6 = new confirmedObservation(observation, yExpected);

            xValues = new List<double>();
            xValues.Add(6.0);
            xValues.Add(6.0);
            observation = new observation(xValues);
            yExpected = 1.0;
            var confirmedObservation7 = new confirmedObservation(observation, yExpected);


            var trainingObservations = new List<confirmedObservation>();
            trainingObservations.Add(confirmedObservation0);
            trainingObservations.Add(confirmedObservation1);
            trainingObservations.Add(confirmedObservation2);
            trainingObservations.Add(confirmedObservation3);
            trainingObservations.Add(confirmedObservation4);
            trainingObservations.Add(confirmedObservation5);
            trainingObservations.Add(confirmedObservation6);
            trainingObservations.Add(confirmedObservation7);

            var random = new Random();
            var weightSeedValue = random.NextDouble();
            var biasSeedValue = random.NextDouble();
            var alpha = .001;
            var maxEpoches = 100;

            var trainInput = new trainInput(trainingObservations, weightSeedValue, biasSeedValue, alpha, maxEpoches);
            var trainOutput = _perceptron.train(trainInput);

            Assert.IsNotNull(trainOutput);

        }

With the training out of the way, I could concentrate on the prediction. The prediction was much easier because there are no adjustments and the rotation is run once. The data structures are also simpler because I don’t have to pass in the knownY values. I also only have 1 covering (all be it long) unit test that looks that the results of the prediction.

    member this.runPredictionCycle (cyclePredictionInput:cyclePredictionInput) = 
        let neuronInput = 
            {neuronInput.weightedObservation=cyclePredictionInput.weightedObservation;
            neuronInput.bias=cyclePredictionInput.bias}
        let neuronResult = this.runNeuron(neuronInput)
        this.runActivation(neuronResult)

    member this.runPredictionRotation (rotationPredictionInput:rotationPredictionInput) = 
        let output = new List<List<float>*float>();
        let weights = rotationPredictionInput.weights
        for i=0 to rotationPredictionInput.observations.Count-1 do
            let currentObservation = rotationPredictionInput.observations.[i];
            let xws = Seq.zip currentObservation.xValues weights
            let xws' = new List<(float*float)>(xws)
            let weightedObservation = {xws=xws'}
            let cyclePredictionInput = {
                            cyclePredictionInput.weightedObservation = weightedObservation;
                            cyclePredictionInput.bias = rotationPredictionInput.bias}
            let cycleOutput = this.runPredictionCycle(cyclePredictionInput)
            output.Add(currentObservation.xValues, cycleOutput)
        output

    member this.predict(predictInput:predictInput) = 
        let rotationPredictionInput = {
                        rotationPredictionInput.observations = predictInput.observations;
                        rotationPredictionInput.weights = predictInput.weights;
                        rotationPredictionInput.bias = predictInput.bias }
        this.runPredictionRotation(rotationPredictionInput)

        [TestMethod]
        public void predictUsingTestData_ReturnsExpected()
        {
            var xValues = new List<double>();
            xValues.Add(3.0);
            xValues.Add(4.0);
            var observation0 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(0.0);
            xValues.Add(1.0);
            var observation1 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(2.0);
            xValues.Add(5.0);
            var observation2 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(5.0);
            xValues.Add(6.0);
            var observation3 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(9.0);
            xValues.Add(9.0);
            var observation4 = new observation(xValues);

            xValues = new List<double>();
            xValues.Add(4.0);
            xValues.Add(6.0);
            var observation5 = new observation(xValues);

            var observations = new List<observation>();
            observations.Add(observation0);
            observations.Add(observation1);
            observations.Add(observation2);
            observations.Add(observation3);
            observations.Add(observation4);
            observations.Add(observation5);


            var weights = new List<double>();
            weights.Add(.0065);
            weights.Add(.0123);

            var bias = -0.0906;

            var predictInput = new predictInput(observations, weights, bias);

            var predictOutput = _perceptron.predict(predictInput);
            Assert.IsNotNull(predictOutput);
        }

When I run all of the unit tests the all run green:

With the Perceptron created, I can now go back and change the code and figure out:

1) Why my weights across the XValues are the same (wrong!)

2) How to implement a more idomatic/recursive way of running rotations so I can remove the mutation

With my unit tests running green, I know I am covered in case I make a mistake

Filed under F#, Machine Learning

One Response to Neural Network Part 3: Perceptrons

Pingback: F# Weekly #32, 2014 | Sergey Tihon's Blog

Jamie Dixon's Home

Neural Network Part 3: Perceptrons

One Response to Neural Network Part 3: Perceptrons

Leave a Reply Cancel reply

Categories

Recent Posts

Archives

Blogroll

Meta

Jamie Dixon's Home

Neural Network Part 3: Perceptrons

Share this:

Like this:

Related

One Response to Neural Network Part 3: Perceptrons

Leave a Reply Cancel reply

Categories

Recent Posts

Archives

Blogroll

Meta