I went back to my code for building a Perceptron and I made some changes. I realized that although McCaffrey combines the code together, there are actually two actions for the perceptron: training and predicting. I created a diagram to help me keep the functions that I need for each in mind:
I also skeletoned out some data structures that I think I need:
With the base diagrams out of the way, I created different data structures that were tailored to each action. These are a bit different than the diagrams –> I didn’t go back and update the diagrams because the code is where you would look to see how the system works:
type observation = {xValues:float List}
type weightedObservation = {xws:(float*float) List}
type confirmedObservation = {observation:observation;yExpected:float}
type weightedConfirmedObservation = {weightedObservation:weightedObservation;yExpected:float}
type neuronInput = {weightedObservation:weightedObservation;bias:float}
type cycleTrainingInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float}
type adjustmentInput = {weightedConfirmedObservation:weightedConfirmedObservation;bias:float;alpha:float;yActual:float}
type adjustmentOutput = {weights:float List; bias:float}
type rotationTrainingInput = {confirmedObservations:confirmedObservation List;weights:float List;bias:float;alpha:float}
type trainInput = {confirmedObservations:confirmedObservation List; weightSeedValue:float;biasSeedValue:float;alpha:float; maxEpoches:int}
type cyclePredictionInput = {weightedObservation:weightedObservation;bias:float}
type rotationPredictionInput = {observations:observation List;weights:float List;bias:float}
type predictInput = {observations:observation List;weights:float List;bias:float}
Note that I am composing data structures with the base being an observation. The observation is a list of different xValues for a given, well, observation. The weighted observation is the XValue paired with the perceptron weights. The confirmedObservation is for training –> given an observation, what was the actual output?
With the data structures out of the way, I went to the Perceptron and added in the basic functions for creating seed values:
member this.initializeWeights(xValues, randomSeedValue) =
let lo = -0.01
let hi = 0.01
let xWeight = (hi-lo) * randomSeedValue + lo
xValues |> Seq.map(fun w -> xWeight)
member this.initializeBias(randomSeedValue) =
let lo = -0.01
let hi = 0.01
(hi-lo) * randomSeedValue + lo
Since I was doing TDD, here are the unit tests I used for these functions:
[TestMethod]
public void initializeWeightsUsingHalfSeedValue_ReturnsExpected()
{
var weights = _perceptron.initializeWeights(_observation.xValues, .5);
var weightsList = new List<double>(weights);
var expected = 0.0;
var actual = weightsList[0];
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void initializeWeightsUsingLessThanHalfSeedValue_ReturnsExpected()
{
var weights = _perceptron.initializeWeights(_observation.xValues, .4699021627);
var weightsList = new List<double>(weights);
var expected = -0.00060;
var actual = Math.Round(weightsList[0],5);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void initializeBiasesUsingHalfSeedValue_ReturnsExpected()
{
var expected = 0.0;
var actual = _perceptron.initializeBias(.5);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void initializeBiasesUsingLessThanHalfSeedValue_ReturnsExpected()
{
var expected = -0.00060;
var bias = _perceptron.initializeBias(.4699021627);
var actual = Math.Round(bias, 5);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void initializeBiasesUsingGreaterThanHalfSeedValue_ReturnsExpected()
{
var expected = 0.00364;
var bias = _perceptron.initializeBias(.6820621978);
var actual = Math.Round(bias,5);
Assert.AreEqual(expected, actual);
}
I then created a base neuron and activation function that would work for both training and predicting:
member this.runNeuron(input:neuronInput) =
let xws = input.weightedObservation.xws
let output = xws
|> Seq.map(fun (xValue,xWeight) -> xValue*xWeight)
|> Seq.sumBy(fun x -> x)
output + input.bias
member this.runActivation(input) =
if input < 0.0 then -1.0 else 1.0
[TestMethod]
public void runNeuronUsingNormalInput_ReturnsExpected()
{
var expected = -0.0219;
var perceptronOutput = _perceptron.runNeuron(_neuronInput);
var actual = Math.Round(perceptronOutput, 4);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void runActivationUsingNormalInput_ReturnsExpected()
{
var expected = -1;
var actual = _perceptron.runActivation(-0.0219);
Assert.AreEqual(expected, actual);
}
I then created the functions for training –> specifically to return adjusted weights and biases based on the result of the activation function
member this.calculateWeightAdjustment(xValue, xWeight, alpha, delta) =
match delta > 0.0, xValue >= 0.0 with
| true,true -> xWeight - (alpha * abs(delta) * xValue)
| false,true -> xWeight + (alpha * abs(delta) * xValue)
| true,false -> xWeight - (alpha * abs(delta) * xValue)
| false,false -> xWeight + (alpha * abs(delta) * xValue)
member this.calculateBiasAdjustment(bias, alpha, delta) =
match delta > 0.0 with
| true -> bias - (alpha * abs(delta))
| false -> bias + (alpha * abs(delta))
member this.runAdjustment (input:adjustmentInput) =
match input.weightedConfirmedObservation.yExpected = input.yActual with
| true ->
let weights = input.weightedConfirmedObservation.weightedObservation.xws |> Seq.map(fun (x,w) -> w)
let weights' = new List<float>(weights)
{adjustmentOutput.weights=weights';adjustmentOutput.bias=input.bias}
| false ->
let delta = input.yActual - input.weightedConfirmedObservation.yExpected
let weights' = input.weightedConfirmedObservation.weightedObservation.xws
|> Seq.map(fun (xValue, xWeight) ->
this.calculateWeightAdjustment(xValue,xWeight,input.alpha,delta))
|> Seq.toList
let weights'' = new List<float>(weights')
let bias' = this.calculateBiasAdjustment(input.bias,input.alpha,delta)
{adjustmentOutput.weights=weights'';adjustmentOutput.bias=bias'}
[TestMethod]
public void calculateWeightAdjustmentUsingPositiveDelta_ReturnsExpected()
{
var xValue = 1.5;
var xWeight = .00060;
var delta = 2;
var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta);
var actual = Math.Round(weightAdjustment, 4);
var expected = -.0024;
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void calculateWeightAdjustmentUsingNegativeDelta_ReturnsExpected()
{
var xValue = 1.5;
var xWeight = .00060;
var delta = -2;
var weightAdjustment = _perceptron.calculateWeightAdjustment(xValue, xWeight, _alpha, delta);
var actual = Math.Round(weightAdjustment, 5);
var expected = .0036;
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void calculateBiasAdjustmentUsingPositiveDelta_ReturnsExpected()
{
var bias = 0.00364;
var delta = 2;
var expected = .00164;
var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void calculateBiasAdjustmentUsingNegativeDelta_ReturnsExpected()
{
var bias = 0.00364;
var delta = -2;
var expected = .00564;
var actual = _perceptron.calculateBiasAdjustment(bias, _alpha, delta);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void runAdjustmentUsingMatchingData_ReturnsExpected()
{
var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, -1.0);
var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
var expected = .0065;
var actual = Math.Round(adjustedWeights.weights[0],4);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void runAdjustmentUsingNegativeData_ReturnsExpected()
{
weightedConfirmedObservation weightedConfirmedObservation =
new NeuralNetworks.weightedConfirmedObservation(_weightedObservation, 1.0);
var adjustmentInput = new adjustmentInput(weightedConfirmedObservation, _bias, _alpha, -1.0);
var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
var expected = .0125;
var actual = Math.Round(adjustedWeights.weights[0], 4);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void runAdjustmentUsingPositiveData_ReturnsExpected()
{
var adjustmentInput = new adjustmentInput(_weightedConfirmedObservation, _bias, _alpha, 1.0);
var adjustedWeights = _perceptron.runAdjustment(adjustmentInput);
var expected = .0005;
var actual = Math.Round(adjustedWeights.weights[0], 4);
Assert.AreEqual(expected, actual);
}
With these functions ready, I could run a training cycle for a given observation
member this.runTrainingCycle (cycleTrainingInput:cycleTrainingInput) =
let neuronTrainingInput =
{neuronInput.weightedObservation=cycleTrainingInput.weightedConfirmedObservation.weightedObservation;
neuronInput.bias=cycleTrainingInput.bias}
let neuronResult = this.runNeuron(neuronTrainingInput)
let activationResult = this.runActivation(neuronResult)
let adjustmentInput = {weightedConfirmedObservation=cycleTrainingInput.weightedConfirmedObservation;
bias=cycleTrainingInput.bias;alpha=cycleTrainingInput.alpha;
yActual=activationResult}
this.runAdjustment(adjustmentInput)
[TestMethod]
public void runTrainingCycleUsingNegativeData_ReturnsExpected()
{
var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha);
var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput);
var expected = .0125;
var actual = Math.Round(adjustmentOutput.weights[0], 4);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void runTrainingCycleUsingPositiveData_ReturnsExpected()
{
var cycleTrainingInput = new cycleTrainingInput(_weightedConfirmedObservation, _bias, _alpha);
var adjustmentOutput = _perceptron.runTrainingCycle(cycleTrainingInput);
var expected = .0065;
var actual = Math.Round(adjustmentOutput.weights[0], 4);
Assert.AreEqual(expected, actual);
}
And then I could run a cycle for each of the observations in the training set, a rotation. I am not happy that I am mutating the weights and biases here, though I am not sure how to fix that. I looked for a Seq.Scan function where the results of a function applied to the 1st element of a Seq is used in the input of the next –> all I could see were examples of threading a collector of int (like Seq.mapi). This will be something I will ask the functional ninjas when I see them again.
member this.runTrainingRotation(rotationTrainingInput: rotationTrainingInput)=
let mutable weights = rotationTrainingInput.weights
let mutable bias = rotationTrainingInput.bias
let alpha = rotationTrainingInput.alpha
for i=0 to rotationTrainingInput.confirmedObservations.Count-1 do
let currentConfirmedObservation = rotationTrainingInput.confirmedObservations.[i]
let xws = Seq.zip currentConfirmedObservation.observation.xValues weights
let xws' = new List<(float*float)>(xws)
let weightedObservation = {xws=xws'}
let weightedTrainingObservation = {weightedObservation=weightedObservation;yExpected=currentConfirmedObservation.yExpected}
let cycleTrainingInput = {
cycleTrainingInput.weightedConfirmedObservation=weightedTrainingObservation;
cycleTrainingInput.bias=bias;
cycleTrainingInput.alpha=alpha}
let cycleOutput = this.runTrainingCycle(cycleTrainingInput)
weights <- cycleOutput.weights
bias <- cycleOutput.bias
{adjustmentOutput.weights=weights; adjustmentOutput.bias=bias}
[TestMethod]
public void runTrainingRotationUsingNegativeData_ReturnsExpected()
{
var xValues = new List<double>();
xValues.Add(3.0);
xValues.Add(4.0);
var observation = new observation(xValues);
var yExpected = -1.0;
var confirmedObservation0 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(1.5);
xValues.Add(2.0);
yExpected = -1.0;
var confirmedObservation1 = new confirmedObservation(observation, yExpected);
var trainingObservations = new List<confirmedObservation>();
trainingObservations.Add(confirmedObservation0);
trainingObservations.Add(confirmedObservation1);
var weights = new List<double>();
weights.Add(.0065);
weights.Add(.0123);
var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha);
var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput);
var expected = -0.09606;
var actual = Math.Round(trainingRotationOutput.bias, 5);
Assert.AreEqual(expected, actual);
}
[TestMethod]
public void runTrainingRotationUsingPositiveData_ReturnsExpected()
{
var xValues = new List<double>();
xValues.Add(3.0);
xValues.Add(4.0);
var observation = new observation(xValues);
var yExpected = 1.0;
var confirmedObservation0 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(1.5);
xValues.Add(2.0);
yExpected = 1.0;
var confirmedObservation1 = new confirmedObservation(observation, yExpected);
var trainingObservations = new List<confirmedObservation>();
trainingObservations.Add(confirmedObservation0);
trainingObservations.Add(confirmedObservation1);
var weights = new List<double>();
weights.Add(.0065);
weights.Add(.0123);
var rotationTrainingInput = new rotationTrainingInput(trainingObservations, weights, _bias, _alpha);
var trainingRotationOutput = _perceptron.runTrainingRotation(rotationTrainingInput);
var expected = -.09206;
var actual = Math.Round(trainingRotationOutput.bias, 5);
Assert.AreEqual(expected, actual);
}
With the rotation done, I could write the train function which runs rotations for N number of times to tune the weights and biases:
member this.train(trainInput:trainInput) =
let currentObservation = trainInput.confirmedObservations.[0].observation
let weights = this.initializeWeights(currentObservation.xValues,trainInput.weightSeedValue)
let weights' = new List<float>(weights)
let mutable bias = this.initializeBias(trainInput.biasSeedValue)
let alpha = trainInput.alpha
for i=0 to trainInput.maxEpoches do
let rotationTrainingInput={rotationTrainingInput.confirmedObservations=trainInput.confirmedObservations;
rotationTrainingInput.weights = weights';
rotationTrainingInput.bias=bias;
rotationTrainingInput.alpha=trainInput.alpha}
this.runTrainingRotation(rotationTrainingInput) |> ignore
{adjustmentOutput.weights=weights'; adjustmentOutput.bias=bias}
[TestMethod]
public void trainUsingTestData_RetunsExpected()
{
var xValues = new List<double>();
xValues.Add(1.5);
xValues.Add(2.0);
var observation = new observation(xValues);
var yExpected = -1.0;
var confirmedObservation0 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(2.0);
xValues.Add(3.5);
observation = new observation(xValues);
yExpected = -1.0;
var confirmedObservation1 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(3.0);
xValues.Add(5.0);
observation = new observation(xValues);
yExpected = -1.0;
var confirmedObservation2 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(3.5);
xValues.Add(2.5);
observation = new observation(xValues);
yExpected = -1.0;
var confirmedObservation3 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(4.5);
xValues.Add(5.0);
observation = new observation(xValues);
yExpected = 1.0;
var confirmedObservation4 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(5.0);
xValues.Add(7.5);
observation = new observation(xValues);
yExpected = 1.0;
var confirmedObservation5 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(5.5);
xValues.Add(8.0);
observation = new observation(xValues);
yExpected = 1.0;
var confirmedObservation6 = new confirmedObservation(observation, yExpected);
xValues = new List<double>();
xValues.Add(6.0);
xValues.Add(6.0);
observation = new observation(xValues);
yExpected = 1.0;
var confirmedObservation7 = new confirmedObservation(observation, yExpected);
var trainingObservations = new List<confirmedObservation>();
trainingObservations.Add(confirmedObservation0);
trainingObservations.Add(confirmedObservation1);
trainingObservations.Add(confirmedObservation2);
trainingObservations.Add(confirmedObservation3);
trainingObservations.Add(confirmedObservation4);
trainingObservations.Add(confirmedObservation5);
trainingObservations.Add(confirmedObservation6);
trainingObservations.Add(confirmedObservation7);
var random = new Random();
var weightSeedValue = random.NextDouble();
var biasSeedValue = random.NextDouble();
var alpha = .001;
var maxEpoches = 100;
var trainInput = new trainInput(trainingObservations, weightSeedValue, biasSeedValue, alpha, maxEpoches);
var trainOutput = _perceptron.train(trainInput);
Assert.IsNotNull(trainOutput);
}
With the training out of the way, I could concentrate on the prediction. The prediction was much easier because there are no adjustments and the rotation is run once. The data structures are also simpler because I don’t have to pass in the knownY values. I also only have 1 covering (all be it long) unit test that looks that the results of the prediction.
member this.runPredictionCycle (cyclePredictionInput:cyclePredictionInput) =
let neuronInput =
{neuronInput.weightedObservation=cyclePredictionInput.weightedObservation;
neuronInput.bias=cyclePredictionInput.bias}
let neuronResult = this.runNeuron(neuronInput)
this.runActivation(neuronResult)
member this.runPredictionRotation (rotationPredictionInput:rotationPredictionInput) =
let output = new List<List<float>*float>();
let weights = rotationPredictionInput.weights
for i=0 to rotationPredictionInput.observations.Count-1 do
let currentObservation = rotationPredictionInput.observations.[i];
let xws = Seq.zip currentObservation.xValues weights
let xws' = new List<(float*float)>(xws)
let weightedObservation = {xws=xws'}
let cyclePredictionInput = {
cyclePredictionInput.weightedObservation = weightedObservation;
cyclePredictionInput.bias = rotationPredictionInput.bias}
let cycleOutput = this.runPredictionCycle(cyclePredictionInput)
output.Add(currentObservation.xValues, cycleOutput)
output
member this.predict(predictInput:predictInput) =
let rotationPredictionInput = {
rotationPredictionInput.observations = predictInput.observations;
rotationPredictionInput.weights = predictInput.weights;
rotationPredictionInput.bias = predictInput.bias }
this.runPredictionRotation(rotationPredictionInput)
[TestMethod]
public void predictUsingTestData_ReturnsExpected()
{
var xValues = new List<double>();
xValues.Add(3.0);
xValues.Add(4.0);
var observation0 = new observation(xValues);
xValues = new List<double>();
xValues.Add(0.0);
xValues.Add(1.0);
var observation1 = new observation(xValues);
xValues = new List<double>();
xValues.Add(2.0);
xValues.Add(5.0);
var observation2 = new observation(xValues);
xValues = new List<double>();
xValues.Add(5.0);
xValues.Add(6.0);
var observation3 = new observation(xValues);
xValues = new List<double>();
xValues.Add(9.0);
xValues.Add(9.0);
var observation4 = new observation(xValues);
xValues = new List<double>();
xValues.Add(4.0);
xValues.Add(6.0);
var observation5 = new observation(xValues);
var observations = new List<observation>();
observations.Add(observation0);
observations.Add(observation1);
observations.Add(observation2);
observations.Add(observation3);
observations.Add(observation4);
observations.Add(observation5);
var weights = new List<double>();
weights.Add(.0065);
weights.Add(.0123);
var bias = -0.0906;
var predictInput = new predictInput(observations, weights, bias);
var predictOutput = _perceptron.predict(predictInput);
Assert.IsNotNull(predictOutput);
}
When I run all of the unit tests the all run green:
With the Perceptron created, I can now go back and change the code and figure out:
1) Why my weights across the XValues are the same (wrong!)
2) How to implement a more idomatic/recursive way of running rotations so I can remove the mutation
With my unit tests running green, I know I am covered in case I make a mistake
), I will need to put the 5.5 million records into a format that can be easily searched. Thinking what we want is:
Jamie Dixon's Home 