Neural Networks

CIO LEVEL SUMMARY

• Neural Networks (NNs) are a useful and flexible tool that can help
  learn patterns from data that humans cannot

• Neural Networks contain layers of nodes which feed directly into the
  following layer. As information moves through the NN, information is
  weighted and combined

• Once a prediction is made, the Neural Network compares its
  predictions to the desired output

• Neural Networks learn by figuring out how to tweak its calculations
  in order to reduce the amount of errors it makes

WHAT ARE NEURAL NETS?

Neural Networks have gotten a lot of attention in the past few decades.
They’re used in social media apps like Snapchat, and they’re being used
more and more in the medical and health fields¹.

Neural Networks–or Neural Nets for short–are an incredibly flexible
tool to have in your analytical tool belt because they are universal
approximators². This means that under certain assumptions, a
neural network can approximate any function.

Imagine that you’re building a toy helicopter. You grab some supplies
like plastic, wood, and glue, and you get started. The first thing you
need to do is make the propellor out of plastic. But you’re not sure how
to do it…if you were an expert, you might know how to cut and shape
the propeller by hand. But you’re not an expert. So you’re stuck.

Someone hands you a box, and tells you that it’s a universal tool
approximator. All you have to do is show it a few examples of the
propellers that you want, input some plastic, and it will learn how to
output the propellers. The box learns the process that takes raw plastic
and turns it into a propeller…or at least it gets close enough.

The *best* thing about this box is that it can learn to make or do
*anything*. When you need cap for your pen, or a buckle for your belt,
a box can learn how to make it.

Neural Networks are similar to these boxes. But their inputs and outputs
are numbers, rather than materials and gadgets.

Often, we want Neural Networks to approximate the relationship between
data (the input) and some kind of prediction (our output). For example,
we may want to approximate the relationship between features from a
digitized image of a fine needle aspirate of a breast mass³ and
whether or not it’s malignant. We can use the variables radius, texture,
perimeter, area, smoothness, and symmetry of the tumor cell nuclei as
our inputs.

We can call the true relationship that maps the inputs (radius, texture,
perimeter, area, smoothness, and symmetry) onto the output (malignancy
diagnosis) (f^*(x)) .

The Neural Network learns what (f^*(x)) is from the data. It won’t always get
(f^*(x)) perfectly correct, but it can come up with some function
(f(x)) that is close. This is a huge help, because often the function that
turns inputs into outputs is complex, and would be hard for a human to
discover on their own.

THE STRUCTURE

Neural Networks use layers of nodes to take data from inputs to
outputs.

For now, let’s think of each node as something that stores a value. The
first layer is our input layer. It’s nodes represent the data that we
feed into our network.

The last layer is the output layer, which has two nodes, one for
Malignant. One for Benign. Each of these nodes will eventually have a
value in them once we turn our inputs into outputs. Whichever node has a
higher value, will be the diagnosis we choose.

So now, let’s focus on the middle layers, called hidden layers.

THE HIDDEN LAYERS

Let’s add one layer between our input and output layer.

This first hidden layer contains four nodes, and we need to fill these
nodes with values. These values are calculated using the values from the
layer before them. Let’s focus on just one of these nodes for now. The
values from the input layer are weighted and added together to form a
single number. In general the weight
(w^l_{(i,j)}%0)
connects the (i^th) node from the (l^th) layer to the (j^th) neuron of the
(l+1^th) layer.

We also add a constant–called the bias–to this equation. In this
form, it might look similar to a linear regression formula.

The final step is to apply an activation function to the result
((w’x)) of this equation. The activation function takes the weighted inputs and
bias, and transforms it. If we wanted the value to remain unchanged, we
could apply the function (f(x) = x). But, there are many different
activation functions we can use.

One popular one is the Rectified Linear Unit, or ReLU (pronounced
Rey-Loo) for short. The ReLU activation function takes all negative
numbers, and turns them into zeroes. It leaves positive numbers
unchanged. Non-linear activation functions like the ReLU are an
important part of what allows Neural Networks to approximate nonlinear
functions. Without them, Neural Nets would only be able to approximate
linear functions.

Once we’ve added up all the weighted inputs and the bias, and applied an
activation function, we finally have a value to put into our node.

And then, we fill in the rest of the nodes for this layer, doing the
same thing, but with different weights and biases.

And then we use the values in the second layer to compute the values in
the third and fourth layers.

The general steps are:

1. Add the weighted values from the previous layer together

2. Add the bias

3. Apply the activation function

We can use different activation functions for each layer. In fact for
the last layer, we’ll want to use a softmax activation. Softmax is
often used when the outputs you’d like to predict are categorical. It
takes all the values from that layer and normalizes them so that they’re
all between 0 and 1, and the sum of all the outputs is 1.

After applying the softmax activation, we can use these new outputs as
probabilities, or as a measure of how strongly the Network believes a
value is the correct answer.

For example, say that the values of our output layer are 1.6 and 4
before we apply the softmax activation.

After applying the activation function, our values will be 0.08 and 0.92
since

In this case, the neural network will guess that the tumor is Benign,
since it has the higher value.

THE MODEL

So now we’ve built a Feed Forward Neural Network. We started with an
input layer which took in our data on breast tumors, and by moving
through two hidden layers, produced an output layer that can predict
whether a tumor is Malignant, or Benign.

From start to finish, we’ve been continually moving information forward,
from one layer directly to the next until we reach our output layer.
Hence, Feed Forward. This architecture is relatively simple, and yet it
has been used to do incredible things. Like help self driving cars make
decisions, and compress files⁴.

The strength of the Neural Network is its ability to learn the weights,
biases, and sometimes the activation functions⁵, necessary to
approximate the relationship between your inputs and outputs.

Error Functions

Before we give the Neural Network any data, we randomly initialize the
weights and biases, just so that it has some numbers to start with. Then
we give it some data–the training data–and look at it’s outputs. At
first, the outputs will be garbage, since our weights were randomly
initialized and don’t mean anything. But the Neural Net, like mature
humans, learns from its mistakes.

We can compare the output the neural net gave us to the output that we
want.

For example, if the neural network gave use the output from before, we’d
end up with a 0.08 for Malignant and 0.92 for Benign:

We know that this particular tumor was Malignant, so our Neural Network
was pretty far off. The correct output would be a 1 for Malignant and 0
for Benign. Which would represent the Network being sure that the tumor
is Malignant.

We need a way to quantify just how wrong our prediction was. One common
way is the mean squared error.

The mean squared error is calculated by taking the difference between
your desired outcomes, and your predicted outcomes, and squaring them,
and adding them together.

There are other ways of measuring the error (also called the loss or
cost), and whichever function you use is called the error function
(also called the loss function or cost function).

We want to be as accurate as possible, so we want to minimize our
error function. And our error function is just a result of all the
processes, including the weights, biases, and activation functions that
we used in our neural network.

In order to minimize our error function, we need to change these
weights, biases, and activation functions so that our prediction is
closer to our desired output. But figuring out which tweaks to make to
these parameters isn’t immediately clear.

Imagine you’re a ball on this line. You want to find the lowest point.

Backpropagation

If you were a particularly math inclined ball, you could take the
derivative and second derivatives of the function, and find the global
minimum. But that’s not always possible or cost effective. Instead, we
find the gradient of the error function.

The gradient is the rate of change of our error function.

It tells us how fast the error function changes when we make small
changes to our model parameters (like the weights, and biases). In other
words, if you fiddled with the weights and biases of the neural network,
how much would the Neural Network’s output change?

We often find the gradient using a method called backpropagation, or
backprop for short.

Backpropagation is a way of applying the chain rule from calculus to
calculate the gradient of our error function. The error function relies
on the output that our Neural Network produces. And that output depends
on the activation function we used. And the value we plugged into our
activation function depends on the weights and biases that we used.

Backpropagation allows us to find our gradient (the derivative of the
error function with respect to our parameters) by looking at how small
changes in our weight and bias parameters change our weighted sum, z.
And then, how small changes in our weighted sum, z, affect the output
of our Neural Network, a. And finally, how small changes in our
output, a, affect the value of the error function, J.

Gradient Descent

So instead of doing a large calculation to find the absolute global
minimum, we can just take one step at a time towards a minimum. At each
step, we’ll look for the direction that is the steepest downhill, and
step there.

One step at a time, we’ll reach a minimum.

It may not be the *global* minimum. But it will be *a* minimum, and
usually that is good enough to help the Neural Network make accurate
predictions. This iterative algorithm for finding the minimum of a
function is called gradient descent, since we are descending the
gradient, and it’s often used to minimize the error function.

Practically, it can be computationally expensive to calculate the
gradient using all the training data. We’re often going to give the
Neural Net Thousands if not millions of data points, so calculating the
error for every one of those points could take a while. To make things
more efficient, we can use Stochastic Gradient Descent. Instead of
calculating the gradient using all of our training data, we use
smaller samples of our training data, to help speed up the
calculations.

We do sacrifice a little bit of accuracy by not using all the data
samples, but Stochastic Gradient Descent is often very effective at
finding an appropriate minimum. Even though it only uses smaller samples
of the data for each step.

CONCLUSION

So, we’ve found our gradient using backpropagation, and chosen which
direction to go using gradient descent or stochastic gradient descent,
we repeat the process again and again until we reach a place where our
error function is minimized. Hopefully, that means that our Neural
Network is making less errors than it was before. It’s learned
something!

We now have a Neural Network that’s pretty good at predicting whether a
breast tumor is Malignant, or Benign. If a new biopsy comes in, we can
feed our Neural Network the data, and ask for predictions. While the
expertise of doctors and specialists is still necessary, Neural Networks
like this one can help screen or triage cases so that doctors have more
time to spend with patients, and can direct their energy towards the
less clear cut cases.

If you can give a Neural Network enough data to learn from, their
performances are incredibly accurate.

PYTHON

Let’s train our own Neural Network in python using the Wisconsin Breast
Cancer Data Set⁶.

We’ll use the package Keras to build our Neural Network in python.
First, let’s import all the functions we’ll need to run the code.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split,KFold
from sklearn.metrics import confusion_matrix, accuracy_score
from sklearn.preprocessing import StandardScaler
from keras.models import Sequential
from keras.optimizers import SGD
from keras.layers import Dense
from keras.utils import to_categorical

Then, let’s read in our data.

data = pd.read_csv("data.csv", sep = ",")

Next, we’ll split our dataset into two parts, the data we’ll use to
train the Neural Network, and the data we’ll use to evaluate the Network
once it’s been trained. We’ll also z-score our features so that they’re
standardized.

x_train, x_test, y_train, y_test = train_test_split(data[list(data)[1:]],data["diagnosis"],
 test_size=0.20)
sc = StandardScaler()
X_sc = sc.fit_transform(x_train)
X_test_sc = sc.transform(x_test)

We’ll create a list that stores the number of nodes that are in each
layer of our network, so that we can reference it later.

layer_sizes = [6,4,3,2] #input, hidden1, hidden2,output

Our first layer has 6 nodes, since we have 6 features (radius, texture,
perimeter, area, smoothness, and symmetry). Our output layer has 2
nodes; one for Malignant and one for Benign. The two layers in the
middle have 4 and 3 nodes respectively.

Now, let’s tell python how we’d like to structure our model. We’ll give
it information about the number of layers, and their sizes, as well as
which activation functions we’d like to use.

model = Sequential()
model.add(Dense(
        input_dim=layer_sizes[0],
        units=layer_sizes[1],
        activation="relu"
    ))
model.add(Dense(units=layer_sizes[2],
        activation="relu"))
model.add(Dense(units=layer_sizes[3],
                    kernel_initializer='uniform',
                    activation="softmax"))

sgd = SGD(lr=0.001, decay=1e-5, momentum=.9)  # Stochastic gradient descent

model.compile(loss='mean_squared_error', optimizer=sgd, metrics=["accuracy"])

Then, we can train our model.

y_train_vectorized = to_categorical(y_train) #change Malignant/Benign to Categorical
model.fit(X_sc, y_train_vectorized, epochs=1000, batch_size=32, verbose = 1)

So now we have a Neural Network that has learned how to tell if biopsies
from breast tumors are Malignant of Benign. But we want to estimate how
accurate it is. If it can only give the correct diagnosis 50% of the
time, we may not want to deploy it.

To check the accuracy, we can run this code.

#grab prediction for each data point
proba = model.predict_proba(X_test_sc, batch_size=32)
classes = np.argmax(proba, axis=1)

#print out accuracy
print("ACC",accuracy_score(y_test, classes))

This particular model achieved 92.98% accuracy, which is pretty
impressive. If we could give the model more data, hopefully it would
improve even further.

CONCLUSION

Now you’ve had a gentle introduction into how Feed Forward Neural
Networks learn, and how you can implement one in python. The material
covered here just scratches the surface of what Neural Networks can do.
Neural Networks of all types play huge roles in new technologies like
self driving cars to recognize objects like traffic signs and
pedestrians, and in medicine to reduce noise on Electrocardiograms⁷.
The flexibility and application of Neural Networks far outstrips our
coverage here.

Interested in exploring how we can learn from your data to automate
your tasks? Get in touch with us!

More Resources

1. 3Blue1Brown Neural Networks,
   Gradient Descent,
   Backpropagation,
   Backprop with Calculus

2. Deep Learning Book

3. Towards Data Science
   1
   2
   3

4. Wisconsin Breast Cancer
   Data

References