TensorFlow MNIST TensorBoard example

Available in Classic and VPC

TensorFlow MNIST TensorBoard example

This is an example of “TensorFlow MNIST Regression Model” with TensorBoard added.

When this example is executed, a log file is created in the TensorBoard log directory (/home/ncp/workspace/tensorboard) that is provided by default. Access [Public IP address: 18889] on your web browser, and you can see logs on TensorBoard. For how to set up your access environment, see "Set access environment."

This example code uses MNIST basic samples provided by TensorFlow for TensorFlow beginners. We will use MNIST dataset to create the Softmax regression model, which predicts numbers using image data. You can see the results of this example code on TensorBoard.

Each concept or term is described only at the required level to understand the example code. For an accurate understanding, you need to study additionally on Machine Learning and Deep Learning.

MNIST dataset description

MNIST dataset consists of an image that represents a handwritten numeric image as a vector and labels that indicate what the image means. The labels in the image below are 5, 0, 4, and 1 respectively, and labels are made up of 10 unique values from 0 to 9.

MNIST dataset also contains 55,000 pieces of learning data (mnist.train), 10,000 pieces of test data (mnist.test), and 5,000 pieces of validation data (mnist.validation), each of which is further divided into images and labels described above.

Since an image consists of 28x28 (=784) pixels, it is stored as a vector of 784 dimensions, and the 784 dimensions contain values between 0 and 1 depending on the degree of enhancement.

Use the code below to download the data provided by TensorFlow and save it in the data folder.
Use the ‘one_hot=True’ option (one hot encoding) to define a label as a 10-dimensional vector rather than defining one numeric value between 0 and 9. One hot encoding data will be described again in the example below.

""" import TensorFlow package:  You can use tf hereafter. """
import tensorflow as tf

""" Download and load data 
Download the data provided by TensorFlow, save it in the data folder and load it. 
The data is downloaded only at first, and after that, the saved data is loaded without being downloaded, in less time.""" 
from tensorflow.examples.tutorials.mnist import input_data
You can keep track of the whole execution time through %time mnist = input_data.read_data_sets("data/", one_hot=True)  # %time. 

Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
Extracting data/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
Extracting data/train-labels-idx1-ubyte.gz
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting data/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting data/t10k-labels-idx1-ubyte.gz
CPU times: user 447 ms, sys: 454 ms, total: 901 ms
Wall time: 36.1 s

As the images are shown as a vector of 784 dimensions and the labels read data using the “one_hot=True” option (one hot encoding), the label, “7” is represented as '[ 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]' as shown in the code below. (0 is represented as [ 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.], 1 as [ 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.], and 2 as [ 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]).

# Check images/labels data structure 
print 'train dataset (55,000):', mnist.train.images.shape, mnist.train.labels.shape
print 'test dataset (10,000):', mnist.test.images.shape, mnist.test.labels.shape
print 'validation dataset (5,000):', mnist.validation.images.shape, mnist.validation.labels.shape

print '\n Check data sample (first image data 7)'
print 'label:', mnist.train.labels[0]

train dataset (55,000): (55000, 784) (55000, 10)
test dataset (10,000): (10000, 784) (10000, 10)
validation dataset (5,000): (5000, 784) (5000, 10)

 Check data sample (first image data 7)
label: [ 0.  0.  0.  0.  0.  0.  0.  0.  0.  1.]

Regression model

This example code covers MNIST basic samples provided by TensorFlow for TensorFlow beginners.
We will create a regression model and train it to predict a label and calculate the accuracy of the model.

Implementing the regression

Define a placeholder to hold an image and a correct label, and Variable to hold the weight and bias, which are the learning results, and define a Softmax regression model.

""" Define placeholder: Where the data will be placed. 
Create a two-dimensional tensor for images and correct labels. 
None means no limits in length. """
# Placeholder for image data
with tf.name_scope("input") as scope:
    x = tf.placeholder(tf.float32, [None, 784])
# Placeholder for a correct answer label    
with tf.name_scope("y_") as scope:                              
    y_ = tf.placeholder(tf.float32, [None, 10])

""" Define Variable: The weight and bias to store learning results """
# Initialize it to 0. 
with tf.name_scope("weight") as scope:
    W = tf.Variable(tf.zeros([784, 10]))
with tf.name_scope("bias") as scope:
    b = tf.Variable(tf.zeros([10]))   

""" Define Model: Softmax regression 
Use Softmax to choose the value with the highest probability among 10 values. """
# Create a model     
with tf.name_scope("layer1") as scope:
    y = tf.nn.softmax(tf.matmul(x, W) + b)

w_hist = tf.summary.histogram("weight", W) 
b_hist = tf.summary.histogram("bias", b) 
y_hist = tf.summary.histogram("y", y)    

Training

Define Loss function and Learning Rate required for model training, and sample every 100 pieces of data to have the model learn 1000 times.
Increasing the sampled data can enhance accuracy, but the learning time is also increased.
Stochastic Training is to learn with a small batch to be randomly sampled and is commonly used because it can produce similar results at a reasonable price.

""" Model training """
# Define Loss function
with tf.name_scope("cost") as scope:
    cross_entropy = tf.reduce_mean(-tf.reduce_sum(y_ * tf.log(y), reduction_indices=[1]))
    #tf.summary.scalar("cost",cross_entropy)
    cost_sum = tf.summary.scalar("cost",cross_entropy)
# Define a learning rate as 0.5.
with tf.name_scope("train") as scope:    
    train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

# Merge all summaries into a single op
merged = tf.summary.merge_all()
    
# Initialize all variables before starting a session.
init = tf.global_variables_initializer()

sess = tf.Session()
sess.run(init)

# Default TensorBoard log directory: /home/ncp/workspace/tensorboard
writer =tf.summary.FileWriter("/home/ncp/workspace/tensorboard", sess.graph)

# Training: # Sample every 100 pieces of data to perform learning 1000 times.
for i in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)  # Import the ‘batch’ consisting of 100 pieces of randomly sampled data from the learning dataset. 
    summary, _ = sess.run([merged, train_step], feed_dict={x: batch_xs, y_: batch_ys})  # Provide the sampled batch_xs and batch_ys to placeholder x, y_. 
    # write summary events to disk
    writer.add_summary(summary,i)

Model evaluation

Find the label with the highest probability through tf.argmax.
Define the correct_prediction and accuracy tensor to get the same predicted value(y) and correct answer(y_) through tf.equal.

To evaluate the model, use test data to determine the accuracy.
Below, you have an accuracy of 0.9163, about 91%, and every time you re-train the model, the result may vary slightly.

" Model Evaluation "    
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))    
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Accuracy 
print sess.run(accuracy, feed_dict={x: mnist.test.images, y_: mnist.test.labels})

# Close the session when it’s done.
writer.close()    
sess.close()

0.9219