Keras text classification

Available in Classic and VPC

This notebook classifies movie review text into positive or negative text. This example is a binary classification exercise, or with- two classes. Binary classification is important and widely used in machine learning.

We're going to use IMDB dataset which contains 50,000 movie reviews collected from Internet movie database. They are divided with 25,000 reviews for training and 25,000 reviews for testing. The classes of the training set and test set are balanced. In other words, the numbers of positive reviews and negative reviews are identical.

This notebook uses tf.keras, which is TensorFlow's high-level Python API, to create and train models. Refer to MLCC text classification guide for the advanced test classification tutorial using tf.keras.

import tensorflow as tf
from tensorflow import keras

import numpy as np

print(tf.__version__)

# 2.3.0

Download IMDB dataset

The IMDB dataset is provided with TensorFlow. The reviews (sequences of words) are preprocessed and converted to sequences of integers in advance. Each integer refers to a specific word in the word dictionary.

The following code downloads the IMDB dataset to the computer (or uses cached copy if it has been downloaded before.):

imdb = keras.datasets.imdb

(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

Parameter num_words=10000 selects the top 10,000 words that appear most frequently in the training data. We'll exclude words that appear less frequently to maintain an adequate data size.

Explore data

Let's have a brief look at the data format. This dataset's samples are preprocessed sequences of integers. These integers indicate words that appear in the movie reviews. A label is integer 0 or 1. 0 means it's a negative review, and 1 means it's a positive review.

print("training sample: {}, label: {}".format(len(train_data), len(train_labels)))

# Training sample: 25000, label: 25000

Review texts are converted to integers that signify specific words in the word dictionary. Let's check the first review:

print(train_data[0])

# [1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]

Movie reviews have different lengths. The following code prints the number of words from the first and second reviews. Inputs to a neural network must have the same lengths, so let's look into this issue later to solve it.

len(train_data[0]), len(train_data[1])

# (218, 189)

Reconvert integer to word again

It'd be useful if there's a way to reconvert the integers to text again. We're going to create a helper function that queries dictionary objects that have integers and strings mapped:

# Dictionary where words and integer indexes are mapped
word_index = imdb.get_word_index()

# The first few indexes are defined in the dictionary.
word_index = {k:(v+3) for k,v in word_index.items()}
word_index["<PAD>"] = 0
word_index["<START>"] = 1
word_index["<UNK>"] = 2  # unknown
word_index["<UNUSED>"] = 3

reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

def decode_review(text):
    return ' '.join([reverse_word_index.get(i, '?') for i in text])


"""
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json
1646592/1641221 [==============================] - 0s 0us/step
"""

Now, you can use the decode_review function to print the first review text:

decode_review(train_data[0])

# "<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy's that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all"

Prepare data

Reviews - integer sequences - have to be converted to tensors before being injected to a neural network. There are a few conversion methods:

• One-hot encoding converts an integer sequence to a vector of 0s and 1s. For example, the sequence [3, 5] can be converted into a 10,000-dimensional vector where only index 3 and 5 are 1, and the rest is all 0s. And then, it uses the layer that can deal with real number vector data, a dense layer, as the first layer of the neural network. This method uses a lot of memory since it requires a matrix the size of num_words * num_reviews.
• As an alternative, you can create an integer tensor the size of max_length * num_reviews by adding pads, so the lengths of all integer sequences would be the same. You can use an embedding layer that can deal with tensors of such a format as the first layer of the neural network.

We're going to use the second method in this tutorial.

We're going to use the pad_sequences function to align lengths because the lengths of all the movie reviews must be the same:

train_data = keras.preprocessing.sequence.pad_sequences(train_data,
                                                        value=word_index["<PAD>"],
                                                        padding='post',
                                                        maxlen=256)

test_data = keras.preprocessing.sequence.pad_sequences(test_data,
                                                       value=word_index["<PAD>"],
                                                       padding='post',
                                                       maxlen=256)
``
Let's check the sample's length:
```python
len(train_data[0]), len(train_data[1])

# (256, 256)

Let's check the first review's (padded) content:

print(train_data[0])

"""
[   1   14   22   16   43  530  973 1622 1385   65  458 4468   66 3941
    4  173   36  256    5   25  100   43  838  112   50  670    2    9
   35  480  284    5  150    4  172  112  167    2  336  385   39    4
  172 4536 1111   17  546   38   13  447    4  192   50   16    6  147
 2025   19   14   22    4 1920 4613  469    4   22   71   87   12   16
   43  530   38   76   15   13 1247    4   22   17  515   17   12   16
  626   18    2    5   62  386   12    8  316    8  106    5    4 2223
 5244   16  480   66 3785   33    4  130   12   16   38  619    5   25
  124   51   36  135   48   25 1415   33    6   22   12  215   28   77
   52    5   14  407   16   82    2    8    4  107  117 5952   15  256
    4    2    7 3766    5  723   36   71   43  530  476   26  400  317
   46    7    4    2 1029   13  104   88    4  381   15  297   98   32
 2071   56   26  141    6  194 7486   18    4  226   22   21  134  476
   26  480    5  144   30 5535   18   51   36   28  224   92   25  104
    4  226   65   16   38 1334   88   12   16  283    5   16 4472  113
  103   32   15   16 5345   19  178   32    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0    0    0    0    0    0    0    0    0    0    0
    0    0    0    0]
"""