Neural network to read short strings - translational invariance in CNNs

Question

I have a series of short strings that each describe some item (one item per string). The people who write these strings can get pretty creative when it comes to spelling.

For each string, I also have the label of the true object the strings refers to.

string          ->  true label
-------------------------------
lemon           ->  lemon
banana          ->  banana
strawberry      ->  strawberry
lemmonn         ->  lemon
llemon          ->  lemon
stawberry       ->  strawberry
ba nana         ->  banana
yellow fruit    ->  banana
small red fruit -> strawberry

There are millions of such examples, and I'm trying to train a convolutional network to identify the true labels given the strings. The main issue then is that it's hard to make the CNN translationally invariant.

Here's an example of when it becomes problematic:

• banana -> banana appear 120k times in the training dataset
• raw strawberry -> strawberry appears 80k times
• raw banana -> banana almost never appears (1 or 2 times only)

Now comes the problem: when trying to predict the string raw banana, the network outputs strawberry. In this case, it looks like the CNN has learned that something that starts with raw usually corresponds to strawberry and there aren't enough contradictory examples (esp. with banana) to challenge this.

My question is, how to make the CNN learn that banana clearly spelled after raw is more indicative of a banana than of a strawberry? More generally, how to make the CNN learn that banana is representative of a banana, even if not at the very beginning of the string?

I've tried prefixing the input strings with random strings (of variable length) so that a portion of the training data becomes f89jbanana -> banana, ah2qo banana -> banana but it doesn't seem to have much effect and the problem still remains.

Note on the structure of the CNN:

The CNN I'm using is made of 3 parallel blocks of Conv1d/BatchNorm/ReLU with respectively 2, 3 and 4 convolution kernels, blocks which then are concatenated together, result which goes into several convolutional steps with AveragePooling1D in between, finishing by a couple of dense layers.

Answer 1

You don't need deep learning for that. You have a list of keywords and need to match them. The problem is that they may be misspelled. Another problem is that sometimes you need to match a keyword with a different value (small red fruit -> strawberry). The first thing to notice is that those are two distinct problems.

1. If you have misspelled keywords, you can just use one of the many available spellchecking algorithms, just use them against your list of keywords instead of the generic language-specific lists.

   Another option is to use a fuzzy search algorithm, there are also many available implementations, depending on your preferences in Python, Go, and other languages. This is mostly about calculating edit distance between strings. If you want to use an out-of-the-box, scalable solution, ElasticSearch has a fuzzy search build-in.

   Yes, you could use deep learning instead, but there are many reasons why this should be your last resort.

   • For a neural network, you would need a lot of data such as the pairs you presented. We are talking of hundreds of thousands if not millions.
   • You would need to train it, tune, debug, maintain it. That's a lot of work.
   • Deep learning algorithm is a black-box algorithm, you don't know why and how exactly it makes its classifications. You have no guarantees that it doesn't invent some kind of crazy, overfitting rules that have nothing to do with your data.
   • Neural network would be considerably slower and more computationally intensive, hence more expensive.

   A neural network would basically be learning to mimic what the fuzzy search algorithms are already doing: finding substrings with the smallest distance to the target keywords or will memorize your data and become a computationally inefficient lookup table. Why re-inventing the wheel?

   Finally, if you had a simple algorithm that works but misses edge cases, you could use it for the majority of the data and then focus on building a machine learning solution just for the edge cases.

2. As for matching keywords with some other value, you can treat this as a two-step algorithm. In the first step, you do the fuzzy matching for the keywords. In the second step, you just use a hash table to map keywords to different values if needed. This is $O(1)$ complexity problem vs computationally intensive solution with deep learning.

   If you still insisted on using deep learning, I'd recommend having two different models: the first one learning to correct misspelled words and the second one doing a classification.

Answer 2

If you really want to use deep learning for this, then I'd consider a character-level recurrent neural network (such as a bidirectional LSTM) or if you want a transformer, which would take as the input the sequence of characters and would use as its output a category (either 1) if you have a fixed list of possible outputs, those categories, 2) categorical output for the words in an appropriate dictionary, or 3) a sequence of characters). Using a model pre-trained on some English language corpus would seem obvious, because that would likely give the model a lot of relevant context beyond your training data (e.g. https://huggingface.co/transformers/v3.4.0/model_doc/longformer.html).

In fact, DistilGPT2 is a pretty good at this task with just a few examples (few shot learning). I provided this input

Input:banana

Output:banana

Input:raw strawberry

Output:strawberry

Input:llemon

Output:lemon

Input:pear

Output:pear

Input:aple

Output:apple

Input:rotten fig

Output:fig

Input:raw banana

Output:

and the model (gpt2-large, temperature 0, max-time 5) auto-completed this to

Input:banana

Output:banana

Input:raw strawberry

Output:strawberry

Input:llemon

Output:lemon

Input:pear

Output:pear

Input:aple

Output:apple

Input:rotten fig

Output:fig

Input:raw banana

Output: banana

Input:raw

So, it seems like a transformer model can deal with this in a few shot fashion.

What you tried in terms of data augmentation (i.e. adding some extra characters) sounds sensible, but I'd also consider other ideas (e.g. adding extra random words from a dictionary - perhaps making sure not to pick any fruit words - and randomly swapping letters). There's various packages for data augmentation with text data that you could try.