SDS 626: Subword Tokenization with Byte-Pair Encoding — with @JonKrohnLearns​

Word Tokenization

Jon Krohn introduces word tokenization as a foundational step in natural language processing (NLP). This method involves breaking down text into individual words using spaces as delimiters, which is straightforward but has limitations. If a word is not frequently present in the training data, it becomes an unknown token during model deployment, potentially ignoring important words 1.

A big drawback with such word level tokenization is that if a word didn't show up enough times in our training data, then when the NLP model encounters that word in production, there's no way to handle it.

--- Jon Krohn

To address this, character-level tokenization is introduced, allowing models to represent words by their constituent characters, though it requires more tokens and can lead to suboptimal performance 1.

   

Character Tokenization

Character tokenization breaks text into individual characters, offering a solution to the unknown token problem in word tokenization. This method allows models to represent any word by its characters, ensuring no word is ignored due to vocabulary limitations 1. However, it requires a large number of tokens and lacks the semantic richness of words, which can hinder model performance.

Unfortunately, character level tokenization also has its own drawbacks. For one, it requires a large number of tokens to represent a sequence of text.

--- Jon Krohn

Despite these drawbacks, character tokenization is used in techniques like ELMo, which leverages character embeddings to improve NLP tasks 1.

   

Subword Tokenization

Subword tokenization, particularly through byte-pair encoding, offers a balanced approach by combining the strengths of word and character tokenization. This method involves splitting words into meaningful subwords, which can be recombined to represent out-of-vocabulary words efficiently 2.

The upshot is that byte pair encoding is indeed so powerful that it is a crucial component behind many of the leading NLP models of today, such as Bertin, GPT-3 and Excelnet.

--- Jon Krohn

By using byte-pair encoding, NLP models can handle new words by leveraging known subwords, enhancing their ability to understand and process language 1.