Tokenizers

April 1, 2020

Fast Tokenizers

Slow Tokenizers

Byte Pair Encodings

Word Piece

Unigram

Sentence Piece