Byte Pair Encoding (BPE)
NLP & Text
Subword segmentation algorithm
What is Byte Pair Encoding (BPE)?
Builds vocabulary of common subword units to handle rare words and languages efficiently.
Real-World Examples
- •Used in many transformer tokenizers
Related Terms
Learn more about concepts related to Byte Pair Encoding (BPE)