> For the complete documentation index, see [llms.txt](https://docs.aigentx.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.aigentx.xyz/overview/ai-fundamentals/tokenization.md).

# Tokenization

In artificial intelligence, tokenization is the process of converting information into small, manageable units or tokens. This method, often referred to as Byte-pair encoding, involves segmenting text into smaller groups of characters and assigning them labels for efficient storage and interpretation by a computer's binary system.

Take, for instance, the sequence of letters "i-n-g." Individually, each letter is a separate token, but combined as "ing," they form a familiar suffix used in forming the present participle of verbs (e.g., ending, meaning, voting). This concept extends to various two-letter combinations within the sequence, like "ig," "ng," or "gi," where each pair becomes a distinct token. Through tokenization, identifying patterns within vast datasets becomes more straightforward than analyzing each character separately. This approach allows for nuanced understanding, distinguishing when "in" stands alone as a word or when it's part of a larger string, thereby enhancing the model's ability to accurately parse and interpret text.

<br>