Token (in LLMs)

In Large Language Models, text is often broken down into smaller units called tokens for processing. Tokens can be whole words, parts of words (subwords), or even individual characters and punctuation. Model context windows and pricing are often measured in tokens.