What is token?

A small piece of information an AI model processes. For text, a token may be a word, part of a word, or a punctuation mark.

Also called Tokens.

For example: A long word may count as several tokens.