GLOSSARY

The conversion of text or other input into model-specific token identifiers that a large language model can process.

Tokenisation affects context usage, latency, cost, multilingual behaviour, input limits and how reliably specialised terms, code or identifiers are represented.

Before sending long documents to an LLM, a pipeline counts tokens with the model's own tokenizer, preserves required instructions and chunks source text within the context limit.

Tokens are not equivalent to words or meaning, and counts vary by model and language. Truncation can remove critical context, while poor segmentation can disadvantage scripts, domains or uncommon strings.

A tokenizer normalises and segments input into words, subwords, characters or bytes, maps segments to vocabulary IDs and adds special tokens. Padding and truncation shape batches; decoding maps generated IDs back to text.