What is Tokenisation?
The process of breaking text into smaller pieces called tokens that an AI model can process.
Why It Matters
Tokenisation affects how much text a model can handle at once and how well it understands different words and languages.
Real-World Example
The word 'understanding' might be split into tokens like 'under', 'stand', and 'ing'.
“Understanding terms like Tokenisation matters because it helps you have better conversations with developers and make smarter decisions about your software. You do not need to be technical. You just need to know enough to ask the right questions.”
Related Terms
Large Language Model (LLM)
An AI system trained on massive amounts of text that can understand and generate human language.
Context Window
The maximum amount of text an AI model can consider at once when generating a response.
Embeddings
A way of representing words, sentences, or other data as lists of numbers that capture their meaning.
From definition to deployment
Knowing the term is step one. Using it in something real is the rest.