In generative AI systems, what does the term "token" refer to?
Choose an answer
Tap an option to check your answer.
Correct answer: The basic units that the model processes as input and output, such as words, subwords, or other linguistic pieces..
Why this is the answer
In generative AI, a token is the fundamental unit of text that the model processes. These can be whole words, parts of words (subwords), or even individual characters, depending on the tokenization strategy. Models break down input text into tokens and generate output token by token. The numeric or vector representation of words or concepts inside the model refers to embeddings, not tokens themselves. Pretrained weight values are model parameters, not tokens. Prompts are the input instructions, which are then broken down into tokens for processing.
Pass your exam — without the endless answer hunt
Get every verified question and explanation for this exam in one place, and save hours of prep. 1,000+ certifications · 20+ languages · free to start.
Pass your exam faster → No card needed