Which large language model parameter determines how many candidate next words or tokens are considered at each generation step?

Top K is a sampling parameter that specifies the number of highest probability tokens to consider at each step of the generation process. For example, if Top K is set to 5, the model will choose the next token from the 5 most probable options. This helps to make the output more diverse while still being coherent. Maximum tokens defines the maximum length of the generated output, not the number of candidates considered at each step. Temperature controls the randomness of the output; higher temperatures lead to more creative but potentially less coherent text, while lower temperatures make the output more deterministic. Batch size refers to the number of sequences processed simultaneously during inference or training, which is unrelated to token selection.