- Coherence: The degree to which the model’s output makes sense in the given context. For example, a language model generating a long passage should maintain logical flow, topic consistency, and narrative coherence.
- Consistency: For conversational AI or story generation models, does the output remain consistent over time? For instance, if the model is generating a dialogue, does it maintain consistent character traits, facts, or storyline elements?
Coherence and Consistency
Metrics:
- Perplexity (for text models): A measure of how well the probability distribution predicted by the model fits the actual next word in a sequence. Lower perplexity means the model's predictions are more likely to match real-world language patterns, thus enhancing coherence.
- BLEU and ROUGE Scores: These are commonly used to measure the consistency and fidelity of generated text in comparison to human-written reference text in tasks like machine translation, summarization, and text generation.