Editors need a search tool that returns articles where queried words are most important and representative in the document corpus. The initial tool returns general word matches and requires manual filtering. Which solution will capture per-document word importance and frequency so the tool highlights articles where queried words matter most?

The correct solution uses TF-IDF (Term Frequency-Inverse Document Frequency), a standard method for evaluating how important a word is to a document in a corpus. Term Frequency (TF) measures how often a word appears in a document, indicating its importance within that specific document. Inverse Document Frequency (IDF) measures how unique or rare a word is across the entire corpus; words appearing in many documents have a lower IDF, while rare words have a higher IDF. Multiplying TF and IDF provides a score that highlights words that are frequent in a specific document but rare across the corpus, making them highly representative of that document's content. LDA (Latent Dirichlet Allocation) is a topic modeling technique that identifies latent topics in documents, but it doesn't directly provide a per-word importance score for search relevance in the same way TF-IDF does. Word embeddings capture semantic relationships but are not designed to directly score word importance for search relevance based on frequency and rarity. A simple raw term frequency (even with stop word removal) doesn't account for the word's rarity across the entire corpus, meaning common words would still get high scores even if they aren't discriminative.