In Retrieval Augmented Generation (RAG), what is the main purpose of splitting large documents into chunks?

Splitting large documents into smaller, manageable chunks is crucial for RAG because it improves the contextual relevance of retrieved information. When a query is made, the system searches for chunks that are most semantically similar to the query. If documents are too large, an entire document might be retrieved even if only a small portion is relevant, introducing noise and diluting the context. Smaller chunks ensure that the retrieved information is more focused and directly addresses the query, leading to more accurate and helpful generations. Chunking does not primarily aim to avoid database storage limits or lower storage costs, nor does it remove the need for vector embeddings; in fact, each chunk is typically converted into its own vector embedding.