RAG frameworks have gained attention for their ability to enhance LLMs by integrating external knowledge sources, helping address limitations like hallucinations and outdated information. Traditional RAG approaches often rely on surface-level document relevance despite their potential, missing deeply embedded insights within texts or overlooking information spread across multiple sources. These methods are also limited in their applicability, primarily catering to simple question-answering tasks and struggling with more complex applications, such as synthesizing insights from varied qualitative data or analyzing intricate legal or business content. While earlier RAG models improved accuracy in tasks like summarization and open-domain QA, their retrieval mechanisms…

Transformers have emerged as foundational tools in machine learning, underpinning models that operate on sequential and structured data. One critical challenge in this setup is enabling the model to understand the position of tokens or inputs since Transformers inherently lack a mechanism for encoding order. Rotary Position Embedding (RoPE) became a popular solution, especially in language and vision tasks, because it efficiently encodes absolute positions to facilitate relative spatial understanding. As these models grow in complexity and application across modalities, enhancing the expressiveness and dimensional flexibility of RoPE has become increasingly significant. A significant challenge arises when scaling RoPE, from…