Introduction
We will guide you through a real-world scenario, elucidating the essential steps involved in constructing the experiment, quantifying performance, and deriving meaningful insights from the results.
We’ll begin by defining the problem and outlining the specific use case for RAG in your application. Next, we’ll walk you through the process of designing and configuring your RAG pipeline within the Dataworkz platform. This will include selecting a suitable retrieval model, integrating a language model, and fine-tuning the pipeline parameters to optimize performance.
Once your pipeline is configured, we’ll show you how to run the experiment and collect performance metrics. We’ll discuss various evaluation metrics that can be used to assess the effectiveness of your RAG pipeline, such as accuracy, precision, recall, and F1 score.
Finally, we’ll interpret the results of your experiment and provide actionable insights for improving the performance of your RAG pipeline. We’ll also discuss common challenges and pitfalls to avoid when evaluating RAG pipelines, and offer tips for overcoming them.
By the end of this blog post, you will have a solid understanding of how to evaluate RAG pipelines using the Dataworkz platform, and be equipped with the knowledge and tools to apply these techniques to your own real-world applications.
Case Study: LegalbenchRAG Dataset
In our research, we utilized the legalbenchRAG repositories as the foundational datasets for our experimental investigations. These datasets were uploaded into an S3 bucket, and baseline Retrieval Augmented Generation (RAG) pipelines were constructed to process this data.
To facilitate the measurement of key metrics, we developed a Python package that seamlessly integrated with our API calls. This package was then leveraged to implement the metrics computation routines, utilizing the capabilities of the continuous-eval package. The continuous-eval package proved to be instrumental in this regard, as it furnished us with all the essential deterministic metrics required for evaluating the chunk retrieval process. Moreover, it offered additional functionalities for computing llm-based metrics, further enriching our analysis.
For the purpose of this evaluation, we relied upon the golden dataset that had been meticulously generated from the legalbenchrag repository. This dataset served as a reliable benchmark against which we could compare the results of our experiments. It is important to note that alternative approaches exist for establishing a ground truth dataset. For instance, the Dataworkz platform provides tools and techniques that enable the creation of questions from chunks, thereby facilitating the generation of a golden truth dataset.
The results from our experiments on the dataset are presented in the table below. The two experiments were conducted with variations in the chunk_size and chunk_overlap parameters. We observed that increasing the chunk_size led to a substantial improvement in both recall and average_precision metrics.
However, there was one exception to this trend: the maud dataset. In this dataset, the recall metric showed a marginal decrease. This anomaly could be attributed to the scarcity of relevant chunks in the top_k with the larger chunk_size. It’s possible that with a smaller chunk_size, these few relevant chunks were captured within a larger portion of the top_k set.
In contrast, the experiment with the increased chunk_size showed improvements in average_precision and ndcg metrics. This suggests that although some relevant chunks may have been missed, the overall ranking of the relevant chunks within the top_k list improved. In other words, the relevant chunks were positioned higher up in the list when a larger chunk_size was used.
These findings highlight the importance of parameter tuning in retrieval augmented generation. The optimal chunk_size and chunk_overlap values can significantly impact the performance of the system, and these values may need to be adjusted depending on the characteristics of the dataset being used.
| source | (chunk_size, chunk_overlap) | context_f1 | context_precision | context_recall | average_precision | ndcg | reciprocal_rank |
|---|---|---|---|---|---|---|---|
| contractnli | (500,0) | 2.74% | 1.55% | 19.50% | 5.75% | 8.58% | 6.32% |
| contractnli | (1500,500) | 8.57% | 5.25% | 41.83% | 20.77% | 23.75% | 21.90% |
| contractnli | (1500,500)+rerank | 19.07% | 13.40% | 39.83% | 37.00% | 33.99% | 37.48% |
| cuad | (2000,250) | 12.42% | 7.15% | 74.38% | 24.37% | 35.14% | 27.23% |
| cuad | (3000,300) | 13.60% | 7.85% | 83.17% | 30.02% | 40.30% | 32.61% |
| cuad | (3000,300)+rerank | 28.28% | 18.60% | 67.93% | 42.55% | 44.19% | 42.63% |
| maud | (3000,300) | 9.08% | 5.75% | 36.81% | 11.74% | 15.79% | 12.94% |
| maud | (6000,600) | 8.23% | 5.40% | 31.80% | 14.88% | 17.87% | 19.18% |
| maud | (6000,600)+rerank | 14.56% | 11.00% | 27.71% | 26.51% | 22.14% | 26.82% |
| privacy_qa | (500,0) | 6.90% | 3.88% | 42.70% | 19.87% | 23.24% | 21.85% |
| privacy_qa | (1500,300) | 13.56% | 7.52% | 95.75% | 53.95% | 64.90% | 62.98% |
| privacy_qa | (1500,300)+rerank | 35.16% | 24.10% | 82.53% | 59.23% | 60.42% | 62.59% |
Case Study: Finance RAG Dataset
In a comprehensive evaluation of Retrieval Augmented Generation (RAG), we utilized a substantial finance benchmark repository, containing over 360 extensive documents, each with 150-200 pages of content. To ensure accuracy and provide a basis for comparison, we employed an open-source “golden truth set” comprising 150 questions, each accompanied by their respective answers and relevant context excerpts.
The core of our RAG experiment involved the systematic alteration of the chunk embedding subroutine. We conducted two primary variations: one utilizing the ‘mpnet’ model and another employing the ‘openai-text-embedding-v3-large’ model. To further refine the results and explore potential enhancements in the retrieval process, we incorporated a re-ranker into the experimental setup specifically for the ‘openai’ embedding variation. This multi-faceted approach allowed us to thoroughly investigate the impact of different embedding models and re-ranking techniques on the overall effectiveness and performance of the RAG framework.
The empirical findings of the study highlight that the performance of a Retrieval Augmented Generation (RAG) system is significantly influenced by the specific configurations employed. For instance, while utilizing OpenAI’s large embedding model in conjunction with a reranking mechanism might lead to an improvement in average precision, it may also result in a decrease in context_f1 when compared to using MPNet embeddings. This suggests that there exist trade-offs between different performance metrics and that the optimal configuration will depend on the specific use case and the relative importance of different evaluation criteria.
| source | chunk_size, chunk_overlap,
embedding_model |
average_precision | context_f1 | context_precision | context_recall | ndcg | reciprocal_rank |
| finance_bench | (2000,300,mpnet) | 95.99% | 41.88% | 95.23% | 27.21% | 3.95% | 97.44% |
| finance_bench_v1 | (2000,300,openAI_large) + rerank | 97.73% | 28.79% | 96.65% | 17.37% | 2.36% | 98.27% |
Results and Interpretation
Our experiments showed that increasing chunk size generally improved recall and average precision metrics. However, we observed some variations depending on the dataset and specific configuration. These findings highlight the importance of parameter tuning in RAG and the value of the Dataworkz platform in facilitating this process.
Conclusion
Evaluating RAG pipelines on Dataworkz is a straightforward and insightful process. By leveraging the platform’s tools and capabilities, you can gain valuable insights into the performance of your RAG models and fine-tune them for optimal results.