Introduction
Traditionally, information retrieval systems relied heavily on keyword matching and manual indexing to fetch relevant data. However, the introduction of LLMs has revolutionized this process by enabling a more nuanced understanding of context, semantics, and user intent. RAG further enhances these capabilities by integrating retrieval mechanisms directly into the generation process, allowing for more accurate and contextually relevant responses. This synergy between LLMs and RAG represents a leap forward in the efficiency and effectiveness of information retrieval, making it possible to access and generate precise information in real-time, even from vast and complex datasets.
Dataworkz RAG Builder

With Dataworkz’s visual RAG builder, you can effortlessly develop GenAI applications without worrying about the complexity of the underlying infrastructure. Start by configuring lexical and semantic search using a knowledge graph for smart routing and employ GenAI to search across all your data sources, whether structured or unstructured, such as Box, Slack, G-Drive, PDF manuals, Sharepoint, Zendesk, RDBMS, No SQL databases, Data warehouse, Object stores like S3, Azure Blob, GCS and JIRA. Ensure end-to-end traceability to observe and optimize performance by monitoring indexing, retrieval and LLM calls in a single tool. Dataworkz provides transparency and allows customization at every stage of the RAG stack, offering a no-code interface to change and visualize chunks, A/B test RAG pipelines with built-in evaluation metrics, and deploy RAG as a Service in flexible models.
To start using Dataworkz visual RAG builder, create an account and refer to the documentation, Build Q&A with RAG.
Dataworkz Evaluation Framework (DEF)
The most popular metrics for evaluating QnA systems are BLEU, ROUGE-N and BertScore. These metrics are designed to measure textual similarity in some way or form, which can sometimes be misleading. Two LLM responses may be phrased differently yet still satisfy the same customer need, while two similarly phrased responses might differ significantly in their effectiveness.
On the other hand it is essential to provide an evaluation model that reflects how well the system is delivering the intended customer experience. Such a model should be able to capture the subjective experience of each individual answer. Implementation and assessment of such a model would need human(s) in the loop, who are trained to evaluate such models and are mostly domain experts.
To provide the best of both worlds, we introduce a state-of-the-art evaluation method (DEF) using LLM-as-a-Judge that captures the essence of each individual answer and also provides a measurable metric. We run our evaluation metrics across a publicly available dataset and QA benchmark. We investigate various popular metrics and provide a discussion on the limitation of the existing metrics. Finally, we provide a detailed introduction and exposition of DEF based on LLM-as-a-Judge.
DEF incorporates 3 evaluation metrics:
- Answer Accuracy – Comparison of Golden answer to Generated answer.
- Context Relevance – Comparison of Golden context to Generated context.
- Faithfulness – Comparison of Golden context to Generated answer.
This blog provides discussion and observation related to the first evaluation metric. The remaining evaluation metrics would be covered in a subsequent blog.
I. Answer Accuracy
Setup
We utilized a publicly available finance dataset, “Apple 10-K filing for the fiscal year 2022” which can be downloaded from the link provided in the References section. We leveraged FinQABench, a QA benchmark specifically designed for financial applications, which is based on Apple’s 2022 10-K filing.
FinQABench comprises 100 questions related to the Apple 10-K 2022 filing, along with golden answers and corresponding golden context. We tested all the FinQABench questions against the Dataworkz (DW) RAG system and evaluated the generated DW answers by comparing them to the FinQABench golden answers using the following metrics: BLEU Score, ROUGE-1, ROUGE-L, BERTScore (Precision, Recall, and F1), and Similarity Score.
Results and Observations with Existing Metrics
We ran all 100 tests in FinQABench and evaluated our answers using the metrics above. The average score were:
| Existing Metrics | ||||||
| Bleu Score | Rouge-1 | Rouge-L | Bert Precision | Bert Recall | Bert F1 | Similarity |
| 0.2951 | 0.5533 | 0.5095 | 0.3158 | 0.5354 | 0.4191 | 0.7302 |
On first look it seems that our system is performing badly however, a deep dive on the various results exposed us to the inherent limitations of evaluation metrics while comparing the golden answer with the Dataworkz generated answer. Although the semantic similarity score seems to be higher, we will observe later that it does not do a good job in capturing the context of each individual answer.
In the table below, although all responses from Dataworkz QnA system are correct, the scores received are substantially lower. This indicates that current metrics fail to adequately verify the Dataworkz responses against the golden responses. A critical limitation of these metrics is exposed: if the data is numerical or not formatted exactly like the golden answer, the scores suffer significantly.
| Fact Based QnA | |||||||
| Question | Golden Response | Dataworkz Response | Bleu Score | Rouge-1 | Rouge-L | Bert Score (F1) | Similarity Score |
| What is the date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. as stated in the document? | 2022-10-27 00:00:00 | The date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. is October 27, 2022. | 0.0000 | 0.1481 | 0.0741 | -0.6088 | 0.3347 |
| What is the jurisdiction of incorporation of Apple South Asia (Thailand) Limited? | Thailand | The jurisdiction of incorporation of Apple South Asia (Thailand) Limited is Thailand. | 0.0000 | 0.1538 | 0.1538 | -0.1914 | 0.5613 |
| What is the date of the report referenced in the document? | 2022-10-27 00:00:00 | The date of the report referenced in the document is October 27, 2022. | 0.0000 | 0.2105 | 0.1053 | -0.4998 | 0.4525 |
Additionally, we observe similar issues with “Descriptive QnA” below. We observe that the similarity score is high in the first question, however, this score is not consistent across all types of questions. The Dataworkz answer to the next question in the table below is evidence to that observation. It captures the main facts of the golden answer but the similarity score is not a good measure of its relevance.
| Descriptive QnA | |||||||
| Question | Golden Response | Dataworkz Response | Bleu Score | Rouge-1 | Rouge-L | Bert Score (F1) | Similarity Score |
| What is Rule 10b5-1 Trading Plans? | Rule 10b5-1 Trading Plans is a written document that preestablishes the amounts, prices and dates (or formula for determining the amounts, prices and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. | According to the context, Rule 10b5-1 Trading Plans refer to written documents that preestablish the amounts, prices, and dates (or formula for determining the amounts, prices, and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. | 0.5538 | 0.9216 | 0.9216 | 0.7631 | 0.9534 |
| Who signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022? | Timothy D. Cook, Luca Maestri, Chris Kondo, James A. Bell, Al Gore, Alex Gorsky, Andrea Jung, Arthur D. Levinson, Monica Lozano, Ronald D. Sugar, and Susan L. Wagner. | The following persons signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022: 1. Timothy D. Cook – Chief Executive Officer and Director (Principal Executive Officer) 2. Luca Maestri – Senior Vice President, Chief Financial Officer (Principal Financial Officer) 3. Chris Kondo – Senior Director of Corporate Accounting (Principal Accounting Officer) 4. James A. Bell – Director 5. Al Gore – Director 6. Alex Gorsky – Director 7. Andrea Jung – Director 8. Arthur D. Levinson – Director and Chair of the Board 9. Monica Lozano – Director 10. Ronald D. Sugar – Director 11. Susan L. Wagner – Director | 0.0092 | 0.4516 | 0.4355 | 0.0993 | 0.5076 |
Our observations indicate that any one metric is not able to consistently reflect the quality of answer score if their explanations do not closely align with the language of the golden response. This misalignment can be misleading, as it fails to accurately reflect the true quality and comprehensiveness of the Q&A system’s responses when compared to the benchmark. Traditional metrics seem to have substantial constraints to correlate with human preferences. Hence, we utilized the current state-of-the-art solution to build a scalable solution which aligns well with human evaluation called “LLM-as-a-Judge”. LLM-as-a-judge is a reference-free metric that directly prompts a powerful LLM to evaluate the quality of another model’s output.
LLM-as-a-Judge
Introduction
Large Language Models (LLMs) have become pivotal in the advancement of Question-Answering (QA) systems, not only for generating responses but also for evaluating them. Leveraging the language understanding capability of LLMs, they can generate metrics free assessment of the contextual accuracy of QA system outputs. By utilizing sophisticated evaluation criteria via Prompt engineering, LLMs can provide nuanced insights into the performance of QA systems, ensuring they meet high standards of precision and reliability.
Methodology
We developed a generic prompt capable of evaluating all types of responses, whether they are descriptive, numerical, or factual. Our evaluation criteria, embedded within the prompt, include the following essential instructions:
- Calculate the total number of claims in the golden response with respect to the question.
- Calculate the total number of claims in the candidate response with respect to the question.
- Calculate the total number of claims common to both the golden response and the candidate response.
This process generates lists of claims for both the golden response and the Dataworkz response. Once these claims are derived from the original responses, we calculate precision, recall, and F1 scores based on the following criteria:
- Precision: Number of claims common to both the golden response and Dataworkz response / Total number of claims in the Dataworkz response.
- Recall: Number of claims common to both the golden response and Dataworkz response / Total number of claims in the golden response.
- F1 Score: Harmonic mean of precision and recall.
Results with LLM-as-a-Judge
After applying these evaluation criteria to the generated answers, we found that it effectively captures the essence of the generated responses in comparison to the golden answers. The overall result for the metrics was:
| DEF Metrics | ||
| LLM Precision | LLM Recall | LLM F1 |
| 0.8252 | 0.7350 | 0.7490 |
The table below indicates that DEF was able to correctly evaluate the fact based answers irrespective of the format of the generated answer.
| Fact Based QnA | ||||||||
| Question | Golden Response | Dataworkz Response | LLM Score (F1) | Bleu Score | Rouge-1 | Rouge-L | Bert Score (F1) | Similarity Score |
| What is the date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. as stated in the document? | 2022-10-27 00:00:00 | The date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. is October 27, 2022. | 1.0000 | 0.0000 | 0.1481 | 0.0741 | -0.6088 | 0.3347 |
| What is the maturity date of the 0.875% 2025 Notes issued by Apple Inc.? | 2025-05-24 00:00:00 | The maturity date of the 0.875% 2025 Notes issued by Apple Inc. is May 24, 2025. | 1.0000 | 0.0000 | 0.1739 | 0.1739 | -0.5808 | 0.4227 |
| What is the date of the report referenced in the document? | 2022-10-27 00:00:00 | The date of the report referenced in the document is October 27, 2022. | 1.0000 | 0.0000 | 0.2105 | 0.1053 | -0.4998 | 0.4525 |
DEF evaluation was also consistent across the “Descriptive QnA” and was able to evaluate the content accurately.
| Descriptive QnA | ||||||||
| Question | Golden Response | Dataworkz Response | LLM Score (F1) | Bleu Score | Rouge-1 | Rouge-L | Bert Score (F1) | Similarity Score |
| What is Rule 10b5-1 Trading Plans? | Rule 10b5-1 Trading Plans is a written document that preestablishes the amounts, prices and dates (or formula for determining the amounts, prices and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. | According to the context, Rule 10b5-1 Trading Plans refer to written documents that preestablish the amounts, prices, and dates (or formula for determining the amounts, prices, and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. | 1.0000 | 0.5538 | 0.9216 | 0.9216 | 0.7631 | 0.9534 |
| Who signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022? | Timothy D. Cook, Luca Maestri, Chris Kondo, James A. Bell, Al Gore, Alex Gorsky, Andrea Jung, Arthur D. Levinson, Monica Lozano, Ronald D. Sugar, and Susan L. Wagner. | The following persons signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022: 1. Timothy D. Cook – Chief Executive Officer and Director (Principal Executive Officer) 2. Luca Maestri – Senior Vice President, Chief Financial Officer (Principal Financial Officer) 3. Chris Kondo – Senior Director of Corporate Accounting (Principal Accounting Officer) 4. James A. Bell – Director 5. Al Gore – Director 6. Alex Gorsky – Director 7. Andrea Jung – Director 8. Arthur D. Levinson – Director and Chair of the Board 9. Monica Lozano – Director 10. Ronald D. Sugar – Director 11. Susan L. Wagner – Director | 1.0000 | 0.0092 | 0.4516 | 0.4355 | 0.0993 | 0.5076 |
Example
To elaborate on how our evaluation system works, let us take the first question from the previous table:
Question: What is Rule 10b5-1 Trading Plans?
Part I – The first part is retrieving all individual claims from each response:
Total Number of claims in Golden response: 6
‘1’: ‘Rule 10b5-1 Trading Plans is a written document.’,
‘2’: “Rule 10b5-1 Trading Plans preestablish the amounts of future purchases or sales of the Company’s stock.”
‘3’: “Rule 10b5-1 Trading Plans preestablish the prices of future purchases or sales of the Company’s stock.”
‘4’: “Rule 10b5-1 Trading Plans preestablish the dates of future purchases or sales of the Company’s stock.”
‘5’: “Rule 10b5-1 Trading Plans include a formula for determining the amounts, prices, and dates of future purchases or sales of the Company’s stock.”
‘6’: “Rule 10b5-1 Trading Plans include sales of shares acquired under the Company’s employee and director equity plans.”
Total Number of claims in Dataworkz response: 6
‘1’: ‘Rule 10b5-1 Trading Plans refer to written documents.’
‘2’: “Rule 10b5-1 Trading Plans preestablish the amounts of future purchases or sales of the Company’s stock.”
‘3’: “Rule 10b5-1 Trading Plans preestablish the prices of future purchases or sales of the Company’s stock.”
‘4’: “Rule 10b5-1 Trading Plans preestablish the dates of future purchases or sales of the Company’s stock.”
‘5’: “Rule 10b5-1 Trading Plans include a formula for determining the amounts, prices, and dates of future purchases or sales of the Company’s stock.”
‘6’: “Rule 10b5-1 Trading Plans include sales of shares acquired under the Company’s employee and director equity plans.”
Total Number of common claims: 6
‘1’: ‘Rule 10b5-1 Trading Plans is a written document.’
‘2’: “Rule 10b5-1 Trading Plans preestablish the amounts of future purchases or sales of the Company’s stock.”
‘3’: “Rule 10b5-1 Trading Plans preestablish the prices of future purchases or sales of the Company’s stock.”
‘4’: “Rule 10b5-1 Trading Plans preestablish the dates of future purchases or sales of the Company’s stock.”
‘5’: “Rule 10b5-1 Trading Plans include a formula for determining the amounts, prices, and dates of future purchases or sales of the Company’s stock.”
‘6’: “Rule 10b5-1 Trading Plans include sales of shares acquired under the Company’s employee and director equity plans.”
Part II – Calculate the metrics based on the total claims.
Subsequently, we calculate the metrics based on the claims retrieved in the previous step.
- Precision: Number of common claims / Number of claims in Dataworkz response i.e. 6/6 = 1.
- Recall: Number of common claims / Number of claims in Golden response i.e. 6/6 = 1.
- F1 Score: Harmonic mean of precision and recall i.e. 1.
Final Thoughts
The results above demonstrate that the Dataworkz evaluation framework provides a comprehensive system for measuring QA systems against existing benchmarks. By leveraging LLM-as-a-Judge, this framework surpasses standard evaluation metrics like BLEU, ROUGE, and BERTScore, offering a more nuanced and accurate assessment of QA system performance. It is evident that traditional metrics like BLEU, ROUGE and BertScore have certain limitations for evaluating QnA systems and LLM-as-a-Judge can be utilized for generating qualitative and contextual metrics. We have open sourced our evaluation framework. The code is available here. For detailed and complete results, please refer to Evaluation Results.
References
- FinQABench
- Apple 10k Filing 2022
- Bleu Score
- Rouge Score
- Bert Score
- Using LLMs for Evaluation
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena