Agentic AI

Dataworkz RAG Builder: Evaluation Framework

Why text-similarity metrics mislead, and how an LLM-as-a-Judge approach scores answers the way people do.

Key takeaways

  • Traditional QA-system metrics — BLEU, ROUGE-N, and BERTScore — measure textual similarity and can be misleading: differently phrased answers can satisfy the same need, while similarly phrased answers can differ greatly in effectiveness.
  • The Dataworkz Evaluation Framework (DEF) uses an LLM-as-a-Judge approach and incorporates three metrics: Answer Accuracy (golden vs. generated answer), Context Relevance (golden vs. generated context), and Faithfulness (golden context vs. generated answer); this post covers the first metric, Answer Accuracy.
  • Testing used FinQABench, a 100-question QA benchmark based on Apple's 2022 10-K filing, with golden answers and golden context; existing metrics scored Dataworkz answers low (e.g., BLEU 0.2951, ROUGE-1 0.5533, Similarity 0.7302 on average) even when answers were correct, because numerical or differently formatted answers are penalized.
  • DEF works by extracting the claims in the golden response and the candidate response, counting common claims, and computing precision, recall, and F1; on the same benchmark DEF scored LLM Precision 0.8252, LLM Recall 0.7350, and LLM F1 0.7490, correctly evaluating fact-based and descriptive answers regardless of format.
  • Dataworkz has open-sourced the evaluation framework code, and the post notes the remaining metrics (Context Relevance and Faithfulness) will be covered in a subsequent blog.
On this pagesections

    Introduction

    Traditionally, information retrieval systems relied heavily on keyword matching and manual indexing to fetch relevant data. However, the introduction of LLMs has revolutionized this process by enabling a more nuanced understanding of context, semantics, and user intent. RAG further enhances these capabilities by integrating retrieval mechanisms directly into the generation process, allowing for more accurate and contextually relevant responses. This synergy between LLMs and RAG represents a leap forward in the efficiency and effectiveness of information retrieval, making it possible to access and generate precise information in real-time, even from vast and complex datasets.

    Dataworkz RAG Builder

    With Dataworkz’s visual RAG builder, you can effortlessly develop GenAI applications without worrying about the complexity of the underlying infrastructure. Start by configuring lexical and semantic search using a knowledge graph for smart routing and employ GenAI to search across all your data sources, whether structured or unstructured, such as Box, Slack, G-Drive, PDF manuals, Sharepoint, Zendesk, RDBMS, No SQL databases, Data warehouse, Object stores like S3, Azure Blob, GCS and JIRA. Ensure end-to-end traceability to observe and optimize performance by monitoring indexing, retrieval and LLM calls in a single tool. Dataworkz provides transparency and allows customization at every stage of the RAG stack, offering a no-code interface to change and visualize chunks, A/B test RAG pipelines with built-in evaluation metrics, and deploy RAG as a Service in flexible models.

    To start using Dataworkz visual RAG builder, create an account and refer to the documentation, Build Q&A with RAG.

    Dataworkz Evaluation Framework (DEF)

    The most popular metrics for evaluating QnA systems are BLEU, ROUGE-N and BertScore. These metrics are designed to measure textual similarity in some way or form, which can sometimes be misleading. Two LLM responses may be phrased differently yet still satisfy the same customer need, while two similarly phrased responses might differ significantly in their effectiveness.

    On the other hand it is essential to provide an evaluation model that reflects how well the system is delivering the intended customer experience. Such a model should be able to capture the subjective experience of each individual answer. Implementation and assessment of such a model would need human(s) in the loop, who are trained to evaluate such models and are mostly domain experts.

    To provide the best of both worlds, we introduce a state-of-the-art evaluation method (DEF) using LLM-as-a-Judge that captures the essence of each individual answer and also provides a measurable metric. We run our evaluation metrics across a publicly available dataset and QA benchmark. We investigate various popular metrics and provide a discussion on the limitation of the existing metrics. Finally, we provide a detailed introduction and exposition of DEF based on LLM-as-a-Judge.

    DEF incorporates 3 evaluation metrics:

    • Answer Accuracy – Comparison of Golden answer to Generated answer.
    • Context Relevance – Comparison of Golden context to Generated context.
    • Faithfulness – Comparison of Golden context to Generated answer.

    This blog provides discussion and observation related to the first evaluation metric. The remaining evaluation metrics would be covered in a subsequent blog.

    I. Answer Accuracy

    Setup

    We utilized a publicly available finance dataset, “Apple 10-K filing for the fiscal year 2022” which can be downloaded from the link provided in the References section. We leveraged FinQABench, a QA benchmark specifically designed for financial applications, which is based on Apple’s 2022 10-K filing.

    FinQABench comprises 100 questions related to the Apple 10-K 2022 filing, along with golden answers and corresponding golden context. We tested all the FinQABench questions against the Dataworkz (DW) RAG system and evaluated the generated DW answers by comparing them to the FinQABench golden answers using the following metrics: BLEU Score, ROUGE-1, ROUGE-L, BERTScore (Precision, Recall, and F1), and Similarity Score.

    Results and Observations with Existing Metrics

    We ran all 100 tests in FinQABench and evaluated our answers using the metrics above. The average score were:

    Existing Metrics
    Bleu Score Rouge-1 Rouge-L Bert Precision Bert Recall Bert F1 Similarity
    0.2951 0.5533 0.5095 0.3158 0.5354 0.4191 0.7302

    On first look it seems that our system is performing badly however, a deep dive on the various results exposed us to the inherent limitations of evaluation metrics while comparing the golden answer with the Dataworkz generated answer. Although the semantic similarity score seems to be higher, we will observe later that it does not do a good job in capturing the context of each individual answer.

    In the table below, although all responses from Dataworkz QnA system are correct, the scores received are substantially lower. This indicates that current metrics fail to adequately verify the Dataworkz responses against the golden responses. A critical limitation of these metrics is exposed: if the data is numerical or not formatted exactly like the golden answer, the scores suffer significantly.

    Fact Based QnA
    Question Golden Response Dataworkz Response Bleu Score Rouge-1 Rouge-L Bert Score (F1) Similarity Score
    What is the date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. as stated in the document? 2022-10-27 00:00:00 The date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. is October 27, 2022. 0.0000 0.1481 0.0741 -0.6088 0.3347
    What is the jurisdiction of incorporation of Apple South Asia (Thailand) Limited? Thailand The jurisdiction of incorporation of Apple South Asia (Thailand) Limited is Thailand. 0.0000 0.1538 0.1538 -0.1914 0.5613
    What is the date of the report referenced in the document? 2022-10-27 00:00:00 The date of the report referenced in the document is October 27, 2022. 0.0000 0.2105 0.1053 -0.4998 0.4525

    Additionally, we observe similar issues with “Descriptive QnA” below. We observe that the similarity score is high in the first question, however, this score is not consistent across all types of questions. The Dataworkz answer to the next question in the table below is evidence to that observation. It captures the main facts of the golden answer but the similarity score is not a good measure of its relevance.

    Descriptive QnA
    Question Golden Response Dataworkz Response Bleu Score Rouge-1 Rouge-L Bert Score (F1) Similarity Score
    What is Rule 10b5-1 Trading Plans? Rule 10b5-1 Trading Plans is a written document that preestablishes the amounts, prices and dates (or formula for determining the amounts, prices and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. According to the context, Rule 10b5-1 Trading Plans refer to written documents that preestablish the amounts, prices, and dates (or formula for determining the amounts, prices, and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. 0.5538 0.9216 0.9216 0.7631 0.9534
    Who signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022? Timothy D. Cook, Luca Maestri, Chris Kondo, James A. Bell, Al Gore, Alex Gorsky, Andrea Jung, Arthur D. Levinson, Monica Lozano, Ronald D. Sugar, and Susan L. Wagner. The following persons signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022:  1. Timothy D. Cook – Chief Executive Officer and Director (Principal Executive Officer) 2. Luca Maestri – Senior Vice President, Chief Financial Officer (Principal Financial Officer) 3. Chris Kondo – Senior Director of Corporate Accounting (Principal Accounting Officer) 4. James A. Bell – Director 5. Al Gore – Director 6. Alex Gorsky – Director 7. Andrea Jung – Director 8. Arthur D. Levinson – Director and Chair of the Board 9. Monica Lozano – Director 10. Ronald D. Sugar – Director 11. Susan L. Wagner – Director 0.0092 0.4516 0.4355 0.0993 0.5076

    Our observations indicate that any one metric is not able to consistently reflect the quality of answer score if their explanations do not closely align with the language of the golden response. This misalignment can be misleading, as it fails to accurately reflect the true quality and comprehensiveness of the Q&A system’s responses when compared to the benchmark. Traditional metrics seem to have substantial constraints to correlate with human preferences. Hence, we utilized the current state-of-the-art solution to build a scalable solution which aligns well with human evaluation called “LLM-as-a-Judge”. LLM-as-a-judge is a reference-free metric that directly prompts a powerful LLM to evaluate the quality of another model’s output.

    LLM-as-a-Judge

    Introduction

    Large Language Models (LLMs) have become pivotal in the advancement of Question-Answering (QA) systems, not only for generating responses but also for evaluating them. Leveraging the language understanding capability of LLMs, they can generate metrics free assessment of the contextual accuracy of QA system outputs. By utilizing sophisticated evaluation criteria via Prompt engineering, LLMs can provide nuanced insights into the performance of QA systems, ensuring they meet high standards of precision and reliability.

    Methodology

    We developed a generic prompt capable of evaluating all types of responses, whether they are descriptive, numerical, or factual. Our evaluation criteria, embedded within the prompt, include the following essential instructions:

    1. Calculate the total number of claims in the golden response with respect to the question.
    2. Calculate the total number of claims in the candidate response with respect to the question.
    3. Calculate the total number of claims common to both the golden response and the candidate response.

    This process generates lists of claims for both the golden response and the Dataworkz response. Once these claims are derived from the original responses, we calculate precision, recall, and F1 scores based on the following criteria:

    • Precision: Number of claims common to both the golden response and Dataworkz response / Total number of claims in the Dataworkz response.
    • Recall: Number of claims common to both the golden response and Dataworkz response / Total number of claims in the golden response.
    • F1 Score: Harmonic mean of precision and recall.

    Results with LLM-as-a-Judge

    After applying these evaluation criteria to the generated answers, we found that it effectively captures the essence of the generated responses in comparison to the golden answers. The overall result for the metrics was:

    DEF Metrics
    LLM Precision LLM Recall LLM F1
    0.8252 0.7350 0.7490

    The table below indicates that DEF was able to correctly evaluate the fact based answers irrespective of the format of the generated answer.

    Fact Based QnA
    Question Golden Response Dataworkz Response LLM Score (F1) Bleu Score Rouge-1 Rouge-L Bert Score (F1) Similarity Score
    What is the date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. as stated in the document? 2022-10-27 00:00:00 The date of the certification of the Chief Executive Officer and Chief Financial Officer of Apple Inc. is October 27, 2022. 1.0000 0.0000 0.1481 0.0741 -0.6088 0.3347
    What is the maturity date of the 0.875% 2025 Notes issued by Apple Inc.? 2025-05-24 00:00:00 The maturity date of the 0.875% 2025 Notes issued by Apple Inc. is May 24, 2025. 1.0000 0.0000 0.1739 0.1739 -0.5808 0.4227
    What is the date of the report referenced in the document? 2022-10-27 00:00:00 The date of the report referenced in the document is October 27, 2022. 1.0000 0.0000 0.2105 0.1053 -0.4998 0.4525

    DEF evaluation was also consistent across the “Descriptive QnA” and was able to evaluate the content accurately.

    Descriptive QnA
    Question Golden Response Dataworkz Response LLM Score (F1) Bleu Score Rouge-1 Rouge-L Bert Score (F1) Similarity Score
    What is Rule 10b5-1 Trading Plans? Rule 10b5-1 Trading Plans is a written document that preestablishes the amounts, prices and dates (or formula for determining the amounts, prices and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. According to the context, Rule 10b5-1 Trading Plans refer to written documents that preestablish the amounts, prices, and dates (or formula for determining the amounts, prices, and dates) of future purchases or sales of the Company’s stock, including sales of shares acquired under the Company’s employee and director equity plans. 1.0000 0.5538 0.9216 0.9216 0.7631 0.9534
    Who signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022? Timothy D. Cook, Luca Maestri, Chris Kondo, James A. Bell, Al Gore, Alex Gorsky, Andrea Jung, Arthur D. Levinson, Monica Lozano, Ronald D. Sugar, and Susan L. Wagner. The following persons signed the Annual Report on Form 10-K on behalf of the Registrant on October 27, 2022:  1. Timothy D. Cook – Chief Executive Officer and Director (Principal Executive Officer) 2. Luca Maestri – Senior Vice President, Chief Financial Officer (Principal Financial Officer) 3. Chris Kondo – Senior Director of Corporate Accounting (Principal Accounting Officer) 4. James A. Bell – Director 5. Al Gore – Director 6. Alex Gorsky – Director 7. Andrea Jung – Director 8. Arthur D. Levinson – Director and Chair of the Board 9. Monica Lozano – Director 10. Ronald D. Sugar – Director 11. Susan L. Wagner – Director 1.0000 0.0092 0.4516 0.4355 0.0993 0.5076

    Example

    To elaborate on how our evaluation system works, let us take the first question from the previous table:

    Question: What is Rule 10b5-1 Trading Plans?

    Part I – The first part is retrieving all individual claims from each response:

    Total Number of claims in Golden response: 6

    ‘1’: ‘Rule 10b5-1 Trading Plans is a written document.’,

    ‘2’: “Rule 10b5-1 Trading Plans preestablish the amounts of future purchases or sales of the Company’s stock.”

    ‘3’: “Rule 10b5-1 Trading Plans preestablish the prices of future purchases or sales of the Company’s stock.”

    ‘4’: “Rule 10b5-1 Trading Plans preestablish the dates of future purchases or sales of the Company’s stock.”

    ‘5’: “Rule 10b5-1 Trading Plans include a formula for determining the amounts, prices, and dates of future purchases or sales of the Company’s stock.”

    ‘6’: “Rule 10b5-1 Trading Plans include sales of shares acquired under the Company’s employee and director equity plans.”

     

    Total Number of claims in Dataworkz response: 6

    ‘1’: ‘Rule 10b5-1 Trading Plans refer to written documents.’

    ‘2’: “Rule 10b5-1 Trading Plans preestablish the amounts of future purchases or sales of the Company’s stock.”

    ‘3’: “Rule 10b5-1 Trading Plans preestablish the prices of future purchases or sales of the Company’s stock.”

    ‘4’: “Rule 10b5-1 Trading Plans preestablish the dates of future purchases or sales of the Company’s stock.”

    ‘5’: “Rule 10b5-1 Trading Plans include a formula for determining the amounts, prices, and dates of future purchases or sales of the Company’s stock.”

    ‘6’: “Rule 10b5-1 Trading Plans include sales of shares acquired under the Company’s employee and director equity plans.”

     

    Total Number of common claims: 6

    ‘1’: ‘Rule 10b5-1 Trading Plans is a written document.’

    ‘2’: “Rule 10b5-1 Trading Plans preestablish the amounts of future purchases or sales of the Company’s stock.”

    ‘3’: “Rule 10b5-1 Trading Plans preestablish the prices of future purchases or sales of the Company’s stock.”

    ‘4’: “Rule 10b5-1 Trading Plans preestablish the dates of future purchases or sales of the Company’s stock.”

    ‘5’: “Rule 10b5-1 Trading Plans include a formula for determining the amounts, prices, and dates of future purchases or sales of the Company’s stock.”

    ‘6’: “Rule 10b5-1 Trading Plans include sales of shares acquired under the Company’s employee and director equity plans.”

     

    Part II – Calculate the metrics based on the total claims.

    Subsequently, we calculate the metrics based on the claims retrieved in the previous step.

    1. Precision: Number of common claims / Number of claims in Dataworkz response i.e. 6/6 = 1.
    2. Recall: Number of  common claims / Number of claims in Golden response i.e. 6/6 = 1.
    3. F1 Score: Harmonic mean of precision and recall i.e. 1.

    Final Thoughts

    The results above demonstrate that the Dataworkz evaluation framework provides a comprehensive system for measuring QA systems against existing benchmarks. By leveraging LLM-as-a-Judge, this framework surpasses standard evaluation metrics like BLEU, ROUGE, and BERTScore, offering a more nuanced and accurate assessment of QA system performance. It is evident that traditional metrics like BLEU, ROUGE and BertScore have certain limitations for evaluating QnA systems and LLM-as-a-Judge can be utilized for generating qualitative and contextual metrics. We have open sourced our evaluation framework. The code is available here. For detailed  and complete results, please refer to Evaluation Results.

    References

    1. FinQABench
    2. Apple 10k Filing 2022
    3. Bleu Score
      1. Wikipedia
      2. Bleu Metrics
    4. Rouge Score
      1. Wikipedia
      2. Rouge Metrics
    5. Bert Score
      1. Paper
      2. Bert Score
    6. Using LLMs for Evaluation
    7. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    Frequently asked questions

    What is the Dataworkz Evaluation Framework (DEF)?

    DEF is an evaluation method that uses LLM-as-a-Judge to assess QA/RAG system answers. It incorporates three metrics — Answer Accuracy, Context Relevance, and Faithfulness — and aims to capture the quality of individual answers while still producing a measurable, scalable metric that aligns with human evaluation.

    Why are traditional metrics like BLEU, ROUGE, and BERTScore insufficient for evaluating QA systems?

    These metrics measure textual similarity, so they can penalize answers that are correct but phrased or formatted differently from the golden answer. The post shows correct Dataworkz answers receiving very low scores, especially when the answer is numerical or not formatted exactly like the golden response.

    How does LLM-as-a-Judge score an answer in DEF?

    It extracts the list of claims in both the golden response and the candidate response, counts the claims common to both, and then computes precision (common claims / candidate claims), recall (common claims / golden claims), and the F1 score as the harmonic mean of the two.

    What dataset and benchmark were used to test DEF?

    The evaluation used a publicly available finance dataset — Apple's 10-K filing for fiscal year 2022 — and FinQABench, a 100-question QA benchmark based on that filing, which provides golden answers and golden context.

    See what your engineers can ship in production

    Dataworkz is the agentic AI platform enterprises trust to put AI agents into production. Teams build production-ready agents in days, every decision stays traceable, and every agent has ROI attached.

    Explore the platform