Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] How to provide context to LLM RAG Evaluator when the documents are chunked and stored in a vector database #1194

Open
2 of 3 tasks
railsstudent opened this issue Dec 10, 2024 · 1 comment
Labels
documentation Improvements or additions to documentation

Comments

@railsstudent
Copy link

Description
I would like to use LLM RAG Evaluator to validate the response against the prompt and the context. This validator expects the context to be a string. However, my RAG application split the PDF document into chunked documents and store them in vector store. If I want to pass the context to the evaluator, I do not know how to do it. It is because qdrant returns the context in an array of JSON objects.

Please advice. Thanks.

Current documentation
https://hub.guardrailsai.com/validator/arize-ai/llm_rag_evaluator

Suggested changes
Please provide an example to provide the context to the validator when the documents are chunked and stored in a Qdrant vector database. The context is an array of JSON objects; do I concatenate the array to a string and assign the result to the context property of the metadata?

Additional context
[Add any other context, screenshots, or examples that could help explain the issue or improvement]

Checklist

  • I have checked that this issue hasn't already been reported
  • I have checked the latest version of the documentation to ensure this issue still exists
  • For simple typos or fixes, I have considered submitting a pull request instead
@railsstudent railsstudent added the documentation Improvements or additions to documentation label Dec 10, 2024
@brianlai98
Copy link

Hi @railsstudent, yes that's correct, you would concatenate the contents of the retrieved chunks from Qdrant and pass that as the context field in the metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

No branches or pull requests

2 participants