PGVector Duplicates Entries

[MichaelMMeskhi]

Describe the bug
When training the RAG layer for PGVector, it duplicates the entires. For instance in ChromaDB, duplicate entries are skipped over.

To Reproduce
Steps to reproduce the behavior:

1. Run script to embed 10 documents into PGVector.
2. Check Vanna app to confirm training data has 10 entries,
3. Rerun training script
4. Training data now has 20 entries (10 duplicates).

Expected behavior
Should skip duplicate embeddings.

Error logs/Screenshots
If applicable, add logs/screenshots to give more information about the issue.

Desktop (please complete the following information where):

• OS: Ubuntu
• Version: 24.04
• Python: 3.11
• Vanna: 0.7.5

[MichaelMMeskhi]

For instance, when using ChromaDB, it warns the user that an existing embedding already exists and it skips it. While PGVector has no such warnings and somehow manages to slip the duplicates in.