Example only using free Hugging Face inference and embeddings

[vanetreg]

Pls. create a working example which only uses free Hugging Face endpoints for both embeddings and completion.
I tried to tweak existing example using
AutoQueryEngine.from_config()
by the docs but it still uses OpenAI's endpoints, so I was wrong.
Probably others also would prefer this, without spending any money for OpenAI API :)

[SeeknnDestroy]

Hi @vanetreg, thank you for suggestion! I will definetely update our examples according your suggestion :)

For quick solution, you can set your llm_model and embed_model parameters as Hugging Face models and you are good to go! If you have hosted it on Huggingface, you also should give the llm_api parameter as your hosted url:

tasks:
  - name: "qa"
    llm_model: "huggingface/WizardLM/WizardCoder-Python-34B-V1.0"
    llm_api_base: "https://my-endpoint.huggingface.cloud"
    embed_model: "huggingface/BAAI/bge-large-zh"  

also you might need to set your HUGGINGFACE_API_KEY env variable:

os.environ['HUGGINGFACE_API_KEY'] = ...

Hope it helps. If you have any more suggestions or questions please share them as well.

[vanetreg]

Hi @SeeknnDestroy ,
I tried with the following code ( .env includes HF API key) but got back the following warning and the process should be interrupted:
WARNING:llama_index.llms.litellm_utils:Retrying llama_index.llms.litellm_utils.completion_with_retry.<locals>._completion_with_retry in 10.0 seconds as it raised APIError: HuggingfaceException - response is not in expected format - [{'error': 'The model mistralai/Mixtral-8x7B-v0.1 is too large to be loaded automatically (93GB > 10GB). Please use Spaces (https://huggingface.co/spaces) or Inference Endpoints (https://huggingface.co/inference-endpoints).'}].

The code used last time is:

from autollm import AutoQueryEngine, read_files_as_documents

from dotenv import load_dotenv, find_dotenv

_ = load_dotenv(find_dotenv())  # read local .env file

documents = read_files_as_documents(input_dir="documents")

query_engine = AutoQueryEngine.from_defaults(
    documents,
    llm_model="huggingface/mistralai/Mixtral-8x7B-v0.1",
    llm_api_base="https://api-inference.huggingface.co/models/mistralai/Mixtral-8x7B-v0.1",
    llm_max_tokens=1900,
    llm_temperature=0.1,
    chunk_size=500,
    chunk_overlap=40  
    )

# query_engine = AutoQueryEngine.from_config(
#     "config.yaml",
#     ".env",
#     documents=documents
# )

# response = query_engine.query("Why did SafeVideo AI develop this project?")
response = query_engine.query("Who is the author and what is this all about?")

print(response.response)
# "Because they wanted to deploy rag based llm apis in no time!"

So the main problems were:

1. figure out "huggingface/" must be added before the HF model name
2. API base URL ( most of the popular models @ HF doesn't have free inference endpoint )
3. I found free inference API endpoint at: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1 when clicking "Deploy" dropdown, but there is not too much for smaller models like Phi-2, so not easy to find one...

So finally I'll generate a HF space and keep testing it there...

[vanetreg]

Hi @SeeknnDestroy

query_engine = AutoQueryEngine.from_defaults(
    documents,
    llm_model="huggingface/HuggingFaceH4/zephyr-7b-beta",
    llm_api_base="https://api-inference.huggingface.co/models/HuggingFaceH4/zephyr-7b-beta",
    llm_max_tokens=1900,
    llm_temperature=0.1,
    embed_model="huggingface/BAAI/bge-large-en-v1.5", 
    chunk_size=500,
    chunk_overlap=40 
    )

worked, though embedding triggered a rate limit error, which I think should be handled properly for every inference, not only for HF.
Rate limit error:

  File "D:\***\Lib\site-packages\litellm\llms\huggingface_restapi.py", line 699, in embedding
    raise HuggingfaceError(status_code=500, message=embeddings["error"])
litellm.llms.huggingface_restapi.HuggingfaceError: Rate limit reached. Please log in or use your apiToken

I have a free HF token, but considering HF Pro (9 USD / month), if eg. AutoLLM works with HF models well.

[SeeknnDestroy]

Thanks for your time for trying 'autollm' @vanetreg!

Yes, 'autollm' indeed works with huggingface models and embeddings well. You can try it and please share your feedback :)