Error while running llama3.2-3B-Instruct Inference on CPU

[rakshit2020]

Hi,
I was successfully able to build the model (llama3.2-3B-Instruct) but while running the model inference using **model-qa.py** I am getting the following error -
RuntimeError: Unknow search option:batch_size
While running the cmd python model-qa.py -m ..... (Running on Pure CPU)

[aciddelgado]

Hello, are you running the latest version of GenAI (built from source)? batch_size is a search option we added recently and is not in the latest release (0.5.2), so if you are running model-qa.py script as it is in main, it will throw this error. Apologies for the incongruence.

[kunal-vaishnavi]

You can see this announcement for more information on which example script to use.