Kv cache APIs

[prashantaithal]

Does ORT Gen Ai have an in built implementation of KV cache , especially for LLM use case? Are there any APIs for KV cache that we can use ?

[baijumeswani]

onnxruntime-genai currently has a built-in implementation of the kv caching.

What kind of APIs are you looking for?

[prashantaithal]

I am using a tiny llama and llama2 7B model in onnx format and run inference on them. I have observed slow inference time after increasing the max number of tokens to be generated, hence looking into kv cache.

I see that these models have past and present kv input and output nodes respectively.
How can I use the kv cache to improve inference times using onnxruntime-genai?