Due to some internal requirements, we are not allowed to use a second feed.

The CUDA DLLs are so huge that they cannot be hosted at pypi. So they must be hosted at somewhere else. However, due to some security concerns we are not allowed to use a second feed. So this went to a dead end.

I will keep this issue open, while we could continue discussing the details offline.

Is the problem that you are you not allowed to consume packages from the external feed https://pypi.ngc.nvidia.com/ in your development environment and/or in your CI tests pipelines (github actions) for security reasons?

In CI.

could you possibly add https://pypi.ngc.nvidia.com/ as an upstream to the feed you are currently using?

Azure DevOps Artifact's upstream feature doesn't support that. It only supports pypi. Correct me if I was wrong.

Indeed it seems I cannot use https://pypi.ngc.nvidia.com/ as an upstream in Azure Devops.
However It seems that all the required nvidia packages to get the required dlls for cuda 12 are actually on pypi.org, so not need to add https://pypi.ngc.nvidia.com/ as an upstream.

Here is a list of package on pypi.org that allowed me to get all the dlls I needed to use onnxruntime-gpu (some might actually not be required)

nvidia-nvjitlink-cu12
nvidia-nvtx-cu12
nvidia-cuda-runtime-cu12
nvidia-cublas-cu12
nvidia-cuda-cupti-cu12
nvidia-cuda-nvrtc-cu12
nvidia-cudnn-cu12
nvidia-cufft-cu12
nvidia-curand-cu12
nvidia-cusolver-cu12
nvidia-cusparse-cu12

so for cuda 12 it seems we could potentially get onnxruntime-gpu to use the dlls from these packages.
Unfortunately onnxruntime-gpu for cuda 12 is not on pypi.org see #19438, which I hope will be solved soon.

Are they real? The file size of nvidia-cuda-runtime-cu12 is less than 1MB.
https://pypi.org/project/nvidia-cuda-runtime-cu12/12.4.127/#files

Are they real? The file size of nvidia-cuda-runtime-cu12 is less than 1MB. https://pypi.org/project/nvidia-cuda-runtime-cu12/12.4.127/#files

I think so. The whl file contains the cudart64_12.dll that is about 540KB, which is about the same as the dll we can get from the official cuda toolkit installer

But the cudnn files looks wired. The latest one(9.0) only has binaries for ARM64, while a previous version(8.9) only has binaries for Windows/Linux x64. And we don't know if the ARM one is for sbsa or jetson.

@snnn These cudnn wheels look reasonable to me: https://pypi.org/project/nvidia-cudnn-cu12/#files
Which ones did you look at ?

Now it is good. Thanks!

@snnn or @jchen351 did you start any work on this yet ? The problem propagates to ORT GenAI as well which is even more python focused. pip deployment will make it much easier with nvidia-* packages.

Sorry the work has not been started.

The first solution is not available to us

Packaged CUDA DLLs with onnxruntime-gpu Wheels:

Because these DLLs are large and we are very tight on space.

We can try the second one.

Dependency Configuration via onnxruntime-gpu Wheel:

@martinResearch Do you know if libcuda.so should to be installed via pip? If so, do you know where can I find it?

@martinResearch Do you know if libcuda.so should to be installed via pip? If so, do you know where can I find it?

I don't know. Is it required for onnxruntime-gpu? it is not not in the python environment I use to run onnxruntime-gpu. You have in libcudart.so.12 in https://files.pythonhosted.org/packages/f0/62/65c05e161eeddbafeca24dc461f47de550d9fa8a7e04eb213e32b55cfd99/nvidia_cuda_runtime_cu12-12.6.77-py3-none-manylinux2014_x86_64.whl

libcuda.so is a driver library and is installed with the nvidia driver. In a docker container it will be mounted when using nvidia docker toolkit.

When installing the packages nvidia-cuda-nvrtc-cu12 and nvidia-cudnn-cu12 the dlls end up respectively in lib\site-packages\nvidia\cuda_nvrtc\bin and lib\site-packages\nvidia\cudnn\bin and I wonder if that makes using these dlls more complicated on the onnxruntime side.
Maybe Nvidia could structure the published packages using namespace packages so that the dlls for the different packages end up in the same lib\site-packages\nvidia\bin folder? I don't know if that solution is pip-compliant and would allow uninstalling a nvidia packages individually though.

A similar effort is planned in the cupy repository cupy/cupy#8013. It might provide some valuable information on how to proceed.

Configure onnxruntime to utilize these DLLs instead of those in CUDA_PATH.

Would you mind explaining more on this part? Do we need to manually preload the libraries, or we just need to setup some paths(like https://docs.python.org/3/library/os.html#os.add_dll_directory)