CMake: Refactor Build Folder Structure

[shaoboyan]

Previous build scripts (e.g. build.bat) generate build folder with multi-config system as:

• Nested config folders (e.g. build/Windows/Debug/Debug)
• Multiple deps sources in each build config folders (e.g. build/Windows/Debug/_deps and build/Windows/Release/_deps)

This PR refactors the build folder by:

• Remove nested config folder structure and use single "bin", "lib" and "arch" folder for each configuration with multi config system.
• Promote "_deps" folder from configuration folder to build root folder (e.g. From build/Windows/Debug/_deps to build/Windows/_deps).
• Created new "deps" folder in each configuration folder. (e.g. build/Windows/Debug/deps).
• The promoted _deps folder holds "deps_src" and "deps_download"(for old FetchContent behaviour).
• The new created "deps" folder holds "deps_build" to avoid redundant build caused by configuration project file overridding.
• A new command option in build.py to control whether update/download deps sources from remote url.

It achieves:

• No nested config folder structure in build folder.
• Single deps sources for each build configuration to avoid redundant downloads and builds.

Description

Motivation and Context

[shaoboyan]

The PR tries not to do large changes on current CmakeList. However, it makes _deps management a bit hard due to:

• FetchContent could check downloaded file with provided hash, but always removed the src folder and extract the zip file to create a new one

This behaviour has no proper flag to control ("DOWNLOAD_NO_EXTRACT" maybe but it requires extra extract process). I have to add a build flag update_deps to control DFETCHCONTENT_FULLY_DISCONNECTED manually.

Not sure any CMake experts could provide comments.

My thoughts on this are:

• Using git submodule to manage all third parties. It works with git pull perfectly but might download third party which may not needed at all in current build env.
• Replace FetchContent with self created download, checking and extract logics. We also need to manage dependencies carefully, It requires lots of work.
• Maybe introduce some package management tool is a good idea. E.g. vcpkg.

WDYT?

[shaoboyan]

@microsoft-github-policy-service agree

[snnn]

You can use https://github.com/microsoft/onnxruntime/blob/main/cmake/deps_update_and_upload.py to manually download artifacts. For example:

python cmake/deps_update_and_upload.py --root-path mirror

Then our cmake scripts will auto pick the files there, even if your switch to a different branch. It can achieve the goal of "Single deps sources for each build configuration to avoid redundant downloads and builds." Or you may also consider using vcpkg instead, which can even cache intermediate build files. Therefore I think there is no urgent need to do such a refactoring.

[shaoboyan]

@snnn Thanks for this info.
So does this mean the current way I build onnxruntime is not perfect? I execute build.bat --use_webgpu --skip_tests always. And it uses fetchContent to update dawn or other third_parties per build folder.

So I think your suggestion is that we should always do deps update, and point the build script to the deps folder and then do build?

[snnn]

You may use the following commands to build ONNX Runtime code:

python cmake/deps_update_and_upload.py --root-path mirror
python tools\ci_build\build.py  --config Debug --build_dir bwebgpu --skip_submodule_sync --build_csharp --update --parallel --cmake_generator "Visual Studio 17 2022" --build_shared_lib --use_webgpu

The first one does not need to be run every time.

[shaoboyan]

Thanks for the suggestion.
I tried the commands.

It indeed saving lots of download time but not meet my requirement 100%. For example, _deps still lives in each config folder and the problem is that it runs
Running fetch_dawn_dependencies: for a while when I delete the folder and do a clean build again.