Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download cpython installer to cache dir on macos #2108

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

trim21
Copy link

@trim21 trim21 commented Dec 7, 2024

When download cpython on macos, download it to CIBW_CACHE_PATH/python/ instead of temp path.

So if user cache CIBW_CACHE_PATH between ci runs, they do not need to download cpython every times.

We have done similar things for pypy, but not cpython.

for example:

Building cp313t-macosx_x86_64 wheel
CPython 3.13t macOS x86_64
Installing Python cp313t...
  + pkgutil --pkgs
  + Download https://www.python.org/ftp/python/3.13.0/python-3.13.0-macos11.pkg to /Users/runner/Library/Caches/cibuildwheel/python/python-3.13.0-macos11.pkg
  + sudo installer -pkg /Users/runner/Library/Caches/cibuildwheel/python/python-3.13.0-macos11.pkg -applyChoiceChangesXML /Users/runner/work/_temp/cibw/lib/python3.12/site-packages/cibuildwheel/resources/free-threaded-enable-313.xml -target /
  installer: Package name is Python
  installer: choices changes file '/Users/runner/work/_temp/cibw/lib/python3.12/site-packages/cibuildwheel/resources/free-threaded-enable-313.xml' applied

close #2081

travis ci failed on unrelated os

image

@trim21 trim21 changed the title WIP: cache cpython install file on macos WIP: cache cpython install on macos Dec 7, 2024
@trim21 trim21 changed the title WIP: cache cpython install on macos cache cpython install on macos Dec 7, 2024
@trim21 trim21 marked this pull request as ready for review December 7, 2024 15:45
@trim21 trim21 force-pushed the cache-macox-cpython branch 3 times, most recently from e8c0a11 to 040e249 Compare December 7, 2024 15:52
@trim21 trim21 force-pushed the cache-macox-cpython branch from 040e249 to eec7b3d Compare December 7, 2024 15:53
@mayeut
Copy link
Member

mayeut commented Dec 8, 2024

Unless the download of the installer is very slow itself, I would guess that it wouldn't change much (just moving the timings from cibuildwheel download to cache download+extract). We'd still need to install which is probably where most of the time is spent.

Given all those are just guesses, seeing the actual benefits might help reason about this. Do you have runs on one of your projects showing the improvement ?

EDIT: I could confirm my guesses by running with and without installer cache https://github.com/mayeut/cibuildwheel/actions/runs/12221740312/usage

@trim21
Copy link
Author

trim21 commented Dec 8, 2024

Unless the download of the installer is very slow itself, I would guess that it wouldn't change much (just moving the timings from cibuildwheel download to cache download+extract). We'd still need to install which is probably where most of the time is spent.

Given all those are just guesses, seeing the actual benefits might help reason about this. Do you have runs on one of your projects showing the improvement ?

EDIT: I could confirm my guesses by running with and without installer cache https://github.com/mayeut/cibuildwheel/actions/runs/12221740312/usage

Thanks for your explanation.

I agree, on GitHub CI it has roughly same speed downloading cache and python pkg, it will only benefit users with low network speed with www.python.org, and sadly I'm one of them.

for example:

$ curl https://www.python.org/ftp/python/3.10.11/python-3.10.11-macos11.pkg -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 10 39.1M   10 4086k    0     0  45543      0  0:15:00  0:01:31  0:13:29 48863

But cache can be really fast in internal network with local S3, using a temp cache make it uncacheable.

@trim21 trim21 changed the title cache cpython install on macos download cpython installer to cache dir on macos Dec 8, 2024
@trim21
Copy link
Author

trim21 commented Dec 8, 2024

I updated the PR title to make it more proper

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

cache macos python installation pkg when possible
2 participants