Allow the use of arbitrary Pyodide versions

[agriyakhetarpal]

Description

This PR updates the Pyodide build procedure (see #1456) that is enabled with CIBW_PLATFORM: "pyodide" (or with the --platform pyodide CLI equivalent) post the changes in pyodide/pyodide#4882, where pyodide/pyodide-build was unvendored from the main Pyodide repository to accommodate faster updates and fixes.

This means that the Pyodide version and pyodide-build are not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install by pyodide-build through a recently added pyodide xbuildenv search command, which prints out this table:

Tap to expand table

┌────────────┬────────────┬────────────┬───────────────────────────┬────────────┐
│ Version    │ Python     │ Emscripten │ pyodide-build             │ Compatible │
├────────────┼────────────┼────────────┼───────────────────────────┼────────────┤
│ 0.27.0a2   │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.4     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.3     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.2     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.1     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.0     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
└────────────┴────────────┴────────────┴───────────────────────────┴────────────┘

Alternatively, one may use pyodide xbuildenv search --all to return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).

---

Additionally, in this PR, support has been added for installing arbitrary Pyodide versions, or, more specifically, arbitrary versions for "Pyodide cross-build environments (xbuildenvs)" – though, only the ones that are supported for a given pyodide-build version. This has been implemented through an environment variable CIBW_PYODIDE_VERSION and an associated configuration variable in the schema (through a table implemented via pyodide/pyodide-build#26).

The rationale behind this is that WebAssembly/Pyodide builds are already experimental, and it would be useful to not tie the available Pyodide version to the cibuildwheel version – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide's alpha releases and/or for the case of greater reproducibility against Pyodide's older releases.

cc: @hoodmane and @ryanking13 for visibility

---

Suggested CHANGELOG entry

Since I didn't find a way to add an entry without the pre-commit hook removing previous entries, I've added a few lines here based on the current state of this PR. Please feel free to suggest changes or modify these lines directly.

- 🛠 Provide [Pyodide version 0.26.4](https://github.com/pyodide/pyodide/releases/tag/0.26.4) with `cp312-pyodide_wasm32` (#2002)
- ✨ Allow the use of a custom Pyodide version to target by the use of the `CIBW_PYODIDE_VERSION` environment variable. This
is an experimental option and users are advised to look at the [compatible Pyodide versions](https://github.com/pyodide/pyodide/blob/main/pyodide-cross-build-environments.json) according to the [`pyodide-build`](https://github.com/pyodide/pyodide-build) version.

[agriyakhetarpal]

The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to.

[agriyakhetarpal]

The failing CircleCI-Linux-Python312 test should be unrelated.

[agriyakhetarpal]

I'm poised to remove this check, since this would allow downstream users to test versions such as Pyodide 0.27.0a2 and see if things are working. Filtering the versions with this check restricts us within the same minor release at times (say, 0.26.2 or 0.26.3, etc.) and the Emscripten version bump/ABI break in Pyodide is never achieved until a new minor release, say, 0.27 or 0.28 (or later) is released. Testing with new Pyodide patch releases thus does not bring much merit.

[agriyakhetarpal]

e551021 removes this check, feedback is welcome.

[hoodmane]

Generally looks good to me. I'd like to see:

1. pyodide_version override handled like all the other options
2. an override for pyodide_build_version

[hoodmane]

Didn't you just tell them what the compatible versions are? So they probably don't need to actually do this.
Also, maybe we want a newline after are:.

[agriyakhetarpal]

We have a writeup about the compatible versions in the docs, but the instance where we tell them about this at runtime is here in case users don't read the docs is here – or did you mean something else? I might have misunderstood.

[hoodmane]

The other options can be passed either as a cli flag e.g., --pyodide-version or as the corresponding environment variable in this case CIBW_PYODIDE_VERSION. They can also come from the cibuildwheel section in pyproject.toml. We should handle this in the same way as all the other options. Then we can just use python_configuration.pyodide_version and it should already have the right value.

[henryiii]

Most options are not available as flags (and I wouldn't add this one), only stuff you often need on the command line, like --platform. Just env var and config.

[hoodmane]

Again we should get the correct value into config.pyodide_version to begin with.

[hoodmane]

Why don't we keep telling people to use x86-64 linux runners? Is there a good reason to use arm64 linux? We don't test it ourselves, are we sure that it works? "Mac builds may succeed but there are known bugs".

[agriyakhetarpal]

We could do that, though since Emscripten has arm64 binaries now, I've started using them in Docker containers and on macOS and they seem to work as intended. Maybe it would be worth adding a pipeline to test Linux arm64 for Pyodide here as well, should I try that? I'll replace the macOS sentence with your suggestion.

[agriyakhetarpal]

Added your phrasing here: 057e542. I'd like to see if there is interest in adding Linux-arm64 and macOS-arm64 Pyodide builds, so I've kept this conversation as "unresolved" for now :)

[hoodmane]

Is there a good reason that it isn't available in the pyproject.toml config?

[henryiii]

It seems like it should be.

[agriyakhetarpal]

Generally looks good to me. I'd like to see:

1. pyodide_version override handled like all the other options
2. an override for pyodide_build_version

I'll check 1) in a bit, but for 2), we can do this with the current structure and use a newer pyodide-build version via the CIBW_DEPENDENCY_VERSIONS option (or the more specific CIBW_DEPENDENCY_VERSIONS_PYODIDE) by supplying a custom constraints file. Is this what you intended?

[agriyakhetarpal]

Also, I think I should add a test workflow (or another job through a matrix) that builds Pyodide wheels with the new CIBW_PYODIDE_VERSION variable set so that the functionality remains intact.

[henryiii]

This file gets generated, I don't see the bin/generate_schema.py being updated?

[agriyakhetarpal]

🤔 Just tried nox -s generate_schema and the changes went away. I'll check how to update the schema properly.

[henryiii]

Doesn't cibuildwheel/options.py also need updating? I don't see a .get("pyodide-versions") being read there.

[henryiii]

Wild idea (probably), but what about allowing the version to be part of the platform? So --platform pyodide would be the default version, but --platform pyodide-0.27.0a2 would be allowed, too. This would tie the pyodide-specific setting to the pyodide platform only. But this would not allow you to require a specific version inside pyproject.toml, I guess. Thoughts?

[henryiii]

RE: an override for pyodide_build_version

Can't this be done like overriding any other constrained package? You could set the versions to latest, or even pip install a specific version in before-build?

[agriyakhetarpal]

Wild idea (probably), but what about allowing the version to be part of the platform? So --platform pyodide would be the default version, but --platform pyodide-0.27.0a2 would be allowed, too. This would tie the pyodide-specific setting to the pyodide platform only. But this would not allow you to require a specific version inside pyproject.toml, I guess. Thoughts?

I would be fine with implementing this, but in principle (and to be slightly pedantic), "pyodide" would generally be the platform and "0.27.0a2" would be the version, so mixing the platform name and the version into one string sounds a bit odd to me – given that we don't have a "linux-cp313" of sorts in https://cibuildwheel.pypa.io/en/stable/options/#platform, right?

RE: an override for pyodide_build_version

Can't this be done like overriding any other constrained package? You could set the versions to latest, or even pip install a specific version in before-build?

Yes, I had the same suggestion in #2002 (comment), but TIL that before-build can also override, that is probably neater for those who would like to explore newer versions without pinning other constraints.

[henryiii]

If linux-cp313 was valid then pyodide-cp313 would be valid too, which is the same thing or helpful; I don't think that's a correct comparison. Maybe linux-manylinux2014 would be a better comparison. (Which we don't allow)

Unless someone really likes the idea, let's keep going with pyodide-version. My only thought is that it's a general config setting only applicable to a specific platform. Though we do have a few of those (the manylinux settings, for example).

[joerick]

RE: an override for pyodide_build_version

Can't this be done like overriding any other constrained package? You could set the versions to latest, or even pip install a specific version in before-build?

Ah, you raise a good point! If we have an option CIBW_PYODIDE_VERSION and the user customises it, well we still have constraints in cibuildwheel/resources/constraints-pyodide312.txt that are for a different version. That's inconsistent/inefficient at best, error-prone at worst.

So, yes, the other approach would be to read the contents of CIBW_DEPENDENCY_VERSIONS to get the pyodide-build version. We already do something similar here, it's fairly easy using packaging.requirements.Requirement

https://github.com/henryiii/cibuildwheel/blob/6f9ad0b8989d9c476e7bb42b1b52536a19a5328b/cibuildwheel/util.py#L648-L692

The other thing that would help here is a way to specify CIBW_DEPENDENCY_VERSIONS inline, without the extra file. Something like:

CIBW_DEPENDENCY_VERSIONS_PYODIDE: "requirements: pyodide-build==0.29.1"

(I don't love the requirements: marker but can't see a way to reliably distinguish from a file otherwise)