Skip build step for changes in test

[fatteneder]

Depends at least on #400

To get this working we need to figure out if the merge-base of a PR provides artifacts which we can then download and use for testing steps.
But it might happen that some artifacts are missing, either due to build errors or timeouts. I guess we should then try to rebuild those again.

[DilumAluthge]

Do any stdlibs still (as of Julia 1.12) go into the default Julia sysimage? If so, then we'd need to run the build CI when any such stdlib is modified.

If the Julia testsuite (the top-level /test directory) has been modified, then we need to run the test CI jobs on all platforms, right? To make sure that the new tests are passing? But in order to run the tests, we first need to build Julia, right?

[fatteneder]

Do any stdlibs still (as of Julia 1.12) go into the default Julia sysimage? If so, then we'd need to run the build CI when any such stdlib is modified.

Oh, didn't think of that. I guess you are right.

If the Julia testsuite (the top-level /test directory) has been modified, then we need to run the test CI jobs on all platforms, right? To make sure that the new tests are passing?

Yes and Yes.

But in order to run the tests, we first need to build Julia, right?

What I am suggesting is that we grab a build artifact from the CI run of the commit on master from which we branched off (if available). That binary should then be the same as if we were to build it again, no?

[DilumAluthge]

But in order to run the tests, we first need to build Julia, right?

What I am suggesting is that we grab a build artifact from the CI run of the commit on master from which we branched off (if available). That binary should then be the same as if we were to build it again, no?

Ah, I see. Yeah, that would be quite clever if we could get it to work.

[DilumAluthge]

So we'd basically need to:

1. Figure out what commit (on master) we branched off of.
2. Use the Buildkite API to find the Buildkite build for that commit on master. We'll need a Buildkite API token for that. We should make sure that the Buildkite API token only has read-only access (because there's no need for write access).
3. Make sure the build passed on that commit on master. If the build failed for some jobs, we have to re-build those jobs (but presumably don't need to re-build the jobs that originally passed).
4. Grab the build artifacts from the previously-passing build jobs, download them, and use them to run the tests.

[fatteneder]

Sounds like this requires some effort (I am not too familiar with step 2) compared to what would be gained, as the number of PRs that only touch test is rather small, I think.
But maybe we could reuse that logic later for a more fine-grained base and stdlib filter that can differentiate between docstring-only changes and changes that touch actual code (using JuliaSyntax.jl).

[fatteneder]

But maybe we could reuse that logic later for a more fine-grained base and stdlib filter that can differentiate between docstring-only changes and changes that touch actual code (using JuliaSyntax.jl).

Ok, maybe that won't work, as the docstrings are typically baked into sysimages too, so we need to always do a full build.

[DilumAluthge]

Yeah, it seems like we'll need to build Julia for docstring changes.

For stdlibs, if there are any stdlibs that are still in the default Julia sysimage, then of course we need to build Julia for PRs that modify those stdlibs.

But what about the stdlibs that are no longer in the default Julia sysimage? We still need to test that pkgimages (.so/.dll/.dylib) for those stdlibs are able to be built, right? And in order to build the stdlib pkgimages, we need to either:

1. Build Julia first (which is what we currently do).
2. Do the above-described process of grabbing the Julia binaries built on the relevant ancestor commit on the master branch.

[fatteneder]

We still need to test that pkgimages (.so/.dll/.dylib) for those stdlibs are able to be built, right?

Yes, I think so too.