🚀 Feature: Test Sharding for CI Matrix Purposes #4958
Comments
|
@amerryma Thx for working on this. It would be fantastic to get sharding support as Jest has it too and it is amazing to be able to use Github Actions Matrix support. It doesn't hurt to just make a PR and see the feedback you get. It will be one step closer to getting this merged in. |
|
I tried your fork @amerryma and as it turns out, we have tests within a test file that depends on the previous test to run in order for it to pass. I imagine many people have also implemented their tests this way as well. Jest shards based on the entire file which won't balance the load as evenly but it helps solve the problem in a safer more general way that works with more projects. If you were to submit a PR, I would suggest an implementation that is similar to how Jest handles it using file based sharding. https://jestjs.io/docs/next/cli#--shard |
|
Thank you so much for your feedback! I will look at Jest's implementation
and get back to you :)
…On Fri, May 19, 2023, 2:32 AM David Siu ***@***.***> wrote:
I tried your fork @amerryma <https://github.com/amerryma> and as it turns
out, we have tests within a test file that depends on the previous test to
run in order for it to pass. I imagine many people have also implemented
their tests this way as well. Jest shards based on the entire file which
won't balance the load as evenly but it helps solve the problem in a safer
more general way that works with more projects. If you were to submit a PR,
I would suggest an implementation that is similar to how Jest handles it
using file based sharding. https://jestjs.io/docs/next/cli#--shard
—
Reply to this email directly, view it on GitHub
<#4958 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABQD4K67GDKJVROOLNFH7FDXG443RANCNFSM6AAAAAATNZVCAM>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Subscribing to this. Would be great to have! |
|
@amerryma did you make any progress? |
|
@amerryma 👀 is there any progress? |
|
Hi all, I've updated my fork with the new file based strategy. @davidsiu If this is the correct direction you were thinking, I can make the PR and see if we can get this reviewed and in! |
JoshuaKGoldberg
changed the title
|
Would be happy to see this as well! |
|
@amerryma i think it's worth creating the PR at this point. |
|
👍 this is definitely a feature gap in Mocha compared to other test runners. I think it definitely qualifies as something we long-term would want to take in. Note that per #5027 it'll be a while before we can really take a look at something as impactful as this. What'd be helpful first, though is to see a comparison of how other popular test runners (Codeception, Jest, Playwright, Vitest, etc.) with sharding approach it. We should make sure considerations those test runners have learned they need to contend with are accounted for in Mocha if needed.
|
JoshuaKGoldberg
added
status: waiting for author
|
We have implemented partial sharding in our project using I'm happy to share more information if it might be valuable to be ported or adapted into mocha. |
Is your feature request related to a problem or a nice-to-have?? Please describe.
I use CodeceptJS which uses Mocha under the hood to programmatically run end-to-end tests for browsers. We've been increasing the amount of tests slowly and we're running into issues where tests are taking too long to run on a single machine.
Describe the solution you'd like
There is a concept of test sharding that I propose we add to Mocha. You can read more about this in a few other test runners such as Playwright and Codeception .
So if we had 100 tests, and I split into groups of 4, then tell one machine to execute group 1, it should execute 25 tests. One method of determining which test are part of which groups are simply running every 4th test.
We can then use the
matrixstrategy in Github Actions to execute each shard on a different machine in parallel.Describe alternatives you've considered
Additional context
I've implemented this in a fork: master...discatech:mocha:test-sharding of Mocha already and would love to get some feedback on whether this is a good change for Mocha itself. I will probably continue to use my forked Mocha until we reach a decision. Thank you so much for considering this!
The text was updated successfully, but these errors were encountered: